Note: This is test shard 1 of 3.
[==========] Running 2 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 2 tests from CatalogManagerConfigurations/MasterStressTest
[ RUN ] CatalogManagerConfigurations/MasterStressTest.Test/0
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20250113 02:51:11.684476 416 test_util.cc:274] Using random seed: -2073929902
2025-01-13T02:51:11Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-01-13T02:51:11Z Disabled control of system clock
I20250113 02:51:11.718482 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskXJrxug/build/asan/bin/kudu
/tmp/dist-test-taskXJrxug/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:44217
--webserver_interface=127.0.104.62
--webserver_port=0
--builtin_ntp_servers=127.0.104.20:45573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:44217,127.0.104.61:35271,127.0.104.60:42067
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20250113 02:51:12.048226 424 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250113 02:51:12.048584 424 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250113 02:51:12.048713 424 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250113 02:51:12.056195 424 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250113 02:51:12.056339 424 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250113 02:51:12.056432 424 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250113 02:51:12.056521 424 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250113 02:51:12.066089 424 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:45573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/master-0/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:44217,127.0.104.61:35271,127.0.104.60:42067
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:44217
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.104.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 13 Jan 2025 02:43:23 UTC on 24a791456cd2
build id 4151
ASAN enabled
I20250113 02:51:12.067809 424 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250113 02:51:12.069702 424 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 02:51:12.080715 430 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 02:51:12.080736 429 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 02:51:12.081980 432 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 02:51:12.082741 424 server_base.cc:1034] running on GCE node
I20250113 02:51:12.083514 424 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250113 02:51:12.085068 424 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250113 02:51:12.086278 424 hybrid_clock.cc:648] HybridClock initialized: now 1736736672086205 us; error 52 us; skew 500 ppm
I20250113 02:51:12.086628 424 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250113 02:51:12.094873 424 webserver.cc:458] Webserver started at http://127.0.104.62:38073/ using document root <none> and password file <none>
I20250113 02:51:12.095652 424 fs_manager.cc:362] Metadata directory not provided
I20250113 02:51:12.095772 424 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 02:51:12.096096 424 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250113 02:51:12.098851 424 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/master-0/data/instance:
uuid: "e13f5bcdcee34225a46c6d2651e5ede9"
format_stamp: "Formatted at 2025-01-13 02:51:12 on dist-test-slave-v0t5"
I20250113 02:51:12.099581 424 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/master-0/wal/instance:
uuid: "e13f5bcdcee34225a46c6d2651e5ede9"
format_stamp: "Formatted at 2025-01-13 02:51:12 on dist-test-slave-v0t5"
I20250113 02:51:12.104781 424 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20250113 02:51:12.107940 438 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 02:51:12.109323 424 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20250113 02:51:12.109473 424 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/master-0/data,/tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/master-0/wal
uuid: "e13f5bcdcee34225a46c6d2651e5ede9"
format_stamp: "Formatted at 2025-01-13 02:51:12 on dist-test-slave-v0t5"
I20250113 02:51:12.110121 424 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 02:51:12.143335 424 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 02:51:12.144207 424 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250113 02:51:12.144496 424 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 02:51:12.170320 424 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.62:44217
I20250113 02:51:12.170356 489 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.62:44217 every 8 connection(s)
I20250113 02:51:12.171626 424 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/master-0/data/info.pb
I20250113 02:51:12.174253 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskXJrxug/build/asan/bin/kudu as pid 424
I20250113 02:51:12.174552 416 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/master-0/wal/instance
I20250113 02:51:12.176208 490 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:12.178274 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskXJrxug/build/asan/bin/kudu
/tmp/dist-test-taskXJrxug/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.61:35271
--webserver_interface=127.0.104.61
--webserver_port=0
--builtin_ntp_servers=127.0.104.20:45573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:44217,127.0.104.61:35271,127.0.104.60:42067
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250113 02:51:12.184263 490 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44217 } has no permanent_uuid. Determining permanent_uuid...
I20250113 02:51:12.200268 490 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.61" port: 35271 } has no permanent_uuid. Determining permanent_uuid...
W20250113 02:51:12.201234 440 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.61:35271: connect: Connection refused (error 111)
W20250113 02:51:12.202622 490 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.104.61:35271: Network error: Client connection negotiation failed: client connection to 127.0.104.61:35271: connect: Connection refused (error 111)
I20250113 02:51:12.251948 490 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.61" port: 35271 } attempt: 1
W20250113 02:51:12.253995 490 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.104.61:35271: Network error: Client connection negotiation failed: client connection to 127.0.104.61:35271: connect: Connection refused (error 111)
I20250113 02:51:12.322237 490 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.61" port: 35271 } attempt: 2
W20250113 02:51:12.324018 490 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.104.61:35271: Network error: Client connection negotiation failed: client connection to 127.0.104.61:35271: connect: Connection refused (error 111)
I20250113 02:51:12.415266 490 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.61" port: 35271 } attempt: 3
W20250113 02:51:12.416996 490 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.104.61:35271: Network error: Client connection negotiation failed: client connection to 127.0.104.61:35271: connect: Connection refused (error 111)
W20250113 02:51:12.519827 492 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250113 02:51:12.520177 492 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250113 02:51:12.520313 492 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250113 02:51:12.528497 492 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250113 02:51:12.528630 492 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250113 02:51:12.528710 492 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250113 02:51:12.528788 492 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250113 02:51:12.538596 492 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:45573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/master-1/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:44217,127.0.104.61:35271,127.0.104.60:42067
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.61:35271
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.104.61
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 13 Jan 2025 02:43:23 UTC on 24a791456cd2
build id 4151
ASAN enabled
I20250113 02:51:12.540253 492 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250113 02:51:12.541913 492 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 02:51:12.551859 503 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 02:51:12.551761 501 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 02:51:12.552004 500 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 02:51:12.552539 492 server_base.cc:1034] running on GCE node
I20250113 02:51:12.553248 492 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250113 02:51:12.554508 492 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250113 02:51:12.555718 492 hybrid_clock.cc:648] HybridClock initialized: now 1736736672555665 us; error 32 us; skew 500 ppm
I20250113 02:51:12.556185 492 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250113 02:51:12.558948 492 webserver.cc:458] Webserver started at http://127.0.104.61:37629/ using document root <none> and password file <none>
I20250113 02:51:12.559682 492 fs_manager.cc:362] Metadata directory not provided
I20250113 02:51:12.559809 492 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 02:51:12.560174 492 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250113 02:51:12.560218 490 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.61" port: 35271 } attempt: 4
W20250113 02:51:12.562006 490 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.104.61:35271: Network error: Client connection negotiation failed: client connection to 127.0.104.61:35271: connect: Connection refused (error 111)
I20250113 02:51:12.563228 492 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/master-1/data/instance:
uuid: "6bbc3541604a455584bb32dfc2cc4efd"
format_stamp: "Formatted at 2025-01-13 02:51:12 on dist-test-slave-v0t5"
I20250113 02:51:12.564009 492 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/master-1/wal/instance:
uuid: "6bbc3541604a455584bb32dfc2cc4efd"
format_stamp: "Formatted at 2025-01-13 02:51:12 on dist-test-slave-v0t5"
I20250113 02:51:12.569450 492 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20250113 02:51:12.572724 509 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 02:51:12.573997 492 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20250113 02:51:12.574142 492 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/master-1/data,/tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/master-1/wal
uuid: "6bbc3541604a455584bb32dfc2cc4efd"
format_stamp: "Formatted at 2025-01-13 02:51:12 on dist-test-slave-v0t5"
I20250113 02:51:12.574318 492 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 02:51:12.610404 492 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 02:51:12.611276 492 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250113 02:51:12.611563 492 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 02:51:12.631817 492 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.61:35271
I20250113 02:51:12.631848 560 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.61:35271 every 8 connection(s)
I20250113 02:51:12.633179 492 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/master-1/data/info.pb
I20250113 02:51:12.635394 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskXJrxug/build/asan/bin/kudu as pid 492
I20250113 02:51:12.635582 416 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/master-1/wal/instance
I20250113 02:51:12.636684 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskXJrxug/build/asan/bin/kudu
/tmp/dist-test-taskXJrxug/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.60:42067
--webserver_interface=127.0.104.60
--webserver_port=0
--builtin_ntp_servers=127.0.104.20:45573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:44217,127.0.104.61:35271,127.0.104.60:42067
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250113 02:51:12.637110 561 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:12.644833 561 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44217 } has no permanent_uuid. Determining permanent_uuid...
I20250113 02:51:12.658484 561 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.61" port: 35271 } has no permanent_uuid. Determining permanent_uuid...
I20250113 02:51:12.667593 561 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42067 } has no permanent_uuid. Determining permanent_uuid...
W20250113 02:51:12.668567 510 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.60:42067: connect: Connection refused (error 111)
W20250113 02:51:12.669878 561 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.104.60:42067: Network error: Client connection negotiation failed: client connection to 127.0.104.60:42067: connect: Connection refused (error 111)
I20250113 02:51:12.719151 561 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42067 } attempt: 1
W20250113 02:51:12.720917 561 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.104.60:42067: Network error: Client connection negotiation failed: client connection to 127.0.104.60:42067: connect: Connection refused (error 111)
I20250113 02:51:12.789207 561 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42067 } attempt: 2
W20250113 02:51:12.791052 561 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.104.60:42067: Network error: Client connection negotiation failed: client connection to 127.0.104.60:42067: connect: Connection refused (error 111)
I20250113 02:51:12.861279 490 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.61" port: 35271 } attempt: 5
I20250113 02:51:12.868120 490 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42067 } has no permanent_uuid. Determining permanent_uuid...
W20250113 02:51:12.870050 490 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.104.60:42067: Network error: Client connection negotiation failed: client connection to 127.0.104.60:42067: connect: Connection refused (error 111)
I20250113 02:51:12.882321 561 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42067 } attempt: 3
W20250113 02:51:12.884075 561 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.104.60:42067: Network error: Client connection negotiation failed: client connection to 127.0.104.60:42067: connect: Connection refused (error 111)
I20250113 02:51:12.921281 490 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42067 } attempt: 1
W20250113 02:51:12.923142 490 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.104.60:42067: Network error: Client connection negotiation failed: client connection to 127.0.104.60:42067: connect: Connection refused (error 111)
W20250113 02:51:12.979629 563 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250113 02:51:12.979930 563 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250113 02:51:12.980060 563 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250113 02:51:12.988266 563 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250113 02:51:12.988410 563 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250113 02:51:12.988480 563 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250113 02:51:12.988564 563 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250113 02:51:12.991343 490 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42067 } attempt: 2
W20250113 02:51:12.993140 490 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.104.60:42067: Network error: Client connection negotiation failed: client connection to 127.0.104.60:42067: connect: Connection refused (error 111)
I20250113 02:51:12.998291 563 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:45573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/master-2/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:44217,127.0.104.61:35271,127.0.104.60:42067
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.60:42067
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.104.60
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 13 Jan 2025 02:43:23 UTC on 24a791456cd2
build id 4151
ASAN enabled
I20250113 02:51:12.999861 563 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250113 02:51:13.001489 563 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 02:51:13.012323 572 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 02:51:13.012281 574 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 02:51:13.012851 571 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 02:51:13.013954 563 server_base.cc:1034] running on GCE node
I20250113 02:51:13.014693 563 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250113 02:51:13.016126 563 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250113 02:51:13.017328 563 hybrid_clock.cc:648] HybridClock initialized: now 1736736673017273 us; error 37 us; skew 500 ppm
I20250113 02:51:13.017647 563 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250113 02:51:13.020313 563 webserver.cc:458] Webserver started at http://127.0.104.60:38411/ using document root <none> and password file <none>
I20250113 02:51:13.021013 563 fs_manager.cc:362] Metadata directory not provided
I20250113 02:51:13.021126 563 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 02:51:13.021402 563 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250113 02:51:13.026707 563 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/master-2/data/instance:
uuid: "45e9d43ecac0401fb2c18fe5eaf089b5"
format_stamp: "Formatted at 2025-01-13 02:51:13 on dist-test-slave-v0t5"
I20250113 02:51:13.027295 561 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42067 } attempt: 4
I20250113 02:51:13.027493 563 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/master-2/wal/instance:
uuid: "45e9d43ecac0401fb2c18fe5eaf089b5"
format_stamp: "Formatted at 2025-01-13 02:51:13 on dist-test-slave-v0t5"
W20250113 02:51:13.028880 561 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.104.60:42067: Network error: Client connection negotiation failed: client connection to 127.0.104.60:42067: connect: Connection refused (error 111)
I20250113 02:51:13.032792 563 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20250113 02:51:13.036325 580 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 02:51:13.037720 563 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20250113 02:51:13.037878 563 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/master-2/data,/tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/master-2/wal
uuid: "45e9d43ecac0401fb2c18fe5eaf089b5"
format_stamp: "Formatted at 2025-01-13 02:51:13 on dist-test-slave-v0t5"
I20250113 02:51:13.038064 563 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 02:51:13.061419 563 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 02:51:13.062228 563 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250113 02:51:13.062510 563 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 02:51:13.082062 563 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.60:42067
I20250113 02:51:13.082091 631 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.60:42067 every 8 connection(s)
I20250113 02:51:13.083567 563 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/master-2/data/info.pb
I20250113 02:51:13.085740 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskXJrxug/build/asan/bin/kudu as pid 563
I20250113 02:51:13.085933 416 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/master-2/wal/instance
I20250113 02:51:13.087473 632 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:13.093343 632 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44217 } has no permanent_uuid. Determining permanent_uuid...
I20250113 02:51:13.099341 490 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42067 } attempt: 3
I20250113 02:51:13.107867 632 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.61" port: 35271 } has no permanent_uuid. Determining permanent_uuid...
I20250113 02:51:13.114065 490 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Bootstrap starting.
I20250113 02:51:13.118880 490 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:13.120313 490 log.cc:826] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Log is configured to *not* fsync() on all Append() calls
I20250113 02:51:13.121274 632 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42067 } has no permanent_uuid. Determining permanent_uuid...
I20250113 02:51:13.123932 490 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: No bootstrap required, opened a new log
I20250113 02:51:13.132730 632 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45e9d43ecac0401fb2c18fe5eaf089b5: Bootstrap starting.
I20250113 02:51:13.132957 490 raft_consensus.cc:357] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e13f5bcdcee34225a46c6d2651e5ede9" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44217 } } peers { permanent_uuid: "6bbc3541604a455584bb32dfc2cc4efd" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 35271 } } peers { permanent_uuid: "45e9d43ecac0401fb2c18fe5eaf089b5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42067 } }
I20250113 02:51:13.133330 490 raft_consensus.cc:383] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:13.133452 490 raft_consensus.cc:738] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e13f5bcdcee34225a46c6d2651e5ede9, State: Initialized, Role: FOLLOWER
I20250113 02:51:13.134136 490 consensus_queue.cc:260] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e13f5bcdcee34225a46c6d2651e5ede9" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44217 } } peers { permanent_uuid: "6bbc3541604a455584bb32dfc2cc4efd" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 35271 } } peers { permanent_uuid: "45e9d43ecac0401fb2c18fe5eaf089b5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42067 } }
I20250113 02:51:13.135035 641 sys_catalog.cc:455] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e13f5bcdcee34225a46c6d2651e5ede9" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44217 } } peers { permanent_uuid: "6bbc3541604a455584bb32dfc2cc4efd" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 35271 } } peers { permanent_uuid: "45e9d43ecac0401fb2c18fe5eaf089b5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42067 } } }
I20250113 02:51:13.135303 641 sys_catalog.cc:458] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9 [sys.catalog]: This master's current role is: FOLLOWER
I20250113 02:51:13.136425 490 sys_catalog.cc:564] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9 [sys.catalog]: configured and running, proceeding with master startup.
I20250113 02:51:13.137429 632 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 45e9d43ecac0401fb2c18fe5eaf089b5: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:13.138984 632 log.cc:826] T 00000000000000000000000000000000 P 45e9d43ecac0401fb2c18fe5eaf089b5: Log is configured to *not* fsync() on all Append() calls
I20250113 02:51:13.142196 632 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45e9d43ecac0401fb2c18fe5eaf089b5: No bootstrap required, opened a new log
W20250113 02:51:13.146905 444 tablet.cc:2367] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250113 02:51:13.149312 653 catalog_manager.cc:1559] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250113 02:51:13.149438 653 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250113 02:51:13.152216 632 raft_consensus.cc:357] T 00000000000000000000000000000000 P 45e9d43ecac0401fb2c18fe5eaf089b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e13f5bcdcee34225a46c6d2651e5ede9" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44217 } } peers { permanent_uuid: "6bbc3541604a455584bb32dfc2cc4efd" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 35271 } } peers { permanent_uuid: "45e9d43ecac0401fb2c18fe5eaf089b5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42067 } }
I20250113 02:51:13.152565 632 raft_consensus.cc:383] T 00000000000000000000000000000000 P 45e9d43ecac0401fb2c18fe5eaf089b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:13.152680 632 raft_consensus.cc:738] T 00000000000000000000000000000000 P 45e9d43ecac0401fb2c18fe5eaf089b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45e9d43ecac0401fb2c18fe5eaf089b5, State: Initialized, Role: FOLLOWER
I20250113 02:51:13.153347 632 consensus_queue.cc:260] T 00000000000000000000000000000000 P 45e9d43ecac0401fb2c18fe5eaf089b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e13f5bcdcee34225a46c6d2651e5ede9" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44217 } } peers { permanent_uuid: "6bbc3541604a455584bb32dfc2cc4efd" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 35271 } } peers { permanent_uuid: "45e9d43ecac0401fb2c18fe5eaf089b5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42067 } }
I20250113 02:51:13.154250 655 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45e9d43ecac0401fb2c18fe5eaf089b5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e13f5bcdcee34225a46c6d2651e5ede9" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44217 } } peers { permanent_uuid: "6bbc3541604a455584bb32dfc2cc4efd" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 35271 } } peers { permanent_uuid: "45e9d43ecac0401fb2c18fe5eaf089b5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42067 } } }
I20250113 02:51:13.154665 655 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45e9d43ecac0401fb2c18fe5eaf089b5 [sys.catalog]: This master's current role is: FOLLOWER
I20250113 02:51:13.155503 632 sys_catalog.cc:564] T 00000000000000000000000000000000 P 45e9d43ecac0401fb2c18fe5eaf089b5 [sys.catalog]: configured and running, proceeding with master startup.
I20250113 02:51:13.151093 424 master_runner.cc:186] Error getting master registration for 127.0.104.61:35271: OK, instance_id { permanent_uuid: "6bbc3541604a455584bb32dfc2cc4efd" instance_seqno: 1736736672626504 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20250113 02:51:13.160406 424 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250113 02:51:13.168951 666 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 45e9d43ecac0401fb2c18fe5eaf089b5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250113 02:51:13.169225 666 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250113 02:51:13.170135 563 master_runner.cc:186] Error getting master registration for 127.0.104.61:35271: OK, instance_id { permanent_uuid: "6bbc3541604a455584bb32dfc2cc4efd" instance_seqno: 1736736672626504 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20250113 02:51:13.182077 563 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250113 02:51:13.314536 586 tablet.cc:2367] T 00000000000000000000000000000000 P 45e9d43ecac0401fb2c18fe5eaf089b5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250113 02:51:13.328104 561 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42067 } attempt: 5
I20250113 02:51:13.340526 561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6bbc3541604a455584bb32dfc2cc4efd: Bootstrap starting.
I20250113 02:51:13.345527 561 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6bbc3541604a455584bb32dfc2cc4efd: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:13.346895 561 log.cc:826] T 00000000000000000000000000000000 P 6bbc3541604a455584bb32dfc2cc4efd: Log is configured to *not* fsync() on all Append() calls
I20250113 02:51:13.350066 561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6bbc3541604a455584bb32dfc2cc4efd: No bootstrap required, opened a new log
I20250113 02:51:13.359864 561 raft_consensus.cc:357] T 00000000000000000000000000000000 P 6bbc3541604a455584bb32dfc2cc4efd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e13f5bcdcee34225a46c6d2651e5ede9" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44217 } } peers { permanent_uuid: "6bbc3541604a455584bb32dfc2cc4efd" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 35271 } } peers { permanent_uuid: "45e9d43ecac0401fb2c18fe5eaf089b5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42067 } }
I20250113 02:51:13.360374 561 raft_consensus.cc:383] T 00000000000000000000000000000000 P 6bbc3541604a455584bb32dfc2cc4efd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:13.360538 561 raft_consensus.cc:738] T 00000000000000000000000000000000 P 6bbc3541604a455584bb32dfc2cc4efd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bbc3541604a455584bb32dfc2cc4efd, State: Initialized, Role: FOLLOWER
I20250113 02:51:13.361424 561 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6bbc3541604a455584bb32dfc2cc4efd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e13f5bcdcee34225a46c6d2651e5ede9" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44217 } } peers { permanent_uuid: "6bbc3541604a455584bb32dfc2cc4efd" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 35271 } } peers { permanent_uuid: "45e9d43ecac0401fb2c18fe5eaf089b5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42067 } }
I20250113 02:51:13.362514 669 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bbc3541604a455584bb32dfc2cc4efd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e13f5bcdcee34225a46c6d2651e5ede9" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44217 } } peers { permanent_uuid: "6bbc3541604a455584bb32dfc2cc4efd" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 35271 } } peers { permanent_uuid: "45e9d43ecac0401fb2c18fe5eaf089b5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42067 } } }
I20250113 02:51:13.362871 669 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bbc3541604a455584bb32dfc2cc4efd [sys.catalog]: This master's current role is: FOLLOWER
I20250113 02:51:13.364017 561 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6bbc3541604a455584bb32dfc2cc4efd [sys.catalog]: configured and running, proceeding with master startup.
W20250113 02:51:13.364143 515 tablet.cc:2367] T 00000000000000000000000000000000 P 6bbc3541604a455584bb32dfc2cc4efd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250113 02:51:13.382022 492 master_runner.cc:263] No leader master found from master 6bbc3541604a455584bb32dfc2cc4efd
I20250113 02:51:13.382208 492 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250113 02:51:13.385963 680 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 6bbc3541604a455584bb32dfc2cc4efd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250113 02:51:13.386217 680 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250113 02:51:13.442065 669 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6bbc3541604a455584bb32dfc2cc4efd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:13.442382 669 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6bbc3541604a455584bb32dfc2cc4efd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e13f5bcdcee34225a46c6d2651e5ede9" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44217 } } peers { permanent_uuid: "6bbc3541604a455584bb32dfc2cc4efd" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 35271 } } peers { permanent_uuid: "45e9d43ecac0401fb2c18fe5eaf089b5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42067 } }
I20250113 02:51:13.444654 669 leader_election.cc:290] T 00000000000000000000000000000000 P 6bbc3541604a455584bb32dfc2cc4efd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e13f5bcdcee34225a46c6d2651e5ede9 (127.0.104.62:44217), 45e9d43ecac0401fb2c18fe5eaf089b5 (127.0.104.60:42067)
I20250113 02:51:13.445885 607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6bbc3541604a455584bb32dfc2cc4efd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45e9d43ecac0401fb2c18fe5eaf089b5" is_pre_election: true
I20250113 02:51:13.446451 607 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 45e9d43ecac0401fb2c18fe5eaf089b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bbc3541604a455584bb32dfc2cc4efd in term 0.
I20250113 02:51:13.447748 465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6bbc3541604a455584bb32dfc2cc4efd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e13f5bcdcee34225a46c6d2651e5ede9" is_pre_election: true
I20250113 02:51:13.448290 465 raft_consensus.cc:2463] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bbc3541604a455584bb32dfc2cc4efd in term 0.
I20250113 02:51:13.449067 511 leader_election.cc:304] T 00000000000000000000000000000000 P 6bbc3541604a455584bb32dfc2cc4efd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bbc3541604a455584bb32dfc2cc4efd, e13f5bcdcee34225a46c6d2651e5ede9; no voters:
I20250113 02:51:13.450081 669 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 6bbc3541604a455584bb32dfc2cc4efd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 02:51:13.450240 669 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6bbc3541604a455584bb32dfc2cc4efd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 02:51:13.450354 669 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 6bbc3541604a455584bb32dfc2cc4efd [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:13.458060 669 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6bbc3541604a455584bb32dfc2cc4efd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e13f5bcdcee34225a46c6d2651e5ede9" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44217 } } peers { permanent_uuid: "6bbc3541604a455584bb32dfc2cc4efd" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 35271 } } peers { permanent_uuid: "45e9d43ecac0401fb2c18fe5eaf089b5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42067 } }
I20250113 02:51:13.461194 465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6bbc3541604a455584bb32dfc2cc4efd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e13f5bcdcee34225a46c6d2651e5ede9"
I20250113 02:51:13.462100 465 raft_consensus.cc:3054] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9 [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:13.462814 607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6bbc3541604a455584bb32dfc2cc4efd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45e9d43ecac0401fb2c18fe5eaf089b5"
I20250113 02:51:13.463045 607 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 45e9d43ecac0401fb2c18fe5eaf089b5 [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:13.465492 465 raft_consensus.cc:2463] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bbc3541604a455584bb32dfc2cc4efd in term 1.
I20250113 02:51:13.466032 511 leader_election.cc:304] T 00000000000000000000000000000000 P 6bbc3541604a455584bb32dfc2cc4efd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bbc3541604a455584bb32dfc2cc4efd, e13f5bcdcee34225a46c6d2651e5ede9; no voters:
I20250113 02:51:13.466352 607 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 45e9d43ecac0401fb2c18fe5eaf089b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bbc3541604a455584bb32dfc2cc4efd in term 1.
I20250113 02:51:13.467016 669 leader_election.cc:290] T 00000000000000000000000000000000 P 6bbc3541604a455584bb32dfc2cc4efd [CANDIDATE]: Term 1 election: Requested vote from peers e13f5bcdcee34225a46c6d2651e5ede9 (127.0.104.62:44217), 45e9d43ecac0401fb2c18fe5eaf089b5 (127.0.104.60:42067)
I20250113 02:51:13.467320 669 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 6bbc3541604a455584bb32dfc2cc4efd [term 1 FOLLOWER]: Leader election won for term 1
I20250113 02:51:13.467531 669 raft_consensus.cc:695] T 00000000000000000000000000000000 P 6bbc3541604a455584bb32dfc2cc4efd [term 1 LEADER]: Becoming Leader. State: Replica: 6bbc3541604a455584bb32dfc2cc4efd, State: Running, Role: LEADER
I20250113 02:51:13.468129 669 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6bbc3541604a455584bb32dfc2cc4efd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e13f5bcdcee34225a46c6d2651e5ede9" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44217 } } peers { permanent_uuid: "6bbc3541604a455584bb32dfc2cc4efd" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 35271 } } peers { permanent_uuid: "45e9d43ecac0401fb2c18fe5eaf089b5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42067 } }
I20250113 02:51:13.482702 682 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bbc3541604a455584bb32dfc2cc4efd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6bbc3541604a455584bb32dfc2cc4efd. Latest consensus state: current_term: 1 leader_uuid: "6bbc3541604a455584bb32dfc2cc4efd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e13f5bcdcee34225a46c6d2651e5ede9" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44217 } } peers { permanent_uuid: "6bbc3541604a455584bb32dfc2cc4efd" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 35271 } } peers { permanent_uuid: "45e9d43ecac0401fb2c18fe5eaf089b5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42067 } } }
I20250113 02:51:13.483129 682 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bbc3541604a455584bb32dfc2cc4efd [sys.catalog]: This master's current role is: LEADER
I20250113 02:51:13.490631 684 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250113 02:51:13.493562 684 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250113 02:51:13.510202 607 raft_consensus.cc:1270] T 00000000000000000000000000000000 P 45e9d43ecac0401fb2c18fe5eaf089b5 [term 1 FOLLOWER]: Refusing update from remote peer 6bbc3541604a455584bb32dfc2cc4efd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250113 02:51:13.511296 682 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6bbc3541604a455584bb32dfc2cc4efd [LEADER]: Connected to new peer: Peer: permanent_uuid: "45e9d43ecac0401fb2c18fe5eaf089b5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 02:51:13.518805 465 raft_consensus.cc:1270] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9 [term 1 FOLLOWER]: Refusing update from remote peer 6bbc3541604a455584bb32dfc2cc4efd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250113 02:51:13.519668 682 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6bbc3541604a455584bb32dfc2cc4efd [LEADER]: Connected to new peer: Peer: permanent_uuid: "e13f5bcdcee34225a46c6d2651e5ede9" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 02:51:13.537431 655 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45e9d43ecac0401fb2c18fe5eaf089b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6bbc3541604a455584bb32dfc2cc4efd. Latest consensus state: current_term: 1 leader_uuid: "6bbc3541604a455584bb32dfc2cc4efd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e13f5bcdcee34225a46c6d2651e5ede9" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44217 } } peers { permanent_uuid: "6bbc3541604a455584bb32dfc2cc4efd" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 35271 } } peers { permanent_uuid: "45e9d43ecac0401fb2c18fe5eaf089b5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42067 } } }
I20250113 02:51:13.537732 655 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45e9d43ecac0401fb2c18fe5eaf089b5 [sys.catalog]: This master's current role is: FOLLOWER
I20250113 02:51:13.540665 686 mvcc.cc:204] Tried to move back new op lower bound from 7113673414683848704 to 7113673414558203904. Current Snapshot: MvccSnapshot[applied={T|T < 7113673414683848704}]
I20250113 02:51:13.542677 682 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bbc3541604a455584bb32dfc2cc4efd [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6bbc3541604a455584bb32dfc2cc4efd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e13f5bcdcee34225a46c6d2651e5ede9" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44217 } } peers { permanent_uuid: "6bbc3541604a455584bb32dfc2cc4efd" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 35271 } } peers { permanent_uuid: "45e9d43ecac0401fb2c18fe5eaf089b5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42067 } } }
I20250113 02:51:13.542941 682 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bbc3541604a455584bb32dfc2cc4efd [sys.catalog]: This master's current role is: LEADER
I20250113 02:51:13.543128 685 mvcc.cc:204] Tried to move back new op lower bound from 7113673414683848704 to 7113673414558203904. Current Snapshot: MvccSnapshot[applied={T|T < 7113673414683848704}]
I20250113 02:51:13.550308 655 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45e9d43ecac0401fb2c18fe5eaf089b5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6bbc3541604a455584bb32dfc2cc4efd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e13f5bcdcee34225a46c6d2651e5ede9" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44217 } } peers { permanent_uuid: "6bbc3541604a455584bb32dfc2cc4efd" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 35271 } } peers { permanent_uuid: "45e9d43ecac0401fb2c18fe5eaf089b5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42067 } } }
I20250113 02:51:13.550606 655 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45e9d43ecac0401fb2c18fe5eaf089b5 [sys.catalog]: This master's current role is: FOLLOWER
I20250113 02:51:13.552223 684 catalog_manager.cc:1348] Generated new cluster ID: 00860a399ece46318a55d1901f0159c4
I20250113 02:51:13.552332 684 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250113 02:51:13.553388 641 sys_catalog.cc:455] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6bbc3541604a455584bb32dfc2cc4efd. Latest consensus state: current_term: 1 leader_uuid: "6bbc3541604a455584bb32dfc2cc4efd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e13f5bcdcee34225a46c6d2651e5ede9" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44217 } } peers { permanent_uuid: "6bbc3541604a455584bb32dfc2cc4efd" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 35271 } } peers { permanent_uuid: "45e9d43ecac0401fb2c18fe5eaf089b5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42067 } } }
I20250113 02:51:13.553706 641 sys_catalog.cc:458] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9 [sys.catalog]: This master's current role is: FOLLOWER
I20250113 02:51:13.554957 669 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bbc3541604a455584bb32dfc2cc4efd [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6bbc3541604a455584bb32dfc2cc4efd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e13f5bcdcee34225a46c6d2651e5ede9" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44217 } } peers { permanent_uuid: "6bbc3541604a455584bb32dfc2cc4efd" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 35271 } } peers { permanent_uuid: "45e9d43ecac0401fb2c18fe5eaf089b5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42067 } } }
I20250113 02:51:13.555245 669 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bbc3541604a455584bb32dfc2cc4efd [sys.catalog]: This master's current role is: LEADER
I20250113 02:51:13.556844 687 mvcc.cc:204] Tried to move back new op lower bound from 7113673414683848704 to 7113673414558203904. Current Snapshot: MvccSnapshot[applied={T|T < 7113673414683848704}]
I20250113 02:51:13.570477 641 sys_catalog.cc:455] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6bbc3541604a455584bb32dfc2cc4efd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e13f5bcdcee34225a46c6d2651e5ede9" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44217 } } peers { permanent_uuid: "6bbc3541604a455584bb32dfc2cc4efd" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 35271 } } peers { permanent_uuid: "45e9d43ecac0401fb2c18fe5eaf089b5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42067 } } }
I20250113 02:51:13.570721 641 sys_catalog.cc:458] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9 [sys.catalog]: This master's current role is: FOLLOWER
I20250113 02:51:13.605331 684 catalog_manager.cc:1371] Generated new certificate authority record
I20250113 02:51:13.607539 684 catalog_manager.cc:1505] Loading token signing keys...
I20250113 02:51:13.648591 684 catalog_manager.cc:5899] T 00000000000000000000000000000000 P 6bbc3541604a455584bb32dfc2cc4efd: Generated new TSK 0
I20250113 02:51:13.649644 684 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250113 02:51:13.673432 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskXJrxug/build/asan/bin/kudu
/tmp/dist-test-taskXJrxug/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.1:0
--local_ip_for_outbound_sockets=127.0.104.1
--webserver_interface=127.0.104.1
--webserver_port=0
--tserver_master_addrs=127.0.104.62:44217,127.0.104.61:35271,127.0.104.60:42067
--builtin_ntp_servers=127.0.104.20:45573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
W20250113 02:51:14.018772 693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250113 02:51:14.019078 693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250113 02:51:14.019208 693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250113 02:51:14.027283 693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250113 02:51:14.027531 693 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.1
I20250113 02:51:14.037067 693 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:45573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.104.1
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.0.104.62:44217,127.0.104.61:35271,127.0.104.60:42067
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.1
--log_dir=/tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 13 Jan 2025 02:43:23 UTC on 24a791456cd2
build id 4151
ASAN enabled
I20250113 02:51:14.038727 693 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250113 02:51:14.040405 693 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 02:51:14.051436 698 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 02:51:14.051652 699 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 02:51:14.051723 701 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 02:51:14.052707 693 server_base.cc:1034] running on GCE node
I20250113 02:51:14.053233 693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250113 02:51:14.054090 693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250113 02:51:14.055765 693 hybrid_clock.cc:648] HybridClock initialized: now 1736736674055246 us; error 38 us; skew 500 ppm
I20250113 02:51:14.056073 693 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250113 02:51:14.058281 693 webserver.cc:458] Webserver started at http://127.0.104.1:35381/ using document root <none> and password file <none>
I20250113 02:51:14.058974 693 fs_manager.cc:362] Metadata directory not provided
I20250113 02:51:14.059089 693 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 02:51:14.059356 693 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250113 02:51:14.061980 693 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/data/instance:
uuid: "1186fa869eb14812832fa6562531f196"
format_stamp: "Formatted at 2025-01-13 02:51:14 on dist-test-slave-v0t5"
I20250113 02:51:14.062692 693 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/instance:
uuid: "1186fa869eb14812832fa6562531f196"
format_stamp: "Formatted at 2025-01-13 02:51:14 on dist-test-slave-v0t5"
I20250113 02:51:14.067788 693 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20250113 02:51:14.071012 707 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 02:51:14.072300 693 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250113 02:51:14.072448 693 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/data,/tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal
uuid: "1186fa869eb14812832fa6562531f196"
format_stamp: "Formatted at 2025-01-13 02:51:14 on dist-test-slave-v0t5"
I20250113 02:51:14.072613 693 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 02:51:14.085871 693 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 02:51:14.086678 693 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250113 02:51:14.086951 693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 02:51:14.088001 693 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250113 02:51:14.089521 693 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250113 02:51:14.089597 693 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250113 02:51:14.089717 693 ts_tablet_manager.cc:610] Registered 0 tablets
I20250113 02:51:14.089776 693 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250113 02:51:14.126789 693 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.1:40095
I20250113 02:51:14.126814 819 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.1:40095 every 8 connection(s)
I20250113 02:51:14.128144 693 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/data/info.pb
I20250113 02:51:14.130786 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskXJrxug/build/asan/bin/kudu as pid 693
I20250113 02:51:14.130963 416 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/instance
I20250113 02:51:14.137621 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskXJrxug/build/asan/bin/kudu
/tmp/dist-test-taskXJrxug/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.2:0
--local_ip_for_outbound_sockets=127.0.104.2
--webserver_interface=127.0.104.2
--webserver_port=0
--tserver_master_addrs=127.0.104.62:44217,127.0.104.61:35271,127.0.104.60:42067
--builtin_ntp_servers=127.0.104.20:45573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20250113 02:51:14.151898 653 catalog_manager.cc:1260] Loaded cluster ID: 00860a399ece46318a55d1901f0159c4
I20250113 02:51:14.152824 653 catalog_manager.cc:1553] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: loading cluster ID for follower catalog manager: success
I20250113 02:51:14.156589 820 heartbeater.cc:346] Connected to a master server at 127.0.104.60:42067
I20250113 02:51:14.157028 820 heartbeater.cc:463] Registering TS with master...
I20250113 02:51:14.158349 820 heartbeater.cc:510] Master 127.0.104.60:42067 requested a full tablet report, sending...
I20250113 02:51:14.161020 597 ts_manager.cc:194] Registered new tserver with Master: 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
I20250113 02:51:14.162652 653 catalog_manager.cc:1575] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: acquiring CA information for follower catalog manager: success
I20250113 02:51:14.165774 653 catalog_manager.cc:1603] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250113 02:51:14.162119 831 raft_consensus.cc:491] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6bbc3541604a455584bb32dfc2cc4efd)
I20250113 02:51:14.166373 831 raft_consensus.cc:513] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e13f5bcdcee34225a46c6d2651e5ede9" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44217 } } peers { permanent_uuid: "6bbc3541604a455584bb32dfc2cc4efd" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 35271 } } peers { permanent_uuid: "45e9d43ecac0401fb2c18fe5eaf089b5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42067 } }
I20250113 02:51:14.168812 536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e13f5bcdcee34225a46c6d2651e5ede9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6bbc3541604a455584bb32dfc2cc4efd" is_pre_election: true
I20250113 02:51:14.170127 832 raft_consensus.cc:491] T 00000000000000000000000000000000 P 45e9d43ecac0401fb2c18fe5eaf089b5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6bbc3541604a455584bb32dfc2cc4efd)
I20250113 02:51:14.170349 832 raft_consensus.cc:513] T 00000000000000000000000000000000 P 45e9d43ecac0401fb2c18fe5eaf089b5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e13f5bcdcee34225a46c6d2651e5ede9" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44217 } } peers { permanent_uuid: "6bbc3541604a455584bb32dfc2cc4efd" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 35271 } } peers { permanent_uuid: "45e9d43ecac0401fb2c18fe5eaf089b5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42067 } }
I20250113 02:51:14.170590 831 leader_election.cc:290] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6bbc3541604a455584bb32dfc2cc4efd (127.0.104.61:35271), 45e9d43ecac0401fb2c18fe5eaf089b5 (127.0.104.60:42067)
I20250113 02:51:14.170909 607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e13f5bcdcee34225a46c6d2651e5ede9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "45e9d43ecac0401fb2c18fe5eaf089b5" is_pre_election: true
I20250113 02:51:14.171089 607 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 45e9d43ecac0401fb2c18fe5eaf089b5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e13f5bcdcee34225a46c6d2651e5ede9 in term 1.
I20250113 02:51:14.171527 439 leader_election.cc:304] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 45e9d43ecac0401fb2c18fe5eaf089b5, e13f5bcdcee34225a46c6d2651e5ede9; no voters: 6bbc3541604a455584bb32dfc2cc4efd
I20250113 02:51:14.171725 666 catalog_manager.cc:1260] Loaded cluster ID: 00860a399ece46318a55d1901f0159c4
I20250113 02:51:14.171806 666 catalog_manager.cc:1553] T 00000000000000000000000000000000 P 45e9d43ecac0401fb2c18fe5eaf089b5: loading cluster ID for follower catalog manager: success
I20250113 02:51:14.172150 831 raft_consensus.cc:2798] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250113 02:51:14.172255 831 raft_consensus.cc:491] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6bbc3541604a455584bb32dfc2cc4efd)
I20250113 02:51:14.172334 831 raft_consensus.cc:3054] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9 [term 1 FOLLOWER]: Advancing to term 2
I20250113 02:51:14.175302 831 raft_consensus.cc:513] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e13f5bcdcee34225a46c6d2651e5ede9" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44217 } } peers { permanent_uuid: "6bbc3541604a455584bb32dfc2cc4efd" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 35271 } } peers { permanent_uuid: "45e9d43ecac0401fb2c18fe5eaf089b5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42067 } }
I20250113 02:51:14.176017 536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "45e9d43ecac0401fb2c18fe5eaf089b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6bbc3541604a455584bb32dfc2cc4efd" is_pre_election: true
I20250113 02:51:14.176159 666 catalog_manager.cc:1575] T 00000000000000000000000000000000 P 45e9d43ecac0401fb2c18fe5eaf089b5: acquiring CA information for follower catalog manager: success
I20250113 02:51:14.177008 536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e13f5bcdcee34225a46c6d2651e5ede9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6bbc3541604a455584bb32dfc2cc4efd"
I20250113 02:51:14.177178 607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e13f5bcdcee34225a46c6d2651e5ede9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "45e9d43ecac0401fb2c18fe5eaf089b5"
I20250113 02:51:14.177362 607 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 45e9d43ecac0401fb2c18fe5eaf089b5 [term 1 FOLLOWER]: Advancing to term 2
I20250113 02:51:14.177767 832 leader_election.cc:290] T 00000000000000000000000000000000 P 45e9d43ecac0401fb2c18fe5eaf089b5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e13f5bcdcee34225a46c6d2651e5ede9 (127.0.104.62:44217), 6bbc3541604a455584bb32dfc2cc4efd (127.0.104.61:35271)
I20250113 02:51:14.178212 464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "45e9d43ecac0401fb2c18fe5eaf089b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e13f5bcdcee34225a46c6d2651e5ede9" is_pre_election: true
I20250113 02:51:14.178475 464 raft_consensus.cc:2388] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 45e9d43ecac0401fb2c18fe5eaf089b5 in current term 2: Already voted for candidate e13f5bcdcee34225a46c6d2651e5ede9 in this term.
I20250113 02:51:14.178678 666 catalog_manager.cc:1603] T 00000000000000000000000000000000 P 45e9d43ecac0401fb2c18fe5eaf089b5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250113 02:51:14.178941 582 leader_election.cc:304] T 00000000000000000000000000000000 P 45e9d43ecac0401fb2c18fe5eaf089b5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 45e9d43ecac0401fb2c18fe5eaf089b5; no voters: 6bbc3541604a455584bb32dfc2cc4efd, e13f5bcdcee34225a46c6d2651e5ede9
I20250113 02:51:14.180238 607 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 45e9d43ecac0401fb2c18fe5eaf089b5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e13f5bcdcee34225a46c6d2651e5ede9 in term 2.
I20250113 02:51:14.180656 439 leader_election.cc:304] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 45e9d43ecac0401fb2c18fe5eaf089b5, e13f5bcdcee34225a46c6d2651e5ede9; no voters: 6bbc3541604a455584bb32dfc2cc4efd
I20250113 02:51:14.183434 832 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 45e9d43ecac0401fb2c18fe5eaf089b5 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250113 02:51:14.183653 464 raft_consensus.cc:1235] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9 [term 2 FOLLOWER]: Rejecting Update request from peer 6bbc3541604a455584bb32dfc2cc4efd for earlier term 1. Current term is 2. Ops: []
I20250113 02:51:14.185391 682 consensus_queue.cc:1046] T 00000000000000000000000000000000 P 6bbc3541604a455584bb32dfc2cc4efd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e13f5bcdcee34225a46c6d2651e5ede9" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44217 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250113 02:51:14.187032 831 leader_election.cc:290] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9 [CANDIDATE]: Term 2 election: Requested vote from peers 6bbc3541604a455584bb32dfc2cc4efd (127.0.104.61:35271), 45e9d43ecac0401fb2c18fe5eaf089b5 (127.0.104.60:42067)
I20250113 02:51:14.187094 833 raft_consensus.cc:2798] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9 [term 2 FOLLOWER]: Leader election won for term 2
I20250113 02:51:14.188891 822 heartbeater.cc:346] Connected to a master server at 127.0.104.61:35271
I20250113 02:51:14.188915 821 heartbeater.cc:346] Connected to a master server at 127.0.104.62:44217
I20250113 02:51:14.189080 821 heartbeater.cc:463] Registering TS with master...
I20250113 02:51:14.189085 822 heartbeater.cc:463] Registering TS with master...
I20250113 02:51:14.190462 822 heartbeater.cc:510] Master 127.0.104.61:35271 requested a full tablet report, sending...
I20250113 02:51:14.190842 821 heartbeater.cc:510] Master 127.0.104.62:44217 requested a full tablet report, sending...
I20250113 02:51:14.192256 682 raft_consensus.cc:3049] T 00000000000000000000000000000000 P 6bbc3541604a455584bb32dfc2cc4efd [term 1 LEADER]: Stepping down as leader of term 1
I20250113 02:51:14.192389 682 raft_consensus.cc:738] T 00000000000000000000000000000000 P 6bbc3541604a455584bb32dfc2cc4efd [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6bbc3541604a455584bb32dfc2cc4efd, State: Running, Role: LEADER
I20250113 02:51:14.192714 682 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6bbc3541604a455584bb32dfc2cc4efd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e13f5bcdcee34225a46c6d2651e5ede9" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44217 } } peers { permanent_uuid: "6bbc3541604a455584bb32dfc2cc4efd" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 35271 } } peers { permanent_uuid: "45e9d43ecac0401fb2c18fe5eaf089b5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42067 } }
I20250113 02:51:14.193060 833 raft_consensus.cc:695] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9 [term 2 LEADER]: Becoming Leader. State: Replica: e13f5bcdcee34225a46c6d2651e5ede9, State: Running, Role: LEADER
I20250113 02:51:14.193261 682 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 6bbc3541604a455584bb32dfc2cc4efd [term 1 FOLLOWER]: Advancing to term 2
I20250113 02:51:14.193712 833 consensus_queue.cc:237] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e13f5bcdcee34225a46c6d2651e5ede9" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44217 } } peers { permanent_uuid: "6bbc3541604a455584bb32dfc2cc4efd" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 35271 } } peers { permanent_uuid: "45e9d43ecac0401fb2c18fe5eaf089b5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42067 } }
I20250113 02:51:14.195791 831 sys_catalog.cc:455] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e13f5bcdcee34225a46c6d2651e5ede9. Latest consensus state: current_term: 2 leader_uuid: "e13f5bcdcee34225a46c6d2651e5ede9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e13f5bcdcee34225a46c6d2651e5ede9" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44217 } } peers { permanent_uuid: "6bbc3541604a455584bb32dfc2cc4efd" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 35271 } } peers { permanent_uuid: "45e9d43ecac0401fb2c18fe5eaf089b5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42067 } } }
I20250113 02:51:14.196100 831 sys_catalog.cc:458] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9 [sys.catalog]: This master's current role is: LEADER
I20250113 02:51:14.197783 526 ts_manager.cc:194] Registered new tserver with Master: 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
I20250113 02:51:14.199550 455 ts_manager.cc:194] Registered new tserver with Master: 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
I20250113 02:51:14.200927 836 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250113 02:51:14.203747 836 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250113 02:51:14.204576 836 catalog_manager.cc:1260] Loaded cluster ID: 00860a399ece46318a55d1901f0159c4
I20250113 02:51:14.204643 836 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250113 02:51:14.205644 836 catalog_manager.cc:1505] Loading token signing keys...
I20250113 02:51:14.206379 836 catalog_manager.cc:5910] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Loaded TSK: 0
I20250113 02:51:14.207123 836 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250113 02:51:14.388377 680 catalog_manager.cc:1603] T 00000000000000000000000000000000 P 6bbc3541604a455584bb32dfc2cc4efd: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20250113 02:51:14.523294 830 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250113 02:51:14.523655 830 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250113 02:51:14.523809 830 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250113 02:51:14.531694 830 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250113 02:51:14.531955 830 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.2
I20250113 02:51:14.541641 830 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:45573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.104.2
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.0.104.62:44217,127.0.104.61:35271,127.0.104.60:42067
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.2
--log_dir=/tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 13 Jan 2025 02:43:23 UTC on 24a791456cd2
build id 4151
ASAN enabled
I20250113 02:51:14.543206 830 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250113 02:51:14.544879 830 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 02:51:14.556617 842 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 02:51:14.556777 844 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 02:51:14.556951 841 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 02:51:14.558351 830 server_base.cc:1034] running on GCE node
I20250113 02:51:14.558889 830 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250113 02:51:14.559737 830 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250113 02:51:14.560956 830 hybrid_clock.cc:648] HybridClock initialized: now 1736736674560884 us; error 46 us; skew 500 ppm
I20250113 02:51:14.561239 830 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250113 02:51:14.563443 830 webserver.cc:458] Webserver started at http://127.0.104.2:40625/ using document root <none> and password file <none>
I20250113 02:51:14.564179 830 fs_manager.cc:362] Metadata directory not provided
I20250113 02:51:14.564296 830 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 02:51:14.564570 830 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250113 02:51:14.567167 830 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/data/instance:
uuid: "df69f48656b44bb890f76587ee69cc0f"
format_stamp: "Formatted at 2025-01-13 02:51:14 on dist-test-slave-v0t5"
I20250113 02:51:14.567993 830 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/instance:
uuid: "df69f48656b44bb890f76587ee69cc0f"
format_stamp: "Formatted at 2025-01-13 02:51:14 on dist-test-slave-v0t5"
I20250113 02:51:14.573587 830 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20250113 02:51:14.576843 850 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 02:51:14.578250 830 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20250113 02:51:14.578385 830 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/data,/tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal
uuid: "df69f48656b44bb890f76587ee69cc0f"
format_stamp: "Formatted at 2025-01-13 02:51:14 on dist-test-slave-v0t5"
I20250113 02:51:14.578547 830 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 02:51:14.587281 682 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6bbc3541604a455584bb32dfc2cc4efd [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:14.587472 682 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6bbc3541604a455584bb32dfc2cc4efd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e13f5bcdcee34225a46c6d2651e5ede9" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44217 } } peers { permanent_uuid: "6bbc3541604a455584bb32dfc2cc4efd" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 35271 } } peers { permanent_uuid: "45e9d43ecac0401fb2c18fe5eaf089b5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42067 } }
I20250113 02:51:14.588156 682 leader_election.cc:290] T 00000000000000000000000000000000 P 6bbc3541604a455584bb32dfc2cc4efd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e13f5bcdcee34225a46c6d2651e5ede9 (127.0.104.62:44217), 45e9d43ecac0401fb2c18fe5eaf089b5 (127.0.104.60:42067)
I20250113 02:51:14.588663 464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6bbc3541604a455584bb32dfc2cc4efd" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e13f5bcdcee34225a46c6d2651e5ede9" is_pre_election: true
I20250113 02:51:14.588704 607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6bbc3541604a455584bb32dfc2cc4efd" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "45e9d43ecac0401fb2c18fe5eaf089b5" is_pre_election: true
I20250113 02:51:14.588963 607 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 45e9d43ecac0401fb2c18fe5eaf089b5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bbc3541604a455584bb32dfc2cc4efd in term 2.
I20250113 02:51:14.589342 510 leader_election.cc:304] T 00000000000000000000000000000000 P 6bbc3541604a455584bb32dfc2cc4efd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 45e9d43ecac0401fb2c18fe5eaf089b5, 6bbc3541604a455584bb32dfc2cc4efd; no voters: e13f5bcdcee34225a46c6d2651e5ede9
I20250113 02:51:14.589625 682 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 6bbc3541604a455584bb32dfc2cc4efd [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250113 02:51:14.589715 682 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6bbc3541604a455584bb32dfc2cc4efd [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 02:51:14.589850 682 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 6bbc3541604a455584bb32dfc2cc4efd [term 2 FOLLOWER]: Advancing to term 3
I20250113 02:51:14.591638 682 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6bbc3541604a455584bb32dfc2cc4efd [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e13f5bcdcee34225a46c6d2651e5ede9" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44217 } } peers { permanent_uuid: "6bbc3541604a455584bb32dfc2cc4efd" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 35271 } } peers { permanent_uuid: "45e9d43ecac0401fb2c18fe5eaf089b5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42067 } }
I20250113 02:51:14.592263 682 leader_election.cc:290] T 00000000000000000000000000000000 P 6bbc3541604a455584bb32dfc2cc4efd [CANDIDATE]: Term 3 election: Requested vote from peers e13f5bcdcee34225a46c6d2651e5ede9 (127.0.104.62:44217), 45e9d43ecac0401fb2c18fe5eaf089b5 (127.0.104.60:42067)
I20250113 02:51:14.592702 464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6bbc3541604a455584bb32dfc2cc4efd" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e13f5bcdcee34225a46c6d2651e5ede9"
I20250113 02:51:14.592718 607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6bbc3541604a455584bb32dfc2cc4efd" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "45e9d43ecac0401fb2c18fe5eaf089b5"
I20250113 02:51:14.592955 607 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 45e9d43ecac0401fb2c18fe5eaf089b5 [term 2 FOLLOWER]: Advancing to term 3
I20250113 02:51:14.596093 607 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 45e9d43ecac0401fb2c18fe5eaf089b5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bbc3541604a455584bb32dfc2cc4efd in term 3.
I20250113 02:51:14.596498 510 leader_election.cc:304] T 00000000000000000000000000000000 P 6bbc3541604a455584bb32dfc2cc4efd [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 45e9d43ecac0401fb2c18fe5eaf089b5, 6bbc3541604a455584bb32dfc2cc4efd; no voters: e13f5bcdcee34225a46c6d2651e5ede9
I20250113 02:51:14.596843 682 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 6bbc3541604a455584bb32dfc2cc4efd [term 3 FOLLOWER]: Leader election won for term 3
I20250113 02:51:14.597044 682 raft_consensus.cc:695] T 00000000000000000000000000000000 P 6bbc3541604a455584bb32dfc2cc4efd [term 3 LEADER]: Becoming Leader. State: Replica: 6bbc3541604a455584bb32dfc2cc4efd, State: Running, Role: LEADER
I20250113 02:51:14.597298 682 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6bbc3541604a455584bb32dfc2cc4efd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e13f5bcdcee34225a46c6d2651e5ede9" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44217 } } peers { permanent_uuid: "6bbc3541604a455584bb32dfc2cc4efd" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 35271 } } peers { permanent_uuid: "45e9d43ecac0401fb2c18fe5eaf089b5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42067 } }
I20250113 02:51:14.598397 834 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bbc3541604a455584bb32dfc2cc4efd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6bbc3541604a455584bb32dfc2cc4efd. Latest consensus state: current_term: 3 leader_uuid: "6bbc3541604a455584bb32dfc2cc4efd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e13f5bcdcee34225a46c6d2651e5ede9" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44217 } } peers { permanent_uuid: "6bbc3541604a455584bb32dfc2cc4efd" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 35271 } } peers { permanent_uuid: "45e9d43ecac0401fb2c18fe5eaf089b5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42067 } } }
I20250113 02:51:14.598634 834 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bbc3541604a455584bb32dfc2cc4efd [sys.catalog]: This master's current role is: LEADER
I20250113 02:51:14.599222 855 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250113 02:51:14.600978 855 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250113 02:51:14.601835 855 catalog_manager.cc:1260] Loaded cluster ID: 00860a399ece46318a55d1901f0159c4
I20250113 02:51:14.601918 855 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250113 02:51:14.603142 855 catalog_manager.cc:1505] Loading token signing keys...
I20250113 02:51:14.604102 855 catalog_manager.cc:5910] T 00000000000000000000000000000000 P 6bbc3541604a455584bb32dfc2cc4efd: Loaded TSK: 0
I20250113 02:51:14.604861 855 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250113 02:51:14.606316 830 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 02:51:14.607122 830 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250113 02:51:14.607381 830 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 02:51:14.608321 830 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250113 02:51:14.609786 830 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250113 02:51:14.609864 830 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250113 02:51:14.609983 830 ts_tablet_manager.cc:610] Registered 0 tablets
I20250113 02:51:14.610038 830 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250113 02:51:14.644830 830 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.2:38467
I20250113 02:51:14.644860 963 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.2:38467 every 8 connection(s)
I20250113 02:51:14.646561 830 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/data/info.pb
I20250113 02:51:14.651229 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskXJrxug/build/asan/bin/kudu as pid 830
I20250113 02:51:14.651432 416 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/instance
I20250113 02:51:14.658193 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskXJrxug/build/asan/bin/kudu
/tmp/dist-test-taskXJrxug/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.3:0
--local_ip_for_outbound_sockets=127.0.104.3
--webserver_interface=127.0.104.3
--webserver_port=0
--tserver_master_addrs=127.0.104.62:44217,127.0.104.61:35271,127.0.104.60:42067
--builtin_ntp_servers=127.0.104.20:45573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20250113 02:51:14.677249 965 heartbeater.cc:346] Connected to a master server at 127.0.104.62:44217
I20250113 02:51:14.677640 965 heartbeater.cc:463] Registering TS with master...
I20250113 02:51:14.679214 965 heartbeater.cc:510] Master 127.0.104.62:44217 requested a full tablet report, sending...
I20250113 02:51:14.680640 455 ts_manager.cc:194] Registered new tserver with Master: df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:14.681461 964 heartbeater.cc:346] Connected to a master server at 127.0.104.60:42067
I20250113 02:51:14.681594 964 heartbeater.cc:463] Registering TS with master...
I20250113 02:51:14.682121 964 heartbeater.cc:510] Master 127.0.104.60:42067 requested a full tablet report, sending...
I20250113 02:51:14.682910 455 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.104.2:39945
I20250113 02:51:14.683211 597 ts_manager.cc:194] Registered new tserver with Master: df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:14.683739 966 heartbeater.cc:346] Connected to a master server at 127.0.104.61:35271
I20250113 02:51:14.683909 966 heartbeater.cc:463] Registering TS with master...
I20250113 02:51:14.684494 966 heartbeater.cc:510] Master 127.0.104.61:35271 requested a full tablet report, sending...
I20250113 02:51:14.685490 526 ts_manager.cc:194] Registered new tserver with Master: df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:14.687125 526 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.104.2:44295
I20250113 02:51:14.702338 526 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.104.1:44085
I20250113 02:51:14.702617 455 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.104.1:45059
I20250113 02:51:14.747493 536 raft_consensus.cc:1235] T 00000000000000000000000000000000 P 6bbc3541604a455584bb32dfc2cc4efd [term 3 LEADER]: Rejecting Update request from peer e13f5bcdcee34225a46c6d2651e5ede9 for earlier term 2. Current term is 3. Ops: []
I20250113 02:51:14.748247 972 consensus_queue.cc:1046] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6bbc3541604a455584bb32dfc2cc4efd" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 35271 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250113 02:51:14.748881 972 consensus_queue.cc:1225] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9 [LEADER]: Peer 6bbc3541604a455584bb32dfc2cc4efd log is divergent from this leader: its last log entry 3.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20250113 02:51:14.749078 973 raft_consensus.cc:3049] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9 [term 2 LEADER]: Stepping down as leader of term 2
I20250113 02:51:14.749150 973 raft_consensus.cc:738] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: e13f5bcdcee34225a46c6d2651e5ede9, State: Running, Role: LEADER
I20250113 02:51:14.749469 973 consensus_queue.cc:260] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e13f5bcdcee34225a46c6d2651e5ede9" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44217 } } peers { permanent_uuid: "6bbc3541604a455584bb32dfc2cc4efd" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 35271 } } peers { permanent_uuid: "45e9d43ecac0401fb2c18fe5eaf089b5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42067 } }
I20250113 02:51:14.749862 973 raft_consensus.cc:3054] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9 [term 2 FOLLOWER]: Advancing to term 3
I20250113 02:51:14.995697 464 raft_consensus.cc:1270] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9 [term 3 FOLLOWER]: Refusing update from remote peer 6bbc3541604a455584bb32dfc2cc4efd: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 5. (term mismatch)
I20250113 02:51:14.995836 464 pending_rounds.cc:77] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Aborting all ops after (but not including) 4
I20250113 02:51:14.995962 464 pending_rounds.cc:99] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Aborting uncommitted NO_OP operation due to leader change: 2.5
I20250113 02:51:14.996096 464 raft_consensus.cc:2883] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250113 02:51:14.996768 834 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6bbc3541604a455584bb32dfc2cc4efd [LEADER]: Connected to new peer: Peer: permanent_uuid: "e13f5bcdcee34225a46c6d2651e5ede9" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250113 02:51:14.997987 973 sys_catalog.cc:455] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6bbc3541604a455584bb32dfc2cc4efd. Latest consensus state: current_term: 3 leader_uuid: "6bbc3541604a455584bb32dfc2cc4efd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e13f5bcdcee34225a46c6d2651e5ede9" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44217 } } peers { permanent_uuid: "6bbc3541604a455584bb32dfc2cc4efd" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 35271 } } peers { permanent_uuid: "45e9d43ecac0401fb2c18fe5eaf089b5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42067 } } }
I20250113 02:51:14.998239 973 sys_catalog.cc:458] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9 [sys.catalog]: This master's current role is: FOLLOWER
I20250113 02:51:14.999413 682 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bbc3541604a455584bb32dfc2cc4efd [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "6bbc3541604a455584bb32dfc2cc4efd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e13f5bcdcee34225a46c6d2651e5ede9" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44217 } } peers { permanent_uuid: "6bbc3541604a455584bb32dfc2cc4efd" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 35271 } } peers { permanent_uuid: "45e9d43ecac0401fb2c18fe5eaf089b5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42067 } } }
I20250113 02:51:14.999661 682 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bbc3541604a455584bb32dfc2cc4efd [sys.catalog]: This master's current role is: LEADER
I20250113 02:51:15.000514 607 raft_consensus.cc:1270] T 00000000000000000000000000000000 P 45e9d43ecac0401fb2c18fe5eaf089b5 [term 3 FOLLOWER]: Refusing update from remote peer 6bbc3541604a455584bb32dfc2cc4efd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20250113 02:51:15.001240 682 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6bbc3541604a455584bb32dfc2cc4efd [LEADER]: Connected to new peer: Peer: permanent_uuid: "45e9d43ecac0401fb2c18fe5eaf089b5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250113 02:51:15.011998 973 sys_catalog.cc:455] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "6bbc3541604a455584bb32dfc2cc4efd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e13f5bcdcee34225a46c6d2651e5ede9" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44217 } } peers { permanent_uuid: "6bbc3541604a455584bb32dfc2cc4efd" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 35271 } } peers { permanent_uuid: "45e9d43ecac0401fb2c18fe5eaf089b5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42067 } } }
I20250113 02:51:15.012382 973 sys_catalog.cc:458] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9 [sys.catalog]: This master's current role is: FOLLOWER
I20250113 02:51:15.015069 983 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45e9d43ecac0401fb2c18fe5eaf089b5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "6bbc3541604a455584bb32dfc2cc4efd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e13f5bcdcee34225a46c6d2651e5ede9" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44217 } } peers { permanent_uuid: "6bbc3541604a455584bb32dfc2cc4efd" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 35271 } } peers { permanent_uuid: "45e9d43ecac0401fb2c18fe5eaf089b5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42067 } } }
I20250113 02:51:15.015347 983 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45e9d43ecac0401fb2c18fe5eaf089b5 [sys.catalog]: This master's current role is: FOLLOWER
I20250113 02:51:15.015727 977 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45e9d43ecac0401fb2c18fe5eaf089b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6bbc3541604a455584bb32dfc2cc4efd. Latest consensus state: current_term: 3 leader_uuid: "6bbc3541604a455584bb32dfc2cc4efd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e13f5bcdcee34225a46c6d2651e5ede9" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44217 } } peers { permanent_uuid: "6bbc3541604a455584bb32dfc2cc4efd" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 35271 } } peers { permanent_uuid: "45e9d43ecac0401fb2c18fe5eaf089b5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42067 } } }
I20250113 02:51:15.015900 977 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45e9d43ecac0401fb2c18fe5eaf089b5 [sys.catalog]: This master's current role is: FOLLOWER
W20250113 02:51:15.022076 971 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250113 02:51:15.022347 971 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250113 02:51:15.022505 971 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250113 02:51:15.030237 971 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250113 02:51:15.030481 971 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.3
I20250113 02:51:15.040495 971 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:45573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.104.3
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.0.104.62:44217,127.0.104.61:35271,127.0.104.60:42067
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.3
--log_dir=/tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 13 Jan 2025 02:43:23 UTC on 24a791456cd2
build id 4151
ASAN enabled
I20250113 02:51:15.042122 971 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250113 02:51:15.043853 971 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 02:51:15.055759 986 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 02:51:15.055800 988 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 02:51:15.055812 985 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 02:51:15.056561 971 server_base.cc:1034] running on GCE node
I20250113 02:51:15.057135 971 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250113 02:51:15.058010 971 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250113 02:51:15.059211 971 hybrid_clock.cc:648] HybridClock initialized: now 1736736675059149 us; error 34 us; skew 500 ppm
I20250113 02:51:15.059499 971 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250113 02:51:15.061661 971 webserver.cc:458] Webserver started at http://127.0.104.3:34429/ using document root <none> and password file <none>
I20250113 02:51:15.062371 971 fs_manager.cc:362] Metadata directory not provided
I20250113 02:51:15.062484 971 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 02:51:15.062755 971 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250113 02:51:15.065322 971 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/data/instance:
uuid: "4216a7f853d642b7bc3e5cff458f3aca"
format_stamp: "Formatted at 2025-01-13 02:51:15 on dist-test-slave-v0t5"
I20250113 02:51:15.066015 971 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/instance:
uuid: "4216a7f853d642b7bc3e5cff458f3aca"
format_stamp: "Formatted at 2025-01-13 02:51:15 on dist-test-slave-v0t5"
I20250113 02:51:15.071384 971 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20250113 02:51:15.074548 994 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 02:51:15.075831 971 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20250113 02:51:15.076023 971 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/data,/tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal
uuid: "4216a7f853d642b7bc3e5cff458f3aca"
format_stamp: "Formatted at 2025-01-13 02:51:15 on dist-test-slave-v0t5"
I20250113 02:51:15.076191 971 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 02:51:15.105629 971 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 02:51:15.106474 971 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250113 02:51:15.106750 971 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 02:51:15.107664 971 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250113 02:51:15.109156 971 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250113 02:51:15.109233 971 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250113 02:51:15.109354 971 ts_tablet_manager.cc:610] Registered 0 tablets
I20250113 02:51:15.109406 971 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250113 02:51:15.143730 971 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.3:42587
I20250113 02:51:15.143770 1106 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.3:42587 every 8 connection(s)
I20250113 02:51:15.145555 971 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/data/info.pb
I20250113 02:51:15.146389 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskXJrxug/build/asan/bin/kudu as pid 971
I20250113 02:51:15.146577 416 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/instance
I20250113 02:51:15.162686 1109 heartbeater.cc:346] Connected to a master server at 127.0.104.61:35271
I20250113 02:51:15.162801 1108 heartbeater.cc:346] Connected to a master server at 127.0.104.62:44217
I20250113 02:51:15.162897 1107 heartbeater.cc:346] Connected to a master server at 127.0.104.60:42067
I20250113 02:51:15.163064 1109 heartbeater.cc:463] Registering TS with master...
I20250113 02:51:15.163065 1108 heartbeater.cc:463] Registering TS with master...
I20250113 02:51:15.163064 1107 heartbeater.cc:463] Registering TS with master...
I20250113 02:51:15.164688 1108 heartbeater.cc:510] Master 127.0.104.62:44217 requested a full tablet report, sending...
I20250113 02:51:15.164739 1109 heartbeater.cc:510] Master 127.0.104.61:35271 requested a full tablet report, sending...
I20250113 02:51:15.164809 1107 heartbeater.cc:510] Master 127.0.104.60:42067 requested a full tablet report, sending...
I20250113 02:51:15.166483 597 ts_manager.cc:194] Registered new tserver with Master: 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:15.167248 455 ts_manager.cc:194] Registered new tserver with Master: 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:15.168100 526 ts_manager.cc:194] Registered new tserver with Master: 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:15.168939 526 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.104.3:47577
I20250113 02:51:15.173432 416 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250113 02:51:15.185948 965 heartbeater.cc:502] Master 127.0.104.62:44217 was elected leader, sending a full tablet report...
I20250113 02:51:15.188519 966 heartbeater.cc:502] Master 127.0.104.61:35271 was elected leader, sending a full tablet report...
I20250113 02:51:15.204512 821 heartbeater.cc:502] Master 127.0.104.62:44217 was elected leader, sending a full tablet report...
I20250113 02:51:15.204552 822 heartbeater.cc:502] Master 127.0.104.61:35271 was elected leader, sending a full tablet report...
I20250113 02:51:15.216147 524 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56852:
name: "default.Table_8e51b064dea74e74af985492584d4c8c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:15.229104 524 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8e51b064dea74e74af985492584d4c8c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:15.216634 522 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56852:
name: "default.Table_16a0ce79b0d24dedb82518e0aecb1be3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:15.232265 522 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_16a0ce79b0d24dedb82518e0aecb1be3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:15.216961 525 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56852:
name: "default.Table_9e454f9bb8904be582a99f7041edd7a8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:15.234416 525 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9e454f9bb8904be582a99f7041edd7a8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:15.217085 526 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56852:
name: "default.Table_ade8f3a8679d4125b7a28d3fd034d414"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:15.236236 526 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ade8f3a8679d4125b7a28d3fd034d414 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:15.221019 519 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56852:
name: "default.Table_75519a5862f9412fb28c7c15bfb20dcc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:15.237993 519 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_75519a5862f9412fb28c7c15bfb20dcc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:15.223356 517 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56852:
name: "default.Table_8548949ff1d1472988f195afb43986d9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:15.239136 517 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8548949ff1d1472988f195afb43986d9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:15.222087 518 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56852:
name: "default.Table_b45d293fa3164c4697fba0ec61c25d8c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:15.240466 518 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b45d293fa3164c4697fba0ec61c25d8c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:15.221746 520 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56852:
name: "default.Table_4e59130f15104ad38c82df889e84b290"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:15.241979 520 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4e59130f15104ad38c82df889e84b290 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:15.221379 521 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56852:
name: "default.Table_730a80c679a540bbbffec1c7e378fa0d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:15.242614 521 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_730a80c679a540bbbffec1c7e378fa0d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:15.216061 523 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56852:
name: "default.Table_ccb6790b25724f8ca2cdd9f6a53c40bd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:15.244102 523 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ccb6790b25724f8ca2cdd9f6a53c40bd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:15.245350 975 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bbc3541604a455584bb32dfc2cc4efd [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "6bbc3541604a455584bb32dfc2cc4efd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e13f5bcdcee34225a46c6d2651e5ede9" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44217 } } peers { permanent_uuid: "6bbc3541604a455584bb32dfc2cc4efd" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 35271 } } peers { permanent_uuid: "45e9d43ecac0401fb2c18fe5eaf089b5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42067 } } }
I20250113 02:51:15.245611 975 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bbc3541604a455584bb32dfc2cc4efd [sys.catalog]: This master's current role is: LEADER
I20250113 02:51:15.290797 524 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56852:
table { table_name: "default.Table_730a80c679a540bbbffec1c7e378fa0d" } new_table_name: "default.Table_6a88830e770641c4bbe405c0ef700d67" modify_external_catalogs: true
I20250113 02:51:15.291033 416 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskXJrxug/build/asan/bin/kudu pid 492
I20250113 02:51:15.291260 525 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56852:
table { table_name: "default.Table_9e454f9bb8904be582a99f7041edd7a8" } new_table_name: "default.Table_c991f9dbd70d4c4682664b8711f05db4" modify_external_catalogs: true
I20250113 02:51:15.292083 519 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56852:
table { table_name: "default.Table_8548949ff1d1472988f195afb43986d9" } modify_external_catalogs: true
I20250113 02:51:15.292261 556 generic_service.cc:196] Checking for leaks (request via RPC)
I20250113 02:51:15.292338 519 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56852:
table { table_name: "default.Table_8548949ff1d1472988f195afb43986d9" } modify_external_catalogs: true
I20250113 02:51:15.404520 521 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56852:
table { table_name: "default.Table_8e51b064dea74e74af985492584d4c8c" } new_table_name: "default.Table_97d5a53a398b4eabbec0655b6fd01a14" modify_external_catalogs: true
I20250113 02:51:15.404896 523 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56852:
table { table_name: "default.Table_ccb6790b25724f8ca2cdd9f6a53c40bd" } modify_external_catalogs: true
I20250113 02:51:15.405087 523 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56852:
table { table_name: "default.Table_ccb6790b25724f8ca2cdd9f6a53c40bd" } modify_external_catalogs: true
I20250113 02:51:15.405035 517 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56852:
table { table_name: "default.Table_b45d293fa3164c4697fba0ec61c25d8c" } modify_external_catalogs: true
I20250113 02:51:15.405234 517 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56852:
table { table_name: "default.Table_b45d293fa3164c4697fba0ec61c25d8c" } modify_external_catalogs: true
I20250113 02:51:15.406440 522 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56852:
table { table_name: "default.Table_16a0ce79b0d24dedb82518e0aecb1be3" } new_table_name: "default.Table_21460e420d7f46a5ac3f778924062761" modify_external_catalogs: true
I20250113 02:51:15.406505 520 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56852:
table { table_name: "default.Table_4e59130f15104ad38c82df889e84b290" } modify_external_catalogs: true
I20250113 02:51:15.406561 526 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56852:
table { table_name: "default.Table_ade8f3a8679d4125b7a28d3fd034d414" } new_table_name: "default.Table_9b8d59feafa947b496dd21573a542482" modify_external_catalogs: true
I20250113 02:51:15.406689 520 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56852:
table { table_name: "default.Table_4e59130f15104ad38c82df889e84b290" } modify_external_catalogs: true
I20250113 02:51:15.407835 518 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56852:
table { table_name: "default.Table_75519a5862f9412fb28c7c15bfb20dcc" } modify_external_catalogs: true
I20250113 02:51:15.408073 518 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56852:
table { table_name: "default.Table_75519a5862f9412fb28c7c15bfb20dcc" } modify_external_catalogs: true
I20250113 02:51:15.413138 416 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskXJrxug/build/asan/bin/kudu with pid 492
W20250113 02:51:15.433959 442 connection.cc:537] server connection from 127.0.0.1:49228 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250113 02:51:15.434871 1115 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.0.104.61:35271)
W20250113 02:51:15.435511 1115 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.0.104.61:35271)
W20250113 02:51:15.465421 1115 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.104.61:35271)
I20250113 02:51:15.671021 1109 heartbeater.cc:502] Master 127.0.104.61:35271 was elected leader, sending a full tablet report...
W20250113 02:51:15.672202 1109 heartbeater.cc:643] Failed to heartbeat to 127.0.104.61:35271 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.104.61:35271: connect: Connection refused (error 111)
W20250113 02:51:15.691267 966 heartbeater.cc:643] Failed to heartbeat to 127.0.104.61:35271 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.104.61:35271: connect: Connection refused (error 111)
W20250113 02:51:15.707723 822 heartbeater.cc:643] Failed to heartbeat to 127.0.104.61:35271 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.104.61:35271: connect: Connection refused (error 111)
I20250113 02:51:15.909943 1166 raft_consensus.cc:491] T 00000000000000000000000000000000 P 45e9d43ecac0401fb2c18fe5eaf089b5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6bbc3541604a455584bb32dfc2cc4efd)
I20250113 02:51:15.910135 1166 raft_consensus.cc:513] T 00000000000000000000000000000000 P 45e9d43ecac0401fb2c18fe5eaf089b5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e13f5bcdcee34225a46c6d2651e5ede9" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44217 } } peers { permanent_uuid: "6bbc3541604a455584bb32dfc2cc4efd" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 35271 } } peers { permanent_uuid: "45e9d43ecac0401fb2c18fe5eaf089b5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42067 } }
I20250113 02:51:15.910863 1166 leader_election.cc:290] T 00000000000000000000000000000000 P 45e9d43ecac0401fb2c18fe5eaf089b5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers e13f5bcdcee34225a46c6d2651e5ede9 (127.0.104.62:44217), 6bbc3541604a455584bb32dfc2cc4efd (127.0.104.61:35271)
I20250113 02:51:15.911229 464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "45e9d43ecac0401fb2c18fe5eaf089b5" candidate_term: 4 candidate_status { last_received { term: 3 index: 18 } } ignore_live_leader: false dest_uuid: "e13f5bcdcee34225a46c6d2651e5ede9" is_pre_election: true
W20250113 02:51:15.911752 582 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.61:35271: connect: Connection refused (error 111)
W20250113 02:51:15.912945 582 leader_election.cc:336] T 00000000000000000000000000000000 P 45e9d43ecac0401fb2c18fe5eaf089b5 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6bbc3541604a455584bb32dfc2cc4efd (127.0.104.61:35271): Network error: Client connection negotiation failed: client connection to 127.0.104.61:35271: connect: Connection refused (error 111)
I20250113 02:51:15.913094 582 leader_election.cc:304] T 00000000000000000000000000000000 P 45e9d43ecac0401fb2c18fe5eaf089b5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 45e9d43ecac0401fb2c18fe5eaf089b5; no voters: 6bbc3541604a455584bb32dfc2cc4efd, e13f5bcdcee34225a46c6d2651e5ede9
I20250113 02:51:15.913326 1166 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 45e9d43ecac0401fb2c18fe5eaf089b5 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250113 02:51:15.915683 1169 raft_consensus.cc:491] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6bbc3541604a455584bb32dfc2cc4efd)
I20250113 02:51:15.915809 1169 raft_consensus.cc:513] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e13f5bcdcee34225a46c6d2651e5ede9" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44217 } } peers { permanent_uuid: "6bbc3541604a455584bb32dfc2cc4efd" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 35271 } } peers { permanent_uuid: "45e9d43ecac0401fb2c18fe5eaf089b5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42067 } }
I20250113 02:51:15.916373 1169 leader_election.cc:290] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6bbc3541604a455584bb32dfc2cc4efd (127.0.104.61:35271), 45e9d43ecac0401fb2c18fe5eaf089b5 (127.0.104.60:42067)
I20250113 02:51:15.916790 607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e13f5bcdcee34225a46c6d2651e5ede9" candidate_term: 4 candidate_status { last_received { term: 3 index: 18 } } ignore_live_leader: false dest_uuid: "45e9d43ecac0401fb2c18fe5eaf089b5" is_pre_election: true
I20250113 02:51:15.917018 607 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 45e9d43ecac0401fb2c18fe5eaf089b5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e13f5bcdcee34225a46c6d2651e5ede9 in term 3.
I20250113 02:51:15.917464 439 leader_election.cc:304] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45e9d43ecac0401fb2c18fe5eaf089b5, e13f5bcdcee34225a46c6d2651e5ede9; no voters:
I20250113 02:51:15.917706 1169 raft_consensus.cc:2798] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250113 02:51:15.917810 1169 raft_consensus.cc:491] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 6bbc3541604a455584bb32dfc2cc4efd)
I20250113 02:51:15.917924 1169 raft_consensus.cc:3054] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9 [term 3 FOLLOWER]: Advancing to term 4
W20250113 02:51:15.918068 440 leader_election.cc:336] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6bbc3541604a455584bb32dfc2cc4efd (127.0.104.61:35271): Network error: Client connection negotiation failed: client connection to 127.0.104.61:35271: connect: Connection refused (error 111)
I20250113 02:51:15.919754 1169 raft_consensus.cc:513] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e13f5bcdcee34225a46c6d2651e5ede9" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44217 } } peers { permanent_uuid: "6bbc3541604a455584bb32dfc2cc4efd" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 35271 } } peers { permanent_uuid: "45e9d43ecac0401fb2c18fe5eaf089b5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42067 } }
I20250113 02:51:15.920270 1169 leader_election.cc:290] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9 [CANDIDATE]: Term 4 election: Requested vote from peers 6bbc3541604a455584bb32dfc2cc4efd (127.0.104.61:35271), 45e9d43ecac0401fb2c18fe5eaf089b5 (127.0.104.60:42067)
I20250113 02:51:15.920719 607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e13f5bcdcee34225a46c6d2651e5ede9" candidate_term: 4 candidate_status { last_received { term: 3 index: 18 } } ignore_live_leader: false dest_uuid: "45e9d43ecac0401fb2c18fe5eaf089b5"
I20250113 02:51:15.920892 607 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 45e9d43ecac0401fb2c18fe5eaf089b5 [term 3 FOLLOWER]: Advancing to term 4
W20250113 02:51:15.921475 440 leader_election.cc:336] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 6bbc3541604a455584bb32dfc2cc4efd (127.0.104.61:35271): Network error: Client connection negotiation failed: client connection to 127.0.104.61:35271: connect: Connection refused (error 111)
I20250113 02:51:15.922920 607 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 45e9d43ecac0401fb2c18fe5eaf089b5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e13f5bcdcee34225a46c6d2651e5ede9 in term 4.
I20250113 02:51:15.923224 439 leader_election.cc:304] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 45e9d43ecac0401fb2c18fe5eaf089b5, e13f5bcdcee34225a46c6d2651e5ede9; no voters: 6bbc3541604a455584bb32dfc2cc4efd
I20250113 02:51:15.923646 1169 raft_consensus.cc:2798] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9 [term 4 FOLLOWER]: Leader election won for term 4
I20250113 02:51:15.924047 1169 raft_consensus.cc:695] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9 [term 4 LEADER]: Becoming Leader. State: Replica: e13f5bcdcee34225a46c6d2651e5ede9, State: Running, Role: LEADER
I20250113 02:51:15.924362 1169 consensus_queue.cc:237] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 3.18, Last appended by leader: 18, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e13f5bcdcee34225a46c6d2651e5ede9" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44217 } } peers { permanent_uuid: "6bbc3541604a455584bb32dfc2cc4efd" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 35271 } } peers { permanent_uuid: "45e9d43ecac0401fb2c18fe5eaf089b5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42067 } }
I20250113 02:51:15.925465 1172 sys_catalog.cc:455] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e13f5bcdcee34225a46c6d2651e5ede9. Latest consensus state: current_term: 4 leader_uuid: "e13f5bcdcee34225a46c6d2651e5ede9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e13f5bcdcee34225a46c6d2651e5ede9" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44217 } } peers { permanent_uuid: "6bbc3541604a455584bb32dfc2cc4efd" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 35271 } } peers { permanent_uuid: "45e9d43ecac0401fb2c18fe5eaf089b5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42067 } } }
I20250113 02:51:15.925652 1172 sys_catalog.cc:458] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9 [sys.catalog]: This master's current role is: LEADER
I20250113 02:51:15.926124 1173 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250113 02:51:15.928824 1173 catalog_manager.cc:670] Loaded metadata for table default.Table_b45d293fa3164c4697fba0ec61c25d8c [id=0e25abba788e44a6acefb18a8241d48c]
I20250113 02:51:15.929296 1173 catalog_manager.cc:670] Loaded metadata for table default.Table_c991f9dbd70d4c4682664b8711f05db4 [id=2c64da64ecdb4bf48e11b523533902e8]
I20250113 02:51:15.929739 1173 catalog_manager.cc:670] Loaded metadata for table default.Table_75519a5862f9412fb28c7c15bfb20dcc [id=325414f311eb466797b3239842975878]
I20250113 02:51:15.930176 1173 catalog_manager.cc:670] Loaded metadata for table default.Table_ade8f3a8679d4125b7a28d3fd034d414 [id=377f041bf95e41f3b7e8ea26f9795b6a]
I20250113 02:51:15.930522 1173 catalog_manager.cc:670] Loaded metadata for table default.Table_4e59130f15104ad38c82df889e84b290 [id=39b7d015a52e45718d74de68c205a41e]
I20250113 02:51:15.930838 1173 catalog_manager.cc:670] Loaded metadata for table default.Table_16a0ce79b0d24dedb82518e0aecb1be3 [id=7caed6f6020f49a5898a2261f653070d]
I20250113 02:51:15.931138 1173 catalog_manager.cc:670] Loaded metadata for table default.Table_8548949ff1d1472988f195afb43986d9 [id=8a0a919f19354d07b615d3ec95c88c01]
I20250113 02:51:15.931456 1173 catalog_manager.cc:670] Loaded metadata for table default.Table_6a88830e770641c4bbe405c0ef700d67 [id=ba1edf2b734e48cb89a77ca59fa717c6]
I20250113 02:51:15.931763 1173 catalog_manager.cc:670] Loaded metadata for table default.Table_ccb6790b25724f8ca2cdd9f6a53c40bd [id=c555dbc12f4d4a60bc45788e4826a971]
I20250113 02:51:15.932107 1173 catalog_manager.cc:670] Loaded metadata for table default.Table_8e51b064dea74e74af985492584d4c8c [id=c5f818ea390c4c1eb1517a61e7f86a83]
I20250113 02:51:15.933509 1173 tablet_loader.cc:96] loaded metadata for tablet 1a942745de0d4238b36d73357da2c562 (table default.Table_75519a5862f9412fb28c7c15bfb20dcc [id=325414f311eb466797b3239842975878])
I20250113 02:51:15.933811 1173 tablet_loader.cc:96] loaded metadata for tablet 1fa104e6bd704e28a1958dad011d6932 (table default.Table_ccb6790b25724f8ca2cdd9f6a53c40bd [id=c555dbc12f4d4a60bc45788e4826a971])
I20250113 02:51:15.934048 1173 tablet_loader.cc:96] loaded metadata for tablet 4345e71c882a4fd1a152c64cb137e824 (table default.Table_4e59130f15104ad38c82df889e84b290 [id=39b7d015a52e45718d74de68c205a41e])
I20250113 02:51:15.934317 1173 tablet_loader.cc:96] loaded metadata for tablet 455b143af8c94f9d8a4111330648cac6 (table default.Table_c991f9dbd70d4c4682664b8711f05db4 [id=2c64da64ecdb4bf48e11b523533902e8])
I20250113 02:51:15.934685 1173 tablet_loader.cc:96] loaded metadata for tablet 8bd5fa88fbd2482eb42a6ac2c5a6d949 (table default.Table_8e51b064dea74e74af985492584d4c8c [id=c5f818ea390c4c1eb1517a61e7f86a83])
I20250113 02:51:15.934963 1173 tablet_loader.cc:96] loaded metadata for tablet a97da72b5a0e4b678a1fbfcee6c32010 (table default.Table_8548949ff1d1472988f195afb43986d9 [id=8a0a919f19354d07b615d3ec95c88c01])
I20250113 02:51:15.935211 1173 tablet_loader.cc:96] loaded metadata for tablet add325b1608b466a9326c69dd8c54c77 (table default.Table_b45d293fa3164c4697fba0ec61c25d8c [id=0e25abba788e44a6acefb18a8241d48c])
I20250113 02:51:15.935443 1173 tablet_loader.cc:96] loaded metadata for tablet b262c70712444c06a61918f21736f339 (table default.Table_ade8f3a8679d4125b7a28d3fd034d414 [id=377f041bf95e41f3b7e8ea26f9795b6a])
I20250113 02:51:15.935711 1173 tablet_loader.cc:96] loaded metadata for tablet eb4a985261f4452db4de74891b7bf65a (table default.Table_16a0ce79b0d24dedb82518e0aecb1be3 [id=7caed6f6020f49a5898a2261f653070d])
I20250113 02:51:15.936030 1173 tablet_loader.cc:96] loaded metadata for tablet ee09d2b74c05432c9d7917b036bf8898 (table default.Table_6a88830e770641c4bbe405c0ef700d67 [id=ba1edf2b734e48cb89a77ca59fa717c6])
I20250113 02:51:15.936240 1173 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250113 02:51:15.937057 1173 catalog_manager.cc:1260] Loaded cluster ID: 00860a399ece46318a55d1901f0159c4
I20250113 02:51:15.937132 1173 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250113 02:51:15.938263 1173 catalog_manager.cc:1505] Loading token signing keys...
I20250113 02:51:15.939056 1173 catalog_manager.cc:5910] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Loaded TSK: 0
I20250113 02:51:15.939426 1173 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250113 02:51:15.970664 455 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_4e59130f15104ad38c82df889e84b290" } modify_external_catalogs: true
I20250113 02:51:15.971007 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_8548949ff1d1472988f195afb43986d9" } modify_external_catalogs: true
I20250113 02:51:15.971076 455 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_4e59130f15104ad38c82df889e84b290" } modify_external_catalogs: true
I20250113 02:51:15.971248 453 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_b45d293fa3164c4697fba0ec61c25d8c" } modify_external_catalogs: true
I20250113 02:51:15.971254 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_8548949ff1d1472988f195afb43986d9" } modify_external_catalogs: true
I20250113 02:51:15.970695 454 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_75519a5862f9412fb28c7c15bfb20dcc" } modify_external_catalogs: true
I20250113 02:51:15.971484 453 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_b45d293fa3164c4697fba0ec61c25d8c" } modify_external_catalogs: true
I20250113 02:51:15.971668 454 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_75519a5862f9412fb28c7c15bfb20dcc" } modify_external_catalogs: true
I20250113 02:51:15.972743 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_ccb6790b25724f8ca2cdd9f6a53c40bd" } modify_external_catalogs: true
I20250113 02:51:15.972971 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_ccb6790b25724f8ca2cdd9f6a53c40bd" } modify_external_catalogs: true
I20250113 02:51:15.972656 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_8e51b064dea74e74af985492584d4c8c" } new_table_name: "default.Table_97d5a53a398b4eabbec0655b6fd01a14" modify_external_catalogs: true
I20250113 02:51:15.975419 448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_16a0ce79b0d24dedb82518e0aecb1be3" } new_table_name: "default.Table_21460e420d7f46a5ac3f778924062761" modify_external_catalogs: true
I20250113 02:51:15.976234 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_ade8f3a8679d4125b7a28d3fd034d414" } new_table_name: "default.Table_9b8d59feafa947b496dd21573a542482" modify_external_catalogs: true
I20250113 02:51:15.976528 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_730a80c679a540bbbffec1c7e378fa0d" } new_table_name: "default.Table_6a88830e770641c4bbe405c0ef700d67" modify_external_catalogs: true
I20250113 02:51:15.976884 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_9e454f9bb8904be582a99f7041edd7a8" } new_table_name: "default.Table_c991f9dbd70d4c4682664b8711f05db4" modify_external_catalogs: true
I20250113 02:51:15.979383 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_2106a7d1900a44df83cf7ec9f377687d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:15.979710 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_a485956f153c4d8b988ff1bf83e2394a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:15.981015 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2106a7d1900a44df83cf7ec9f377687d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250113 02:51:15.981066 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a485956f153c4d8b988ff1bf83e2394a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:15.982827 607 raft_consensus.cc:1270] T 00000000000000000000000000000000 P 45e9d43ecac0401fb2c18fe5eaf089b5 [term 4 FOLLOWER]: Refusing update from remote peer e13f5bcdcee34225a46c6d2651e5ede9: Log matching property violated. Preceding OpId in replica: term: 3 index: 18. Preceding OpId from leader: term: 4 index: 20. (index mismatch)
I20250113 02:51:15.983431 1169 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "45e9d43ecac0401fb2c18fe5eaf089b5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 15, Time since last communication: 0.000s
W20250113 02:51:15.984997 440 consensus_peers.cc:487] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9 -> Peer 6bbc3541604a455584bb32dfc2cc4efd (127.0.104.61:35271): Couldn't send request to peer 6bbc3541604a455584bb32dfc2cc4efd. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.61:35271: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250113 02:51:15.994087 1166 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45e9d43ecac0401fb2c18fe5eaf089b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e13f5bcdcee34225a46c6d2651e5ede9. Latest consensus state: current_term: 4 leader_uuid: "e13f5bcdcee34225a46c6d2651e5ede9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e13f5bcdcee34225a46c6d2651e5ede9" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44217 } } peers { permanent_uuid: "6bbc3541604a455584bb32dfc2cc4efd" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 35271 } } peers { permanent_uuid: "45e9d43ecac0401fb2c18fe5eaf089b5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42067 } } }
I20250113 02:51:15.994387 1166 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45e9d43ecac0401fb2c18fe5eaf089b5 [sys.catalog]: This master's current role is: FOLLOWER
I20250113 02:51:15.996882 455 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet 4345e71c882a4fd1a152c64cb137e824
I20250113 02:51:15.996985 451 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet 1fa104e6bd704e28a1958dad011d6932
I20250113 02:51:15.999325 1166 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45e9d43ecac0401fb2c18fe5eaf089b5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "e13f5bcdcee34225a46c6d2651e5ede9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e13f5bcdcee34225a46c6d2651e5ede9" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44217 } } peers { permanent_uuid: "6bbc3541604a455584bb32dfc2cc4efd" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 35271 } } peers { permanent_uuid: "45e9d43ecac0401fb2c18fe5eaf089b5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42067 } } }
I20250113 02:51:15.999553 1166 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45e9d43ecac0401fb2c18fe5eaf089b5 [sys.catalog]: This master's current role is: FOLLOWER
I20250113 02:51:16.000221 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_d664a11db73146e38b79b065b9642015"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:16.000569 1169 sys_catalog.cc:455] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 4 leader_uuid: "e13f5bcdcee34225a46c6d2651e5ede9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e13f5bcdcee34225a46c6d2651e5ede9" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44217 } } peers { permanent_uuid: "6bbc3541604a455584bb32dfc2cc4efd" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 35271 } } peers { permanent_uuid: "45e9d43ecac0401fb2c18fe5eaf089b5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42067 } } }
I20250113 02:51:16.000809 1169 sys_catalog.cc:458] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9 [sys.catalog]: This master's current role is: LEADER
W20250113 02:51:16.000854 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d664a11db73146e38b79b065b9642015 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:16.001965 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_6016a17614354067893bba398a0814bc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:16.002437 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6016a17614354067893bba398a0814bc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:16.007272 454 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet 1a942745de0d4238b36d73357da2c562
I20250113 02:51:16.007597 452 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet a97da72b5a0e4b678a1fbfcee6c32010
I20250113 02:51:16.008548 453 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet add325b1608b466a9326c69dd8c54c77
W20250113 02:51:16.009380 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8bd5fa88fbd2482eb42a6ac2c5a6d949 consensus state has no leader
W20250113 02:51:16.009999 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b262c70712444c06a61918f21736f339 has no consensus state
W20250113 02:51:16.010593 448 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet eb4a985261f4452db4de74891b7bf65a consensus state has no leader
I20250113 02:51:16.016902 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_2b0a9e9f93054ef49660c5ce145a0d75"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:16.017504 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2b0a9e9f93054ef49660c5ce145a0d75 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:16.018041 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_9b8d59feafa947b496dd21573a542482" } new_table_name: "default.Table_5e0263e22f1e4d2b983ae6b6ef03fca5" modify_external_catalogs: true
I20250113 02:51:16.018944 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_6a96fb47f7444dcfb90d4463de217db6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:16.019418 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6a96fb47f7444dcfb90d4463de217db6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:16.018900 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_6a4e0c489c1944a69d3f46a874357dd6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:16.020143 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6a4e0c489c1944a69d3f46a874357dd6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:16.020852 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_97d5a53a398b4eabbec0655b6fd01a14" } new_table_name: "default.Table_52d61a5d1dfd491091959c5426e3cc7d" modify_external_catalogs: true
I20250113 02:51:16.021287 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_dfb2da2848a94fe4b5d89023934e8740"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:16.021425 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_351e5f6de7c64dafb0d9ee4d86510d01"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:16.022060 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_351e5f6de7c64dafb0d9ee4d86510d01 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250113 02:51:16.022435 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dfb2da2848a94fe4b5d89023934e8740 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:16.023924 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_cacef89852fd4576937e26bc2da7b632"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:16.024507 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cacef89852fd4576937e26bc2da7b632 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250113 02:51:16.028451 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b262c70712444c06a61918f21736f339 has no consensus state
W20250113 02:51:16.034119 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8bd5fa88fbd2482eb42a6ac2c5a6d949 consensus state has no leader
W20250113 02:51:16.039763 441 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 8bd5fa88fbd2482eb42a6ac2c5a6d949 (table default.Table_52d61a5d1dfd491091959c5426e3cc7d [id=c5f818ea390c4c1eb1517a61e7f86a83], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 8bd5fa88fbd2482eb42a6ac2c5a6d949 consensus state has no leader
I20250113 02:51:16.068084 1038 tablet_service.cc:1467] Processing CreateTablet for tablet 455b143af8c94f9d8a4111330648cac6 (DEFAULT_TABLE table=default.Table_c991f9dbd70d4c4682664b8711f05db4 [id=2c64da64ecdb4bf48e11b523533902e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:16.067849 1042 tablet_service.cc:1467] Processing CreateTablet for tablet 6f1c2ae654bd46c1a8772fa62c8111ab (DEFAULT_TABLE table=default.Table_6016a17614354067893bba398a0814bc [id=7dc8c3e55ce843009ade78391e271091]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:16.069496 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 455b143af8c94f9d8a4111330648cac6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:16.068007 1041 tablet_service.cc:1467] Processing CreateTablet for tablet c1ab705ebcf1430e9e0c376b18841d27 (DEFAULT_TABLE table=default.Table_a485956f153c4d8b988ff1bf83e2394a [id=7c2bc6d4232f4585b84d71e3e1f85794]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:16.067921 1036 tablet_service.cc:1467] Processing CreateTablet for tablet 44686b9a58c9440e9932b504ad7957dc (DEFAULT_TABLE table=default.Table_351e5f6de7c64dafb0d9ee4d86510d01 [id=a383dc51c39e4c8ba4b421119c752470]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:16.072623 1039 tablet_service.cc:1467] Processing CreateTablet for tablet 015447488b734f6a93b8c36be975c031 (DEFAULT_TABLE table=default.Table_2106a7d1900a44df83cf7ec9f377687d [id=a2c08f472dd7494ab8aff87e9aa43028]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:16.072813 751 tablet_service.cc:1467] Processing CreateTablet for tablet 455b143af8c94f9d8a4111330648cac6 (DEFAULT_TABLE table=default.Table_c991f9dbd70d4c4682664b8711f05db4 [id=2c64da64ecdb4bf48e11b523533902e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:16.072959 1037 tablet_service.cc:1467] Processing CreateTablet for tablet b262c70712444c06a61918f21736f339 (DEFAULT_TABLE table=default.Table_5e0263e22f1e4d2b983ae6b6ef03fca5 [id=377f041bf95e41f3b7e8ea26f9795b6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:16.073618 1037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b262c70712444c06a61918f21736f339. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:16.074018 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 455b143af8c94f9d8a4111330648cac6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:16.074118 746 tablet_service.cc:1467] Processing CreateTablet for tablet 4e61c6a807884a17843955be24de2975 (DEFAULT_TABLE table=default.Table_6a96fb47f7444dcfb90d4463de217db6 [id=5eea6db2313a4b9abc61ea0545091e17]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:16.074498 1036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44686b9a58c9440e9932b504ad7957dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:16.074664 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e61c6a807884a17843955be24de2975. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:16.075171 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 015447488b734f6a93b8c36be975c031. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:16.075596 1035 tablet_service.cc:1467] Processing CreateTablet for tablet 5ae18cbff90d4fa88f6b9b8746fc2027 (DEFAULT_TABLE table=default.Table_6a4e0c489c1944a69d3f46a874357dd6 [id=b6376e7b605c4b6f8d1373018a6c207e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:16.075645 749 tablet_service.cc:1467] Processing CreateTablet for tablet 44686b9a58c9440e9932b504ad7957dc (DEFAULT_TABLE table=default.Table_351e5f6de7c64dafb0d9ee4d86510d01 [id=a383dc51c39e4c8ba4b421119c752470]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:16.076159 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44686b9a58c9440e9932b504ad7957dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:16.076036 1040 tablet_service.cc:1467] Processing CreateTablet for tablet 9d2e5efaac7f42c998cda21df209c29a (DEFAULT_TABLE table=default.Table_d664a11db73146e38b79b065b9642015 [id=d9b085b0fd8c418aa964aaea87079200]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:16.076740 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ae18cbff90d4fa88f6b9b8746fc2027. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:16.076980 747 tablet_service.cc:1467] Processing CreateTablet for tablet 23e3cc4d6004439bb95ec3ab2504b1a1 (DEFAULT_TABLE table=default.Table_dfb2da2848a94fe4b5d89023934e8740 [id=b409b6c2b10e471aba03feb404311e3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:16.077337 1040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d2e5efaac7f42c998cda21df209c29a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:16.077458 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23e3cc4d6004439bb95ec3ab2504b1a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:16.077306 899 tablet_service.cc:1467] Processing CreateTablet for tablet 6f1c2ae654bd46c1a8772fa62c8111ab (DEFAULT_TABLE table=default.Table_6016a17614354067893bba398a0814bc [id=7dc8c3e55ce843009ade78391e271091]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:16.077903 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c1ab705ebcf1430e9e0c376b18841d27. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:16.078172 750 tablet_service.cc:1467] Processing CreateTablet for tablet b262c70712444c06a61918f21736f339 (DEFAULT_TABLE table=default.Table_5e0263e22f1e4d2b983ae6b6ef03fca5 [id=377f041bf95e41f3b7e8ea26f9795b6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:16.078713 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b262c70712444c06a61918f21736f339. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:16.078948 1033 tablet_service.cc:1467] Processing CreateTablet for tablet 4e61c6a807884a17843955be24de2975 (DEFAULT_TABLE table=default.Table_6a96fb47f7444dcfb90d4463de217db6 [id=5eea6db2313a4b9abc61ea0545091e17]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:16.079382 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e61c6a807884a17843955be24de2975. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:16.081504 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f1c2ae654bd46c1a8772fa62c8111ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:16.081512 1034 tablet_service.cc:1467] Processing CreateTablet for tablet 23e3cc4d6004439bb95ec3ab2504b1a1 (DEFAULT_TABLE table=default.Table_dfb2da2848a94fe4b5d89023934e8740 [id=b409b6c2b10e471aba03feb404311e3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:16.082119 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23e3cc4d6004439bb95ec3ab2504b1a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:16.083227 1032 tablet_service.cc:1467] Processing CreateTablet for tablet 89517af3545c47a5a3a247a411190a47 (DEFAULT_TABLE table=default.Table_2b0a9e9f93054ef49660c5ce145a0d75 [id=e66cde9b8d9b4763b33e7641601c47da]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:16.083757 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 89517af3545c47a5a3a247a411190a47. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:16.092440 1031 tablet_service.cc:1467] Processing CreateTablet for tablet 7666ece05d334bac88fb7b58a4350b63 (DEFAULT_TABLE table=default.Table_cacef89852fd4576937e26bc2da7b632 [id=1379e5486da84a00b6682ae48a8cfe87]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:16.092882 897 tablet_service.cc:1467] Processing CreateTablet for tablet 9d2e5efaac7f42c998cda21df209c29a (DEFAULT_TABLE table=default.Table_d664a11db73146e38b79b065b9642015 [id=d9b085b0fd8c418aa964aaea87079200]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:16.093040 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7666ece05d334bac88fb7b58a4350b63. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:16.093345 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d2e5efaac7f42c998cda21df209c29a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:16.096398 898 tablet_service.cc:1467] Processing CreateTablet for tablet c1ab705ebcf1430e9e0c376b18841d27 (DEFAULT_TABLE table=default.Table_a485956f153c4d8b988ff1bf83e2394a [id=7c2bc6d4232f4585b84d71e3e1f85794]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:16.096526 744 tablet_service.cc:1467] Processing CreateTablet for tablet 7666ece05d334bac88fb7b58a4350b63 (DEFAULT_TABLE table=default.Table_cacef89852fd4576937e26bc2da7b632 [id=1379e5486da84a00b6682ae48a8cfe87]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:16.096904 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c1ab705ebcf1430e9e0c376b18841d27. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:16.097216 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7666ece05d334bac88fb7b58a4350b63. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:16.100432 745 tablet_service.cc:1467] Processing CreateTablet for tablet 89517af3545c47a5a3a247a411190a47 (DEFAULT_TABLE table=default.Table_2b0a9e9f93054ef49660c5ce145a0d75 [id=e66cde9b8d9b4763b33e7641601c47da]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:16.100600 896 tablet_service.cc:1467] Processing CreateTablet for tablet 015447488b734f6a93b8c36be975c031 (DEFAULT_TABLE table=default.Table_2106a7d1900a44df83cf7ec9f377687d [id=a2c08f472dd7494ab8aff87e9aa43028]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:16.101008 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 89517af3545c47a5a3a247a411190a47. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:16.101050 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 015447488b734f6a93b8c36be975c031. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:16.104316 752 tablet_service.cc:1467] Processing CreateTablet for tablet 015447488b734f6a93b8c36be975c031 (DEFAULT_TABLE table=default.Table_2106a7d1900a44df83cf7ec9f377687d [id=a2c08f472dd7494ab8aff87e9aa43028]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:16.104779 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 015447488b734f6a93b8c36be975c031. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:16.107990 754 tablet_service.cc:1467] Processing CreateTablet for tablet c1ab705ebcf1430e9e0c376b18841d27 (DEFAULT_TABLE table=default.Table_a485956f153c4d8b988ff1bf83e2394a [id=7c2bc6d4232f4585b84d71e3e1f85794]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:16.108059 895 tablet_service.cc:1467] Processing CreateTablet for tablet 455b143af8c94f9d8a4111330648cac6 (DEFAULT_TABLE table=default.Table_c991f9dbd70d4c4682664b8711f05db4 [id=2c64da64ecdb4bf48e11b523533902e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:16.108537 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 455b143af8c94f9d8a4111330648cac6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:16.108562 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c1ab705ebcf1430e9e0c376b18841d27. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:16.112272 753 tablet_service.cc:1467] Processing CreateTablet for tablet 9d2e5efaac7f42c998cda21df209c29a (DEFAULT_TABLE table=default.Table_d664a11db73146e38b79b065b9642015 [id=d9b085b0fd8c418aa964aaea87079200]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:16.112777 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d2e5efaac7f42c998cda21df209c29a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:16.115263 894 tablet_service.cc:1467] Processing CreateTablet for tablet 44686b9a58c9440e9932b504ad7957dc (DEFAULT_TABLE table=default.Table_351e5f6de7c64dafb0d9ee4d86510d01 [id=a383dc51c39e4c8ba4b421119c752470]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:16.115742 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44686b9a58c9440e9932b504ad7957dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:16.072489 748 tablet_service.cc:1467] Processing CreateTablet for tablet 5ae18cbff90d4fa88f6b9b8746fc2027 (DEFAULT_TABLE table=default.Table_6a4e0c489c1944a69d3f46a874357dd6 [id=b6376e7b605c4b6f8d1373018a6c207e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:16.117612 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ae18cbff90d4fa88f6b9b8746fc2027. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:16.122311 893 tablet_service.cc:1467] Processing CreateTablet for tablet 23e3cc4d6004439bb95ec3ab2504b1a1 (DEFAULT_TABLE table=default.Table_dfb2da2848a94fe4b5d89023934e8740 [id=b409b6c2b10e471aba03feb404311e3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:16.122857 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23e3cc4d6004439bb95ec3ab2504b1a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:16.123392 755 tablet_service.cc:1467] Processing CreateTablet for tablet 6f1c2ae654bd46c1a8772fa62c8111ab (DEFAULT_TABLE table=default.Table_6016a17614354067893bba398a0814bc [id=7dc8c3e55ce843009ade78391e271091]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:16.123881 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f1c2ae654bd46c1a8772fa62c8111ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:16.138021 1042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f1c2ae654bd46c1a8772fa62c8111ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:16.141191 448 master_service.cc:946] ReplaceTablet: received request to replace tablet 015447488b734f6a93b8c36be975c031 from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:16.144246 453 master_service.cc:946] ReplaceTablet: received request to replace tablet 015447488b734f6a93b8c36be975c031 from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:16.144580 446 master_service.cc:946] ReplaceTablet: received request to replace tablet 455b143af8c94f9d8a4111330648cac6 from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:16.147950 447 master_service.cc:946] ReplaceTablet: received request to replace tablet 44686b9a58c9440e9932b504ad7957dc from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:16.152801 452 master_service.cc:946] ReplaceTablet: received request to replace tablet 44686b9a58c9440e9932b504ad7957dc from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:16.166316 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 015447488b734f6a93b8c36be975c031
I20250113 02:51:16.168076 448 catalog_manager.cc:6596] ReplaceTablet: tablet 015447488b734f6a93b8c36be975c031 deleted and replaced by tablet 9a323451cdfe4fd69ec62cda230f9afb
I20250113 02:51:16.168948 894 tablet_service.cc:1467] Processing CreateTablet for tablet 5ae18cbff90d4fa88f6b9b8746fc2027 (DEFAULT_TABLE table=default.Table_6a4e0c489c1944a69d3f46a874357dd6 [id=b6376e7b605c4b6f8d1373018a6c207e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:16.169382 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ae18cbff90d4fa88f6b9b8746fc2027. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:16.170356 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 455b143af8c94f9d8a4111330648cac6
I20250113 02:51:16.170794 446 catalog_manager.cc:6596] ReplaceTablet: tablet 455b143af8c94f9d8a4111330648cac6 deleted and replaced by tablet bc88676bf738442d82c25e56f30c4ada
I20250113 02:51:16.171698 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 44686b9a58c9440e9932b504ad7957dc
I20250113 02:51:16.172128 447 catalog_manager.cc:6596] ReplaceTablet: tablet 44686b9a58c9440e9932b504ad7957dc deleted and replaced by tablet c97a6ad98ddc4680a0f071cb91f05402
I20250113 02:51:16.172664 895 tablet_service.cc:1467] Processing CreateTablet for tablet b262c70712444c06a61918f21736f339 (DEFAULT_TABLE table=default.Table_5e0263e22f1e4d2b983ae6b6ef03fca5 [id=377f041bf95e41f3b7e8ea26f9795b6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:16.173059 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b262c70712444c06a61918f21736f339. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:16.176721 892 tablet_service.cc:1467] Processing CreateTablet for tablet 7666ece05d334bac88fb7b58a4350b63 (DEFAULT_TABLE table=default.Table_cacef89852fd4576937e26bc2da7b632 [id=1379e5486da84a00b6682ae48a8cfe87]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:16.177227 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7666ece05d334bac88fb7b58a4350b63. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:16.183349 893 tablet_service.cc:1467] Processing CreateTablet for tablet 4e61c6a807884a17843955be24de2975 (DEFAULT_TABLE table=default.Table_6a96fb47f7444dcfb90d4463de217db6 [id=5eea6db2313a4b9abc61ea0545091e17]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:16.183390 752 tablet_service.cc:1514] Processing DeleteTablet for tablet 44686b9a58c9440e9932b504ad7957dc with delete_type TABLET_DATA_DELETED (replaced by tablet c97a6ad98ddc4680a0f071cb91f05402) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:16.183844 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e61c6a807884a17843955be24de2975. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:16.184227 898 tablet_service.cc:1467] Processing CreateTablet for tablet 89517af3545c47a5a3a247a411190a47 (DEFAULT_TABLE table=default.Table_2b0a9e9f93054ef49660c5ce145a0d75 [id=e66cde9b8d9b4763b33e7641601c47da]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:16.184566 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 89517af3545c47a5a3a247a411190a47. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:16.188135 754 tablet_service.cc:1514] Processing DeleteTablet for tablet 015447488b734f6a93b8c36be975c031 with delete_type TABLET_DATA_DELETED (replaced by tablet 9a323451cdfe4fd69ec62cda230f9afb) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:16.189157 891 tablet_service.cc:1514] Processing DeleteTablet for tablet 455b143af8c94f9d8a4111330648cac6 with delete_type TABLET_DATA_DELETED (replaced by tablet bc88676bf738442d82c25e56f30c4ada) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:16.191591 890 tablet_service.cc:1514] Processing DeleteTablet for tablet 44686b9a58c9440e9932b504ad7957dc with delete_type TABLET_DATA_DELETED (replaced by tablet c97a6ad98ddc4680a0f071cb91f05402) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:16.192353 892 tablet_service.cc:1514] Processing DeleteTablet for tablet 015447488b734f6a93b8c36be975c031 with delete_type TABLET_DATA_DELETED (replaced by tablet 9a323451cdfe4fd69ec62cda230f9afb) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:16.194510 744 tablet_service.cc:1514] Processing DeleteTablet for tablet 455b143af8c94f9d8a4111330648cac6 with delete_type TABLET_DATA_DELETED (replaced by tablet bc88676bf738442d82c25e56f30c4ada) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:16.201537 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_2106a7d1900a44df83cf7ec9f377687d" } modify_external_catalogs: true
I20250113 02:51:16.201591 447 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_6a96fb47f7444dcfb90d4463de217db6" } modify_external_catalogs: true
I20250113 02:51:16.201798 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_2106a7d1900a44df83cf7ec9f377687d" } modify_external_catalogs: true
I20250113 02:51:16.201799 447 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_6a96fb47f7444dcfb90d4463de217db6" } modify_external_catalogs: true
I20250113 02:51:16.202718 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet 015447488b734f6a93b8c36be975c031 with delete_type TABLET_DATA_DELETED (replaced by tablet 9a323451cdfe4fd69ec62cda230f9afb) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:16.203254 755 tablet_service.cc:1467] Processing CreateTablet for tablet 9a323451cdfe4fd69ec62cda230f9afb (DEFAULT_TABLE table=default.Table_2106a7d1900a44df83cf7ec9f377687d [id=a2c08f472dd7494ab8aff87e9aa43028]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:16.203738 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a323451cdfe4fd69ec62cda230f9afb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:16.203750 898 tablet_service.cc:1467] Processing CreateTablet for tablet 9a323451cdfe4fd69ec62cda230f9afb (DEFAULT_TABLE table=default.Table_2106a7d1900a44df83cf7ec9f377687d [id=a2c08f472dd7494ab8aff87e9aa43028]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:16.204205 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a323451cdfe4fd69ec62cda230f9afb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:16.209045 446 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_dfb2da2848a94fe4b5d89023934e8740" } modify_external_catalogs: true
I20250113 02:51:16.209326 446 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_dfb2da2848a94fe4b5d89023934e8740" } modify_external_catalogs: true
I20250113 02:51:16.209894 448 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_6a4e0c489c1944a69d3f46a874357dd6" } modify_external_catalogs: true
I20250113 02:51:16.210086 448 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_6a4e0c489c1944a69d3f46a874357dd6" } modify_external_catalogs: true
I20250113 02:51:16.212764 453 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_52d61a5d1dfd491091959c5426e3cc7d" } modify_external_catalogs: true
I20250113 02:51:16.212944 453 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_52d61a5d1dfd491091959c5426e3cc7d" } modify_external_catalogs: true
I20250113 02:51:16.213413 454 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_6016a17614354067893bba398a0814bc" } new_table_name: "default.Table_f62804de5455434a9c3496f72b6d94d8" modify_external_catalogs: true
I20250113 02:51:16.213493 1042 tablet_service.cc:1514] Processing DeleteTablet for tablet 455b143af8c94f9d8a4111330648cac6 with delete_type TABLET_DATA_DELETED (replaced by tablet bc88676bf738442d82c25e56f30c4ada) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:16.213541 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_d664a11db73146e38b79b065b9642015" } new_table_name: "default.Table_7f0e4331a3aa457b95c912f7ab644e0d" modify_external_catalogs: true
I20250113 02:51:16.212936 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_21460e420d7f46a5ac3f778924062761" } new_table_name: "default.Table_1c9fae2ae0b14e7ca0b0ed24b12c0313" modify_external_catalogs: true
I20250113 02:51:16.216809 1192 tablet_bootstrap.cc:492] T 9d2e5efaac7f42c998cda21df209c29a P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:16.218175 1194 tablet_bootstrap.cc:492] T 015447488b734f6a93b8c36be975c031 P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
W20250113 02:51:16.220789 454 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6f1c2ae654bd46c1a8772fa62c8111ab consensus state has no leader
I20250113 02:51:16.221762 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_87cfc114b15047b49fd25446ff7aa441"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:16.222361 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_87cfc114b15047b49fd25446ff7aa441 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:16.222664 1194 tablet_bootstrap.cc:654] T 015447488b734f6a93b8c36be975c031 P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:16.223600 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_e98cbfadda17490b9078136422d0fa57"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:16.224128 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e98cbfadda17490b9078136422d0fa57 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250113 02:51:16.224609 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9d2e5efaac7f42c998cda21df209c29a consensus state has no leader
W20250113 02:51:16.225040 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet eb4a985261f4452db4de74891b7bf65a consensus state has no leader
W20250113 02:51:16.226853 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 455b143af8c94f9d8a4111330648cac6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 455b143af8c94f9d8a4111330648cac6 already in progress: creating tablet
I20250113 02:51:16.227932 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_fff2447caa364c3381a3ba0abbde1c50"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:16.227726 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_8e22a139d87f407da7668fde4cf6c67a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:16.229851 752 tablet_service.cc:1467] Processing CreateTablet for tablet c97a6ad98ddc4680a0f071cb91f05402 (DEFAULT_TABLE table=default.Table_351e5f6de7c64dafb0d9ee4d86510d01 [id=a383dc51c39e4c8ba4b421119c752470]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:16.230275 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c97a6ad98ddc4680a0f071cb91f05402. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:16.230494 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fff2447caa364c3381a3ba0abbde1c50 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:16.232769 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskXJrxug/build/asan/bin/kudu
/tmp/dist-test-taskXJrxug/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.61:35271
--webserver_interface=127.0.104.61
--webserver_port=37629
--builtin_ntp_servers=127.0.104.20:45573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:44217,127.0.104.61:35271,127.0.104.60:42067
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20250113 02:51:16.237680 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 44686b9a58c9440e9932b504ad7957dc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 44686b9a58c9440e9932b504ad7957dc already in progress: creating tablet
W20250113 02:51:16.238212 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 015447488b734f6a93b8c36be975c031 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 015447488b734f6a93b8c36be975c031 already in progress: creating tablet
I20250113 02:51:16.217952 1038 tablet_service.cc:1514] Processing DeleteTablet for tablet 44686b9a58c9440e9932b504ad7957dc with delete_type TABLET_DATA_DELETED (replaced by tablet c97a6ad98ddc4680a0f071cb91f05402) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:16.238830 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_2b0a9e9f93054ef49660c5ce145a0d75" } new_table_name: "default.Table_5b16d8b5a936486fa405acf8dcda4674" modify_external_catalogs: true
W20250113 02:51:16.243757 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8e22a139d87f407da7668fde4cf6c67a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250113 02:51:16.244006 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 455b143af8c94f9d8a4111330648cac6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 455b143af8c94f9d8a4111330648cac6 already in progress: creating tablet
I20250113 02:51:16.246726 752 tablet_service.cc:1467] Processing CreateTablet for tablet bc88676bf738442d82c25e56f30c4ada (DEFAULT_TABLE table=default.Table_c991f9dbd70d4c4682664b8711f05db4 [id=2c64da64ecdb4bf48e11b523533902e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:16.247190 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc88676bf738442d82c25e56f30c4ada. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:16.253253 1037 tablet_service.cc:1467] Processing CreateTablet for tablet 9a323451cdfe4fd69ec62cda230f9afb (DEFAULT_TABLE table=default.Table_2106a7d1900a44df83cf7ec9f377687d [id=a2c08f472dd7494ab8aff87e9aa43028]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:16.256268 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_062d5a72f882408a815a7b9251705c11"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:16.256878 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_062d5a72f882408a815a7b9251705c11 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:16.257268 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 9a323451cdfe4fd69ec62cda230f9afb
I20250113 02:51:16.257320 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 4e61c6a807884a17843955be24de2975
I20250113 02:51:16.257983 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_f62804de5455434a9c3496f72b6d94d8" } new_table_name: "default.Table_1d9e98bb72674d99bfb3a2265bc57d65" modify_external_catalogs: true
I20250113 02:51:16.258129 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 8bd5fa88fbd2482eb42a6ac2c5a6d949
I20250113 02:51:16.258069 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_6bf4b05618a2461e87fd3d2901ef09bc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:16.258572 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6bf4b05618a2461e87fd3d2901ef09bc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:16.258793 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_eec2c35d4a8a4196983a872a0fa4bda5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:16.259219 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_eec2c35d4a8a4196983a872a0fa4bda5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:16.259562 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 23e3cc4d6004439bb95ec3ab2504b1a1
I20250113 02:51:16.260660 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_24e009db5fcf4e97b04f9bd96cd120e6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:16.261093 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_24e009db5fcf4e97b04f9bd96cd120e6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:16.261284 744 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e61c6a807884a17843955be24de2975 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:16 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:16.261555 754 tablet_service.cc:1514] Processing DeleteTablet for tablet 8bd5fa88fbd2482eb42a6ac2c5a6d949 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:16 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:16.261585 1037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a323451cdfe4fd69ec62cda230f9afb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:16.261788 744 tablet_service.cc:1514] Processing DeleteTablet for tablet 23e3cc4d6004439bb95ec3ab2504b1a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:16 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:16.262009 752 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a323451cdfe4fd69ec62cda230f9afb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:16 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:16.262259 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 5ae18cbff90d4fa88f6b9b8746fc2027
I20250113 02:51:16.262913 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_449479ad3dbb4c1aaa5cd954816b9935"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:16.263317 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_449479ad3dbb4c1aaa5cd954816b9935 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250113 02:51:16.263849 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 89517af3545c47a5a3a247a411190a47 consensus state has no leader
I20250113 02:51:16.264112 752 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ae18cbff90d4fa88f6b9b8746fc2027 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:16 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:16.264278 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_d12d804163994a63a236be44401aa53b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:16.264714 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d12d804163994a63a236be44401aa53b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:16.265305 1192 tablet_bootstrap.cc:654] T 9d2e5efaac7f42c998cda21df209c29a P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:16.223959 1194 log.cc:826] T 015447488b734f6a93b8c36be975c031 P df69f48656b44bb890f76587ee69cc0f: Log is configured to *not* fsync() on all Append() calls
I20250113 02:51:16.252566 894 tablet_service.cc:1467] Processing CreateTablet for tablet bc88676bf738442d82c25e56f30c4ada (DEFAULT_TABLE table=default.Table_c991f9dbd70d4c4682664b8711f05db4 [id=2c64da64ecdb4bf48e11b523533902e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:16.268905 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc88676bf738442d82c25e56f30c4ada. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:16.269340 1192 log.cc:826] T 9d2e5efaac7f42c998cda21df209c29a P 4216a7f853d642b7bc3e5cff458f3aca: Log is configured to *not* fsync() on all Append() calls
I20250113 02:51:16.252862 899 tablet_service.cc:1467] Processing CreateTablet for tablet c97a6ad98ddc4680a0f071cb91f05402 (DEFAULT_TABLE table=default.Table_351e5f6de7c64dafb0d9ee4d86510d01 [id=a383dc51c39e4c8ba4b421119c752470]), partition=RANGE (key) PARTITION UNBOUNDED
W20250113 02:51:16.273239 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 4e61c6a807884a17843955be24de2975 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4e61c6a807884a17843955be24de2975 already in progress: creating tablet
W20250113 02:51:16.274168 439 catalog_manager.cc:4839] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 8bd5fa88fbd2482eb42a6ac2c5a6d949 because the tablet was not found. No further retry: Not found: Tablet not found: 8bd5fa88fbd2482eb42a6ac2c5a6d949
I20250113 02:51:16.275494 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c97a6ad98ddc4680a0f071cb91f05402. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:16.278086 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 23e3cc4d6004439bb95ec3ab2504b1a1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 23e3cc4d6004439bb95ec3ab2504b1a1 already in progress: creating tablet
W20250113 02:51:16.280239 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6f1c2ae654bd46c1a8772fa62c8111ab consensus state has no leader
I20250113 02:51:16.280675 1191 tablet_bootstrap.cc:492] T 44686b9a58c9440e9932b504ad7957dc P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:16.281656 1192 tablet_bootstrap.cc:492] T 9d2e5efaac7f42c998cda21df209c29a P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:16.281960 1192 ts_tablet_manager.cc:1397] T 9d2e5efaac7f42c998cda21df209c29a P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.065s user 0.012s sys 0.000s
W20250113 02:51:16.282164 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 9a323451cdfe4fd69ec62cda230f9afb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9a323451cdfe4fd69ec62cda230f9afb already in progress: creating tablet
W20250113 02:51:16.282559 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 5ae18cbff90d4fa88f6b9b8746fc2027 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5ae18cbff90d4fa88f6b9b8746fc2027 already in progress: creating tablet
I20250113 02:51:16.286212 1194 tablet_bootstrap.cc:492] T 015447488b734f6a93b8c36be975c031 P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:16.286554 1194 ts_tablet_manager.cc:1397] T 015447488b734f6a93b8c36be975c031 P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.069s user 0.003s sys 0.009s
W20250113 02:51:16.286713 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 455b143af8c94f9d8a4111330648cac6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 455b143af8c94f9d8a4111330648cac6 already in progress: creating tablet
W20250113 02:51:16.287154 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 44686b9a58c9440e9932b504ad7957dc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 44686b9a58c9440e9932b504ad7957dc already in progress: creating tablet
I20250113 02:51:16.287302 893 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a323451cdfe4fd69ec62cda230f9afb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:16 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:16.287719 898 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e61c6a807884a17843955be24de2975 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:16 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:16.288077 896 tablet_service.cc:1514] Processing DeleteTablet for tablet 8bd5fa88fbd2482eb42a6ac2c5a6d949 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:16 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:16.288269 895 tablet_service.cc:1514] Processing DeleteTablet for tablet 23e3cc4d6004439bb95ec3ab2504b1a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:16 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:16.288271 1035 tablet_service.cc:1467] Processing CreateTablet for tablet c97a6ad98ddc4680a0f071cb91f05402 (DEFAULT_TABLE table=default.Table_351e5f6de7c64dafb0d9ee4d86510d01 [id=a383dc51c39e4c8ba4b421119c752470]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:16.288458 897 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ae18cbff90d4fa88f6b9b8746fc2027 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:16 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:16.288654 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c97a6ad98ddc4680a0f071cb91f05402. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:16.297312 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 44686b9a58c9440e9932b504ad7957dc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 44686b9a58c9440e9932b504ad7957dc already in progress: creating tablet
I20250113 02:51:16.297293 1194 raft_consensus.cc:357] T 015447488b734f6a93b8c36be975c031 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:16.297792 1194 raft_consensus.cc:383] T 015447488b734f6a93b8c36be975c031 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:16.297930 1194 raft_consensus.cc:738] T 015447488b734f6a93b8c36be975c031 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:16.298682 1194 consensus_queue.cc:260] T 015447488b734f6a93b8c36be975c031 P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:16.304572 1038 tablet_service.cc:1467] Processing CreateTablet for tablet bc88676bf738442d82c25e56f30c4ada (DEFAULT_TABLE table=default.Table_c991f9dbd70d4c4682664b8711f05db4 [id=2c64da64ecdb4bf48e11b523533902e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:16.304690 1042 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e61c6a807884a17843955be24de2975 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:16 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:16.305039 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc88676bf738442d82c25e56f30c4ada. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:16.305184 1040 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a323451cdfe4fd69ec62cda230f9afb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:16 UTC) from {username='slave'} at 127.0.0.1:60230
W20250113 02:51:16.309244 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 9a323451cdfe4fd69ec62cda230f9afb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9a323451cdfe4fd69ec62cda230f9afb already in progress: creating tablet
W20250113 02:51:16.309566 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 4e61c6a807884a17843955be24de2975 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4e61c6a807884a17843955be24de2975 already in progress: creating tablet
W20250113 02:51:16.309795 442 catalog_manager.cc:4839] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 8bd5fa88fbd2482eb42a6ac2c5a6d949 because the tablet was not found. No further retry: Not found: Tablet not found: 8bd5fa88fbd2482eb42a6ac2c5a6d949
I20250113 02:51:16.311053 1042 tablet_service.cc:1514] Processing DeleteTablet for tablet 23e3cc4d6004439bb95ec3ab2504b1a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:16 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:16.311300 1040 tablet_service.cc:1514] Processing DeleteTablet for tablet 8bd5fa88fbd2482eb42a6ac2c5a6d949 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:16 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:16.312083 749 tablet_service.cc:1467] Processing CreateTablet for tablet 7f3cbef7d91d4576822d913071a88248 (DEFAULT_TABLE table=default.Table_e98cbfadda17490b9078136422d0fa57 [id=54e4e3810b3b412bbec3b5bf5b28c1a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:16.312302 1042 tablet_service.cc:1467] Processing CreateTablet for tablet 7f3cbef7d91d4576822d913071a88248 (DEFAULT_TABLE table=default.Table_e98cbfadda17490b9078136422d0fa57 [id=54e4e3810b3b412bbec3b5bf5b28c1a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:16.312314 744 tablet_service.cc:1467] Processing CreateTablet for tablet 1643b3e1607f42fc82886d0b09e2fe8c (DEFAULT_TABLE table=default.Table_fff2447caa364c3381a3ba0abbde1c50 [id=15d9be3deeb84c1494965848dfb804ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:16.312620 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f3cbef7d91d4576822d913071a88248. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:16.312685 1042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f3cbef7d91d4576822d913071a88248. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:16.312868 752 tablet_service.cc:1467] Processing CreateTablet for tablet 7dbda00269cf4f7095b3504ae87ac243 (DEFAULT_TABLE table=default.Table_8e22a139d87f407da7668fde4cf6c67a [id=46fce62be9184eea82a2fdc88b9aff21]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:16.313239 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7dbda00269cf4f7095b3504ae87ac243. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:16.316253 755 tablet_service.cc:1467] Processing CreateTablet for tablet db266a1848094b7a98aa75f7afabd471 (DEFAULT_TABLE table=default.Table_87cfc114b15047b49fd25446ff7aa441 [id=ca79853d13ee4a76a18d9fa68c1feb31]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:16.316569 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet db266a1848094b7a98aa75f7afabd471. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:16.316623 1036 tablet_service.cc:1467] Processing CreateTablet for tablet 7dbda00269cf4f7095b3504ae87ac243 (DEFAULT_TABLE table=default.Table_8e22a139d87f407da7668fde4cf6c67a [id=46fce62be9184eea82a2fdc88b9aff21]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:16.316948 1036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7dbda00269cf4f7095b3504ae87ac243. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:16.320327 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1643b3e1607f42fc82886d0b09e2fe8c. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:16.323273 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 4e61c6a807884a17843955be24de2975 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4e61c6a807884a17843955be24de2975 already in progress: creating tablet
W20250113 02:51:16.323570 440 catalog_manager.cc:4839] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 9a323451cdfe4fd69ec62cda230f9afb because the tablet was not found. No further retry: Not found: Tablet not found: 9a323451cdfe4fd69ec62cda230f9afb
I20250113 02:51:16.323616 1038 tablet_service.cc:1467] Processing CreateTablet for tablet db266a1848094b7a98aa75f7afabd471 (DEFAULT_TABLE table=default.Table_87cfc114b15047b49fd25446ff7aa441 [id=ca79853d13ee4a76a18d9fa68c1feb31]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:16.323999 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet db266a1848094b7a98aa75f7afabd471. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:16.323959 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 23e3cc4d6004439bb95ec3ab2504b1a1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 23e3cc4d6004439bb95ec3ab2504b1a1 already in progress: creating tablet
W20250113 02:51:16.324214 440 catalog_manager.cc:4839] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 8bd5fa88fbd2482eb42a6ac2c5a6d949 because the tablet was not found. No further retry: Not found: Tablet not found: 8bd5fa88fbd2482eb42a6ac2c5a6d949
I20250113 02:51:16.328143 1040 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ae18cbff90d4fa88f6b9b8746fc2027 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:16 UTC) from {username='slave'} at 127.0.0.1:60230
W20250113 02:51:16.328495 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 23e3cc4d6004439bb95ec3ab2504b1a1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 23e3cc4d6004439bb95ec3ab2504b1a1 already in progress: creating tablet
W20250113 02:51:16.328806 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 5ae18cbff90d4fa88f6b9b8746fc2027 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5ae18cbff90d4fa88f6b9b8746fc2027 already in progress: creating tablet
I20250113 02:51:16.331246 891 tablet_service.cc:1467] Processing CreateTablet for tablet 7f3cbef7d91d4576822d913071a88248 (DEFAULT_TABLE table=default.Table_e98cbfadda17490b9078136422d0fa57 [id=54e4e3810b3b412bbec3b5bf5b28c1a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:16.331470 895 tablet_service.cc:1467] Processing CreateTablet for tablet 7dbda00269cf4f7095b3504ae87ac243 (DEFAULT_TABLE table=default.Table_8e22a139d87f407da7668fde4cf6c67a [id=46fce62be9184eea82a2fdc88b9aff21]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:16.331908 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7dbda00269cf4f7095b3504ae87ac243. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:16.332149 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f3cbef7d91d4576822d913071a88248. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:16.332504 896 tablet_service.cc:1467] Processing CreateTablet for tablet 1643b3e1607f42fc82886d0b09e2fe8c (DEFAULT_TABLE table=default.Table_fff2447caa364c3381a3ba0abbde1c50 [id=15d9be3deeb84c1494965848dfb804ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:16.332854 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1643b3e1607f42fc82886d0b09e2fe8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:16.331246 897 tablet_service.cc:1467] Processing CreateTablet for tablet db266a1848094b7a98aa75f7afabd471 (DEFAULT_TABLE table=default.Table_87cfc114b15047b49fd25446ff7aa441 [id=ca79853d13ee4a76a18d9fa68c1feb31]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:16.336374 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet db266a1848094b7a98aa75f7afabd471. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:16.339872 1192 raft_consensus.cc:357] T 9d2e5efaac7f42c998cda21df209c29a P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:16.340500 1192 raft_consensus.cc:383] T 9d2e5efaac7f42c998cda21df209c29a P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:16.340628 1192 raft_consensus.cc:738] T 9d2e5efaac7f42c998cda21df209c29a P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:16.341897 1192 consensus_queue.cc:260] T 9d2e5efaac7f42c998cda21df209c29a P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:16.349386 744 tablet_service.cc:1467] Processing CreateTablet for tablet ec0750fd2fa247feb3ea310ffabe05e5 (DEFAULT_TABLE table=default.Table_6bf4b05618a2461e87fd3d2901ef09bc [id=de6b4835f6c04c20b6a5ef8404dc78f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:16.349386 747 tablet_service.cc:1467] Processing CreateTablet for tablet d6675bc3cb5f4f809320e28026cdf4b0 (DEFAULT_TABLE table=default.Table_d12d804163994a63a236be44401aa53b [id=f806136f234042af82e9831467a69caa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:16.350025 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ec0750fd2fa247feb3ea310ffabe05e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:16.350193 1108 heartbeater.cc:502] Master 127.0.104.62:44217 was elected leader, sending a full tablet report...
I20250113 02:51:16.350628 898 tablet_service.cc:1467] Processing CreateTablet for tablet 78689f7ed2a64405adb75f372465a792 (DEFAULT_TABLE table=default.Table_24e009db5fcf4e97b04f9bd96cd120e6 [id=d5354129133b446b8a4a5613db8bc411]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:16.350993 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 78689f7ed2a64405adb75f372465a792. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:16.351796 890 tablet_service.cc:1467] Processing CreateTablet for tablet d6675bc3cb5f4f809320e28026cdf4b0 (DEFAULT_TABLE table=default.Table_d12d804163994a63a236be44401aa53b [id=f806136f234042af82e9831467a69caa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:16.352396 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d6675bc3cb5f4f809320e28026cdf4b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:16.352509 749 tablet_service.cc:1467] Processing CreateTablet for tablet a028cc02273644608f046135851da126 (DEFAULT_TABLE table=default.Table_eec2c35d4a8a4196983a872a0fa4bda5 [id=c7f464a7e8f14a0892d145a69a148e58]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:16.352845 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a028cc02273644608f046135851da126. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:16.353298 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d6675bc3cb5f4f809320e28026cdf4b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:16.356292 893 tablet_service.cc:1467] Processing CreateTablet for tablet a028cc02273644608f046135851da126 (DEFAULT_TABLE table=default.Table_eec2c35d4a8a4196983a872a0fa4bda5 [id=c7f464a7e8f14a0892d145a69a148e58]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:16.356304 754 tablet_service.cc:1467] Processing CreateTablet for tablet 0c57a5add4914835ac366efdcde4ee28 (DEFAULT_TABLE table=default.Table_062d5a72f882408a815a7b9251705c11 [id=1a69ebe950b846c78ec36ca96a1f9e6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:16.356644 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c57a5add4914835ac366efdcde4ee28. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:16.356724 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a028cc02273644608f046135851da126. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:16.360347 892 tablet_service.cc:1467] Processing CreateTablet for tablet 0c57a5add4914835ac366efdcde4ee28 (DEFAULT_TABLE table=default.Table_062d5a72f882408a815a7b9251705c11 [id=1a69ebe950b846c78ec36ca96a1f9e6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:16.360932 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c57a5add4914835ac366efdcde4ee28. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:16.365432 895 tablet_service.cc:1467] Processing CreateTablet for tablet ad6b21ed2b824cb0837fc1d8a39d2472 (DEFAULT_TABLE table=default.Table_449479ad3dbb4c1aaa5cd954816b9935 [id=61e1ae59b7db4fd0a33fe3f73a712028]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:16.365760 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ad6b21ed2b824cb0837fc1d8a39d2472. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:16.367265 752 tablet_service.cc:1467] Processing CreateTablet for tablet 78689f7ed2a64405adb75f372465a792 (DEFAULT_TABLE table=default.Table_24e009db5fcf4e97b04f9bd96cd120e6 [id=d5354129133b446b8a4a5613db8bc411]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:16.367746 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 78689f7ed2a64405adb75f372465a792. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:16.371165 891 tablet_service.cc:1467] Processing CreateTablet for tablet ec0750fd2fa247feb3ea310ffabe05e5 (DEFAULT_TABLE table=default.Table_6bf4b05618a2461e87fd3d2901ef09bc [id=de6b4835f6c04c20b6a5ef8404dc78f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:16.371614 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ec0750fd2fa247feb3ea310ffabe05e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:16.371941 755 tablet_service.cc:1467] Processing CreateTablet for tablet ad6b21ed2b824cb0837fc1d8a39d2472 (DEFAULT_TABLE table=default.Table_449479ad3dbb4c1aaa5cd954816b9935 [id=61e1ae59b7db4fd0a33fe3f73a712028]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:16.372423 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ad6b21ed2b824cb0837fc1d8a39d2472. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:16.374310 1036 tablet_service.cc:1467] Processing CreateTablet for tablet 1643b3e1607f42fc82886d0b09e2fe8c (DEFAULT_TABLE table=default.Table_fff2447caa364c3381a3ba0abbde1c50 [id=15d9be3deeb84c1494965848dfb804ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:16.374706 1036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1643b3e1607f42fc82886d0b09e2fe8c. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:16.381852 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 5ae18cbff90d4fa88f6b9b8746fc2027 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5ae18cbff90d4fa88f6b9b8746fc2027 already in progress: creating tablet
I20250113 02:51:16.382820 447 master_service.cc:946] ReplaceTablet: received request to replace tablet 5ae18cbff90d4fa88f6b9b8746fc2027 from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:16.383584 448 master_service.cc:946] ReplaceTablet: received request to replace tablet b262c70712444c06a61918f21736f339 from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:16.387944 1040 tablet_service.cc:1467] Processing CreateTablet for tablet ad6b21ed2b824cb0837fc1d8a39d2472 (DEFAULT_TABLE table=default.Table_449479ad3dbb4c1aaa5cd954816b9935 [id=61e1ae59b7db4fd0a33fe3f73a712028]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:16.388476 1040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ad6b21ed2b824cb0837fc1d8a39d2472. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:16.392416 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet b262c70712444c06a61918f21736f339
I20250113 02:51:16.393033 448 catalog_manager.cc:6596] ReplaceTablet: tablet b262c70712444c06a61918f21736f339 deleted and replaced by tablet c135506190f849008ce6542d569b402a
I20250113 02:51:16.393564 455 catalog_manager.cc:5618] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587) does not have the latest schema for tablet bc88676bf738442d82c25e56f30c4ada (table default.Table_c991f9dbd70d4c4682664b8711f05db4 [id=2c64da64ecdb4bf48e11b523533902e8]). Expected version 1 got 0
I20250113 02:51:16.393872 455 catalog_manager.cc:5618] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587) does not have the latest schema for tablet 89517af3545c47a5a3a247a411190a47 (table default.Table_5b16d8b5a936486fa405acf8dcda4674 [id=e66cde9b8d9b4763b33e7641601c47da]). Expected version 1 got 0
I20250113 02:51:16.393899 747 tablet_service.cc:1514] Processing DeleteTablet for tablet b262c70712444c06a61918f21736f339 with delete_type TABLET_DATA_DELETED (replaced by tablet c135506190f849008ce6542d569b402a) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:16.394217 455 catalog_manager.cc:5618] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587) does not have the latest schema for tablet 6f1c2ae654bd46c1a8772fa62c8111ab (table default.Table_1d9e98bb72674d99bfb3a2265bc57d65 [id=7dc8c3e55ce843009ade78391e271091]). Expected version 2 got 0
I20250113 02:51:16.394434 455 catalog_manager.cc:5618] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587) does not have the latest schema for tablet 9d2e5efaac7f42c998cda21df209c29a (table default.Table_7f0e4331a3aa457b95c912f7ab644e0d [id=d9b085b0fd8c418aa964aaea87079200]). Expected version 1 got 0
W20250113 02:51:16.394632 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet b262c70712444c06a61918f21736f339 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b262c70712444c06a61918f21736f339 already in progress: creating tablet
I20250113 02:51:16.394963 891 tablet_service.cc:1514] Processing DeleteTablet for tablet b262c70712444c06a61918f21736f339 with delete_type TABLET_DATA_DELETED (replaced by tablet c135506190f849008ce6542d569b402a) from {username='slave'} at 127.0.0.1:37366
W20250113 02:51:16.402653 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet b262c70712444c06a61918f21736f339 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b262c70712444c06a61918f21736f339 already in progress: creating tablet
W20250113 02:51:16.403968 455 catalog_manager.cc:5718] Failed to send AlterTable RPC for tablet bc88676bf738442d82c25e56f30c4ada (table default.Table_c991f9dbd70d4c4682664b8711f05db4 [id=2c64da64ecdb4bf48e11b523533902e8], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet bc88676bf738442d82c25e56f30c4ada consensus state has no leader
I20250113 02:51:16.406765 747 tablet_service.cc:1467] Processing CreateTablet for tablet c135506190f849008ce6542d569b402a (DEFAULT_TABLE table=default.Table_5e0263e22f1e4d2b983ae6b6ef03fca5 [id=377f041bf95e41f3b7e8ea26f9795b6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:16.407163 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c135506190f849008ce6542d569b402a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:16.409157 1191 tablet_bootstrap.cc:700] T 44686b9a58c9440e9932b504ad7957dc P 1186fa869eb14812832fa6562531f196: Time spent opening tablet: real 0.102s user 0.003s sys 0.000s
I20250113 02:51:16.409703 1191 tablet_bootstrap.cc:654] T 44686b9a58c9440e9932b504ad7957dc P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:16.411312 1191 log.cc:826] T 44686b9a58c9440e9932b504ad7957dc P 1186fa869eb14812832fa6562531f196: Log is configured to *not* fsync() on all Append() calls
I20250113 02:51:16.412325 891 tablet_service.cc:1467] Processing CreateTablet for tablet c135506190f849008ce6542d569b402a (DEFAULT_TABLE table=default.Table_5e0263e22f1e4d2b983ae6b6ef03fca5 [id=377f041bf95e41f3b7e8ea26f9795b6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:16.412770 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c135506190f849008ce6542d569b402a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:16.417026 965 heartbeater.cc:502] Master 127.0.104.62:44217 was elected leader, sending a full tablet report...
I20250113 02:51:16.418298 1194 ts_tablet_manager.cc:1428] T 015447488b734f6a93b8c36be975c031 P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.132s user 0.036s sys 0.073s
I20250113 02:51:16.418697 1194 tablet_bootstrap.cc:492] T 455b143af8c94f9d8a4111330648cac6 P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:16.418820 1203 raft_consensus.cc:491] T 015447488b734f6a93b8c36be975c031 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:16.419131 1041 tablet_service.cc:1467] Processing CreateTablet for tablet 0c57a5add4914835ac366efdcde4ee28 (DEFAULT_TABLE table=default.Table_062d5a72f882408a815a7b9251705c11 [id=1a69ebe950b846c78ec36ca96a1f9e6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:16.419034 1203 raft_consensus.cc:513] T 015447488b734f6a93b8c36be975c031 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:16.419514 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c57a5add4914835ac366efdcde4ee28. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:16.425149 455 catalog_manager.cc:5618] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467) does not have the latest schema for tablet 89517af3545c47a5a3a247a411190a47 (table default.Table_5b16d8b5a936486fa405acf8dcda4674 [id=e66cde9b8d9b4763b33e7641601c47da]). Expected version 1 got 0
I20250113 02:51:16.426940 1042 tablet_service.cc:1467] Processing CreateTablet for tablet ec0750fd2fa247feb3ea310ffabe05e5 (DEFAULT_TABLE table=default.Table_6bf4b05618a2461e87fd3d2901ef09bc [id=de6b4835f6c04c20b6a5ef8404dc78f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:16.428361 448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_6bf4b05618a2461e87fd3d2901ef09bc" } new_table_name: "default.Table_38516e9f569d4f58b983b0692edb2a21" modify_external_catalogs: true
I20250113 02:51:16.429651 1035 tablet_service.cc:1467] Processing CreateTablet for tablet a028cc02273644608f046135851da126 (DEFAULT_TABLE table=default.Table_eec2c35d4a8a4196983a872a0fa4bda5 [id=c7f464a7e8f14a0892d145a69a148e58]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:16.430068 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_5b16d8b5a936486fa405acf8dcda4674" } new_table_name: "default.Table_5d3da53a6d984a7ea593b340ad3e1368" modify_external_catalogs: true
I20250113 02:51:16.430486 1194 tablet_bootstrap.cc:654] T 455b143af8c94f9d8a4111330648cac6 P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:16.431877 1042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ec0750fd2fa247feb3ea310ffabe05e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:16.435086 1041 tablet_service.cc:1467] Processing CreateTablet for tablet d6675bc3cb5f4f809320e28026cdf4b0 (DEFAULT_TABLE table=default.Table_d12d804163994a63a236be44401aa53b [id=f806136f234042af82e9831467a69caa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:16.435503 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d6675bc3cb5f4f809320e28026cdf4b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:16.436602 1037 tablet_service.cc:1467] Processing CreateTablet for tablet 78689f7ed2a64405adb75f372465a792 (DEFAULT_TABLE table=default.Table_24e009db5fcf4e97b04f9bd96cd120e6 [id=d5354129133b446b8a4a5613db8bc411]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:16.437057 1037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 78689f7ed2a64405adb75f372465a792. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:16.436961 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_24e009db5fcf4e97b04f9bd96cd120e6" } new_table_name: "default.Table_314d320460b64bccb16b0cd28a07a70f" modify_external_catalogs: true
I20250113 02:51:16.438386 453 master_service.cc:946] ReplaceTablet: received request to replace tablet c1ab705ebcf1430e9e0c376b18841d27 from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:16.441444 1032 tablet_service.cc:1514] Processing DeleteTablet for tablet 015447488b734f6a93b8c36be975c031 with delete_type TABLET_DATA_DELETED (replaced by tablet 9a323451cdfe4fd69ec62cda230f9afb) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:16.443550 452 master_service.cc:946] ReplaceTablet: received request to replace tablet 44686b9a58c9440e9932b504ad7957dc from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:16.443590 450 master_service.cc:946] ReplaceTablet: received request to replace tablet c97a6ad98ddc4680a0f071cb91f05402 from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:16.444177 1032 tablet_service.cc:1514] Processing DeleteTablet for tablet 455b143af8c94f9d8a4111330648cac6 with delete_type TABLET_DATA_DELETED (replaced by tablet bc88676bf738442d82c25e56f30c4ada) from {username='slave'} at 127.0.0.1:60230
W20250113 02:51:16.447402 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 015447488b734f6a93b8c36be975c031 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 015447488b734f6a93b8c36be975c031 already in progress: creating tablet
W20250113 02:51:16.447877 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 455b143af8c94f9d8a4111330648cac6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 455b143af8c94f9d8a4111330648cac6 already in progress: creating tablet
I20250113 02:51:16.448519 1032 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ae18cbff90d4fa88f6b9b8746fc2027 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:16 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:16.448901 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e61c6a807884a17843955be24de2975 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:16 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:16.449203 1039 tablet_service.cc:1514] Processing DeleteTablet for tablet 44686b9a58c9440e9932b504ad7957dc with delete_type TABLET_DATA_DELETED (replaced by tablet c97a6ad98ddc4680a0f071cb91f05402) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:16.449419 1031 tablet_service.cc:1514] Processing DeleteTablet for tablet 23e3cc4d6004439bb95ec3ab2504b1a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:16 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:16.449847 1033 tablet_service.cc:1467] Processing CreateTablet for tablet c135506190f849008ce6542d569b402a (DEFAULT_TABLE table=default.Table_5e0263e22f1e4d2b983ae6b6ef03fca5 [id=377f041bf95e41f3b7e8ea26f9795b6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:16.451789 455 catalog_manager.cc:5618] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467) does not have the latest schema for tablet 6f1c2ae654bd46c1a8772fa62c8111ab (table default.Table_1d9e98bb72674d99bfb3a2265bc57d65 [id=7dc8c3e55ce843009ade78391e271091]). Expected version 2 got 0
I20250113 02:51:16.452024 455 catalog_manager.cc:5618] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467) does not have the latest schema for tablet 9d2e5efaac7f42c998cda21df209c29a (table default.Table_7f0e4331a3aa457b95c912f7ab644e0d [id=d9b085b0fd8c418aa964aaea87079200]). Expected version 1 got 0
I20250113 02:51:16.452354 455 catalog_manager.cc:5618] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467) does not have the latest schema for tablet bc88676bf738442d82c25e56f30c4ada (table default.Table_c991f9dbd70d4c4682664b8711f05db4 [id=2c64da64ecdb4bf48e11b523533902e8]). Expected version 1 got 0
W20250113 02:51:16.453195 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 78689f7ed2a64405adb75f372465a792 consensus state has no leader
W20250113 02:51:16.454003 448 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ec0750fd2fa247feb3ea310ffabe05e5 consensus state has no leader
I20250113 02:51:16.454545 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a028cc02273644608f046135851da126. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:16.455143 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c135506190f849008ce6542d569b402a. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:16.458191 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 89517af3545c47a5a3a247a411190a47 consensus state has no leader
I20250113 02:51:16.454174 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "015447488b734f6a93b8c36be975c031" candidate_uuid: "df69f48656b44bb890f76587ee69cc0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
W20250113 02:51:16.461249 851 leader_election.cc:343] T 015447488b734f6a93b8c36be975c031 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:16.464644 1203 leader_election.cc:290] T 015447488b734f6a93b8c36be975c031 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:16.464898 891 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e61c6a807884a17843955be24de2975 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:16 UTC) from {username='slave'} at 127.0.0.1:37366
W20250113 02:51:16.465740 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 4e61c6a807884a17843955be24de2975 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4e61c6a807884a17843955be24de2975 already in progress: creating tablet
I20250113 02:51:16.466989 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet c1ab705ebcf1430e9e0c376b18841d27
I20250113 02:51:16.467463 453 catalog_manager.cc:6596] ReplaceTablet: tablet c1ab705ebcf1430e9e0c376b18841d27 deleted and replaced by tablet ce92c2b45a274eb09e72a96b7ec133be
I20250113 02:51:16.469419 447 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_cacef89852fd4576937e26bc2da7b632" } modify_external_catalogs: true
I20250113 02:51:16.469609 447 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_cacef89852fd4576937e26bc2da7b632" } modify_external_catalogs: true
I20250113 02:51:16.469637 1192 ts_tablet_manager.cc:1428] T 9d2e5efaac7f42c998cda21df209c29a P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.162s user 0.012s sys 0.007s
I20250113 02:51:16.470121 891 tablet_service.cc:1514] Processing DeleteTablet for tablet c1ab705ebcf1430e9e0c376b18841d27 with delete_type TABLET_DATA_DELETED (replaced by tablet ce92c2b45a274eb09e72a96b7ec133be) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:16.470474 448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_1d9e98bb72674d99bfb3a2265bc57d65" } new_table_name: "default.Table_ec4ee7380446448293eaa15db62d34de" modify_external_catalogs: true
I20250113 02:51:16.470918 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_7f0e4331a3aa457b95c912f7ab644e0d" } modify_external_catalogs: true
I20250113 02:51:16.471063 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_7f0e4331a3aa457b95c912f7ab644e0d" } modify_external_catalogs: true
I20250113 02:51:16.471546 449 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_351e5f6de7c64dafb0d9ee4d86510d01" } modify_external_catalogs: true
I20250113 02:51:16.471729 449 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_351e5f6de7c64dafb0d9ee4d86510d01" } modify_external_catalogs: true
I20250113 02:51:16.471863 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_b31190bf4fa4499da09233dbdd1c9139"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:16.472201 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet c97a6ad98ddc4680a0f071cb91f05402
W20250113 02:51:16.472376 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b31190bf4fa4499da09233dbdd1c9139 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:16.472625 450 catalog_manager.cc:6596] ReplaceTablet: tablet c97a6ad98ddc4680a0f071cb91f05402 deleted and replaced by tablet 2fe235d5280944ef92cf15fb458f2f81
I20250113 02:51:16.474424 453 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_8e22a139d87f407da7668fde4cf6c67a" } modify_external_catalogs: true
I20250113 02:51:16.474597 453 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_8e22a139d87f407da7668fde4cf6c67a" } modify_external_catalogs: true
I20250113 02:51:16.474756 752 tablet_service.cc:1514] Processing DeleteTablet for tablet c97a6ad98ddc4680a0f071cb91f05402 with delete_type TABLET_DATA_DELETED (replaced by tablet 2fe235d5280944ef92cf15fb458f2f81) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:16.475529 891 tablet_service.cc:1514] Processing DeleteTablet for tablet c97a6ad98ddc4680a0f071cb91f05402 with delete_type TABLET_DATA_DELETED (replaced by tablet 2fe235d5280944ef92cf15fb458f2f81) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:16.475718 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_20ca05c407b14ba0a92f76f4596ae2ef"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:16.475858 755 tablet_service.cc:1514] Processing DeleteTablet for tablet c1ab705ebcf1430e9e0c376b18841d27 with delete_type TABLET_DATA_DELETED (replaced by tablet ce92c2b45a274eb09e72a96b7ec133be) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:16.476126 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_fff2447caa364c3381a3ba0abbde1c50" } modify_external_catalogs: true
W20250113 02:51:16.476289 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_20ca05c407b14ba0a92f76f4596ae2ef in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:16.476311 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_fff2447caa364c3381a3ba0abbde1c50" } modify_external_catalogs: true
W20250113 02:51:16.476926 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet c97a6ad98ddc4680a0f071cb91f05402 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c97a6ad98ddc4680a0f071cb91f05402 already in progress: creating tablet
W20250113 02:51:16.477231 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet c1ab705ebcf1430e9e0c376b18841d27 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c1ab705ebcf1430e9e0c376b18841d27 already in progress: creating tablet
I20250113 02:51:16.477357 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_7536c30edcec4848b4334695d022517b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:16.475811 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet c1ab705ebcf1430e9e0c376b18841d27 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c1ab705ebcf1430e9e0c376b18841d27 already in progress: creating tablet
W20250113 02:51:16.477722 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet c97a6ad98ddc4680a0f071cb91f05402 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c97a6ad98ddc4680a0f071cb91f05402 already in progress: creating tablet
W20250113 02:51:16.477828 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7536c30edcec4848b4334695d022517b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:16.474905 1038 tablet_service.cc:1514] Processing DeleteTablet for tablet b262c70712444c06a61918f21736f339 with delete_type TABLET_DATA_DELETED (replaced by tablet c135506190f849008ce6542d569b402a) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:16.484397 1191 tablet_bootstrap.cc:492] T 44686b9a58c9440e9932b504ad7957dc P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:16.484776 1191 ts_tablet_manager.cc:1397] T 44686b9a58c9440e9932b504ad7957dc P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.204s user 0.007s sys 0.007s
W20250113 02:51:16.486202 448 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6f1c2ae654bd46c1a8772fa62c8111ab consensus state has no leader
I20250113 02:51:16.486624 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_3865c1517889457a87be9ea12b3a156f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:16.487155 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3865c1517889457a87be9ea12b3a156f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:16.492280 1191 raft_consensus.cc:357] T 44686b9a58c9440e9932b504ad7957dc P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:16.492805 1191 raft_consensus.cc:383] T 44686b9a58c9440e9932b504ad7957dc P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:16.492997 1191 raft_consensus.cc:738] T 44686b9a58c9440e9932b504ad7957dc P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:16.493878 1191 consensus_queue.cc:260] T 44686b9a58c9440e9932b504ad7957dc P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:16.494907 891 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ae18cbff90d4fa88f6b9b8746fc2027 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:16 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:16.495792 891 tablet_service.cc:1514] Processing DeleteTablet for tablet b262c70712444c06a61918f21736f339 with delete_type TABLET_DATA_DELETED (replaced by tablet c135506190f849008ce6542d569b402a) from {username='slave'} at 127.0.0.1:37366
W20250113 02:51:16.495837 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 5ae18cbff90d4fa88f6b9b8746fc2027 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5ae18cbff90d4fa88f6b9b8746fc2027 already in progress: creating tablet
I20250113 02:51:16.497251 449 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet 2fe235d5280944ef92cf15fb458f2f81
I20250113 02:51:16.497165 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_da7e6ea4fd81406b94864ce3e0d9b14d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:16.497357 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_062d5a72f882408a815a7b9251705c11" } new_table_name: "default.Table_fb5e880807124a7f8a203b5798c056b0" modify_external_catalogs: true
W20250113 02:51:16.497706 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_da7e6ea4fd81406b94864ce3e0d9b14d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:16.497658 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_f9aa5df6ba3d43999ebd6570df3108d1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:16.498013 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 9d2e5efaac7f42c998cda21df209c29a
W20250113 02:51:16.498163 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f9aa5df6ba3d43999ebd6570df3108d1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:16.499217 755 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d2e5efaac7f42c998cda21df209c29a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:16 UTC) from {username='slave'} at 127.0.0.1:55398
W20250113 02:51:16.501252 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 9d2e5efaac7f42c998cda21df209c29a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9d2e5efaac7f42c998cda21df209c29a already in progress: creating tablet
W20250113 02:51:16.502319 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet b262c70712444c06a61918f21736f339 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b262c70712444c06a61918f21736f339 already in progress: creating tablet
I20250113 02:51:16.502944 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_146245c7e89e4b01b6b8e2f5f21a9c72"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:16.503461 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_146245c7e89e4b01b6b8e2f5f21a9c72 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:16.503578 1194 tablet_bootstrap.cc:492] T 455b143af8c94f9d8a4111330648cac6 P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:16.503752 1194 ts_tablet_manager.cc:1397] T 455b143af8c94f9d8a4111330648cac6 P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.085s user 0.022s sys 0.040s
I20250113 02:51:16.504351 899 tablet_service.cc:1514] Processing DeleteTablet for tablet 44686b9a58c9440e9932b504ad7957dc with delete_type TABLET_DATA_DELETED (replaced by tablet c97a6ad98ddc4680a0f071cb91f05402) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:16.504611 894 tablet_service.cc:1514] Processing DeleteTablet for tablet 23e3cc4d6004439bb95ec3ab2504b1a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:16 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:16.504825 892 tablet_service.cc:1514] Processing DeleteTablet for tablet 015447488b734f6a93b8c36be975c031 with delete_type TABLET_DATA_DELETED (replaced by tablet 9a323451cdfe4fd69ec62cda230f9afb) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:16.505041 895 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a323451cdfe4fd69ec62cda230f9afb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:16 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:16.504933 1194 raft_consensus.cc:357] T 455b143af8c94f9d8a4111330648cac6 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:16.505244 898 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d2e5efaac7f42c998cda21df209c29a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:16 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:16.505496 1194 raft_consensus.cc:383] T 455b143af8c94f9d8a4111330648cac6 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:16.505635 1194 raft_consensus.cc:738] T 455b143af8c94f9d8a4111330648cac6 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:16.505923 1194 consensus_queue.cc:260] T 455b143af8c94f9d8a4111330648cac6 P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:16.506296 891 tablet_service.cc:1514] Processing DeleteTablet for tablet 455b143af8c94f9d8a4111330648cac6 with delete_type TABLET_DATA_DELETED (replaced by tablet bc88676bf738442d82c25e56f30c4ada) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:16.508414 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 7666ece05d334bac88fb7b58a4350b63
I20250113 02:51:16.508591 1194 ts_tablet_manager.cc:1428] T 455b143af8c94f9d8a4111330648cac6 P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.005s user 0.000s sys 0.003s
I20250113 02:51:16.509370 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 7dbda00269cf4f7095b3504ae87ac243
I20250113 02:51:16.511735 1192 tablet_bootstrap.cc:492] T 44686b9a58c9440e9932b504ad7957dc P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:16.512984 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 1643b3e1607f42fc82886d0b09e2fe8c
I20250113 02:51:16.512025 1062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "015447488b734f6a93b8c36be975c031" candidate_uuid: "df69f48656b44bb890f76587ee69cc0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
W20250113 02:51:16.517443 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 5ae18cbff90d4fa88f6b9b8746fc2027 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5ae18cbff90d4fa88f6b9b8746fc2027 already in progress: creating tablet
W20250113 02:51:16.517808 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 4e61c6a807884a17843955be24de2975 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4e61c6a807884a17843955be24de2975 already in progress: creating tablet
W20250113 02:51:16.518193 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 23e3cc4d6004439bb95ec3ab2504b1a1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 23e3cc4d6004439bb95ec3ab2504b1a1 already in progress: creating tablet
W20250113 02:51:16.518435 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet b262c70712444c06a61918f21736f339 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b262c70712444c06a61918f21736f339 already in progress: creating tablet
W20250113 02:51:16.519682 852 leader_election.cc:343] T 015447488b734f6a93b8c36be975c031 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:16.520349 747 tablet_service.cc:1514] Processing DeleteTablet for tablet 1643b3e1607f42fc82886d0b09e2fe8c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:16 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:16.520382 752 tablet_service.cc:1514] Processing DeleteTablet for tablet 7dbda00269cf4f7095b3504ae87ac243 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:16 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:16.520665 755 tablet_service.cc:1514] Processing DeleteTablet for tablet 7666ece05d334bac88fb7b58a4350b63 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:16 UTC) from {username='slave'} at 127.0.0.1:55398
W20250113 02:51:16.521620 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 1643b3e1607f42fc82886d0b09e2fe8c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1643b3e1607f42fc82886d0b09e2fe8c already in progress: creating tablet
W20250113 02:51:16.521898 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 7666ece05d334bac88fb7b58a4350b63 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7666ece05d334bac88fb7b58a4350b63 already in progress: creating tablet
W20250113 02:51:16.522147 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 7dbda00269cf4f7095b3504ae87ac243 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7dbda00269cf4f7095b3504ae87ac243 already in progress: creating tablet
I20250113 02:51:16.505934 1196 tablet_replica.cc:331] T 015447488b734f6a93b8c36be975c031 P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:16.520092 852 leader_election.cc:304] T 015447488b734f6a93b8c36be975c031 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df69f48656b44bb890f76587ee69cc0f; no voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca
I20250113 02:51:16.522876 1196 raft_consensus.cc:2238] T 015447488b734f6a93b8c36be975c031 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:16.523329 1203 raft_consensus.cc:2701] T 015447488b734f6a93b8c36be975c031 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Received pre-election callback for term 1 while not running: Illegal state: RaftConsensus is not running: State = Stopping
I20250113 02:51:16.523530 1196 raft_consensus.cc:2267] T 015447488b734f6a93b8c36be975c031 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
W20250113 02:51:16.527251 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0c57a5add4914835ac366efdcde4ee28 consensus state has no leader
I20250113 02:51:16.528306 1031 tablet_service.cc:1514] Processing DeleteTablet for tablet 7666ece05d334bac88fb7b58a4350b63 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:16 UTC) from {username='slave'} at 127.0.0.1:60230
W20250113 02:51:16.528532 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 23e3cc4d6004439bb95ec3ab2504b1a1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 23e3cc4d6004439bb95ec3ab2504b1a1 already in progress: creating tablet
I20250113 02:51:16.529054 1031 tablet_service.cc:1514] Processing DeleteTablet for tablet 7dbda00269cf4f7095b3504ae87ac243 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:16 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:16.529186 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_1b75b69f143d4fb2a97f31fd75657b96"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:16.529567 1031 tablet_service.cc:1514] Processing DeleteTablet for tablet 1643b3e1607f42fc82886d0b09e2fe8c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:16 UTC) from {username='slave'} at 127.0.0.1:60230
W20250113 02:51:16.529760 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1b75b69f143d4fb2a97f31fd75657b96 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250113 02:51:16.530951 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 7666ece05d334bac88fb7b58a4350b63 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7666ece05d334bac88fb7b58a4350b63 already in progress: creating tablet
I20250113 02:51:16.531112 895 tablet_service.cc:1514] Processing DeleteTablet for tablet 1643b3e1607f42fc82886d0b09e2fe8c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:16 UTC) from {username='slave'} at 127.0.0.1:37366
W20250113 02:51:16.531368 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 7dbda00269cf4f7095b3504ae87ac243 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7dbda00269cf4f7095b3504ae87ac243 already in progress: creating tablet
I20250113 02:51:16.531270 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_9ccc7f6e9611440db76e053ff2d60f45"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:16.531399 898 tablet_service.cc:1514] Processing DeleteTablet for tablet 7dbda00269cf4f7095b3504ae87ac243 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:16 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:16.531608 891 tablet_service.cc:1514] Processing DeleteTablet for tablet 7666ece05d334bac88fb7b58a4350b63 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:16 UTC) from {username='slave'} at 127.0.0.1:37366
W20250113 02:51:16.531771 440 catalog_manager.cc:4839] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 1643b3e1607f42fc82886d0b09e2fe8c because the tablet was not found. No further retry: Not found: Tablet not found: 1643b3e1607f42fc82886d0b09e2fe8c
W20250113 02:51:16.531809 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9ccc7f6e9611440db76e053ff2d60f45 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:16.532135 1033 tablet_service.cc:1514] Processing DeleteTablet for tablet c97a6ad98ddc4680a0f071cb91f05402 with delete_type TABLET_DATA_DELETED (replaced by tablet 2fe235d5280944ef92cf15fb458f2f81) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:16.532238 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_32cc476bc0c44beeae94b3c6a36c3a6b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:16.532670 1038 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d2e5efaac7f42c998cda21df209c29a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:16 UTC) from {username='slave'} at 127.0.0.1:60230
W20250113 02:51:16.532691 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_32cc476bc0c44beeae94b3c6a36c3a6b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:16.532963 1198 tablet_replica.cc:331] T 9d2e5efaac7f42c998cda21df209c29a P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:16.533427 1198 raft_consensus.cc:2238] T 9d2e5efaac7f42c998cda21df209c29a P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:16.533470 1196 ts_tablet_manager.cc:1905] T 015447488b734f6a93b8c36be975c031 P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:16.533834 1198 raft_consensus.cc:2267] T 9d2e5efaac7f42c998cda21df209c29a P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:16.535439 1198 ts_tablet_manager.cc:1905] T 9d2e5efaac7f42c998cda21df209c29a P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:16.543267 752 tablet_service.cc:1467] Processing CreateTablet for tablet ce92c2b45a274eb09e72a96b7ec133be (DEFAULT_TABLE table=default.Table_a485956f153c4d8b988ff1bf83e2394a [id=7c2bc6d4232f4585b84d71e3e1f85794]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:16.543732 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce92c2b45a274eb09e72a96b7ec133be. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:16.543787 1194 tablet_bootstrap.cc:492] T 44686b9a58c9440e9932b504ad7957dc P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:16.545583 1194 tablet_bootstrap.cc:654] T 44686b9a58c9440e9932b504ad7957dc P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:16.545856 1040 tablet_service.cc:1514] Processing DeleteTablet for tablet c1ab705ebcf1430e9e0c376b18841d27 with delete_type TABLET_DATA_DELETED (replaced by tablet ce92c2b45a274eb09e72a96b7ec133be) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:16.546674 1196 ts_tablet_manager.cc:1918] T 015447488b734f6a93b8c36be975c031 P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:16.546875 1196 log.cc:1198] T 015447488b734f6a93b8c36be975c031 P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/015447488b734f6a93b8c36be975c031
I20250113 02:51:16.547408 1196 ts_tablet_manager.cc:1939] T 015447488b734f6a93b8c36be975c031 P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:16.550997 1194 tablet_bootstrap.cc:492] T 44686b9a58c9440e9932b504ad7957dc P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:16.551200 1194 ts_tablet_manager.cc:1397] T 44686b9a58c9440e9932b504ad7957dc P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20250113 02:51:16.552174 1194 raft_consensus.cc:357] T 44686b9a58c9440e9932b504ad7957dc P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:16.552459 1194 raft_consensus.cc:383] T 44686b9a58c9440e9932b504ad7957dc P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:16.552557 1194 raft_consensus.cc:738] T 44686b9a58c9440e9932b504ad7957dc P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:16.553036 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet 015447488b734f6a93b8c36be975c031 (table default.Table_2106a7d1900a44df83cf7ec9f377687d [id=a2c08f472dd7494ab8aff87e9aa43028]) successfully deleted
W20250113 02:51:16.553318 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 9a323451cdfe4fd69ec62cda230f9afb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9a323451cdfe4fd69ec62cda230f9afb already in progress: creating tablet
I20250113 02:51:16.553561 891 tablet_service.cc:1467] Processing CreateTablet for tablet ce92c2b45a274eb09e72a96b7ec133be (DEFAULT_TABLE table=default.Table_a485956f153c4d8b988ff1bf83e2394a [id=7c2bc6d4232f4585b84d71e3e1f85794]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:16.553685 1036 tablet_service.cc:1467] Processing CreateTablet for tablet ce92c2b45a274eb09e72a96b7ec133be (DEFAULT_TABLE table=default.Table_a485956f153c4d8b988ff1bf83e2394a [id=7c2bc6d4232f4585b84d71e3e1f85794]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:16.554052 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce92c2b45a274eb09e72a96b7ec133be. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:16.554199 1036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce92c2b45a274eb09e72a96b7ec133be. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:16.552837 1194 consensus_queue.cc:260] T 44686b9a58c9440e9932b504ad7957dc P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:16.562847 1194 ts_tablet_manager.cc:1428] T 44686b9a58c9440e9932b504ad7957dc P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.012s user 0.003s sys 0.000s
I20250113 02:51:16.563150 1194 tablet_bootstrap.cc:492] T c1ab705ebcf1430e9e0c376b18841d27 P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:16.565264 1192 tablet_bootstrap.cc:654] T 44686b9a58c9440e9932b504ad7957dc P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:16.566323 1196 tablet_replica.cc:331] T 455b143af8c94f9d8a4111330648cac6 P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:16.566588 1196 raft_consensus.cc:2238] T 455b143af8c94f9d8a4111330648cac6 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
W20250113 02:51:16.566741 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 9d2e5efaac7f42c998cda21df209c29a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9d2e5efaac7f42c998cda21df209c29a already in progress: creating tablet
I20250113 02:51:16.566790 1196 raft_consensus.cc:2267] T 455b143af8c94f9d8a4111330648cac6 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:16.567247 891 tablet_service.cc:1514] Processing DeleteTablet for tablet 44686b9a58c9440e9932b504ad7957dc with delete_type TABLET_DATA_DELETED (replaced by tablet c97a6ad98ddc4680a0f071cb91f05402) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:16.567534 1196 ts_tablet_manager.cc:1905] T 455b143af8c94f9d8a4111330648cac6 P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:16.567953 1194 tablet_bootstrap.cc:654] T c1ab705ebcf1430e9e0c376b18841d27 P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:16.569813 1194 tablet_bootstrap.cc:492] T c1ab705ebcf1430e9e0c376b18841d27 P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:16.569988 1194 ts_tablet_manager.cc:1397] T c1ab705ebcf1430e9e0c376b18841d27 P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20250113 02:51:16.570773 1194 raft_consensus.cc:357] T c1ab705ebcf1430e9e0c376b18841d27 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:16.570984 1194 raft_consensus.cc:383] T c1ab705ebcf1430e9e0c376b18841d27 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:16.571081 1194 raft_consensus.cc:738] T c1ab705ebcf1430e9e0c376b18841d27 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:16.571347 1194 consensus_queue.cc:260] T c1ab705ebcf1430e9e0c376b18841d27 P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:16.571892 1194 ts_tablet_manager.cc:1428] T c1ab705ebcf1430e9e0c376b18841d27 P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250113 02:51:16.572217 1194 tablet_bootstrap.cc:492] T 23e3cc4d6004439bb95ec3ab2504b1a1 P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:16.574829 1196 ts_tablet_manager.cc:1918] T 455b143af8c94f9d8a4111330648cac6 P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:16.575073 1196 log.cc:1198] T 455b143af8c94f9d8a4111330648cac6 P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/455b143af8c94f9d8a4111330648cac6
I20250113 02:51:16.575543 1196 ts_tablet_manager.cc:1939] T 455b143af8c94f9d8a4111330648cac6 P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:16.576063 891 tablet_service.cc:1514] Processing DeleteTablet for tablet c1ab705ebcf1430e9e0c376b18841d27 with delete_type TABLET_DATA_DELETED (replaced by tablet ce92c2b45a274eb09e72a96b7ec133be) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:16.576874 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet 455b143af8c94f9d8a4111330648cac6 (table default.Table_c991f9dbd70d4c4682664b8711f05db4 [id=2c64da64ecdb4bf48e11b523533902e8]) successfully deleted
I20250113 02:51:16.577440 1194 tablet_bootstrap.cc:654] T 23e3cc4d6004439bb95ec3ab2504b1a1 P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
W20250113 02:51:16.577728 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 1643b3e1607f42fc82886d0b09e2fe8c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1643b3e1607f42fc82886d0b09e2fe8c already in progress: creating tablet
W20250113 02:51:16.582949 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 7dbda00269cf4f7095b3504ae87ac243 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7dbda00269cf4f7095b3504ae87ac243 already in progress: creating tablet
W20250113 02:51:16.584036 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 7666ece05d334bac88fb7b58a4350b63 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7666ece05d334bac88fb7b58a4350b63 already in progress: creating tablet
I20250113 02:51:16.584496 1196 tablet_replica.cc:331] T 44686b9a58c9440e9932b504ad7957dc P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:16.584800 1196 raft_consensus.cc:2238] T 44686b9a58c9440e9932b504ad7957dc P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:16.585083 1196 raft_consensus.cc:2267] T 44686b9a58c9440e9932b504ad7957dc P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:16.586086 1196 ts_tablet_manager.cc:1905] T 44686b9a58c9440e9932b504ad7957dc P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:16.591687 891 tablet_service.cc:1467] Processing CreateTablet for tablet 15a05a32febd443dbb519c186048428a (DEFAULT_TABLE table=default.Table_f9aa5df6ba3d43999ebd6570df3108d1 [id=fa5a1cda58f54103ac637c12d99e4e38]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:16.592813 898 tablet_service.cc:1467] Processing CreateTablet for tablet f0e7cc161f354f86b8c2fef303e87c5b (DEFAULT_TABLE table=default.Table_20ca05c407b14ba0a92f76f4596ae2ef [id=75cb0745bbb84e1c883cb13b546ecb11]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:16.594002 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f0e7cc161f354f86b8c2fef303e87c5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:16.595296 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 15a05a32febd443dbb519c186048428a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:16.598449 895 tablet_service.cc:1467] Processing CreateTablet for tablet ad9b95c5450a4c9c89a0930862348d49 (DEFAULT_TABLE table=default.Table_7536c30edcec4848b4334695d022517b [id=219c739889d04a97aafa4fe1b55b0a24]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:16.599092 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ad9b95c5450a4c9c89a0930862348d49. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:16.599114 752 tablet_service.cc:1467] Processing CreateTablet for tablet 15a05a32febd443dbb519c186048428a (DEFAULT_TABLE table=default.Table_f9aa5df6ba3d43999ebd6570df3108d1 [id=fa5a1cda58f54103ac637c12d99e4e38]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:16.599385 749 tablet_service.cc:1467] Processing CreateTablet for tablet 3d132c29cb2c46108e5877e38068ef17 (DEFAULT_TABLE table=default.Table_b31190bf4fa4499da09233dbdd1c9139 [id=44f4b85774f644aabd241d6ecd397a37]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:16.599574 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 15a05a32febd443dbb519c186048428a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:16.599786 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3d132c29cb2c46108e5877e38068ef17. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:16.599745 755 tablet_service.cc:1467] Processing CreateTablet for tablet f0e7cc161f354f86b8c2fef303e87c5b (DEFAULT_TABLE table=default.Table_20ca05c407b14ba0a92f76f4596ae2ef [id=75cb0745bbb84e1c883cb13b546ecb11]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:16.600100 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f0e7cc161f354f86b8c2fef303e87c5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:16.599929 748 tablet_service.cc:1467] Processing CreateTablet for tablet 3455f010e51b4db1a9804a57fd1a4fb8 (DEFAULT_TABLE table=default.Table_da7e6ea4fd81406b94864ce3e0d9b14d [id=a9e0aae9f5af42b796d29a75140f9c29]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:16.600199 747 tablet_service.cc:1467] Processing CreateTablet for tablet ad9b95c5450a4c9c89a0930862348d49 (DEFAULT_TABLE table=default.Table_7536c30edcec4848b4334695d022517b [id=219c739889d04a97aafa4fe1b55b0a24]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:16.600348 744 tablet_service.cc:1467] Processing CreateTablet for tablet 0c55662a71444a3683a825bcec6c5b68 (DEFAULT_TABLE table=default.Table_3865c1517889457a87be9ea12b3a156f [id=79b4c84e05ce4571b42e2b4e40fae978]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:16.600427 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3455f010e51b4db1a9804a57fd1a4fb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:16.600664 754 tablet_service.cc:1467] Processing CreateTablet for tablet 8193357427f4491c8c6c1396ceeef2dc (DEFAULT_TABLE table=default.Table_146245c7e89e4b01b6b8e2f5f21a9c72 [id=f85f0c435ffa4fdca81d86de4aa4fd5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:16.601009 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c55662a71444a3683a825bcec6c5b68. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:16.601148 446 master_service.cc:946] ReplaceTablet: received request to replace tablet 78689f7ed2a64405adb75f372465a792 from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:16.604502 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8193357427f4491c8c6c1396ceeef2dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:16.608194 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ad9b95c5450a4c9c89a0930862348d49. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:16.610203 892 tablet_service.cc:1467] Processing CreateTablet for tablet 3d132c29cb2c46108e5877e38068ef17 (DEFAULT_TABLE table=default.Table_b31190bf4fa4499da09233dbdd1c9139 [id=44f4b85774f644aabd241d6ecd397a37]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:16.610239 1038 tablet_service.cc:1467] Processing CreateTablet for tablet 3d132c29cb2c46108e5877e38068ef17 (DEFAULT_TABLE table=default.Table_b31190bf4fa4499da09233dbdd1c9139 [id=44f4b85774f644aabd241d6ecd397a37]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:16.610685 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3d132c29cb2c46108e5877e38068ef17. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:16.611053 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3d132c29cb2c46108e5877e38068ef17. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:16.613879 894 tablet_service.cc:1467] Processing CreateTablet for tablet 8193357427f4491c8c6c1396ceeef2dc (DEFAULT_TABLE table=default.Table_146245c7e89e4b01b6b8e2f5f21a9c72 [id=f85f0c435ffa4fdca81d86de4aa4fd5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:16.614809 899 tablet_service.cc:1467] Processing CreateTablet for tablet 0c55662a71444a3683a825bcec6c5b68 (DEFAULT_TABLE table=default.Table_3865c1517889457a87be9ea12b3a156f [id=79b4c84e05ce4571b42e2b4e40fae978]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:16.615782 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8193357427f4491c8c6c1396ceeef2dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:16.617977 1040 tablet_service.cc:1467] Processing CreateTablet for tablet ad9b95c5450a4c9c89a0930862348d49 (DEFAULT_TABLE table=default.Table_7536c30edcec4848b4334695d022517b [id=219c739889d04a97aafa4fe1b55b0a24]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:16.618568 1040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ad9b95c5450a4c9c89a0930862348d49. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:16.624104 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c55662a71444a3683a825bcec6c5b68. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:16.628026 897 tablet_service.cc:1467] Processing CreateTablet for tablet 3455f010e51b4db1a9804a57fd1a4fb8 (DEFAULT_TABLE table=default.Table_da7e6ea4fd81406b94864ce3e0d9b14d [id=a9e0aae9f5af42b796d29a75140f9c29]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:16.628417 1198 ts_tablet_manager.cc:1918] T 9d2e5efaac7f42c998cda21df209c29a P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:16.628600 1198 log.cc:1198] T 9d2e5efaac7f42c998cda21df209c29a P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/9d2e5efaac7f42c998cda21df209c29a
I20250113 02:51:16.628947 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3455f010e51b4db1a9804a57fd1a4fb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:16.629078 1198 ts_tablet_manager.cc:1939] T 9d2e5efaac7f42c998cda21df209c29a P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:16.633713 1042 tablet_service.cc:1467] Processing CreateTablet for tablet f0e7cc161f354f86b8c2fef303e87c5b (DEFAULT_TABLE table=default.Table_20ca05c407b14ba0a92f76f4596ae2ef [id=75cb0745bbb84e1c883cb13b546ecb11]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:16.640076 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 78689f7ed2a64405adb75f372465a792
I20250113 02:51:16.640733 446 catalog_manager.cc:6596] ReplaceTablet: tablet 78689f7ed2a64405adb75f372465a792 deleted and replaced by tablet 0433cfe0cb524511954e1012d4d137e6
I20250113 02:51:16.645071 446 master_service.cc:946] ReplaceTablet: received request to replace tablet 4e61c6a807884a17843955be24de2975 from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:16.646375 748 tablet_service.cc:1514] Processing DeleteTablet for tablet 78689f7ed2a64405adb75f372465a792 with delete_type TABLET_DATA_DELETED (replaced by tablet 0433cfe0cb524511954e1012d4d137e6) from {username='slave'} at 127.0.0.1:55398
W20250113 02:51:16.647212 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 78689f7ed2a64405adb75f372465a792 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 78689f7ed2a64405adb75f372465a792 already in progress: creating tablet
I20250113 02:51:16.657141 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_314d320460b64bccb16b0cd28a07a70f" } new_table_name: "default.Table_f59080d9d91849deafd22506a1af8ff5" modify_external_catalogs: true
I20250113 02:51:16.660519 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_38516e9f569d4f58b983b0692edb2a21" } new_table_name: "default.Table_aafe833f4dfe4ccba9e3d1f86031379c" modify_external_catalogs: true
W20250113 02:51:16.664825 967 tablet.cc:2367] T c1ab705ebcf1430e9e0c376b18841d27 P df69f48656b44bb890f76587ee69cc0f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250113 02:51:16.667362 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_9ccc7f6e9611440db76e053ff2d60f45" } new_table_name: "default.Table_c46ffb30f29b44eea3c09b33f6697efe" modify_external_catalogs: true
I20250113 02:51:16.671447 451 master_service.cc:946] ReplaceTablet: received request to replace tablet 7666ece05d334bac88fb7b58a4350b63 from {username='slave'} at 127.0.0.1:49258
W20250113 02:51:16.673960 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0433cfe0cb524511954e1012d4d137e6 has no consensus state
W20250113 02:51:16.681429 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ec0750fd2fa247feb3ea310ffabe05e5 consensus state has no leader
I20250113 02:51:16.673974 1196 ts_tablet_manager.cc:1918] T 44686b9a58c9440e9932b504ad7957dc P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:16.684808 449 master_service.cc:946] ReplaceTablet: received request to replace tablet 44686b9a58c9440e9932b504ad7957dc from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:16.685009 899 tablet_service.cc:1514] Processing DeleteTablet for tablet 78689f7ed2a64405adb75f372465a792 with delete_type TABLET_DATA_DELETED (replaced by tablet 0433cfe0cb524511954e1012d4d137e6) from {username='slave'} at 127.0.0.1:37366
W20250113 02:51:16.685859 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c62a5e9e43d14b5c94eb87260528469d has no consensus state
I20250113 02:51:16.689213 1196 log.cc:1198] T 44686b9a58c9440e9932b504ad7957dc P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/44686b9a58c9440e9932b504ad7957dc
I20250113 02:51:16.689853 1196 ts_tablet_manager.cc:1939] T 44686b9a58c9440e9932b504ad7957dc P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:16.691334 449 master_service.cc:946] ReplaceTablet: received request to replace tablet b262c70712444c06a61918f21736f339 from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:16.692271 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_a485956f153c4d8b988ff1bf83e2394a" } new_table_name: "default.Table_373ceb7643d241b5a32ab6446397eb95" modify_external_catalogs: true
I20250113 02:51:16.696065 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet 44686b9a58c9440e9932b504ad7957dc (table default.Table_351e5f6de7c64dafb0d9ee4d86510d01 [id=a383dc51c39e4c8ba4b421119c752470]) successfully deleted
W20250113 02:51:16.702205 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ce92c2b45a274eb09e72a96b7ec133be consensus state has no leader
I20250113 02:51:16.702617 1196 tablet_replica.cc:331] T c1ab705ebcf1430e9e0c376b18841d27 P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:16.702929 1196 raft_consensus.cc:2238] T c1ab705ebcf1430e9e0c376b18841d27 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:16.703277 1196 raft_consensus.cc:2267] T c1ab705ebcf1430e9e0c376b18841d27 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
W20250113 02:51:16.703558 1203 raft_consensus.cc:668] T c1ab705ebcf1430e9e0c376b18841d27 P df69f48656b44bb890f76587ee69cc0f: failed to trigger leader election: Illegal state: RaftConsensus is not running: State = Stopped
I20250113 02:51:16.704689 1196 ts_tablet_manager.cc:1905] T c1ab705ebcf1430e9e0c376b18841d27 P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:16.705413 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_c967241e21d948699fdaff1ea61c0d0d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:16.705489 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_97abfde1fbb143d2908bab58cd04645b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:16.706158 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_97abfde1fbb143d2908bab58cd04645b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250113 02:51:16.706197 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c967241e21d948699fdaff1ea61c0d0d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:16.707722 449 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_449479ad3dbb4c1aaa5cd954816b9935" } modify_external_catalogs: true
I20250113 02:51:16.707962 449 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_449479ad3dbb4c1aaa5cd954816b9935" } modify_external_catalogs: true
I20250113 02:51:16.710939 821 heartbeater.cc:502] Master 127.0.104.62:44217 was elected leader, sending a full tablet report...
I20250113 02:51:16.714047 448 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_3865c1517889457a87be9ea12b3a156f" } modify_external_catalogs: true
I20250113 02:51:16.714269 448 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_3865c1517889457a87be9ea12b3a156f" } modify_external_catalogs: true
I20250113 02:51:16.716931 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_bbf0447d9481408cab33e236e7546fd3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:16.717572 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bbf0447d9481408cab33e236e7546fd3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:16.718328 1196 ts_tablet_manager.cc:1918] T c1ab705ebcf1430e9e0c376b18841d27 P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:16.718489 1196 log.cc:1198] T c1ab705ebcf1430e9e0c376b18841d27 P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/c1ab705ebcf1430e9e0c376b18841d27
I20250113 02:51:16.718919 1196 ts_tablet_manager.cc:1939] T c1ab705ebcf1430e9e0c376b18841d27 P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:16.720175 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet c1ab705ebcf1430e9e0c376b18841d27 (table default.Table_373ceb7643d241b5a32ab6446397eb95 [id=7c2bc6d4232f4585b84d71e3e1f85794]) successfully deleted
W20250113 02:51:16.727346 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 78689f7ed2a64405adb75f372465a792 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 78689f7ed2a64405adb75f372465a792 already in progress: creating tablet
I20250113 02:51:16.731195 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_eec2c35d4a8a4196983a872a0fa4bda5" } modify_external_catalogs: true
I20250113 02:51:16.731465 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_eec2c35d4a8a4196983a872a0fa4bda5" } modify_external_catalogs: true
I20250113 02:51:16.731799 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_f59080d9d91849deafd22506a1af8ff5" } modify_external_catalogs: true
I20250113 02:51:16.732122 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_f59080d9d91849deafd22506a1af8ff5" } modify_external_catalogs: true
I20250113 02:51:16.733959 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_fb5e880807124a7f8a203b5798c056b0" } new_table_name: "default.Table_9ea968c0d615400aa500d7bde8d94e3f" modify_external_catalogs: true
I20250113 02:51:16.734234 453 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_5e0263e22f1e4d2b983ae6b6ef03fca5" } modify_external_catalogs: true
I20250113 02:51:16.734480 453 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_5e0263e22f1e4d2b983ae6b6ef03fca5" } modify_external_catalogs: true
I20250113 02:51:16.735143 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_8b98cd7fbceb4d3f8b138e4da9bb0e05"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:16.735844 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8b98cd7fbceb4d3f8b138e4da9bb0e05 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:16.737326 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_7c6817c7dcb74872a38ec4446e4750a9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:16.737922 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7c6817c7dcb74872a38ec4446e4750a9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:16.747349 748 tablet_service.cc:1467] Processing CreateTablet for tablet c62a5e9e43d14b5c94eb87260528469d (DEFAULT_TABLE table=default.Table_c46ffb30f29b44eea3c09b33f6697efe [id=aadd55280b654ceb96c290786a0bfce0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:16.748072 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c62a5e9e43d14b5c94eb87260528469d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:16.759598 1042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f0e7cc161f354f86b8c2fef303e87c5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:16.760959 1033 tablet_service.cc:1467] Processing CreateTablet for tablet 3455f010e51b4db1a9804a57fd1a4fb8 (DEFAULT_TABLE table=default.Table_da7e6ea4fd81406b94864ce3e0d9b14d [id=a9e0aae9f5af42b796d29a75140f9c29]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:16.761798 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3455f010e51b4db1a9804a57fd1a4fb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:16.761929 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet 9d2e5efaac7f42c998cda21df209c29a (table default.Table_7f0e4331a3aa457b95c912f7ab644e0d [id=d9b085b0fd8c418aa964aaea87079200]) successfully deleted
W20250113 02:51:16.762310 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet c1ab705ebcf1430e9e0c376b18841d27 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c1ab705ebcf1430e9e0c376b18841d27 already in progress: creating tablet
I20250113 02:51:16.762526 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_5cf92fa585024c5e905626e0b760ae0d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:16.762881 1031 tablet_service.cc:1514] Processing DeleteTablet for tablet 78689f7ed2a64405adb75f372465a792 with delete_type TABLET_DATA_DELETED (replaced by tablet 0433cfe0cb524511954e1012d4d137e6) from {username='slave'} at 127.0.0.1:60230
W20250113 02:51:16.763274 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5cf92fa585024c5e905626e0b760ae0d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:16.763278 1037 tablet_service.cc:1467] Processing CreateTablet for tablet 0c55662a71444a3683a825bcec6c5b68 (DEFAULT_TABLE table=default.Table_3865c1517889457a87be9ea12b3a156f [id=79b4c84e05ce4571b42e2b4e40fae978]), partition=RANGE (key) PARTITION UNBOUNDED
W20250113 02:51:16.764539 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 78689f7ed2a64405adb75f372465a792 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 78689f7ed2a64405adb75f372465a792 already in progress: creating tablet
I20250113 02:51:16.765002 1035 tablet_service.cc:1467] Processing CreateTablet for tablet 15a05a32febd443dbb519c186048428a (DEFAULT_TABLE table=default.Table_f9aa5df6ba3d43999ebd6570df3108d1 [id=fa5a1cda58f54103ac637c12d99e4e38]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:16.765451 1031 tablet_service.cc:1467] Processing CreateTablet for tablet c62a5e9e43d14b5c94eb87260528469d (DEFAULT_TABLE table=default.Table_c46ffb30f29b44eea3c09b33f6697efe [id=aadd55280b654ceb96c290786a0bfce0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:16.765796 1038 tablet_service.cc:1467] Processing CreateTablet for tablet 8193357427f4491c8c6c1396ceeef2dc (DEFAULT_TABLE table=default.Table_146245c7e89e4b01b6b8e2f5f21a9c72 [id=f85f0c435ffa4fdca81d86de4aa4fd5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:16.765964 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_28d44f0d1c6c4a2cbd27c902820195b9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:16.766579 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_28d44f0d1c6c4a2cbd27c902820195b9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:16.771386 891 tablet_service.cc:1467] Processing CreateTablet for tablet c62a5e9e43d14b5c94eb87260528469d (DEFAULT_TABLE table=default.Table_c46ffb30f29b44eea3c09b33f6697efe [id=aadd55280b654ceb96c290786a0bfce0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:16.771896 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c62a5e9e43d14b5c94eb87260528469d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:16.772730 1039 tablet_service.cc:1467] Processing CreateTablet for tablet fb183b4588e84212a3b4b096f430d47d (DEFAULT_TABLE table=default.Table_32cc476bc0c44beeae94b3c6a36c3a6b [id=33bf7fd42ab5458db0f14347353c0db4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:16.776916 747 tablet_service.cc:1467] Processing CreateTablet for tablet fb183b4588e84212a3b4b096f430d47d (DEFAULT_TABLE table=default.Table_32cc476bc0c44beeae94b3c6a36c3a6b [id=33bf7fd42ab5458db0f14347353c0db4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:16.777396 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb183b4588e84212a3b4b096f430d47d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:16.780249 898 tablet_service.cc:1467] Processing CreateTablet for tablet fb183b4588e84212a3b4b096f430d47d (DEFAULT_TABLE table=default.Table_32cc476bc0c44beeae94b3c6a36c3a6b [id=33bf7fd42ab5458db0f14347353c0db4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:16.780686 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb183b4588e84212a3b4b096f430d47d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:16.782835 1034 tablet_service.cc:1467] Processing CreateTablet for tablet 9041b28402664a08b8761fa155c9c229 (DEFAULT_TABLE table=default.Table_1b75b69f143d4fb2a97f31fd75657b96 [id=7e2f491b17c94a869837d457888bf71b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:16.791249 1191 ts_tablet_manager.cc:1428] T 44686b9a58c9440e9932b504ad7957dc P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.306s user 0.115s sys 0.139s
I20250113 02:51:16.792898 897 tablet_service.cc:1467] Processing CreateTablet for tablet 9041b28402664a08b8761fa155c9c229 (DEFAULT_TABLE table=default.Table_1b75b69f143d4fb2a97f31fd75657b96 [id=7e2f491b17c94a869837d457888bf71b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:16.793097 1191 tablet_bootstrap.cc:492] T 4e61c6a807884a17843955be24de2975 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:16.793370 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9041b28402664a08b8761fa155c9c229. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:16.794780 1191 tablet_bootstrap.cc:654] T 4e61c6a807884a17843955be24de2975 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:16.803499 451 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet 0433cfe0cb524511954e1012d4d137e6
I20250113 02:51:16.807375 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_0d2ea2075ecf40b797bf4579e6700bca"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:16.808272 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0d2ea2075ecf40b797bf4579e6700bca in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:16.810950 748 tablet_service.cc:1467] Processing CreateTablet for tablet 9041b28402664a08b8761fa155c9c229 (DEFAULT_TABLE table=default.Table_1b75b69f143d4fb2a97f31fd75657b96 [id=7e2f491b17c94a869837d457888bf71b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:16.811542 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9041b28402664a08b8761fa155c9c229. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:16.818964 1191 tablet_bootstrap.cc:492] T 4e61c6a807884a17843955be24de2975 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:16.819284 1191 ts_tablet_manager.cc:1397] T 4e61c6a807884a17843955be24de2975 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.026s user 0.006s sys 0.018s
I20250113 02:51:16.821092 1191 raft_consensus.cc:357] T 4e61c6a807884a17843955be24de2975 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:16.821537 1191 raft_consensus.cc:383] T 4e61c6a807884a17843955be24de2975 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:16.821682 1191 raft_consensus.cc:738] T 4e61c6a807884a17843955be24de2975 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:16.822183 1191 consensus_queue.cc:260] T 4e61c6a807884a17843955be24de2975 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:16.823379 1191 ts_tablet_manager.cc:1428] T 4e61c6a807884a17843955be24de2975 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20250113 02:51:16.823930 1191 tablet_bootstrap.cc:492] T 9d2e5efaac7f42c998cda21df209c29a P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:16.826118 1192 tablet_bootstrap.cc:492] T 44686b9a58c9440e9932b504ad7957dc P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:16.826345 1192 ts_tablet_manager.cc:1397] T 44686b9a58c9440e9932b504ad7957dc P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.355s user 0.011s sys 0.074s
I20250113 02:51:16.827418 1191 tablet_bootstrap.cc:654] T 9d2e5efaac7f42c998cda21df209c29a P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:16.827275 1192 raft_consensus.cc:357] T 44686b9a58c9440e9932b504ad7957dc P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:16.827631 1192 raft_consensus.cc:383] T 44686b9a58c9440e9932b504ad7957dc P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:16.827783 1192 raft_consensus.cc:738] T 44686b9a58c9440e9932b504ad7957dc P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:16.828151 1192 consensus_queue.cc:260] T 44686b9a58c9440e9932b504ad7957dc P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:16.830600 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_ce0f77e091564c29bf54791dda458d78"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:16.829938 1037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c55662a71444a3683a825bcec6c5b68. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:16.830977 1192 ts_tablet_manager.cc:1428] T 44686b9a58c9440e9932b504ad7957dc P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
W20250113 02:51:16.831200 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ce0f77e091564c29bf54791dda458d78 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:16.833801 1194 tablet_bootstrap.cc:492] T 23e3cc4d6004439bb95ec3ab2504b1a1 P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:16.834147 1194 ts_tablet_manager.cc:1397] T 23e3cc4d6004439bb95ec3ab2504b1a1 P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.262s user 0.035s sys 0.069s
I20250113 02:51:16.835125 1194 raft_consensus.cc:357] T 23e3cc4d6004439bb95ec3ab2504b1a1 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:16.835439 1194 raft_consensus.cc:383] T 23e3cc4d6004439bb95ec3ab2504b1a1 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:16.835564 1194 raft_consensus.cc:738] T 23e3cc4d6004439bb95ec3ab2504b1a1 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:16.835938 1194 consensus_queue.cc:260] T 23e3cc4d6004439bb95ec3ab2504b1a1 P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:16.839588 1194 ts_tablet_manager.cc:1428] T 23e3cc4d6004439bb95ec3ab2504b1a1 P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250113 02:51:16.842154 1194 tablet_bootstrap.cc:492] T 7666ece05d334bac88fb7b58a4350b63 P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:16.846560 1194 tablet_bootstrap.cc:654] T 7666ece05d334bac88fb7b58a4350b63 P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:16.852910 1192 tablet_bootstrap.cc:492] T 4e61c6a807884a17843955be24de2975 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:16.856184 1194 tablet_bootstrap.cc:492] T 7666ece05d334bac88fb7b58a4350b63 P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:16.856387 1194 ts_tablet_manager.cc:1397] T 7666ece05d334bac88fb7b58a4350b63 P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.001s
I20250113 02:51:16.856817 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c62a5e9e43d14b5c94eb87260528469d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:16.857311 1194 raft_consensus.cc:357] T 7666ece05d334bac88fb7b58a4350b63 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:16.857568 1194 raft_consensus.cc:383] T 7666ece05d334bac88fb7b58a4350b63 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:16.857676 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 15a05a32febd443dbb519c186048428a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:16.857712 1194 raft_consensus.cc:738] T 7666ece05d334bac88fb7b58a4350b63 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:16.857940 1194 consensus_queue.cc:260] T 7666ece05d334bac88fb7b58a4350b63 P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:16.860519 1194 ts_tablet_manager.cc:1428] T 7666ece05d334bac88fb7b58a4350b63 P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250113 02:51:16.860657 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9041b28402664a08b8761fa155c9c229. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:16.861593 1194 tablet_bootstrap.cc:492] T 9d2e5efaac7f42c998cda21df209c29a P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:16.863286 1194 tablet_bootstrap.cc:654] T 9d2e5efaac7f42c998cda21df209c29a P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:16.864442 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb183b4588e84212a3b4b096f430d47d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:16.866372 1192 tablet_bootstrap.cc:654] T 4e61c6a807884a17843955be24de2975 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:16.871891 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8193357427f4491c8c6c1396ceeef2dc. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:16.882205 823 tablet.cc:2367] T 44686b9a58c9440e9932b504ad7957dc P 1186fa869eb14812832fa6562531f196: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250113 02:51:16.881713 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_21dd1d15c2bb4849af81bb4d8e1a5f04"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:16.883518 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_21dd1d15c2bb4849af81bb4d8e1a5f04 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:16.888993 451 catalog_manager.cc:5618] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095) does not have the latest schema for tablet bc88676bf738442d82c25e56f30c4ada (table default.Table_c991f9dbd70d4c4682664b8711f05db4 [id=2c64da64ecdb4bf48e11b523533902e8]). Expected version 1 got 0
I20250113 02:51:16.889304 451 catalog_manager.cc:5618] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095) does not have the latest schema for tablet 89517af3545c47a5a3a247a411190a47 (table default.Table_5d3da53a6d984a7ea593b340ad3e1368 [id=e66cde9b8d9b4763b33e7641601c47da]). Expected version 2 got 0
I20250113 02:51:16.889550 451 catalog_manager.cc:5618] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095) does not have the latest schema for tablet ec0750fd2fa247feb3ea310ffabe05e5 (table default.Table_aafe833f4dfe4ccba9e3d1f86031379c [id=de6b4835f6c04c20b6a5ef8404dc78f7]). Expected version 2 got 0
I20250113 02:51:16.889856 451 catalog_manager.cc:5618] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095) does not have the latest schema for tablet 6f1c2ae654bd46c1a8772fa62c8111ab (table default.Table_ec4ee7380446448293eaa15db62d34de [id=7dc8c3e55ce843009ade78391e271091]). Expected version 3 got 0
I20250113 02:51:16.890363 451 catalog_manager.cc:5618] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095) does not have the latest schema for tablet ce92c2b45a274eb09e72a96b7ec133be (table default.Table_373ceb7643d241b5a32ab6446397eb95 [id=7c2bc6d4232f4585b84d71e3e1f85794]). Expected version 1 got 0
I20250113 02:51:16.890714 451 catalog_manager.cc:5618] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095) does not have the latest schema for tablet 0c57a5add4914835ac366efdcde4ee28 (table default.Table_fb5e880807124a7f8a203b5798c056b0 [id=1a69ebe950b846c78ec36ca96a1f9e6c]). Expected version 1 got 0
I20250113 02:51:16.891557 446 master_service.cc:946] ReplaceTablet: received request to replace tablet 78689f7ed2a64405adb75f372465a792 from {username='slave'} at 127.0.0.1:49258
W20250113 02:51:16.893620 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0c57a5add4914835ac366efdcde4ee28 consensus state has no leader
I20250113 02:51:16.893878 452 master_service.cc:946] ReplaceTablet: received request to replace tablet ec0750fd2fa247feb3ea310ffabe05e5 from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:16.894649 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet c135506190f849008ce6542d569b402a
I20250113 02:51:16.895366 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_da7e6ea4fd81406b94864ce3e0d9b14d" } new_table_name: "default.Table_44f1c4e8cc2b4e83adce8d411ae53448" modify_external_catalogs: true
I20250113 02:51:16.897738 897 tablet_service.cc:1514] Processing DeleteTablet for tablet c135506190f849008ce6542d569b402a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:16 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:16.898442 455 master_service.cc:946] ReplaceTablet: received request to replace tablet 5ae18cbff90d4fa88f6b9b8746fc2027 from {username='slave'} at 127.0.0.1:49258
W20250113 02:51:16.898628 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet c135506190f849008ce6542d569b402a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c135506190f849008ce6542d569b402a already in progress: creating tablet
I20250113 02:51:16.898936 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet ad6b21ed2b824cb0837fc1d8a39d2472
I20250113 02:51:16.899670 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_bbf0447d9481408cab33e236e7546fd3" } new_table_name: "default.Table_4fcd958510914007817364e5a8e8ebec" modify_external_catalogs: true
I20250113 02:51:16.900159 747 tablet_service.cc:1514] Processing DeleteTablet for tablet ad6b21ed2b824cb0837fc1d8a39d2472 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:16 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:16.900265 897 tablet_service.cc:1514] Processing DeleteTablet for tablet ad6b21ed2b824cb0837fc1d8a39d2472 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:16 UTC) from {username='slave'} at 127.0.0.1:37366
W20250113 02:51:16.901053 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet ad6b21ed2b824cb0837fc1d8a39d2472 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ad6b21ed2b824cb0837fc1d8a39d2472 already in progress: creating tablet
W20250113 02:51:16.901338 1110 tablet.cc:2367] T 44686b9a58c9440e9932b504ad7957dc P 4216a7f853d642b7bc3e5cff458f3aca: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250113 02:51:16.905505 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet ad6b21ed2b824cb0837fc1d8a39d2472 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ad6b21ed2b824cb0837fc1d8a39d2472 already in progress: creating tablet
I20250113 02:51:16.906113 1042 tablet_service.cc:1514] Processing DeleteTablet for tablet ad6b21ed2b824cb0837fc1d8a39d2472 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:16 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:16.909580 1191 tablet_bootstrap.cc:492] T 9d2e5efaac7f42c998cda21df209c29a P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:16.909752 1191 ts_tablet_manager.cc:1397] T 9d2e5efaac7f42c998cda21df209c29a P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.086s user 0.022s sys 0.049s
I20250113 02:51:16.910492 1037 tablet_service.cc:1514] Processing DeleteTablet for tablet c135506190f849008ce6542d569b402a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:16 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:16.910590 1191 raft_consensus.cc:357] T 9d2e5efaac7f42c998cda21df209c29a P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:16.910754 748 tablet_service.cc:1514] Processing DeleteTablet for tablet c135506190f849008ce6542d569b402a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:16 UTC) from {username='slave'} at 127.0.0.1:55398
W20250113 02:51:16.911805 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet ad6b21ed2b824cb0837fc1d8a39d2472 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ad6b21ed2b824cb0837fc1d8a39d2472 already in progress: creating tablet
W20250113 02:51:16.912307 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 98f038ad31ff4e05bb88ac68ae341cc2 has no consensus state
I20250113 02:51:16.910843 1191 raft_consensus.cc:383] T 9d2e5efaac7f42c998cda21df209c29a P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:16.912890 1191 raft_consensus.cc:738] T 9d2e5efaac7f42c998cda21df209c29a P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:16.912767 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_aafe833f4dfe4ccba9e3d1f86031379c" } new_table_name: "default.Table_74627d276edf4d75a2fcaa627a7f921b" modify_external_catalogs: true
I20250113 02:51:16.913220 1191 consensus_queue.cc:260] T 9d2e5efaac7f42c998cda21df209c29a P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
W20250113 02:51:16.913620 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet c135506190f849008ce6542d569b402a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c135506190f849008ce6542d569b402a already in progress: creating tablet
I20250113 02:51:16.913812 1191 ts_tablet_manager.cc:1428] T 9d2e5efaac7f42c998cda21df209c29a P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250113 02:51:16.914141 1191 tablet_bootstrap.cc:492] T c1ab705ebcf1430e9e0c376b18841d27 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:16.915503 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 0c55662a71444a3683a825bcec6c5b68
I20250113 02:51:16.915875 1191 tablet_bootstrap.cc:654] T c1ab705ebcf1430e9e0c376b18841d27 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:16.916224 448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_c967241e21d948699fdaff1ea61c0d0d" } new_table_name: "default.Table_870a194df20442a1ae84487760f30246" modify_external_catalogs: true
I20250113 02:51:16.916626 897 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c55662a71444a3683a825bcec6c5b68 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:16 UTC) from {username='slave'} at 127.0.0.1:37366
W20250113 02:51:16.917735 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 0c55662a71444a3683a825bcec6c5b68 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0c55662a71444a3683a825bcec6c5b68 already in progress: creating tablet
I20250113 02:51:16.918886 748 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c55662a71444a3683a825bcec6c5b68 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:16 UTC) from {username='slave'} at 127.0.0.1:55398
W20250113 02:51:16.920450 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 0c55662a71444a3683a825bcec6c5b68 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0c55662a71444a3683a825bcec6c5b68 already in progress: creating tablet
W20250113 02:51:16.922940 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet c135506190f849008ce6542d569b402a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c135506190f849008ce6542d569b402a already in progress: creating tablet
I20250113 02:51:16.924237 1038 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c55662a71444a3683a825bcec6c5b68 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:16 UTC) from {username='slave'} at 127.0.0.1:60230
W20250113 02:51:16.925201 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 0c55662a71444a3683a825bcec6c5b68 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0c55662a71444a3683a825bcec6c5b68 already in progress: creating tablet
I20250113 02:51:16.926589 1191 tablet_bootstrap.cc:492] T c1ab705ebcf1430e9e0c376b18841d27 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:16.926736 1191 ts_tablet_manager.cc:1397] T c1ab705ebcf1430e9e0c376b18841d27 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.004s
I20250113 02:51:16.927498 1191 raft_consensus.cc:357] T c1ab705ebcf1430e9e0c376b18841d27 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:16.927716 1191 raft_consensus.cc:383] T c1ab705ebcf1430e9e0c376b18841d27 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:16.927803 1191 raft_consensus.cc:738] T c1ab705ebcf1430e9e0c376b18841d27 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:16.928071 1191 consensus_queue.cc:260] T c1ab705ebcf1430e9e0c376b18841d27 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
W20250113 02:51:16.928236 448 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9b5dc06ba3ea4056822ca3e4b1249587 has no consensus state
I20250113 02:51:16.928503 448 master_service.cc:946] ReplaceTablet: received request to replace tablet ce92c2b45a274eb09e72a96b7ec133be from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:16.928593 1191 ts_tablet_manager.cc:1428] T c1ab705ebcf1430e9e0c376b18841d27 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250113 02:51:16.928896 1191 tablet_bootstrap.cc:492] T 455b143af8c94f9d8a4111330648cac6 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:16.930730 1191 tablet_bootstrap.cc:654] T 455b143af8c94f9d8a4111330648cac6 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:16.932804 1191 tablet_bootstrap.cc:492] T 455b143af8c94f9d8a4111330648cac6 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:16.933039 1191 ts_tablet_manager.cc:1397] T 455b143af8c94f9d8a4111330648cac6 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20250113 02:51:16.934062 1191 raft_consensus.cc:357] T 455b143af8c94f9d8a4111330648cac6 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:16.934327 1191 raft_consensus.cc:383] T 455b143af8c94f9d8a4111330648cac6 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:16.934456 1191 raft_consensus.cc:738] T 455b143af8c94f9d8a4111330648cac6 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:16.934482 1192 tablet_bootstrap.cc:492] T 4e61c6a807884a17843955be24de2975 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:16.934711 1192 ts_tablet_manager.cc:1397] T 4e61c6a807884a17843955be24de2975 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.103s user 0.002s sys 0.002s
I20250113 02:51:16.934826 1191 consensus_queue.cc:260] T 455b143af8c94f9d8a4111330648cac6 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:16.935472 1191 ts_tablet_manager.cc:1428] T 455b143af8c94f9d8a4111330648cac6 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250113 02:51:16.935909 1192 raft_consensus.cc:357] T 4e61c6a807884a17843955be24de2975 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:16.936228 1192 raft_consensus.cc:383] T 4e61c6a807884a17843955be24de2975 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:16.936309 1192 raft_consensus.cc:738] T 4e61c6a807884a17843955be24de2975 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:16.936966 1191 tablet_bootstrap.cc:492] T 7666ece05d334bac88fb7b58a4350b63 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:16.938675 1191 tablet_bootstrap.cc:654] T 7666ece05d334bac88fb7b58a4350b63 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:16.940557 1192 consensus_queue.cc:260] T 4e61c6a807884a17843955be24de2975 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
W20250113 02:51:16.941118 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3455f010e51b4db1a9804a57fd1a4fb8 consensus state has no leader
I20250113 02:51:16.941529 453 master_service.cc:946] ReplaceTablet: received request to replace tablet 15a05a32febd443dbb519c186048428a from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:16.942960 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet a028cc02273644608f046135851da126
I20250113 02:51:16.943920 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_5d38fa5123014e919f82dafd64c997c1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:16.944644 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5d38fa5123014e919f82dafd64c997c1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:16.946309 1038 tablet_service.cc:1514] Processing DeleteTablet for tablet a028cc02273644608f046135851da126 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:16 UTC) from {username='slave'} at 127.0.0.1:60230
W20250113 02:51:16.948277 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet a028cc02273644608f046135851da126 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a028cc02273644608f046135851da126 already in progress: creating tablet
I20250113 02:51:16.948329 897 tablet_service.cc:1514] Processing DeleteTablet for tablet a028cc02273644608f046135851da126 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:16 UTC) from {username='slave'} at 127.0.0.1:37366
W20250113 02:51:16.948988 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet a028cc02273644608f046135851da126 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a028cc02273644608f046135851da126 already in progress: creating tablet
W20250113 02:51:16.950553 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ec0750fd2fa247feb3ea310ffabe05e5 consensus state has no leader
I20250113 02:51:16.955660 1192 ts_tablet_manager.cc:1428] T 4e61c6a807884a17843955be24de2975 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.021s user 0.003s sys 0.000s
I20250113 02:51:16.959510 897 tablet_service.cc:1514] Processing DeleteTablet for tablet 23e3cc4d6004439bb95ec3ab2504b1a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:16 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:16.959841 1196 tablet_replica.cc:331] T 23e3cc4d6004439bb95ec3ab2504b1a1 P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:16.960047 1196 raft_consensus.cc:2238] T 23e3cc4d6004439bb95ec3ab2504b1a1 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:16.960259 1196 raft_consensus.cc:2267] T 23e3cc4d6004439bb95ec3ab2504b1a1 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:16.962272 1196 ts_tablet_manager.cc:1905] T 23e3cc4d6004439bb95ec3ab2504b1a1 P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:16.966842 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_bee5bcef1f534c6c812e6075291ff90d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:16.967401 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bee5bcef1f534c6c812e6075291ff90d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:16.968945 748 tablet_service.cc:1514] Processing DeleteTablet for tablet a028cc02273644608f046135851da126 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:16 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:16.971318 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_8606b4caf08b43b18b25ee8849eebd07"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:16.972174 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8606b4caf08b43b18b25ee8849eebd07 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:16.974031 1196 ts_tablet_manager.cc:1918] T 23e3cc4d6004439bb95ec3ab2504b1a1 P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250113 02:51:16.974125 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet a028cc02273644608f046135851da126 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a028cc02273644608f046135851da126 already in progress: creating tablet
I20250113 02:51:16.974301 1196 log.cc:1198] T 23e3cc4d6004439bb95ec3ab2504b1a1 P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/23e3cc4d6004439bb95ec3ab2504b1a1
I20250113 02:51:16.975921 897 tablet_service.cc:1514] Processing DeleteTablet for tablet 7666ece05d334bac88fb7b58a4350b63 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:16 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:16.976421 1196 ts_tablet_manager.cc:1939] T 23e3cc4d6004439bb95ec3ab2504b1a1 P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:16.977869 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet 23e3cc4d6004439bb95ec3ab2504b1a1 (table default.Table_dfb2da2848a94fe4b5d89023934e8740 [id=b409b6c2b10e471aba03feb404311e3d]) successfully deleted
I20250113 02:51:16.978232 1196 tablet_replica.cc:331] T 7666ece05d334bac88fb7b58a4350b63 P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:16.978596 1196 raft_consensus.cc:2238] T 7666ece05d334bac88fb7b58a4350b63 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:16.978847 1196 raft_consensus.cc:2267] T 7666ece05d334bac88fb7b58a4350b63 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:16.963227 1038 tablet_service.cc:1514] Processing DeleteTablet for tablet 44686b9a58c9440e9932b504ad7957dc with delete_type TABLET_DATA_DELETED (replaced by tablet c97a6ad98ddc4680a0f071cb91f05402) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:16.970307 1039 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e61c6a807884a17843955be24de2975 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:16 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:16.981263 1198 tablet_replica.cc:331] T 44686b9a58c9440e9932b504ad7957dc P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:16.981504 1196 ts_tablet_manager.cc:1905] T 7666ece05d334bac88fb7b58a4350b63 P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:16.981505 1198 raft_consensus.cc:2238] T 44686b9a58c9440e9932b504ad7957dc P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:16.981710 1198 raft_consensus.cc:2267] T 44686b9a58c9440e9932b504ad7957dc P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:16.982458 1198 ts_tablet_manager.cc:1905] T 44686b9a58c9440e9932b504ad7957dc P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:16.987741 1196 ts_tablet_manager.cc:1918] T 7666ece05d334bac88fb7b58a4350b63 P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:16.987880 1196 log.cc:1198] T 7666ece05d334bac88fb7b58a4350b63 P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/7666ece05d334bac88fb7b58a4350b63
I20250113 02:51:16.988288 1196 ts_tablet_manager.cc:1939] T 7666ece05d334bac88fb7b58a4350b63 P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:16.989121 1198 ts_tablet_manager.cc:1918] T 44686b9a58c9440e9932b504ad7957dc P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:16.989317 1198 log.cc:1198] T 44686b9a58c9440e9932b504ad7957dc P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/44686b9a58c9440e9932b504ad7957dc
I20250113 02:51:16.989780 1198 ts_tablet_manager.cc:1939] T 44686b9a58c9440e9932b504ad7957dc P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:16.990010 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet 7666ece05d334bac88fb7b58a4350b63 (table default.Table_cacef89852fd4576937e26bc2da7b632 [id=1379e5486da84a00b6682ae48a8cfe87]) successfully deleted
I20250113 02:51:16.990999 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet 44686b9a58c9440e9932b504ad7957dc (table default.Table_351e5f6de7c64dafb0d9ee4d86510d01 [id=a383dc51c39e4c8ba4b421119c752470]) successfully deleted
W20250113 02:51:16.991362 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 4e61c6a807884a17843955be24de2975 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4e61c6a807884a17843955be24de2975 already in progress: creating tablet
I20250113 02:51:16.997364 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet ce92c2b45a274eb09e72a96b7ec133be
I20250113 02:51:16.997977 448 catalog_manager.cc:6596] ReplaceTablet: tablet ce92c2b45a274eb09e72a96b7ec133be deleted and replaced by tablet 53cc528d282c47788d5d558569f074c1
I20250113 02:51:16.998489 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet ec0750fd2fa247feb3ea310ffabe05e5
I20250113 02:51:16.999028 452 catalog_manager.cc:6596] ReplaceTablet: tablet ec0750fd2fa247feb3ea310ffabe05e5 deleted and replaced by tablet 107e6e9e2d84496f8fce92b9ba377189
I20250113 02:51:17.000344 1039 tablet_service.cc:1514] Processing DeleteTablet for tablet ce92c2b45a274eb09e72a96b7ec133be with delete_type TABLET_DATA_DELETED (replaced by tablet 53cc528d282c47788d5d558569f074c1) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:17.000599 1038 tablet_service.cc:1514] Processing DeleteTablet for tablet ec0750fd2fa247feb3ea310ffabe05e5 with delete_type TABLET_DATA_DELETED (replaced by tablet 107e6e9e2d84496f8fce92b9ba377189) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:17.001461 898 tablet_service.cc:1514] Processing DeleteTablet for tablet ec0750fd2fa247feb3ea310ffabe05e5 with delete_type TABLET_DATA_DELETED (replaced by tablet 107e6e9e2d84496f8fce92b9ba377189) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:17.001816 897 tablet_service.cc:1514] Processing DeleteTablet for tablet ce92c2b45a274eb09e72a96b7ec133be with delete_type TABLET_DATA_DELETED (replaced by tablet 53cc528d282c47788d5d558569f074c1) from {username='slave'} at 127.0.0.1:37366
W20250113 02:51:17.002565 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet ce92c2b45a274eb09e72a96b7ec133be because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ce92c2b45a274eb09e72a96b7ec133be already in progress: creating tablet
W20250113 02:51:17.002889 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet ec0750fd2fa247feb3ea310ffabe05e5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ec0750fd2fa247feb3ea310ffabe05e5 already in progress: creating tablet
I20250113 02:51:17.003396 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 15a05a32febd443dbb519c186048428a
I20250113 02:51:17.003887 453 catalog_manager.cc:6596] ReplaceTablet: tablet 15a05a32febd443dbb519c186048428a deleted and replaced by tablet 18e2d79d731142f89b9089869bc736e5
I20250113 02:51:17.004491 1038 tablet_service.cc:1514] Processing DeleteTablet for tablet 15a05a32febd443dbb519c186048428a with delete_type TABLET_DATA_DELETED (replaced by tablet 18e2d79d731142f89b9089869bc736e5) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:17.004659 897 tablet_service.cc:1514] Processing DeleteTablet for tablet 15a05a32febd443dbb519c186048428a with delete_type TABLET_DATA_DELETED (replaced by tablet 18e2d79d731142f89b9089869bc736e5) from {username='slave'} at 127.0.0.1:37366
W20250113 02:51:17.005015 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet ec0750fd2fa247feb3ea310ffabe05e5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ec0750fd2fa247feb3ea310ffabe05e5 already in progress: creating tablet
I20250113 02:51:17.005501 748 tablet_service.cc:1514] Processing DeleteTablet for tablet ce92c2b45a274eb09e72a96b7ec133be with delete_type TABLET_DATA_DELETED (replaced by tablet 53cc528d282c47788d5d558569f074c1) from {username='slave'} at 127.0.0.1:55398
W20250113 02:51:17.005053 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 15a05a32febd443dbb519c186048428a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 15a05a32febd443dbb519c186048428a already in progress: creating tablet
W20250113 02:51:17.006062 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet ce92c2b45a274eb09e72a96b7ec133be because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ce92c2b45a274eb09e72a96b7ec133be already in progress: creating tablet
I20250113 02:51:17.006603 748 tablet_service.cc:1514] Processing DeleteTablet for tablet ec0750fd2fa247feb3ea310ffabe05e5 with delete_type TABLET_DATA_DELETED (replaced by tablet 107e6e9e2d84496f8fce92b9ba377189) from {username='slave'} at 127.0.0.1:55398
W20250113 02:51:17.006692 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 15a05a32febd443dbb519c186048428a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 15a05a32febd443dbb519c186048428a already in progress: creating tablet
I20250113 02:51:17.007426 748 tablet_service.cc:1514] Processing DeleteTablet for tablet 15a05a32febd443dbb519c186048428a with delete_type TABLET_DATA_DELETED (replaced by tablet 18e2d79d731142f89b9089869bc736e5) from {username='slave'} at 127.0.0.1:55398
W20250113 02:51:17.008633 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet ce92c2b45a274eb09e72a96b7ec133be because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ce92c2b45a274eb09e72a96b7ec133be already in progress: creating tablet
W20250113 02:51:17.008965 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet ec0750fd2fa247feb3ea310ffabe05e5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ec0750fd2fa247feb3ea310ffabe05e5 already in progress: creating tablet
W20250113 02:51:17.009299 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 15a05a32febd443dbb519c186048428a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 15a05a32febd443dbb519c186048428a already in progress: creating tablet
I20250113 02:51:17.013127 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_4a8b3f444ec345ad8a0c424f291e6d77"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:17.017952 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4a8b3f444ec345ad8a0c424f291e6d77 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:17.014683 453 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_d12d804163994a63a236be44401aa53b" } modify_external_catalogs: true
I20250113 02:51:17.020188 453 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_d12d804163994a63a236be44401aa53b" } modify_external_catalogs: true
I20250113 02:51:17.028836 1192 tablet_bootstrap.cc:492] T b262c70712444c06a61918f21736f339 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:17.032572 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_5727b5d8bf51442e8ae5f7c97f4bfbc8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:17.033296 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5727b5d8bf51442e8ae5f7c97f4bfbc8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:17.036067 1038 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e61c6a807884a17843955be24de2975 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:16 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:17.036413 1198 tablet_replica.cc:331] T 4e61c6a807884a17843955be24de2975 P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:17.036643 1198 raft_consensus.cc:2238] T 4e61c6a807884a17843955be24de2975 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:17.036849 1198 raft_consensus.cc:2267] T 4e61c6a807884a17843955be24de2975 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:17.037875 1198 ts_tablet_manager.cc:1905] T 4e61c6a807884a17843955be24de2975 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:17.046538 1198 ts_tablet_manager.cc:1918] T 4e61c6a807884a17843955be24de2975 P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:17.046872 1198 log.cc:1198] T 4e61c6a807884a17843955be24de2975 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/4e61c6a807884a17843955be24de2975
W20250113 02:51:17.048342 439 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 9b5dc06ba3ea4056822ca3e4b1249587 (table default.Table_870a194df20442a1ae84487760f30246 [id=8d889764fb3046369d87cef8998a5a87], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 9b5dc06ba3ea4056822ca3e4b1249587 has no consensus state
I20250113 02:51:17.052201 1192 tablet_bootstrap.cc:654] T b262c70712444c06a61918f21736f339 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:17.052474 1198 ts_tablet_manager.cc:1939] T 4e61c6a807884a17843955be24de2975 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:17.054701 898 tablet_service.cc:1467] Processing CreateTablet for tablet 2fb62cc83f9c4dcfa9aa01ae8b46c025 (DEFAULT_TABLE table=default.Table_97abfde1fbb143d2908bab58cd04645b [id=dd19d0450abb4862a408f2f5f5922a4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:17.055156 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2fb62cc83f9c4dcfa9aa01ae8b46c025. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:17.055197 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet 4e61c6a807884a17843955be24de2975 (table default.Table_6a96fb47f7444dcfb90d4463de217db6 [id=5eea6db2313a4b9abc61ea0545091e17]) successfully deleted
I20250113 02:51:17.058265 748 tablet_service.cc:1467] Processing CreateTablet for tablet 98f038ad31ff4e05bb88ac68ae341cc2 (DEFAULT_TABLE table=default.Table_4fcd958510914007817364e5a8e8ebec [id=6666a614063942c99a78b95d35add037]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:17.054756 891 tablet_service.cc:1467] Processing CreateTablet for tablet 9b5dc06ba3ea4056822ca3e4b1249587 (DEFAULT_TABLE table=default.Table_870a194df20442a1ae84487760f30246 [id=8d889764fb3046369d87cef8998a5a87]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:17.058746 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 98f038ad31ff4e05bb88ac68ae341cc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:17.058688 1039 tablet_service.cc:1467] Processing CreateTablet for tablet 9b5dc06ba3ea4056822ca3e4b1249587 (DEFAULT_TABLE table=default.Table_870a194df20442a1ae84487760f30246 [id=8d889764fb3046369d87cef8998a5a87]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:17.058775 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b5dc06ba3ea4056822ca3e4b1249587. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:17.059094 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b5dc06ba3ea4056822ca3e4b1249587. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:17.054701 897 tablet_service.cc:1467] Processing CreateTablet for tablet 98f038ad31ff4e05bb88ac68ae341cc2 (DEFAULT_TABLE table=default.Table_4fcd958510914007817364e5a8e8ebec [id=6666a614063942c99a78b95d35add037]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:17.062225 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 98f038ad31ff4e05bb88ac68ae341cc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:17.064288 747 tablet_service.cc:1467] Processing CreateTablet for tablet 2fb62cc83f9c4dcfa9aa01ae8b46c025 (DEFAULT_TABLE table=default.Table_97abfde1fbb143d2908bab58cd04645b [id=dd19d0450abb4862a408f2f5f5922a4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:17.064675 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2fb62cc83f9c4dcfa9aa01ae8b46c025. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:17.068245 744 tablet_service.cc:1467] Processing CreateTablet for tablet 9b5dc06ba3ea4056822ca3e4b1249587 (DEFAULT_TABLE table=default.Table_870a194df20442a1ae84487760f30246 [id=8d889764fb3046369d87cef8998a5a87]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:17.068625 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b5dc06ba3ea4056822ca3e4b1249587. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:17.072062 1038 tablet_service.cc:1467] Processing CreateTablet for tablet 2fb62cc83f9c4dcfa9aa01ae8b46c025 (DEFAULT_TABLE table=default.Table_97abfde1fbb143d2908bab58cd04645b [id=dd19d0450abb4862a408f2f5f5922a4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:17.072477 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2fb62cc83f9c4dcfa9aa01ae8b46c025. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:17.072527 1194 tablet_bootstrap.cc:492] T 9d2e5efaac7f42c998cda21df209c29a P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:17.072669 1194 ts_tablet_manager.cc:1397] T 9d2e5efaac7f42c998cda21df209c29a P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.211s user 0.036s sys 0.053s
I20250113 02:51:17.073490 1194 raft_consensus.cc:357] T 9d2e5efaac7f42c998cda21df209c29a P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:17.073691 1194 raft_consensus.cc:383] T 9d2e5efaac7f42c998cda21df209c29a P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:17.073752 1194 raft_consensus.cc:738] T 9d2e5efaac7f42c998cda21df209c29a P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:17.075060 1192 tablet_bootstrap.cc:492] T b262c70712444c06a61918f21736f339 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:17.075227 1192 ts_tablet_manager.cc:1397] T b262c70712444c06a61918f21736f339 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.047s user 0.003s sys 0.000s
I20250113 02:51:17.075779 1209 raft_consensus.cc:491] T 44686b9a58c9440e9932b504ad7957dc P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:17.076017 1209 raft_consensus.cc:513] T 44686b9a58c9440e9932b504ad7957dc P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:17.076267 1192 raft_consensus.cc:357] T b262c70712444c06a61918f21736f339 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:17.076531 1192 raft_consensus.cc:383] T b262c70712444c06a61918f21736f339 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:17.076638 1192 raft_consensus.cc:738] T b262c70712444c06a61918f21736f339 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:17.076957 1192 consensus_queue.cc:260] T b262c70712444c06a61918f21736f339 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:17.077625 1192 ts_tablet_manager.cc:1428] T b262c70712444c06a61918f21736f339 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20250113 02:51:17.077788 1209 leader_election.cc:290] T 44686b9a58c9440e9932b504ad7957dc P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:17.081902 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_f1d42e491cb34117a3bed634de503a6d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:17.082705 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f1d42e491cb34117a3bed634de503a6d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:17.084357 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_c94783b201764702b6de4f3219090efe"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:17.085005 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c94783b201764702b6de4f3219090efe in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:17.093524 1194 consensus_queue.cc:260] T 9d2e5efaac7f42c998cda21df209c29a P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:17.094445 754 tablet_service.cc:1514] Processing DeleteTablet for tablet c97a6ad98ddc4680a0f071cb91f05402 with delete_type TABLET_DATA_DELETED (replaced by tablet 2fe235d5280944ef92cf15fb458f2f81) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:17.094781 754 tablet_service.cc:1514] Processing DeleteTablet for tablet b262c70712444c06a61918f21736f339 with delete_type TABLET_DATA_DELETED (replaced by tablet c135506190f849008ce6542d569b402a) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:17.094978 754 tablet_service.cc:1514] Processing DeleteTablet for tablet 7dbda00269cf4f7095b3504ae87ac243 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:16 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:17.095206 752 tablet_service.cc:1514] Processing DeleteTablet for tablet c1ab705ebcf1430e9e0c376b18841d27 with delete_type TABLET_DATA_DELETED (replaced by tablet ce92c2b45a274eb09e72a96b7ec133be) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:17.095417 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a323451cdfe4fd69ec62cda230f9afb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:16 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:17.095633 755 tablet_service.cc:1514] Processing DeleteTablet for tablet 015447488b734f6a93b8c36be975c031 with delete_type TABLET_DATA_DELETED (replaced by tablet 9a323451cdfe4fd69ec62cda230f9afb) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:17.095827 745 tablet_service.cc:1514] Processing DeleteTablet for tablet 23e3cc4d6004439bb95ec3ab2504b1a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:16 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:17.096078 750 tablet_service.cc:1514] Processing DeleteTablet for tablet a028cc02273644608f046135851da126 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:16 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:17.096272 751 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e61c6a807884a17843955be24de2975 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:16 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:17.096475 753 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d2e5efaac7f42c998cda21df209c29a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:16 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:17.096295 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_4863cef058bb4fc7b7f38735eb20482c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:17.096648 1194 ts_tablet_manager.cc:1428] T 9d2e5efaac7f42c998cda21df209c29a P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.024s user 0.000s sys 0.002s
I20250113 02:51:17.096676 746 tablet_service.cc:1514] Processing DeleteTablet for tablet 44686b9a58c9440e9932b504ad7957dc with delete_type TABLET_DATA_DELETED (replaced by tablet c97a6ad98ddc4680a0f071cb91f05402) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:17.096886 743 tablet_service.cc:1514] Processing DeleteTablet for tablet 1643b3e1607f42fc82886d0b09e2fe8c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:16 UTC) from {username='slave'} at 127.0.0.1:55398
W20250113 02:51:17.097057 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4863cef058bb4fc7b7f38735eb20482c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:17.097126 742 tablet_service.cc:1514] Processing DeleteTablet for tablet 7666ece05d334bac88fb7b58a4350b63 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:16 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:17.097396 741 tablet_service.cc:1514] Processing DeleteTablet for tablet c135506190f849008ce6542d569b402a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:16 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:17.097656 740 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c55662a71444a3683a825bcec6c5b68 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:16 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:17.097908 739 tablet_service.cc:1514] Processing DeleteTablet for tablet ad6b21ed2b824cb0837fc1d8a39d2472 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:16 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:17.098032 1194 tablet_bootstrap.cc:492] T 89517af3545c47a5a3a247a411190a47 P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:17.098191 738 tablet_service.cc:1514] Processing DeleteTablet for tablet 78689f7ed2a64405adb75f372465a792 with delete_type TABLET_DATA_DELETED (replaced by tablet 0433cfe0cb524511954e1012d4d137e6) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:17.098479 737 tablet_service.cc:1514] Processing DeleteTablet for tablet 455b143af8c94f9d8a4111330648cac6 with delete_type TABLET_DATA_DELETED (replaced by tablet bc88676bf738442d82c25e56f30c4ada) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:17.098582 454 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_21dd1d15c2bb4849af81bb4d8e1a5f04" } modify_external_catalogs: true
I20250113 02:51:17.098805 736 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ae18cbff90d4fa88f6b9b8746fc2027 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:16 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:17.098805 454 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_21dd1d15c2bb4849af81bb4d8e1a5f04" } modify_external_catalogs: true
I20250113 02:51:17.099885 1195 tablet_replica.cc:331] T c1ab705ebcf1430e9e0c376b18841d27 P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:17.101092 1194 tablet_bootstrap.cc:654] T 89517af3545c47a5a3a247a411190a47 P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:17.104146 1195 raft_consensus.cc:2238] T c1ab705ebcf1430e9e0c376b18841d27 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:17.104614 1195 raft_consensus.cc:2267] T c1ab705ebcf1430e9e0c376b18841d27 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:17.106701 1195 ts_tablet_manager.cc:1905] T c1ab705ebcf1430e9e0c376b18841d27 P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:17.106999 897 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d2e5efaac7f42c998cda21df209c29a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:16 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:17.107292 1196 tablet_replica.cc:331] T 9d2e5efaac7f42c998cda21df209c29a P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:17.107483 1196 raft_consensus.cc:2238] T 9d2e5efaac7f42c998cda21df209c29a P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:17.107697 1196 raft_consensus.cc:2267] T 9d2e5efaac7f42c998cda21df209c29a P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:17.108573 1196 ts_tablet_manager.cc:1905] T 9d2e5efaac7f42c998cda21df209c29a P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:17.115092 1195 ts_tablet_manager.cc:1918] T c1ab705ebcf1430e9e0c376b18841d27 P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:17.115417 1195 log.cc:1198] T c1ab705ebcf1430e9e0c376b18841d27 P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/c1ab705ebcf1430e9e0c376b18841d27
I20250113 02:51:17.115906 1195 ts_tablet_manager.cc:1939] T c1ab705ebcf1430e9e0c376b18841d27 P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:17.117132 1196 ts_tablet_manager.cc:1918] T 9d2e5efaac7f42c998cda21df209c29a P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:17.117259 1196 log.cc:1198] T 9d2e5efaac7f42c998cda21df209c29a P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/9d2e5efaac7f42c998cda21df209c29a
I20250113 02:51:17.117580 1196 ts_tablet_manager.cc:1939] T 9d2e5efaac7f42c998cda21df209c29a P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:17.118607 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet 9d2e5efaac7f42c998cda21df209c29a (table default.Table_7f0e4331a3aa457b95c912f7ab644e0d [id=d9b085b0fd8c418aa964aaea87079200]) successfully deleted
I20250113 02:51:17.119374 1209 raft_consensus.cc:491] T 4e61c6a807884a17843955be24de2975 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:17.119612 1209 raft_consensus.cc:513] T 4e61c6a807884a17843955be24de2975 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:17.120530 1209 leader_election.cc:290] T 4e61c6a807884a17843955be24de2975 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:17.122069 1195 tablet_replica.cc:331] T 4e61c6a807884a17843955be24de2975 P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:17.122354 1195 raft_consensus.cc:2238] T 4e61c6a807884a17843955be24de2975 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:17.122596 1195 raft_consensus.cc:2267] T 4e61c6a807884a17843955be24de2975 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:17.123453 1195 ts_tablet_manager.cc:1905] T 4e61c6a807884a17843955be24de2975 P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:17.126071 1191 tablet_bootstrap.cc:492] T 7666ece05d334bac88fb7b58a4350b63 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:17.126236 1191 ts_tablet_manager.cc:1397] T 7666ece05d334bac88fb7b58a4350b63 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.189s user 0.061s sys 0.073s
I20250113 02:51:17.127010 1191 raft_consensus.cc:357] T 7666ece05d334bac88fb7b58a4350b63 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
W20250113 02:51:17.127151 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet c97a6ad98ddc4680a0f071cb91f05402 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c97a6ad98ddc4680a0f071cb91f05402 already in progress: creating tablet
I20250113 02:51:17.127269 1191 raft_consensus.cc:383] T 7666ece05d334bac88fb7b58a4350b63 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:17.127362 1191 raft_consensus.cc:738] T 7666ece05d334bac88fb7b58a4350b63 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
W20250113 02:51:17.127404 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet b262c70712444c06a61918f21736f339 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b262c70712444c06a61918f21736f339 already in progress: creating tablet
W20250113 02:51:17.127653 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 7dbda00269cf4f7095b3504ae87ac243 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7dbda00269cf4f7095b3504ae87ac243 already in progress: creating tablet
I20250113 02:51:17.128966 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_a27963e06eed4ec08989e43969f3bbbb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:17.129719 1191 consensus_queue.cc:260] T 7666ece05d334bac88fb7b58a4350b63 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:17.132373 1191 ts_tablet_manager.cc:1428] T 7666ece05d334bac88fb7b58a4350b63 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20250113 02:51:17.129875 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_7536c30edcec4848b4334695d022517b" } new_table_name: "default.Table_5e955e7a2ca54f1f8ee6012902e3d98c" modify_external_catalogs: true
I20250113 02:51:17.130756 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet c1ab705ebcf1430e9e0c376b18841d27 (table default.Table_373ceb7643d241b5a32ab6446397eb95 [id=7c2bc6d4232f4585b84d71e3e1f85794]) successfully deleted
W20250113 02:51:17.133797 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 9a323451cdfe4fd69ec62cda230f9afb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9a323451cdfe4fd69ec62cda230f9afb already in progress: creating tablet
I20250113 02:51:17.134989 1191 tablet_bootstrap.cc:492] T 23e3cc4d6004439bb95ec3ab2504b1a1 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
W20250113 02:51:17.137526 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 015447488b734f6a93b8c36be975c031 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 015447488b734f6a93b8c36be975c031 already in progress: creating tablet
W20250113 02:51:17.138049 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 23e3cc4d6004439bb95ec3ab2504b1a1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 23e3cc4d6004439bb95ec3ab2504b1a1 already in progress: creating tablet
W20250113 02:51:17.139696 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet a028cc02273644608f046135851da126 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a028cc02273644608f046135851da126 already in progress: creating tablet
W20250113 02:51:17.132387 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a27963e06eed4ec08989e43969f3bbbb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:17.144387 1191 tablet_bootstrap.cc:654] T 23e3cc4d6004439bb95ec3ab2504b1a1 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:17.144727 748 tablet_service.cc:1514] Processing DeleteTablet for tablet c1ab705ebcf1430e9e0c376b18841d27 with delete_type TABLET_DATA_DELETED (replaced by tablet ce92c2b45a274eb09e72a96b7ec133be) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:17.146229 1031 tablet_service.cc:1467] Processing CreateTablet for tablet 98f038ad31ff4e05bb88ac68ae341cc2 (DEFAULT_TABLE table=default.Table_4fcd958510914007817364e5a8e8ebec [id=6666a614063942c99a78b95d35add037]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:17.141084 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_98a51e7ca6354ce0958333364e11adaf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:17.150449 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e61c6a807884a17843955be24de2975" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:17.151289 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_98a51e7ca6354ce0958333364e11adaf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250113 02:51:17.151449 711 leader_election.cc:343] T 44686b9a58c9440e9932b504ad7957dc P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Not found: Tablet not found: 44686b9a58c9440e9932b504ad7957dc
W20250113 02:51:17.151746 711 leader_election.cc:343] T 4e61c6a807884a17843955be24de2975 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:17.156201 1195 ts_tablet_manager.cc:1918] T 4e61c6a807884a17843955be24de2975 P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:17.156649 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_0d2ea2075ecf40b797bf4579e6700bca" } modify_external_catalogs: true
I20250113 02:51:17.159229 1195 log.cc:1198] T 4e61c6a807884a17843955be24de2975 P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/4e61c6a807884a17843955be24de2975
I20250113 02:51:17.159262 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_0d2ea2075ecf40b797bf4579e6700bca" } modify_external_catalogs: true
I20250113 02:51:17.159605 1195 ts_tablet_manager.cc:1939] T 4e61c6a807884a17843955be24de2975 P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:17.160252 1195 tablet_replica.cc:331] T 9d2e5efaac7f42c998cda21df209c29a P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:17.160411 1195 raft_consensus.cc:2238] T 9d2e5efaac7f42c998cda21df209c29a P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:17.157429 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_373ceb7643d241b5a32ab6446397eb95" } new_table_name: "default.Table_c5c12ef64b7b438eb52313e87793e601" modify_external_catalogs: true
I20250113 02:51:17.162354 1195 raft_consensus.cc:2267] T 9d2e5efaac7f42c998cda21df209c29a P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:17.164371 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet 4e61c6a807884a17843955be24de2975 (table default.Table_6a96fb47f7444dcfb90d4463de217db6 [id=5eea6db2313a4b9abc61ea0545091e17]) successfully deleted
I20250113 02:51:17.164465 1191 tablet_bootstrap.cc:492] T 23e3cc4d6004439bb95ec3ab2504b1a1 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:17.164624 1191 ts_tablet_manager.cc:1397] T 23e3cc4d6004439bb95ec3ab2504b1a1 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.032s user 0.004s sys 0.004s
I20250113 02:51:17.165041 1195 ts_tablet_manager.cc:1905] T 9d2e5efaac7f42c998cda21df209c29a P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:17.165481 1191 raft_consensus.cc:357] T 23e3cc4d6004439bb95ec3ab2504b1a1 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:17.165696 1191 raft_consensus.cc:383] T 23e3cc4d6004439bb95ec3ab2504b1a1 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:17.165835 1191 raft_consensus.cc:738] T 23e3cc4d6004439bb95ec3ab2504b1a1 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:17.166479 1191 consensus_queue.cc:260] T 23e3cc4d6004439bb95ec3ab2504b1a1 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:17.167124 1191 ts_tablet_manager.cc:1428] T 23e3cc4d6004439bb95ec3ab2504b1a1 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250113 02:51:17.167382 1191 tablet_bootstrap.cc:492] T 015447488b734f6a93b8c36be975c031 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:17.169024 1191 tablet_bootstrap.cc:654] T 015447488b734f6a93b8c36be975c031 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:17.186841 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 98f038ad31ff4e05bb88ac68ae341cc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:17.193065 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet d6675bc3cb5f4f809320e28026cdf4b0
I20250113 02:51:17.193516 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_b31190bf4fa4499da09233dbdd1c9139" } modify_external_catalogs: true
I20250113 02:51:17.193750 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_b31190bf4fa4499da09233dbdd1c9139" } modify_external_catalogs: true
I20250113 02:51:17.193850 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_32cc476bc0c44beeae94b3c6a36c3a6b" } new_table_name: "default.Table_06e860dcf006410ea4258e4ebe7e3104" modify_external_catalogs: true
I20250113 02:51:17.195444 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_e98cbfadda17490b9078136422d0fa57" } new_table_name: "default.Table_b581b1a063a54551a8535adf01983fb8" modify_external_catalogs: true
I20250113 02:51:17.195967 1039 tablet_service.cc:1514] Processing DeleteTablet for tablet d6675bc3cb5f4f809320e28026cdf4b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:17 UTC) from {username='slave'} at 127.0.0.1:60230
W20250113 02:51:17.197115 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet d6675bc3cb5f4f809320e28026cdf4b0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d6675bc3cb5f4f809320e28026cdf4b0 already in progress: creating tablet
I20250113 02:51:17.197474 1038 tablet_service.cc:1514] Processing DeleteTablet for tablet b262c70712444c06a61918f21736f339 with delete_type TABLET_DATA_DELETED (replaced by tablet c135506190f849008ce6542d569b402a) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:17.197769 898 tablet_service.cc:1514] Processing DeleteTablet for tablet d6675bc3cb5f4f809320e28026cdf4b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:17 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:17.198125 1198 tablet_replica.cc:331] T b262c70712444c06a61918f21736f339 P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:17.198318 1198 raft_consensus.cc:2238] T b262c70712444c06a61918f21736f339 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:17.198482 1198 raft_consensus.cc:2267] T b262c70712444c06a61918f21736f339 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:17.198585 448 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_ec4ee7380446448293eaa15db62d34de" } modify_external_catalogs: true
I20250113 02:51:17.198760 448 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_ec4ee7380446448293eaa15db62d34de" } modify_external_catalogs: true
W20250113 02:51:17.199465 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ad9b95c5450a4c9c89a0930862348d49 consensus state has no leader
W20250113 02:51:17.199739 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet d6675bc3cb5f4f809320e28026cdf4b0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d6675bc3cb5f4f809320e28026cdf4b0 already in progress: creating tablet
I20250113 02:51:17.199853 1198 ts_tablet_manager.cc:1905] T b262c70712444c06a61918f21736f339 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:17.200093 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_146245c7e89e4b01b6b8e2f5f21a9c72" } new_table_name: "default.Table_6d493198af4046609058d1f1a51ab823" modify_external_catalogs: true
I20250113 02:51:17.203122 447 master_service.cc:946] ReplaceTablet: received request to replace tablet b262c70712444c06a61918f21736f339 from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:17.203814 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_b147d4f32fa04b849ff6b75c9382a4e5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:17.204430 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b147d4f32fa04b849ff6b75c9382a4e5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:17.219028 1198 ts_tablet_manager.cc:1918] T b262c70712444c06a61918f21736f339 P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:17.219203 1198 log.cc:1198] T b262c70712444c06a61918f21736f339 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/b262c70712444c06a61918f21736f339
I20250113 02:51:17.219537 1198 ts_tablet_manager.cc:1939] T b262c70712444c06a61918f21736f339 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:17.220280 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet b262c70712444c06a61918f21736f339 (table default.Table_5e0263e22f1e4d2b983ae6b6ef03fca5 [id=377f041bf95e41f3b7e8ea26f9795b6a]) successfully deleted
I20250113 02:51:17.221035 1192 tablet_bootstrap.cc:492] T 7666ece05d334bac88fb7b58a4350b63 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:17.221417 748 tablet_service.cc:1514] Processing DeleteTablet for tablet d6675bc3cb5f4f809320e28026cdf4b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:17 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:17.227542 1195 ts_tablet_manager.cc:1918] T 9d2e5efaac7f42c998cda21df209c29a P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:17.241217 1191 tablet_bootstrap.cc:492] T 015447488b734f6a93b8c36be975c031 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:17.242374 1191 ts_tablet_manager.cc:1397] T 015447488b734f6a93b8c36be975c031 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.075s user 0.016s sys 0.020s
I20250113 02:51:17.242563 1195 log.cc:1198] T 9d2e5efaac7f42c998cda21df209c29a P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/9d2e5efaac7f42c998cda21df209c29a
I20250113 02:51:17.243177 1195 ts_tablet_manager.cc:1939] T 9d2e5efaac7f42c998cda21df209c29a P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:17.246351 452 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet d901ca06d083440aa8e67378126e00b7
I20250113 02:51:17.248844 1038 tablet_service.cc:1514] Processing DeleteTablet for tablet b262c70712444c06a61918f21736f339 with delete_type TABLET_DATA_DELETED (replaced by tablet c135506190f849008ce6542d569b402a) from {username='slave'} at 127.0.0.1:60230
W20250113 02:51:17.249564 440 catalog_manager.cc:4839] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet b262c70712444c06a61918f21736f339 because the tablet was not found. No further retry: Not found: Tablet not found: b262c70712444c06a61918f21736f339
I20250113 02:51:17.249743 452 master_service.cc:946] ReplaceTablet: received request to replace tablet c62a5e9e43d14b5c94eb87260528469d from {username='slave'} at 127.0.0.1:49258
W20250113 02:51:17.250301 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 53cc528d282c47788d5d558569f074c1 has no consensus state
I20250113 02:51:17.250849 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_e1ae667a93ee48e8a3d233eeb6335364"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:17.251574 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e1ae667a93ee48e8a3d233eeb6335364 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250113 02:51:17.253240 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fb183b4588e84212a3b4b096f430d47d consensus state has no leader
I20250113 02:51:17.253574 453 master_service.cc:946] ReplaceTablet: received request to replace tablet ad6b21ed2b824cb0837fc1d8a39d2472 from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:17.254087 453 master_service.cc:946] ReplaceTablet: received request to replace tablet c97a6ad98ddc4680a0f071cb91f05402 from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:17.254503 453 master_service.cc:946] ReplaceTablet: received request to replace tablet 3d132c29cb2c46108e5877e38068ef17 from {username='slave'} at 127.0.0.1:49258
W20250113 02:51:17.255024 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7f3cbef7d91d4576822d913071a88248 consensus state has no leader
I20250113 02:51:17.243248 1191 raft_consensus.cc:357] T 015447488b734f6a93b8c36be975c031 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:17.255511 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_c59cac1b4e0249129c7671a4919283c8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:17.255707 1191 raft_consensus.cc:383] T 015447488b734f6a93b8c36be975c031 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:17.255815 1191 raft_consensus.cc:738] T 015447488b734f6a93b8c36be975c031 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
W20250113 02:51:17.256088 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c59cac1b4e0249129c7671a4919283c8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:17.256109 1191 consensus_queue.cc:260] T 015447488b734f6a93b8c36be975c031 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:17.256736 1191 ts_tablet_manager.cc:1428] T 015447488b734f6a93b8c36be975c031 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20250113 02:51:17.257082 1191 tablet_bootstrap.cc:492] T b262c70712444c06a61918f21736f339 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:17.258884 1195 tablet_replica.cc:331] T 44686b9a58c9440e9932b504ad7957dc P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:17.258956 1191 tablet_bootstrap.cc:654] T b262c70712444c06a61918f21736f339 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:17.259079 1195 raft_consensus.cc:2238] T 44686b9a58c9440e9932b504ad7957dc P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:17.259239 1195 raft_consensus.cc:2267] T 44686b9a58c9440e9932b504ad7957dc P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:17.259931 1195 ts_tablet_manager.cc:1905] T 44686b9a58c9440e9932b504ad7957dc P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:17.264631 748 tablet_service.cc:1514] Processing DeleteTablet for tablet 23e3cc4d6004439bb95ec3ab2504b1a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:16 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:17.268654 1191 tablet_bootstrap.cc:492] T b262c70712444c06a61918f21736f339 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:17.268826 1191 ts_tablet_manager.cc:1397] T b262c70712444c06a61918f21736f339 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.003s
I20250113 02:51:17.269591 1191 raft_consensus.cc:357] T b262c70712444c06a61918f21736f339 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:17.269829 1191 raft_consensus.cc:383] T b262c70712444c06a61918f21736f339 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:17.269933 1191 raft_consensus.cc:738] T b262c70712444c06a61918f21736f339 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:17.270205 1191 consensus_queue.cc:260] T b262c70712444c06a61918f21736f339 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:17.271811 1191 ts_tablet_manager.cc:1428] T b262c70712444c06a61918f21736f339 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250113 02:51:17.273653 1191 tablet_bootstrap.cc:492] T 89517af3545c47a5a3a247a411190a47 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
W20250113 02:51:17.273938 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8193357427f4491c8c6c1396ceeef2dc consensus state has no leader
I20250113 02:51:17.266521 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet 9d2e5efaac7f42c998cda21df209c29a (table default.Table_7f0e4331a3aa457b95c912f7ab644e0d [id=d9b085b0fd8c418aa964aaea87079200]) successfully deleted
I20250113 02:51:17.275393 1191 tablet_bootstrap.cc:654] T 89517af3545c47a5a3a247a411190a47 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:17.279268 1195 ts_tablet_manager.cc:1918] T 44686b9a58c9440e9932b504ad7957dc P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:17.282533 748 tablet_service.cc:1514] Processing DeleteTablet for tablet 015447488b734f6a93b8c36be975c031 with delete_type TABLET_DATA_DELETED (replaced by tablet 9a323451cdfe4fd69ec62cda230f9afb) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:17.284149 1195 log.cc:1198] T 44686b9a58c9440e9932b504ad7957dc P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/44686b9a58c9440e9932b504ad7957dc
I20250113 02:51:17.284595 1195 ts_tablet_manager.cc:1939] T 44686b9a58c9440e9932b504ad7957dc P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:17.289151 1191 tablet_bootstrap.cc:492] T 89517af3545c47a5a3a247a411190a47 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:17.291849 1191 ts_tablet_manager.cc:1397] T 89517af3545c47a5a3a247a411190a47 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.018s user 0.000s sys 0.007s
I20250113 02:51:17.292788 1191 raft_consensus.cc:357] T 89517af3545c47a5a3a247a411190a47 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:17.293144 1191 raft_consensus.cc:383] T 89517af3545c47a5a3a247a411190a47 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:17.293265 1191 raft_consensus.cc:738] T 89517af3545c47a5a3a247a411190a47 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:17.293915 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet 44686b9a58c9440e9932b504ad7957dc (table default.Table_351e5f6de7c64dafb0d9ee4d86510d01 [id=a383dc51c39e4c8ba4b421119c752470]) successfully deleted
I20250113 02:51:17.293582 1191 consensus_queue.cc:260] T 89517af3545c47a5a3a247a411190a47 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
W20250113 02:51:17.294886 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 1643b3e1607f42fc82886d0b09e2fe8c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1643b3e1607f42fc82886d0b09e2fe8c already in progress: creating tablet
I20250113 02:51:17.295179 1195 tablet_replica.cc:331] T 7666ece05d334bac88fb7b58a4350b63 P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:17.295509 748 tablet_service.cc:1514] Processing DeleteTablet for tablet b262c70712444c06a61918f21736f339 with delete_type TABLET_DATA_DELETED (replaced by tablet c135506190f849008ce6542d569b402a) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:17.297644 1192 tablet_bootstrap.cc:654] T 7666ece05d334bac88fb7b58a4350b63 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:17.298617 1191 ts_tablet_manager.cc:1428] T 89517af3545c47a5a3a247a411190a47 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20250113 02:51:17.298979 1191 tablet_bootstrap.cc:492] T 5ae18cbff90d4fa88f6b9b8746fc2027 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:17.301036 1195 raft_consensus.cc:2238] T 7666ece05d334bac88fb7b58a4350b63 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:17.301242 1195 raft_consensus.cc:2267] T 7666ece05d334bac88fb7b58a4350b63 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:17.303336 1195 ts_tablet_manager.cc:1905] T 7666ece05d334bac88fb7b58a4350b63 P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250113 02:51:17.303861 709 leader_election.cc:343] T 44686b9a58c9440e9932b504ad7957dc P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Not found: Tablet not found: 44686b9a58c9440e9932b504ad7957dc
I20250113 02:51:17.304129 709 leader_election.cc:304] T 44686b9a58c9440e9932b504ad7957dc P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
W20250113 02:51:17.305012 709 raft_consensus.cc:2681] T 44686b9a58c9440e9932b504ad7957dc P 1186fa869eb14812832fa6562531f196: unable to run election callback: Service unavailable: Thread pool token was shut down
W20250113 02:51:17.305868 709 leader_election.cc:343] T 4e61c6a807884a17843955be24de2975 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Not found: Tablet not found: 4e61c6a807884a17843955be24de2975
I20250113 02:51:17.306001 709 leader_election.cc:304] T 4e61c6a807884a17843955be24de2975 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
W20250113 02:51:17.306181 709 raft_consensus.cc:2681] T 4e61c6a807884a17843955be24de2975 P 1186fa869eb14812832fa6562531f196: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250113 02:51:17.307694 1194 tablet_bootstrap.cc:492] T 89517af3545c47a5a3a247a411190a47 P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:17.308786 1191 tablet_bootstrap.cc:654] T 5ae18cbff90d4fa88f6b9b8746fc2027 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:17.308792 1194 ts_tablet_manager.cc:1397] T 89517af3545c47a5a3a247a411190a47 P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.211s user 0.008s sys 0.066s
I20250113 02:51:17.313175 1194 raft_consensus.cc:357] T 89517af3545c47a5a3a247a411190a47 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:17.313464 1194 raft_consensus.cc:383] T 89517af3545c47a5a3a247a411190a47 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:17.313617 1194 raft_consensus.cc:738] T 89517af3545c47a5a3a247a411190a47 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:17.313901 1194 consensus_queue.cc:260] T 89517af3545c47a5a3a247a411190a47 P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:17.315687 1194 ts_tablet_manager.cc:1428] T 89517af3545c47a5a3a247a411190a47 P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250113 02:51:17.316602 1194 tablet_bootstrap.cc:492] T b262c70712444c06a61918f21736f339 P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:17.318374 1194 tablet_bootstrap.cc:654] T b262c70712444c06a61918f21736f339 P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:17.322396 1191 tablet_bootstrap.cc:492] T 5ae18cbff90d4fa88f6b9b8746fc2027 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:17.322571 1191 ts_tablet_manager.cc:1397] T 5ae18cbff90d4fa88f6b9b8746fc2027 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.024s user 0.003s sys 0.002s
I20250113 02:51:17.323390 1191 raft_consensus.cc:357] T 5ae18cbff90d4fa88f6b9b8746fc2027 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:17.323655 1191 raft_consensus.cc:383] T 5ae18cbff90d4fa88f6b9b8746fc2027 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:17.323760 1191 raft_consensus.cc:738] T 5ae18cbff90d4fa88f6b9b8746fc2027 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:17.324105 1191 consensus_queue.cc:260] T 5ae18cbff90d4fa88f6b9b8746fc2027 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:17.324764 1191 ts_tablet_manager.cc:1428] T 5ae18cbff90d4fa88f6b9b8746fc2027 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250113 02:51:17.325099 1191 tablet_bootstrap.cc:492] T 6f1c2ae654bd46c1a8772fa62c8111ab P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:17.327507 1191 tablet_bootstrap.cc:654] T 6f1c2ae654bd46c1a8772fa62c8111ab P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:17.333668 447 catalog_manager.cc:5618] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095) does not have the latest schema for tablet 89517af3545c47a5a3a247a411190a47 (table default.Table_5d3da53a6d984a7ea593b340ad3e1368 [id=e66cde9b8d9b4763b33e7641601c47da]). Expected version 2 got 0
I20250113 02:51:17.334661 449 catalog_manager.cc:5618] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467) does not have the latest schema for tablet 89517af3545c47a5a3a247a411190a47 (table default.Table_5d3da53a6d984a7ea593b340ad3e1368 [id=e66cde9b8d9b4763b33e7641601c47da]). Expected version 2 got 0
I20250113 02:51:17.338867 898 tablet_service.cc:1467] Processing CreateTablet for tablet c71511a100f14322b5091a9804720fa0 (DEFAULT_TABLE table=default.Table_8606b4caf08b43b18b25ee8849eebd07 [id=85bca730e1274dbebb2f7c2e6eaea7c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:17.339368 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c71511a100f14322b5091a9804720fa0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:17.339903 449 catalog_manager.cc:5618] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095) does not have the latest schema for tablet 89517af3545c47a5a3a247a411190a47 (table default.Table_5d3da53a6d984a7ea593b340ad3e1368 [id=e66cde9b8d9b4763b33e7641601c47da]). Expected version 2 got 0
I20250113 02:51:17.341650 748 tablet_service.cc:1467] Processing CreateTablet for tablet c71511a100f14322b5091a9804720fa0 (DEFAULT_TABLE table=default.Table_8606b4caf08b43b18b25ee8849eebd07 [id=85bca730e1274dbebb2f7c2e6eaea7c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:17.342054 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c71511a100f14322b5091a9804720fa0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:17.343535 897 tablet_service.cc:1467] Processing CreateTablet for tablet 68023c30a7f644eaa5b4ddd60d6c5ef5 (DEFAULT_TABLE table=default.Table_5d38fa5123014e919f82dafd64c997c1 [id=e5831758ca184c1998f551fe0285d010]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:17.343904 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 68023c30a7f644eaa5b4ddd60d6c5ef5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:17.344758 744 tablet_service.cc:1467] Processing CreateTablet for tablet 68023c30a7f644eaa5b4ddd60d6c5ef5 (DEFAULT_TABLE table=default.Table_5d38fa5123014e919f82dafd64c997c1 [id=e5831758ca184c1998f551fe0285d010]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:17.345119 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 68023c30a7f644eaa5b4ddd60d6c5ef5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:17.348279 747 tablet_service.cc:1467] Processing CreateTablet for tablet 3d3a6d9f687e45218d5875a7c5597648 (DEFAULT_TABLE table=default.Table_21dd1d15c2bb4849af81bb4d8e1a5f04 [id=219903e6141d465bbf5396197210c420]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:17.348685 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3d3a6d9f687e45218d5875a7c5597648. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:17.348765 891 tablet_service.cc:1467] Processing CreateTablet for tablet 3d3a6d9f687e45218d5875a7c5597648 (DEFAULT_TABLE table=default.Table_21dd1d15c2bb4849af81bb4d8e1a5f04 [id=219903e6141d465bbf5396197210c420]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:17.349120 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3d3a6d9f687e45218d5875a7c5597648. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:17.349445 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 3d132c29cb2c46108e5877e38068ef17
I20250113 02:51:17.350880 1038 tablet_service.cc:1467] Processing CreateTablet for tablet 68023c30a7f644eaa5b4ddd60d6c5ef5 (DEFAULT_TABLE table=default.Table_5d38fa5123014e919f82dafd64c997c1 [id=e5831758ca184c1998f551fe0285d010]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:17.351274 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 68023c30a7f644eaa5b4ddd60d6c5ef5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:17.351907 736 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d132c29cb2c46108e5877e38068ef17 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:17 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:17.352396 1039 tablet_service.cc:1467] Processing CreateTablet for tablet 3d3a6d9f687e45218d5875a7c5597648 (DEFAULT_TABLE table=default.Table_21dd1d15c2bb4849af81bb4d8e1a5f04 [id=219903e6141d465bbf5396197210c420]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:17.352737 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3d3a6d9f687e45218d5875a7c5597648. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:17.355278 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 3d3a6d9f687e45218d5875a7c5597648
I20250113 02:51:17.357618 736 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d3a6d9f687e45218d5875a7c5597648 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:17 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:17.357888 892 tablet_service.cc:1467] Processing CreateTablet for tablet 8547b63c40ec4ccebd962b5a3b2c5e0f (DEFAULT_TABLE table=default.Table_28d44f0d1c6c4a2cbd27c902820195b9 [id=1396c4482ee646d18330d200fd3f9816]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:17.358173 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_2565c0551f9943a084ace8f7e463e59d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:17.358723 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2565c0551f9943a084ace8f7e463e59d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:17.359210 1037 tablet_service.cc:1467] Processing CreateTablet for tablet 8547b63c40ec4ccebd962b5a3b2c5e0f (DEFAULT_TABLE table=default.Table_28d44f0d1c6c4a2cbd27c902820195b9 [id=1396c4482ee646d18330d200fd3f9816]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:17.359764 1037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8547b63c40ec4ccebd962b5a3b2c5e0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:17.358630 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8547b63c40ec4ccebd962b5a3b2c5e0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:17.360347 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_b1e77e8246ce414a8f49d24b205d0995"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:17.360846 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b1e77e8246ce414a8f49d24b205d0995 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:17.360845 899 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d132c29cb2c46108e5877e38068ef17 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:17 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:17.361631 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d132c29cb2c46108e5877e38068ef17 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:17 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:17.361951 1209 raft_consensus.cc:491] T 89517af3545c47a5a3a247a411190a47 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:17.361968 1042 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d3a6d9f687e45218d5875a7c5597648 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:17 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:17.362092 1209 raft_consensus.cc:513] T 89517af3545c47a5a3a247a411190a47 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:17.362818 1209 leader_election.cc:290] T 89517af3545c47a5a3a247a411190a47 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:17.363320 1191 tablet_bootstrap.cc:492] T 6f1c2ae654bd46c1a8772fa62c8111ab P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:17.363474 1191 ts_tablet_manager.cc:1397] T 6f1c2ae654bd46c1a8772fa62c8111ab P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.038s user 0.004s sys 0.000s
I20250113 02:51:17.363477 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89517af3545c47a5a3a247a411190a47" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:17.363785 918 raft_consensus.cc:2463] T 89517af3545c47a5a3a247a411190a47 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1186fa869eb14812832fa6562531f196 in term 0.
I20250113 02:51:17.364392 1191 raft_consensus.cc:357] T 6f1c2ae654bd46c1a8772fa62c8111ab P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:17.364601 1191 raft_consensus.cc:383] T 6f1c2ae654bd46c1a8772fa62c8111ab P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:17.364666 1191 raft_consensus.cc:738] T 6f1c2ae654bd46c1a8772fa62c8111ab P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:17.364666 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 6f1c2ae654bd46c1a8772fa62c8111ab
I20250113 02:51:17.364903 1191 consensus_queue.cc:260] T 6f1c2ae654bd46c1a8772fa62c8111ab P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:17.365428 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet c62a5e9e43d14b5c94eb87260528469d
I20250113 02:51:17.365830 452 catalog_manager.cc:6596] ReplaceTablet: tablet c62a5e9e43d14b5c94eb87260528469d deleted and replaced by tablet 6c8ca30720664f3a819c4bff2493934d
I20250113 02:51:17.367230 1209 raft_consensus.cc:491] T 015447488b734f6a93b8c36be975c031 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:17.367404 1209 raft_consensus.cc:513] T 015447488b734f6a93b8c36be975c031 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:17.367725 711 leader_election.cc:304] T 89517af3545c47a5a3a247a411190a47 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1186fa869eb14812832fa6562531f196, df69f48656b44bb890f76587ee69cc0f; no voters:
I20250113 02:51:17.368057 1209 leader_election.cc:290] T 015447488b734f6a93b8c36be975c031 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:17.368237 1209 raft_consensus.cc:2798] T 89517af3545c47a5a3a247a411190a47 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 02:51:17.368364 1209 raft_consensus.cc:491] T 89517af3545c47a5a3a247a411190a47 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 02:51:17.368492 1209 raft_consensus.cc:3054] T 89517af3545c47a5a3a247a411190a47 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:17.368702 894 tablet_service.cc:1514] Processing DeleteTablet for tablet c62a5e9e43d14b5c94eb87260528469d with delete_type TABLET_DATA_DELETED (replaced by tablet 6c8ca30720664f3a819c4bff2493934d) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:17.368930 895 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d3a6d9f687e45218d5875a7c5597648 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:17 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:17.369446 899 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f1c2ae654bd46c1a8772fa62c8111ab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:17 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:17.370906 1042 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f1c2ae654bd46c1a8772fa62c8111ab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:17 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:17.371170 1209 raft_consensus.cc:513] T 89517af3545c47a5a3a247a411190a47 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
W20250113 02:51:17.371522 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 3d132c29cb2c46108e5877e38068ef17 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3d132c29cb2c46108e5877e38068ef17 already in progress: creating tablet
I20250113 02:51:17.371788 1209 leader_election.cc:290] T 89517af3545c47a5a3a247a411190a47 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 election: Requested vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:17.371620 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89517af3545c47a5a3a247a411190a47" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
W20250113 02:51:17.371824 440 catalog_manager.cc:4839] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 3d3a6d9f687e45218d5875a7c5597648 because the tablet was not found. No further retry: Not found: Tablet not found: 3d3a6d9f687e45218d5875a7c5597648
I20250113 02:51:17.371949 748 tablet_service.cc:1514] Processing DeleteTablet for tablet c62a5e9e43d14b5c94eb87260528469d with delete_type TABLET_DATA_DELETED (replaced by tablet 6c8ca30720664f3a819c4bff2493934d) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:17.372241 744 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f1c2ae654bd46c1a8772fa62c8111ab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:17 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:17.372277 1062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "015447488b734f6a93b8c36be975c031" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:17.372494 1191 ts_tablet_manager.cc:1428] T 6f1c2ae654bd46c1a8772fa62c8111ab P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20250113 02:51:17.372792 1191 tablet_bootstrap.cc:492] T c97a6ad98ddc4680a0f071cb91f05402 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
W20250113 02:51:17.374210 709 leader_election.cc:343] T 89517af3545c47a5a3a247a411190a47 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
W20250113 02:51:17.374449 709 leader_election.cc:343] T 015447488b734f6a93b8c36be975c031 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:17.374910 1062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89517af3545c47a5a3a247a411190a47" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca"
I20250113 02:51:17.375366 1194 tablet_bootstrap.cc:492] T b262c70712444c06a61918f21736f339 P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:17.375519 1194 ts_tablet_manager.cc:1397] T b262c70712444c06a61918f21736f339 P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.059s user 0.006s sys 0.000s
I20250113 02:51:17.376317 1194 raft_consensus.cc:357] T b262c70712444c06a61918f21736f339 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:17.376644 1194 raft_consensus.cc:383] T b262c70712444c06a61918f21736f339 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:17.376734 1194 raft_consensus.cc:738] T b262c70712444c06a61918f21736f339 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:17.377044 1194 consensus_queue.cc:260] T b262c70712444c06a61918f21736f339 P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:17.377437 1191 tablet_bootstrap.cc:654] T c97a6ad98ddc4680a0f071cb91f05402 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
W20250113 02:51:17.378876 709 leader_election.cc:343] T 89517af3545c47a5a3a247a411190a47 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:17.379607 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet c62a5e9e43d14b5c94eb87260528469d with delete_type TABLET_DATA_DELETED (replaced by tablet 6c8ca30720664f3a819c4bff2493934d) from {username='slave'} at 127.0.0.1:60230
W20250113 02:51:17.380930 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 6f1c2ae654bd46c1a8772fa62c8111ab because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6f1c2ae654bd46c1a8772fa62c8111ab already in progress: creating tablet
W20250113 02:51:17.381382 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet c62a5e9e43d14b5c94eb87260528469d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c62a5e9e43d14b5c94eb87260528469d already in progress: creating tablet
W20250113 02:51:17.381794 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 3d132c29cb2c46108e5877e38068ef17 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3d132c29cb2c46108e5877e38068ef17 already in progress: creating tablet
W20250113 02:51:17.382030 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet c62a5e9e43d14b5c94eb87260528469d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c62a5e9e43d14b5c94eb87260528469d already in progress: creating tablet
W20250113 02:51:17.382253 442 catalog_manager.cc:4839] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 3d3a6d9f687e45218d5875a7c5597648 because the tablet was not found. No further retry: Not found: Tablet not found: 3d3a6d9f687e45218d5875a7c5597648
W20250113 02:51:17.382499 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 6f1c2ae654bd46c1a8772fa62c8111ab because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6f1c2ae654bd46c1a8772fa62c8111ab already in progress: creating tablet
I20250113 02:51:17.386461 1194 ts_tablet_manager.cc:1428] T b262c70712444c06a61918f21736f339 P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.011s user 0.003s sys 0.000s
I20250113 02:51:17.387029 895 tablet_service.cc:1467] Processing CreateTablet for tablet d101e98b72544c30a3a4bc60d1576aa3 (DEFAULT_TABLE table=default.Table_ce0f77e091564c29bf54791dda458d78 [id=86a46eed92bc4bddad870e392232cb51]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:17.386482 899 tablet_service.cc:1514] Processing DeleteTablet for tablet b262c70712444c06a61918f21736f339 with delete_type TABLET_DATA_DELETED (replaced by tablet c135506190f849008ce6542d569b402a) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:17.387415 744 tablet_service.cc:1467] Processing CreateTablet for tablet 8547b63c40ec4ccebd962b5a3b2c5e0f (DEFAULT_TABLE table=default.Table_28d44f0d1c6c4a2cbd27c902820195b9 [id=1396c4482ee646d18330d200fd3f9816]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:17.387719 1196 tablet_replica.cc:331] T b262c70712444c06a61918f21736f339 P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:17.387826 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8547b63c40ec4ccebd962b5a3b2c5e0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:17.387930 1196 raft_consensus.cc:2238] T b262c70712444c06a61918f21736f339 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:17.388130 1196 raft_consensus.cc:2267] T b262c70712444c06a61918f21736f339 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:17.389513 1042 tablet_service.cc:1467] Processing CreateTablet for tablet 1d81eec487dd4016b3b804451da31c49 (DEFAULT_TABLE table=default.Table_7c6817c7dcb74872a38ec4446e4750a9 [id=2c7925ec0d8d4cd7b8761268d3962f15]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:17.389935 1042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d81eec487dd4016b3b804451da31c49. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:17.392318 1034 tablet_service.cc:1467] Processing CreateTablet for tablet 076ed7744da54ddebd8fc99686beb6ae (DEFAULT_TABLE table=default.Table_8b98cd7fbceb4d3f8b138e4da9bb0e05 [id=4100a37b30624d0bab2ec027b783be00]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:17.392696 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 076ed7744da54ddebd8fc99686beb6ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:17.392623 748 tablet_service.cc:1467] Processing CreateTablet for tablet d101e98b72544c30a3a4bc60d1576aa3 (DEFAULT_TABLE table=default.Table_ce0f77e091564c29bf54791dda458d78 [id=86a46eed92bc4bddad870e392232cb51]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:17.393038 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d101e98b72544c30a3a4bc60d1576aa3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:17.393147 1036 tablet_service.cc:1467] Processing CreateTablet for tablet 53cc528d282c47788d5d558569f074c1 (DEFAULT_TABLE table=default.Table_c5c12ef64b7b438eb52313e87793e601 [id=7c2bc6d4232f4585b84d71e3e1f85794]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:17.393462 1036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 53cc528d282c47788d5d558569f074c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:17.394476 736 tablet_service.cc:1467] Processing CreateTablet for tablet 076ed7744da54ddebd8fc99686beb6ae (DEFAULT_TABLE table=default.Table_8b98cd7fbceb4d3f8b138e4da9bb0e05 [id=4100a37b30624d0bab2ec027b783be00]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:17.395035 736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 076ed7744da54ddebd8fc99686beb6ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:17.396224 1035 tablet_service.cc:1467] Processing CreateTablet for tablet d101e98b72544c30a3a4bc60d1576aa3 (DEFAULT_TABLE table=default.Table_ce0f77e091564c29bf54791dda458d78 [id=86a46eed92bc4bddad870e392232cb51]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:17.396224 747 tablet_service.cc:1467] Processing CreateTablet for tablet 1d81eec487dd4016b3b804451da31c49 (DEFAULT_TABLE table=default.Table_7c6817c7dcb74872a38ec4446e4750a9 [id=2c7925ec0d8d4cd7b8761268d3962f15]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:17.396586 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d101e98b72544c30a3a4bc60d1576aa3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:17.396607 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d81eec487dd4016b3b804451da31c49. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:17.401417 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_f4dcd40f883c4b81a5a850fa2cfc2499"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:17.401341 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_1031f897b6e24d3abd972aca00ddb48b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:17.401964 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f4dcd40f883c4b81a5a850fa2cfc2499 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:17.402179 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_98a51e7ca6354ce0958333364e11adaf" } modify_external_catalogs: true
I20250113 02:51:17.402339 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_98a51e7ca6354ce0958333364e11adaf" } modify_external_catalogs: true
I20250113 02:51:17.402675 894 tablet_service.cc:1467] Processing CreateTablet for tablet 076ed7744da54ddebd8fc99686beb6ae (DEFAULT_TABLE table=default.Table_8b98cd7fbceb4d3f8b138e4da9bb0e05 [id=4100a37b30624d0bab2ec027b783be00]), partition=RANGE (key) PARTITION UNBOUNDED
W20250113 02:51:17.401965 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1031f897b6e24d3abd972aca00ddb48b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:17.403306 899 tablet_service.cc:1467] Processing CreateTablet for tablet 1d81eec487dd4016b3b804451da31c49 (DEFAULT_TABLE table=default.Table_7c6817c7dcb74872a38ec4446e4750a9 [id=2c7925ec0d8d4cd7b8761268d3962f15]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:17.405047 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_9ea968c0d615400aa500d7bde8d94e3f" } new_table_name: "default.Table_3b52330dd7f04b9db3ef41770dc87651" modify_external_catalogs: true
I20250113 02:51:17.406363 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_93d706c664024860855d1c9d91992817"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:17.406607 1196 ts_tablet_manager.cc:1905] T b262c70712444c06a61918f21736f339 P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:17.406615 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_5a2e4e24ce9b4618813ca6d023d05664"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:17.406843 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_93d706c664024860855d1c9d91992817 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250113 02:51:17.406985 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5a2e4e24ce9b4618813ca6d023d05664 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:17.409994 893 tablet_service.cc:1467] Processing CreateTablet for tablet 53cc528d282c47788d5d558569f074c1 (DEFAULT_TABLE table=default.Table_c5c12ef64b7b438eb52313e87793e601 [id=7c2bc6d4232f4585b84d71e3e1f85794]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:17.411835 896 tablet_service.cc:1467] Processing CreateTablet for tablet 18e2d79d731142f89b9089869bc736e5 (DEFAULT_TABLE table=default.Table_f9aa5df6ba3d43999ebd6570df3108d1 [id=fa5a1cda58f54103ac637c12d99e4e38]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:17.412210 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 18e2d79d731142f89b9089869bc736e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:17.412307 455 master_service.cc:946] ReplaceTablet: received request to replace tablet 8193357427f4491c8c6c1396ceeef2dc from {username='slave'} at 127.0.0.1:49258
W20250113 02:51:17.414058 711 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:38467, user_credentials={real_user=slave}} blocked reactor thread for 46369.9us
I20250113 02:51:17.415727 1040 tablet_service.cc:1467] Processing CreateTablet for tablet 18e2d79d731142f89b9089869bc736e5 (DEFAULT_TABLE table=default.Table_f9aa5df6ba3d43999ebd6570df3108d1 [id=fa5a1cda58f54103ac637c12d99e4e38]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:17.416147 1040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 18e2d79d731142f89b9089869bc736e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:17.419000 890 tablet_service.cc:1467] Processing CreateTablet for tablet d5bd09a1ede644c59e400e49c8125793 (DEFAULT_TABLE table=default.Table_bee5bcef1f534c6c812e6075291ff90d [id=e0ced6f3dc0049eaa0e76974916ff63c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:17.419420 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d5bd09a1ede644c59e400e49c8125793. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:17.419467 1041 tablet_service.cc:1467] Processing CreateTablet for tablet d5bd09a1ede644c59e400e49c8125793 (DEFAULT_TABLE table=default.Table_bee5bcef1f534c6c812e6075291ff90d [id=e0ced6f3dc0049eaa0e76974916ff63c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:17.419799 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d5bd09a1ede644c59e400e49c8125793. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:17.423422 1195 ts_tablet_manager.cc:1918] T 7666ece05d334bac88fb7b58a4350b63 P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:17.423555 1195 log.cc:1198] T 7666ece05d334bac88fb7b58a4350b63 P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/7666ece05d334bac88fb7b58a4350b63
I20250113 02:51:17.423602 1033 tablet_service.cc:1467] Processing CreateTablet for tablet d35ec7e4a5084a74bea172ef8d9990b5 (DEFAULT_TABLE table=default.Table_5cf92fa585024c5e905626e0b760ae0d [id=36ff239c08a54be99ca307354810fff3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:17.423959 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d35ec7e4a5084a74bea172ef8d9990b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:17.424001 1195 ts_tablet_manager.cc:1939] T 7666ece05d334bac88fb7b58a4350b63 P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:17.424418 889 tablet_service.cc:1467] Processing CreateTablet for tablet d35ec7e4a5084a74bea172ef8d9990b5 (DEFAULT_TABLE table=default.Table_5cf92fa585024c5e905626e0b760ae0d [id=36ff239c08a54be99ca307354810fff3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:17.424962 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d35ec7e4a5084a74bea172ef8d9990b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:17.427322 888 tablet_service.cc:1467] Processing CreateTablet for tablet 107e6e9e2d84496f8fce92b9ba377189 (DEFAULT_TABLE table=default.Table_74627d276edf4d75a2fcaa627a7f921b [id=de6b4835f6c04c20b6a5ef8404dc78f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:17.427852 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 107e6e9e2d84496f8fce92b9ba377189. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:17.428300 1032 tablet_service.cc:1467] Processing CreateTablet for tablet 107e6e9e2d84496f8fce92b9ba377189 (DEFAULT_TABLE table=default.Table_74627d276edf4d75a2fcaa627a7f921b [id=de6b4835f6c04c20b6a5ef8404dc78f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:17.428710 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 107e6e9e2d84496f8fce92b9ba377189. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:17.429006 1195 tablet_replica.cc:331] T 455b143af8c94f9d8a4111330648cac6 P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:17.429214 1195 raft_consensus.cc:2238] T 455b143af8c94f9d8a4111330648cac6 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:17.429373 1195 raft_consensus.cc:2267] T 455b143af8c94f9d8a4111330648cac6 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:17.432667 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89517af3545c47a5a3a247a411190a47" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f"
I20250113 02:51:17.432957 918 raft_consensus.cc:3054] T 89517af3545c47a5a3a247a411190a47 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Advancing to term 1
W20250113 02:51:17.437461 711 leader_election.cc:343] T 015447488b734f6a93b8c36be975c031 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Not found: Tablet not found: 015447488b734f6a93b8c36be975c031
I20250113 02:51:17.437644 711 leader_election.cc:304] T 015447488b734f6a93b8c36be975c031 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:17.438140 1239 raft_consensus.cc:2743] T 015447488b734f6a93b8c36be975c031 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:17.438854 918 raft_consensus.cc:2463] T 89517af3545c47a5a3a247a411190a47 P df69f48656b44bb890f76587ee69cc0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1186fa869eb14812832fa6562531f196 in term 1.
I20250113 02:51:17.439391 711 leader_election.cc:304] T 89517af3545c47a5a3a247a411190a47 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1186fa869eb14812832fa6562531f196, df69f48656b44bb890f76587ee69cc0f; no voters: 4216a7f853d642b7bc3e5cff458f3aca
I20250113 02:51:17.439814 1239 raft_consensus.cc:2798] T 89517af3545c47a5a3a247a411190a47 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Leader election won for term 1
I20250113 02:51:17.440050 1239 raft_consensus.cc:695] T 89517af3545c47a5a3a247a411190a47 P 1186fa869eb14812832fa6562531f196 [term 1 LEADER]: Becoming Leader. State: Replica: 1186fa869eb14812832fa6562531f196, State: Running, Role: LEADER
I20250113 02:51:17.440203 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 53cc528d282c47788d5d558569f074c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:17.440553 1239 consensus_queue.cc:237] T 89517af3545c47a5a3a247a411190a47 P 1186fa869eb14812832fa6562531f196 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:17.440892 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d101e98b72544c30a3a4bc60d1576aa3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:17.443048 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 076ed7744da54ddebd8fc99686beb6ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:17.443599 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d81eec487dd4016b3b804451da31c49. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:17.445595 736 tablet_service.cc:1467] Processing CreateTablet for tablet 53cc528d282c47788d5d558569f074c1 (DEFAULT_TABLE table=default.Table_c5c12ef64b7b438eb52313e87793e601 [id=7c2bc6d4232f4585b84d71e3e1f85794]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:17.446039 736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 53cc528d282c47788d5d558569f074c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:17.446318 737 tablet_service.cc:1467] Processing CreateTablet for tablet d35ec7e4a5084a74bea172ef8d9990b5 (DEFAULT_TABLE table=default.Table_5cf92fa585024c5e905626e0b760ae0d [id=36ff239c08a54be99ca307354810fff3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:17.446892 737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d35ec7e4a5084a74bea172ef8d9990b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:17.449959 747 tablet_service.cc:1467] Processing CreateTablet for tablet 18e2d79d731142f89b9089869bc736e5 (DEFAULT_TABLE table=default.Table_f9aa5df6ba3d43999ebd6570df3108d1 [id=fa5a1cda58f54103ac637c12d99e4e38]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:17.450352 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 18e2d79d731142f89b9089869bc736e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:17.454653 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet 7666ece05d334bac88fb7b58a4350b63 (table default.Table_cacef89852fd4576937e26bc2da7b632 [id=1379e5486da84a00b6682ae48a8cfe87]) successfully deleted
W20250113 02:51:17.454901 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet c135506190f849008ce6542d569b402a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c135506190f849008ce6542d569b402a already in progress: creating tablet
W20250113 02:51:17.455600 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 0c55662a71444a3683a825bcec6c5b68 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0c55662a71444a3683a825bcec6c5b68 already in progress: creating tablet
I20250113 02:51:17.445655 748 tablet_service.cc:1467] Processing CreateTablet for tablet d5bd09a1ede644c59e400e49c8125793 (DEFAULT_TABLE table=default.Table_bee5bcef1f534c6c812e6075291ff90d [id=e0ced6f3dc0049eaa0e76974916ff63c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:17.456234 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_5cf92fa585024c5e905626e0b760ae0d" } new_table_name: "default.Table_4369a143c72643b084df58d25314c6dd" modify_external_catalogs: true
I20250113 02:51:17.456638 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d5bd09a1ede644c59e400e49c8125793. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:17.457454 454 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_97abfde1fbb143d2908bab58cd04645b" } new_table_name: "default.Table_059eaed9949d445198527eb449fd1447" modify_external_catalogs: true
I20250113 02:51:17.458765 737 tablet_service.cc:1467] Processing CreateTablet for tablet 107e6e9e2d84496f8fce92b9ba377189 (DEFAULT_TABLE table=default.Table_74627d276edf4d75a2fcaa627a7f921b [id=de6b4835f6c04c20b6a5ef8404dc78f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:17.459111 737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 107e6e9e2d84496f8fce92b9ba377189. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:17.464720 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet ad6b21ed2b824cb0837fc1d8a39d2472 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ad6b21ed2b824cb0837fc1d8a39d2472 already in progress: creating tablet
W20250113 02:51:17.464979 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 78689f7ed2a64405adb75f372465a792 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 78689f7ed2a64405adb75f372465a792 already in progress: creating tablet
I20250113 02:51:17.469219 448 master_service.cc:946] ReplaceTablet: received request to replace tablet ec0750fd2fa247feb3ea310ffabe05e5 from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:17.473647 1195 ts_tablet_manager.cc:1905] T 455b143af8c94f9d8a4111330648cac6 P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:17.476320 1031 tablet_service.cc:1467] Processing CreateTablet for tablet c71511a100f14322b5091a9804720fa0 (DEFAULT_TABLE table=default.Table_8606b4caf08b43b18b25ee8849eebd07 [id=85bca730e1274dbebb2f7c2e6eaea7c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:17.476871 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c71511a100f14322b5091a9804720fa0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:17.480428 452 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet 672185866d714f38b1e3043e0e12b3f3
I20250113 02:51:17.481958 1192 tablet_bootstrap.cc:492] T 7666ece05d334bac88fb7b58a4350b63 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:17.482110 1192 ts_tablet_manager.cc:1397] T 7666ece05d334bac88fb7b58a4350b63 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.261s user 0.007s sys 0.012s
I20250113 02:51:17.480731 452 master_service.cc:946] ReplaceTablet: received request to replace tablet 68023c30a7f644eaa5b4ddd60d6c5ef5 from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:17.483867 1192 raft_consensus.cc:357] T 7666ece05d334bac88fb7b58a4350b63 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:17.484133 1192 raft_consensus.cc:383] T 7666ece05d334bac88fb7b58a4350b63 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:17.484226 1192 raft_consensus.cc:738] T 7666ece05d334bac88fb7b58a4350b63 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:17.484593 1192 consensus_queue.cc:260] T 7666ece05d334bac88fb7b58a4350b63 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:17.492494 1192 ts_tablet_manager.cc:1428] T 7666ece05d334bac88fb7b58a4350b63 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.010s user 0.004s sys 0.000s
W20250113 02:51:17.499454 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0c57a5add4914835ac366efdcde4ee28 consensus state has no leader
W20250113 02:51:17.502202 440 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.61:35271: connect: Connection refused (error 111) [suppressed 12 similar messages]
I20250113 02:51:17.503679 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_3b57de11e75e42fbb2cbe60078f76c8d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:17.504287 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3b57de11e75e42fbb2cbe60078f76c8d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:17.505801 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_43bb1aa67eec4cb89272fcec397e08b6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:17.506273 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_43bb1aa67eec4cb89272fcec397e08b6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:17.507436 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_4863cef058bb4fc7b7f38735eb20482c" } new_table_name: "default.Table_b7158f0e7b8a4fd1a65dc21f11365153" modify_external_catalogs: true
I20250113 02:51:17.507676 1191 tablet_bootstrap.cc:492] T c97a6ad98ddc4680a0f071cb91f05402 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:17.507838 1191 ts_tablet_manager.cc:1397] T c97a6ad98ddc4680a0f071cb91f05402 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.135s user 0.033s sys 0.042s
I20250113 02:51:17.508184 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 8193357427f4491c8c6c1396ceeef2dc
I20250113 02:51:17.508671 455 catalog_manager.cc:6596] ReplaceTablet: tablet 8193357427f4491c8c6c1396ceeef2dc deleted and replaced by tablet d3b695ac20ba4205b2d8732db4740531
I20250113 02:51:17.508785 1191 raft_consensus.cc:357] T c97a6ad98ddc4680a0f071cb91f05402 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:17.508992 1191 raft_consensus.cc:383] T c97a6ad98ddc4680a0f071cb91f05402 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:17.509079 1191 raft_consensus.cc:738] T c97a6ad98ddc4680a0f071cb91f05402 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:17.509320 1191 consensus_queue.cc:260] T c97a6ad98ddc4680a0f071cb91f05402 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:17.510133 1191 ts_tablet_manager.cc:1428] T c97a6ad98ddc4680a0f071cb91f05402 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250113 02:51:17.510398 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_5d3da53a6d984a7ea593b340ad3e1368" } modify_external_catalogs: true
I20250113 02:51:17.510547 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_5d3da53a6d984a7ea593b340ad3e1368" } modify_external_catalogs: true
I20250113 02:51:17.513245 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_47dd2257e94f494fbc848efdd878d9c8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:17.513501 1191 tablet_bootstrap.cc:492] T bc88676bf738442d82c25e56f30c4ada P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
W20250113 02:51:17.513773 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_47dd2257e94f494fbc848efdd878d9c8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:17.515055 1204 raft_consensus.cc:491] T 7666ece05d334bac88fb7b58a4350b63 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:17.515273 1204 raft_consensus.cc:513] T 7666ece05d334bac88fb7b58a4350b63 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:17.515483 1191 tablet_bootstrap.cc:654] T bc88676bf738442d82c25e56f30c4ada P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:17.515715 1195 ts_tablet_manager.cc:1918] T 455b143af8c94f9d8a4111330648cac6 P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:17.515861 1195 log.cc:1198] T 455b143af8c94f9d8a4111330648cac6 P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/455b143af8c94f9d8a4111330648cac6
I20250113 02:51:17.516374 1195 ts_tablet_manager.cc:1939] T 455b143af8c94f9d8a4111330648cac6 P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:17.517004 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_f86569599f95484fb5cf0a5dccb6935c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:17.517598 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f86569599f95484fb5cf0a5dccb6935c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:17.519717 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_b1e77e8246ce414a8f49d24b205d0995" } new_table_name: "default.Table_f8af5d3e0dc64d3fae054f82a7b33ccf" modify_external_catalogs: true
I20250113 02:51:17.521775 1195 tablet_replica.cc:331] T 5ae18cbff90d4fa88f6b9b8746fc2027 P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:17.521986 1195 raft_consensus.cc:2238] T 5ae18cbff90d4fa88f6b9b8746fc2027 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:17.521988 1204 leader_election.cc:290] T 7666ece05d334bac88fb7b58a4350b63 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:17.522212 1195 raft_consensus.cc:2267] T 5ae18cbff90d4fa88f6b9b8746fc2027 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:17.523015 1195 ts_tablet_manager.cc:1905] T 5ae18cbff90d4fa88f6b9b8746fc2027 P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:17.529984 888 tablet_service.cc:1514] Processing DeleteTablet for tablet 8193357427f4491c8c6c1396ceeef2dc with delete_type TABLET_DATA_DELETED (replaced by tablet d3b695ac20ba4205b2d8732db4740531) from {username='slave'} at 127.0.0.1:37366
W20250113 02:51:17.532895 998 leader_election.cc:343] T 7666ece05d334bac88fb7b58a4350b63 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Not found: Tablet not found: 7666ece05d334bac88fb7b58a4350b63
I20250113 02:51:17.534173 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet 455b143af8c94f9d8a4111330648cac6 (table default.Table_c991f9dbd70d4c4682664b8711f05db4 [id=2c64da64ecdb4bf48e11b523533902e8]) successfully deleted
I20250113 02:51:17.538789 1194 tablet_bootstrap.cc:492] T 9a323451cdfe4fd69ec62cda230f9afb P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:17.540776 1194 tablet_bootstrap.cc:654] T 9a323451cdfe4fd69ec62cda230f9afb P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
W20250113 02:51:17.541190 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d35ec7e4a5084a74bea172ef8d9990b5 consensus state has no leader
W20250113 02:51:17.543488 454 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2fb62cc83f9c4dcfa9aa01ae8b46c025 consensus state has no leader
I20250113 02:51:17.546006 1196 ts_tablet_manager.cc:1918] T b262c70712444c06a61918f21736f339 P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:17.547262 1036 tablet_service.cc:1514] Processing DeleteTablet for tablet 7666ece05d334bac88fb7b58a4350b63 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:16 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:17.547602 1198 tablet_replica.cc:331] T 7666ece05d334bac88fb7b58a4350b63 P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:17.547811 1198 raft_consensus.cc:2238] T 7666ece05d334bac88fb7b58a4350b63 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:17.548024 1198 raft_consensus.cc:2267] T 7666ece05d334bac88fb7b58a4350b63 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:17.548846 1198 ts_tablet_manager.cc:1905] T 7666ece05d334bac88fb7b58a4350b63 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:17.549192 1191 tablet_bootstrap.cc:492] T bc88676bf738442d82c25e56f30c4ada P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:17.549369 1191 ts_tablet_manager.cc:1397] T bc88676bf738442d82c25e56f30c4ada P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.036s user 0.002s sys 0.005s
I20250113 02:51:17.550231 1191 raft_consensus.cc:357] T bc88676bf738442d82c25e56f30c4ada P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:17.550462 1191 raft_consensus.cc:383] T bc88676bf738442d82c25e56f30c4ada P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:17.550544 1191 raft_consensus.cc:738] T bc88676bf738442d82c25e56f30c4ada P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:17.550928 1191 consensus_queue.cc:260] T bc88676bf738442d82c25e56f30c4ada P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:17.551493 1191 ts_tablet_manager.cc:1428] T bc88676bf738442d82c25e56f30c4ada P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250113 02:51:17.554404 1191 tablet_bootstrap.cc:492] T 9a323451cdfe4fd69ec62cda230f9afb P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:17.556142 1191 tablet_bootstrap.cc:654] T 9a323451cdfe4fd69ec62cda230f9afb P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:17.557154 1196 log.cc:1198] T b262c70712444c06a61918f21736f339 P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/b262c70712444c06a61918f21736f339
I20250113 02:51:17.557358 1032 tablet_service.cc:1514] Processing DeleteTablet for tablet 8193357427f4491c8c6c1396ceeef2dc with delete_type TABLET_DATA_DELETED (replaced by tablet d3b695ac20ba4205b2d8732db4740531) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:17.558101 1196 ts_tablet_manager.cc:1939] T b262c70712444c06a61918f21736f339 P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:17.558440 1192 tablet_bootstrap.cc:492] T 015447488b734f6a93b8c36be975c031 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:17.558748 454 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_f4dcd40f883c4b81a5a850fa2cfc2499" } modify_external_catalogs: true
I20250113 02:51:17.558827 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_3b52330dd7f04b9db3ef41770dc87651" } modify_external_catalogs: true
I20250113 02:51:17.558959 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_3b52330dd7f04b9db3ef41770dc87651" } modify_external_catalogs: true
I20250113 02:51:17.558959 454 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_f4dcd40f883c4b81a5a850fa2cfc2499" } modify_external_catalogs: true
I20250113 02:51:17.560824 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet b262c70712444c06a61918f21736f339 (table default.Table_5e0263e22f1e4d2b983ae6b6ef03fca5 [id=377f041bf95e41f3b7e8ea26f9795b6a]) successfully deleted
I20250113 02:51:17.561187 1198 ts_tablet_manager.cc:1918] T 7666ece05d334bac88fb7b58a4350b63 P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:17.564615 1198 log.cc:1198] T 7666ece05d334bac88fb7b58a4350b63 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/7666ece05d334bac88fb7b58a4350b63
I20250113 02:51:17.564765 449 master_service.cc:946] ReplaceTablet: received request to replace tablet 1d81eec487dd4016b3b804451da31c49 from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:17.565246 1198 ts_tablet_manager.cc:1939] T 7666ece05d334bac88fb7b58a4350b63 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
W20250113 02:51:17.565295 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 8193357427f4491c8c6c1396ceeef2dc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8193357427f4491c8c6c1396ceeef2dc already in progress: creating tablet
I20250113 02:51:17.566449 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet 7666ece05d334bac88fb7b58a4350b63 (table default.Table_cacef89852fd4576937e26bc2da7b632 [id=1379e5486da84a00b6682ae48a8cfe87]) successfully deleted
I20250113 02:51:17.568535 1192 tablet_bootstrap.cc:654] T 015447488b734f6a93b8c36be975c031 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
W20250113 02:51:17.569280 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 8193357427f4491c8c6c1396ceeef2dc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8193357427f4491c8c6c1396ceeef2dc already in progress: creating tablet
I20250113 02:51:17.572369 1194 tablet_bootstrap.cc:492] T 9a323451cdfe4fd69ec62cda230f9afb P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:17.572569 1194 ts_tablet_manager.cc:1397] T 9a323451cdfe4fd69ec62cda230f9afb P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.034s user 0.007s sys 0.000s
I20250113 02:51:17.573391 1194 raft_consensus.cc:357] T 9a323451cdfe4fd69ec62cda230f9afb P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:17.573688 1194 raft_consensus.cc:383] T 9a323451cdfe4fd69ec62cda230f9afb P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:17.573828 1194 raft_consensus.cc:738] T 9a323451cdfe4fd69ec62cda230f9afb P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:17.574159 1194 consensus_queue.cc:260] T 9a323451cdfe4fd69ec62cda230f9afb P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:17.576591 448 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_c59cac1b4e0249129c7671a4919283c8" } modify_external_catalogs: true
I20250113 02:51:17.576839 448 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_c59cac1b4e0249129c7671a4919283c8" } modify_external_catalogs: true
W20250113 02:51:17.577634 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4fe54131aafb40ee9d8bc5ea299af193 has no consensus state
I20250113 02:51:17.578020 1194 ts_tablet_manager.cc:1428] T 9a323451cdfe4fd69ec62cda230f9afb P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
W20250113 02:51:17.579905 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f1a02c37cb4f43f8a9de6484f4431958 has no consensus state
I20250113 02:51:17.580973 1194 tablet_bootstrap.cc:492] T 4e61c6a807884a17843955be24de2975 P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:17.584940 454 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet fe4320c0b63b40adae8aefebb82d73dc
I20250113 02:51:17.585973 1194 tablet_bootstrap.cc:654] T 4e61c6a807884a17843955be24de2975 P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:17.587925 1192 tablet_bootstrap.cc:492] T 015447488b734f6a93b8c36be975c031 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:17.588105 1192 ts_tablet_manager.cc:1397] T 015447488b734f6a93b8c36be975c031 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.055s user 0.004s sys 0.000s
I20250113 02:51:17.589077 1192 raft_consensus.cc:357] T 015447488b734f6a93b8c36be975c031 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:17.589329 1192 raft_consensus.cc:383] T 015447488b734f6a93b8c36be975c031 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:17.589426 1192 raft_consensus.cc:738] T 015447488b734f6a93b8c36be975c031 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:17.589818 1192 consensus_queue.cc:260] T 015447488b734f6a93b8c36be975c031 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:17.590495 1192 ts_tablet_manager.cc:1428] T 015447488b734f6a93b8c36be975c031 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250113 02:51:17.592190 895 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a323451cdfe4fd69ec62cda230f9afb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:16 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:17.593079 454 master_service.cc:946] ReplaceTablet: received request to replace tablet 5ae18cbff90d4fa88f6b9b8746fc2027 from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:17.593405 1196 tablet_replica.cc:331] T 9a323451cdfe4fd69ec62cda230f9afb P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:17.593611 1196 raft_consensus.cc:2238] T 9a323451cdfe4fd69ec62cda230f9afb P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:17.596710 1196 raft_consensus.cc:2267] T 9a323451cdfe4fd69ec62cda230f9afb P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:17.597605 1196 ts_tablet_manager.cc:1905] T 9a323451cdfe4fd69ec62cda230f9afb P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:17.604187 1031 tablet_service.cc:1514] Processing DeleteTablet for tablet 015447488b734f6a93b8c36be975c031 with delete_type TABLET_DATA_DELETED (replaced by tablet 9a323451cdfe4fd69ec62cda230f9afb) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:17.604596 1198 tablet_replica.cc:331] T 015447488b734f6a93b8c36be975c031 P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:17.604889 1198 raft_consensus.cc:2238] T 015447488b734f6a93b8c36be975c031 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:17.605130 1198 raft_consensus.cc:2267] T 015447488b734f6a93b8c36be975c031 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:17.605888 1198 ts_tablet_manager.cc:1905] T 015447488b734f6a93b8c36be975c031 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:17.610431 1192 tablet_bootstrap.cc:492] T 23e3cc4d6004439bb95ec3ab2504b1a1 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:17.611714 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 89517af3545c47a5a3a247a411190a47
I20250113 02:51:17.612768 1031 tablet_service.cc:1467] Processing CreateTablet for tablet 21c4b419bf2148e0b140301fa80622bb (DEFAULT_TABLE table=default.Table_f1d42e491cb34117a3bed634de503a6d [id=6fd9215b11274a76b06813227889927b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:17.613329 1032 tablet_service.cc:1514] Processing DeleteTablet for tablet 89517af3545c47a5a3a247a411190a47 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:17 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:17.613703 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 21c4b419bf2148e0b140301fa80622bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:17.613684 1036 tablet_service.cc:1467] Processing CreateTablet for tablet fc92f19390fd48a4a45ba8fb839020fd (DEFAULT_TABLE table=default.Table_c94783b201764702b6de4f3219090efe [id=285c6d11c414470f8c40eefdf6a8a9e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:17.614864 1032 tablet_service.cc:1467] Processing CreateTablet for tablet 6c8ca30720664f3a819c4bff2493934d (DEFAULT_TABLE table=default.Table_c46ffb30f29b44eea3c09b33f6697efe [id=aadd55280b654ceb96c290786a0bfce0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:17.615092 1042 tablet_service.cc:1467] Processing CreateTablet for tablet 317c6e8013b745c290bf3190dc237ab8 (DEFAULT_TABLE table=default.Table_e1ae667a93ee48e8a3d233eeb6335364 [id=6bfb67a96b2b47849f7fa12c57a2f4a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:17.615571 1035 tablet_service.cc:1467] Processing CreateTablet for tablet 41bcb2eafff14fa8a040982c723f99c1 (DEFAULT_TABLE table=default.Table_4a8b3f444ec345ad8a0c424f291e6d77 [id=2f9412fc42494fa68bee8ae23550beec]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:17.615700 1037 tablet_service.cc:1467] Processing CreateTablet for tablet 34ab7ee573284550aaac9f814e8bd7ab (DEFAULT_TABLE table=default.Table_c59cac1b4e0249129c7671a4919283c8 [id=1cd50765f4964c15a341d307ff2d449f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:17.616253 895 tablet_service.cc:1467] Processing CreateTablet for tablet 21c4b419bf2148e0b140301fa80622bb (DEFAULT_TABLE table=default.Table_f1d42e491cb34117a3bed634de503a6d [id=6fd9215b11274a76b06813227889927b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:17.616679 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 21c4b419bf2148e0b140301fa80622bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:17.618230 1038 tablet_service.cc:1467] Processing CreateTablet for tablet d7c4c7b0cfc44ac1952bc73cbe1bced8 (DEFAULT_TABLE table=default.Table_5727b5d8bf51442e8ae5f7c97f4bfbc8 [id=0c69597e8c5b4ec89179169a7a84e048]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:17.618618 891 tablet_service.cc:1514] Processing DeleteTablet for tablet 89517af3545c47a5a3a247a411190a47 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:17 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:17.619235 446 catalog_manager.cc:5618] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095) does not have the latest schema for tablet bc88676bf738442d82c25e56f30c4ada (table default.Table_c991f9dbd70d4c4682664b8711f05db4 [id=2c64da64ecdb4bf48e11b523533902e8]). Expected version 1 got 0
I20250113 02:51:17.620397 1040 tablet_service.cc:1467] Processing CreateTablet for tablet 1c26074fdd804bd79cb0c295622d11ff (DEFAULT_TABLE table=default.Table_b147d4f32fa04b849ff6b75c9382a4e5 [id=bd8bd572ed524f2cae01f820f07e3c36]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:17.621335 1034 tablet_service.cc:1467] Processing CreateTablet for tablet 4fe54131aafb40ee9d8bc5ea299af193 (DEFAULT_TABLE table=default.Table_b7158f0e7b8a4fd1a65dc21f11365153 [id=70089f757a1c45f6b983f216fb167478]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:17.621881 1041 tablet_service.cc:1467] Processing CreateTablet for tablet be80b631ccd747f1b9355a00787a7b55 (DEFAULT_TABLE table=default.Table_2565c0551f9943a084ace8f7e463e59d [id=fe6f8d302c3e489d8bb832503fd05642]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:17.623301 888 tablet_service.cc:1467] Processing CreateTablet for tablet 41bcb2eafff14fa8a040982c723f99c1 (DEFAULT_TABLE table=default.Table_4a8b3f444ec345ad8a0c424f291e6d77 [id=2f9412fc42494fa68bee8ae23550beec]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:17.623698 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 41bcb2eafff14fa8a040982c723f99c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:17.623644 897 tablet_service.cc:1467] Processing CreateTablet for tablet d7c4c7b0cfc44ac1952bc73cbe1bced8 (DEFAULT_TABLE table=default.Table_5727b5d8bf51442e8ae5f7c97f4bfbc8 [id=0c69597e8c5b4ec89179169a7a84e048]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:17.624022 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7c4c7b0cfc44ac1952bc73cbe1bced8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:17.624274 898 tablet_service.cc:1467] Processing CreateTablet for tablet 6c8ca30720664f3a819c4bff2493934d (DEFAULT_TABLE table=default.Table_c46ffb30f29b44eea3c09b33f6697efe [id=aadd55280b654ceb96c290786a0bfce0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:17.624655 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c8ca30720664f3a819c4bff2493934d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:17.626554 893 tablet_service.cc:1467] Processing CreateTablet for tablet 4fe54131aafb40ee9d8bc5ea299af193 (DEFAULT_TABLE table=default.Table_b7158f0e7b8a4fd1a65dc21f11365153 [id=70089f757a1c45f6b983f216fb167478]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:17.626428 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_36c8a9314f6a4da4a1c0dbb51369d5bd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:17.626899 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4fe54131aafb40ee9d8bc5ea299af193. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:17.626957 1033 tablet_service.cc:1467] Processing CreateTablet for tablet 4efa63b8be794e1db4b340f4c0c4cd5a (DEFAULT_TABLE table=default.Table_a27963e06eed4ec08989e43969f3bbbb [id=4b2bd999368c4b628c70cb3ee8c2b1f3]), partition=RANGE (key) PARTITION UNBOUNDED
W20250113 02:51:17.627238 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_36c8a9314f6a4da4a1c0dbb51369d5bd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:17.629770 892 tablet_service.cc:1467] Processing CreateTablet for tablet be80b631ccd747f1b9355a00787a7b55 (DEFAULT_TABLE table=default.Table_2565c0551f9943a084ace8f7e463e59d [id=fe6f8d302c3e489d8bb832503fd05642]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:17.630082 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be80b631ccd747f1b9355a00787a7b55. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:17.632578 894 tablet_service.cc:1467] Processing CreateTablet for tablet 4efa63b8be794e1db4b340f4c0c4cd5a (DEFAULT_TABLE table=default.Table_a27963e06eed4ec08989e43969f3bbbb [id=4b2bd999368c4b628c70cb3ee8c2b1f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:17.632705 890 tablet_service.cc:1467] Processing CreateTablet for tablet 1c26074fdd804bd79cb0c295622d11ff (DEFAULT_TABLE table=default.Table_b147d4f32fa04b849ff6b75c9382a4e5 [id=bd8bd572ed524f2cae01f820f07e3c36]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:17.632961 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4efa63b8be794e1db4b340f4c0c4cd5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:17.633445 891 tablet_service.cc:1467] Processing CreateTablet for tablet fc92f19390fd48a4a45ba8fb839020fd (DEFAULT_TABLE table=default.Table_c94783b201764702b6de4f3219090efe [id=285c6d11c414470f8c40eefdf6a8a9e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:17.633888 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc92f19390fd48a4a45ba8fb839020fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:17.636430 889 tablet_service.cc:1467] Processing CreateTablet for tablet 34ab7ee573284550aaac9f814e8bd7ab (DEFAULT_TABLE table=default.Table_c59cac1b4e0249129c7671a4919283c8 [id=1cd50765f4964c15a341d307ff2d449f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:17.636839 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34ab7ee573284550aaac9f814e8bd7ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:17.640246 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c26074fdd804bd79cb0c295622d11ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:17.642691 899 tablet_service.cc:1467] Processing CreateTablet for tablet 317c6e8013b745c290bf3190dc237ab8 (DEFAULT_TABLE table=default.Table_e1ae667a93ee48e8a3d233eeb6335364 [id=6bfb67a96b2b47849f7fa12c57a2f4a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:17.643174 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 317c6e8013b745c290bf3190dc237ab8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:17.660481 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 68023c30a7f644eaa5b4ddd60d6c5ef5
I20250113 02:51:17.661230 452 catalog_manager.cc:6596] ReplaceTablet: tablet 68023c30a7f644eaa5b4ddd60d6c5ef5 deleted and replaced by tablet 32351d6de1574353a6e8073d333da477
I20250113 02:51:17.661902 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 1d81eec487dd4016b3b804451da31c49
I20250113 02:51:17.662590 449 catalog_manager.cc:6596] ReplaceTablet: tablet 1d81eec487dd4016b3b804451da31c49 deleted and replaced by tablet 438d1d2300e044f39a15586f335935ba
I20250113 02:51:17.663094 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 34ab7ee573284550aaac9f814e8bd7ab
I20250113 02:51:17.665719 1039 tablet_service.cc:1514] Processing DeleteTablet for tablet 68023c30a7f644eaa5b4ddd60d6c5ef5 with delete_type TABLET_DATA_DELETED (replaced by tablet 32351d6de1574353a6e8073d333da477) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:17.665911 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet 34ab7ee573284550aaac9f814e8bd7ab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:17 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:17.667562 1030 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d81eec487dd4016b3b804451da31c49 with delete_type TABLET_DATA_DELETED (replaced by tablet 438d1d2300e044f39a15586f335935ba) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:17.667835 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 0c57a5add4914835ac366efdcde4ee28
I20250113 02:51:17.668017 889 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d81eec487dd4016b3b804451da31c49 with delete_type TABLET_DATA_DELETED (replaced by tablet 438d1d2300e044f39a15586f335935ba) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:17.668697 889 tablet_service.cc:1514] Processing DeleteTablet for tablet 34ab7ee573284550aaac9f814e8bd7ab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:17 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:17.668949 1030 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c57a5add4914835ac366efdcde4ee28 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:17 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:17.670351 889 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c57a5add4914835ac366efdcde4ee28 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:17 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:17.672703 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_f771ada0bfa5404e819d999f416d30cb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:17.673384 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f771ada0bfa5404e819d999f416d30cb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:17.684677 737 tablet_service.cc:1514] Processing DeleteTablet for tablet 8193357427f4491c8c6c1396ceeef2dc with delete_type TABLET_DATA_DELETED (replaced by tablet d3b695ac20ba4205b2d8732db4740531) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:17.685596 737 tablet_service.cc:1514] Processing DeleteTablet for tablet 89517af3545c47a5a3a247a411190a47 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:17 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:17.686487 737 tablet_service.cc:1467] Processing CreateTablet for tablet 21c4b419bf2148e0b140301fa80622bb (DEFAULT_TABLE table=default.Table_f1d42e491cb34117a3bed634de503a6d [id=6fd9215b11274a76b06813227889927b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:17.687147 737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 21c4b419bf2148e0b140301fa80622bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:17.691072 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_37b1d9d091924f8d9a8a2cbf1d06af1a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:17.691839 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_37b1d9d091924f8d9a8a2cbf1d06af1a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:17.692240 449 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_8b98cd7fbceb4d3f8b138e4da9bb0e05" } modify_external_catalogs: true
I20250113 02:51:17.693104 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_3f75f23afe2f449fa7a1f9df1c0734fb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:17.693732 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3f75f23afe2f449fa7a1f9df1c0734fb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:17.696578 449 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_8b98cd7fbceb4d3f8b138e4da9bb0e05" } modify_external_catalogs: true
I20250113 02:51:17.703569 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 41bcb2eafff14fa8a040982c723f99c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:17.703825 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4fe54131aafb40ee9d8bc5ea299af193. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:17.704144 1040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c26074fdd804bd79cb0c295622d11ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:17.704455 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7c4c7b0cfc44ac1952bc73cbe1bced8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:17.708103 1042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 317c6e8013b745c290bf3190dc237ab8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:17.708992 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4efa63b8be794e1db4b340f4c0c4cd5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:17.716107 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be80b631ccd747f1b9355a00787a7b55. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:17.720045 1036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc92f19390fd48a4a45ba8fb839020fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:17.727128 1037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34ab7ee573284550aaac9f814e8bd7ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:17.727852 747 tablet_service.cc:1467] Processing CreateTablet for tablet 6c8ca30720664f3a819c4bff2493934d (DEFAULT_TABLE table=default.Table_c46ffb30f29b44eea3c09b33f6697efe [id=aadd55280b654ceb96c290786a0bfce0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:17.728634 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_a79eb97979f9403b97fa08ffd00c6513"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:17.729210 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a79eb97979f9403b97fa08ffd00c6513 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:17.729759 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_7c6817c7dcb74872a38ec4446e4750a9" } new_table_name: "default.Table_8fb0d7bc0499400a85e729a9dd749fad" modify_external_catalogs: true
I20250113 02:51:17.730507 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_be2bc384adeb441093bc75976bb51c0b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:17.731002 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_be2bc384adeb441093bc75976bb51c0b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:17.733439 894 tablet_service.cc:1514] Processing DeleteTablet for tablet 68023c30a7f644eaa5b4ddd60d6c5ef5 with delete_type TABLET_DATA_DELETED (replaced by tablet 32351d6de1574353a6e8073d333da477) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:17.737470 1039 tablet_service.cc:1467] Processing CreateTablet for tablet 23e9321f08ee4ce7b2252ad5c76c0a08 (DEFAULT_TABLE table=default.Table_93d706c664024860855d1c9d91992817 [id=6e014400813c4402891b32bcbe3555e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:17.738000 1038 tablet_service.cc:1467] Processing CreateTablet for tablet d304b65eba8c40c6922333506a60540c (DEFAULT_TABLE table=default.Table_f86569599f95484fb5cf0a5dccb6935c [id=e62f8e4c51b04729903b1943275c5f7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:17.738886 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c8ca30720664f3a819c4bff2493934d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:17.739944 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c8ca30720664f3a819c4bff2493934d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:17.739948 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23e9321f08ee4ce7b2252ad5c76c0a08. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:17.742493 1195 ts_tablet_manager.cc:1918] T 5ae18cbff90d4fa88f6b9b8746fc2027 P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:17.747956 1195 log.cc:1198] T 5ae18cbff90d4fa88f6b9b8746fc2027 P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/5ae18cbff90d4fa88f6b9b8746fc2027
I20250113 02:51:17.750263 1194 tablet_bootstrap.cc:492] T 4e61c6a807884a17843955be24de2975 P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:17.736765 1029 tablet_service.cc:1467] Processing CreateTablet for tablet 93a2729c5a604a27a9bbc451ccb4e128 (DEFAULT_TABLE table=default.Table_43bb1aa67eec4cb89272fcec397e08b6 [id=5d1feb17ef344c9287e571496b2b405f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:17.751022 1194 ts_tablet_manager.cc:1397] T 4e61c6a807884a17843955be24de2975 P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.170s user 0.004s sys 0.001s
I20250113 02:51:17.751657 1196 ts_tablet_manager.cc:1918] T 9a323451cdfe4fd69ec62cda230f9afb P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:17.751855 1196 log.cc:1198] T 9a323451cdfe4fd69ec62cda230f9afb P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/9a323451cdfe4fd69ec62cda230f9afb
I20250113 02:51:17.752635 891 tablet_service.cc:1467] Processing CreateTablet for tablet 93a2729c5a604a27a9bbc451ccb4e128 (DEFAULT_TABLE table=default.Table_43bb1aa67eec4cb89272fcec397e08b6 [id=5d1feb17ef344c9287e571496b2b405f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:17.753067 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 93a2729c5a604a27a9bbc451ccb4e128. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:17.754539 888 tablet_service.cc:1467] Processing CreateTablet for tablet 23e9321f08ee4ce7b2252ad5c76c0a08 (DEFAULT_TABLE table=default.Table_93d706c664024860855d1c9d91992817 [id=6e014400813c4402891b32bcbe3555e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:17.759908 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 076ed7744da54ddebd8fc99686beb6ae
I20250113 02:51:17.760234 1196 ts_tablet_manager.cc:1939] T 9a323451cdfe4fd69ec62cda230f9afb P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
W20250113 02:51:17.761296 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 438d1d2300e044f39a15586f335935ba has no consensus state
I20250113 02:51:17.761653 1196 tablet_replica.cc:331] T 89517af3545c47a5a3a247a411190a47 P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:17.762382 1196 raft_consensus.cc:2238] T 89517af3545c47a5a3a247a411190a47 P df69f48656b44bb890f76587ee69cc0f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:17.762641 1196 raft_consensus.cc:2267] T 89517af3545c47a5a3a247a411190a47 P df69f48656b44bb890f76587ee69cc0f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:17.754405 897 tablet_service.cc:1467] Processing CreateTablet for tablet d304b65eba8c40c6922333506a60540c (DEFAULT_TABLE table=default.Table_f86569599f95484fb5cf0a5dccb6935c [id=e62f8e4c51b04729903b1943275c5f7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:17.764559 736 tablet_service.cc:1467] Processing CreateTablet for tablet 317c6e8013b745c290bf3190dc237ab8 (DEFAULT_TABLE table=default.Table_e1ae667a93ee48e8a3d233eeb6335364 [id=6bfb67a96b2b47849f7fa12c57a2f4a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:17.764990 736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 317c6e8013b745c290bf3190dc237ab8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:17.766569 1191 tablet_bootstrap.cc:492] T 9a323451cdfe4fd69ec62cda230f9afb P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:17.766744 1191 ts_tablet_manager.cc:1397] T 9a323451cdfe4fd69ec62cda230f9afb P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.212s user 0.053s sys 0.098s
I20250113 02:51:17.767575 1191 raft_consensus.cc:357] T 9a323451cdfe4fd69ec62cda230f9afb P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:17.767822 1191 raft_consensus.cc:383] T 9a323451cdfe4fd69ec62cda230f9afb P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:17.767932 1191 raft_consensus.cc:738] T 9a323451cdfe4fd69ec62cda230f9afb P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:17.768174 1191 consensus_queue.cc:260] T 9a323451cdfe4fd69ec62cda230f9afb P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:17.768595 748 tablet_service.cc:1467] Processing CreateTablet for tablet 41bcb2eafff14fa8a040982c723f99c1 (DEFAULT_TABLE table=default.Table_4a8b3f444ec345ad8a0c424f291e6d77 [id=2f9412fc42494fa68bee8ae23550beec]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:17.768829 738 tablet_service.cc:1467] Processing CreateTablet for tablet d7c4c7b0cfc44ac1952bc73cbe1bced8 (DEFAULT_TABLE table=default.Table_5727b5d8bf51442e8ae5f7c97f4bfbc8 [id=0c69597e8c5b4ec89179169a7a84e048]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:17.769061 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 41bcb2eafff14fa8a040982c723f99c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:17.769285 1191 ts_tablet_manager.cc:1428] T 9a323451cdfe4fd69ec62cda230f9afb P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20250113 02:51:17.769349 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7c4c7b0cfc44ac1952bc73cbe1bced8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:17.769574 1191 tablet_bootstrap.cc:492] T 7f3cbef7d91d4576822d913071a88248 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:17.771268 894 tablet_service.cc:1467] Processing CreateTablet for tablet d7920c0544c0466bad6bd8c09aae1a7d (DEFAULT_TABLE table=default.Table_47dd2257e94f494fbc848efdd878d9c8 [id=05755562cfc24370a3e6fb245a573dd4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:17.771507 1194 raft_consensus.cc:357] T 4e61c6a807884a17843955be24de2975 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:17.768647 744 tablet_service.cc:1467] Processing CreateTablet for tablet 34ab7ee573284550aaac9f814e8bd7ab (DEFAULT_TABLE table=default.Table_c59cac1b4e0249129c7671a4919283c8 [id=1cd50765f4964c15a341d307ff2d449f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:17.740190 1027 tablet_service.cc:1467] Processing CreateTablet for tablet ebc5f1ea1e1648d49904b17603d3cd2a (DEFAULT_TABLE table=default.Table_3b57de11e75e42fbb2cbe60078f76c8d [id=f4e09f1ea2184aa78abeb307bfc00d37]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:17.740764 1026 tablet_service.cc:1467] Processing CreateTablet for tablet f1a02c37cb4f43f8a9de6484f4431958 (DEFAULT_TABLE table=default.Table_f8af5d3e0dc64d3fae054f82a7b33ccf [id=4b739228564344b8b394a4718750dbb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:17.773375 898 tablet_service.cc:1467] Processing CreateTablet for tablet d3b695ac20ba4205b2d8732db4740531 (DEFAULT_TABLE table=default.Table_6d493198af4046609058d1f1a51ab823 [id=f85f0c435ffa4fdca81d86de4aa4fd5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:17.773567 1196 ts_tablet_manager.cc:1905] T 89517af3545c47a5a3a247a411190a47 P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:17.741367 1025 tablet_service.cc:1467] Processing CreateTablet for tablet b0361f11d0974ed19e7d4eb1c32a2f47 (DEFAULT_TABLE table=default.Table_1031f897b6e24d3abd972aca00ddb48b [id=f9202f2a27134b4984d824717b8c1232]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:17.741704 1030 tablet_service.cc:1467] Processing CreateTablet for tablet d7920c0544c0466bad6bd8c09aae1a7d (DEFAULT_TABLE table=default.Table_47dd2257e94f494fbc848efdd878d9c8 [id=05755562cfc24370a3e6fb245a573dd4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:17.771759 1024 tablet_service.cc:1467] Processing CreateTablet for tablet d3b695ac20ba4205b2d8732db4740531 (DEFAULT_TABLE table=default.Table_6d493198af4046609058d1f1a51ab823 [id=f85f0c435ffa4fdca81d86de4aa4fd5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:17.775177 893 tablet_service.cc:1467] Processing CreateTablet for tablet b0361f11d0974ed19e7d4eb1c32a2f47 (DEFAULT_TABLE table=default.Table_1031f897b6e24d3abd972aca00ddb48b [id=f9202f2a27134b4984d824717b8c1232]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:17.775432 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34ab7ee573284550aaac9f814e8bd7ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:17.775743 890 tablet_service.cc:1467] Processing CreateTablet for tablet f1a02c37cb4f43f8a9de6484f4431958 (DEFAULT_TABLE table=default.Table_f8af5d3e0dc64d3fae054f82a7b33ccf [id=4b739228564344b8b394a4718750dbb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:17.776216 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23e9321f08ee4ce7b2252ad5c76c0a08. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:17.776742 1194 raft_consensus.cc:383] T 4e61c6a807884a17843955be24de2975 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:17.776893 1194 raft_consensus.cc:738] T 4e61c6a807884a17843955be24de2975 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:17.777020 737 tablet_service.cc:1467] Processing CreateTablet for tablet fc92f19390fd48a4a45ba8fb839020fd (DEFAULT_TABLE table=default.Table_c94783b201764702b6de4f3219090efe [id=285c6d11c414470f8c40eefdf6a8a9e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:17.777484 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d3b695ac20ba4205b2d8732db4740531. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:17.778172 1194 consensus_queue.cc:260] T 4e61c6a807884a17843955be24de2975 P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:17.778731 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d304b65eba8c40c6922333506a60540c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:17.778759 1194 ts_tablet_manager.cc:1428] T 4e61c6a807884a17843955be24de2975 P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.028s user 0.002s sys 0.000s
I20250113 02:51:17.779417 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b0361f11d0974ed19e7d4eb1c32a2f47. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:17.779753 737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc92f19390fd48a4a45ba8fb839020fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:17.779847 740 tablet_service.cc:1514] Processing DeleteTablet for tablet c97a6ad98ddc4680a0f071cb91f05402 with delete_type TABLET_DATA_DELETED (replaced by tablet 2fe235d5280944ef92cf15fb458f2f81) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:17.780028 892 tablet_service.cc:1467] Processing CreateTablet for tablet ebc5f1ea1e1648d49904b17603d3cd2a (DEFAULT_TABLE table=default.Table_3b57de11e75e42fbb2cbe60078f76c8d [id=f4e09f1ea2184aa78abeb307bfc00d37]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:17.780316 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1a02c37cb4f43f8a9de6484f4431958. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:17.781023 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d304b65eba8c40c6922333506a60540c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:17.782001 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7920c0544c0466bad6bd8c09aae1a7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:17.783634 1195 ts_tablet_manager.cc:1939] T 5ae18cbff90d4fa88f6b9b8746fc2027 P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:17.783949 889 tablet_service.cc:1514] Processing DeleteTablet for tablet 076ed7744da54ddebd8fc99686beb6ae with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:17 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:17.784807 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 93a2729c5a604a27a9bbc451ccb4e128. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:17.784305 739 tablet_service.cc:1467] Processing CreateTablet for tablet 1c26074fdd804bd79cb0c295622d11ff (DEFAULT_TABLE table=default.Table_b147d4f32fa04b849ff6b75c9382a4e5 [id=bd8bd572ed524f2cae01f820f07e3c36]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:17.785225 1028 tablet_service.cc:1467] Processing CreateTablet for tablet b834f0acf5314bcb9d81c73de1b31842 (DEFAULT_TABLE table=default.Table_5a2e4e24ce9b4618813ca6d023d05664 [id=124669ccadaf49e19d2660b1495f9647]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:17.786839 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7920c0544c0466bad6bd8c09aae1a7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:17.787114 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1a02c37cb4f43f8a9de6484f4431958. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:17.787227 1040 tablet_service.cc:1514] Processing DeleteTablet for tablet 076ed7744da54ddebd8fc99686beb6ae with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:17 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:17.797540 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b0361f11d0974ed19e7d4eb1c32a2f47. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:17.799088 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ebc5f1ea1e1648d49904b17603d3cd2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:17.800668 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d3b695ac20ba4205b2d8732db4740531. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:17.801198 747 tablet_service.cc:1467] Processing CreateTablet for tablet be80b631ccd747f1b9355a00787a7b55 (DEFAULT_TABLE table=default.Table_2565c0551f9943a084ace8f7e463e59d [id=fe6f8d302c3e489d8bb832503fd05642]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:17.783365 891 tablet_service.cc:1467] Processing CreateTablet for tablet b834f0acf5314bcb9d81c73de1b31842 (DEFAULT_TABLE table=default.Table_5a2e4e24ce9b4618813ca6d023d05664 [id=124669ccadaf49e19d2660b1495f9647]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:17.804625 1192 tablet_bootstrap.cc:654] T 23e3cc4d6004439bb95ec3ab2504b1a1 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:17.804592 1041 tablet_service.cc:1467] Processing CreateTablet for tablet 438d1d2300e044f39a15586f335935ba (DEFAULT_TABLE table=default.Table_8fb0d7bc0499400a85e729a9dd749fad [id=2c7925ec0d8d4cd7b8761268d3962f15]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:17.804975 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 438d1d2300e044f39a15586f335935ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:17.805570 1195 tablet_replica.cc:331] T 23e3cc4d6004439bb95ec3ab2504b1a1 P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:17.805677 742 tablet_service.cc:1514] Processing DeleteTablet for tablet 68023c30a7f644eaa5b4ddd60d6c5ef5 with delete_type TABLET_DATA_DELETED (replaced by tablet 32351d6de1574353a6e8073d333da477) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:17.807194 746 tablet_service.cc:1514] Processing DeleteTablet for tablet 34ab7ee573284550aaac9f814e8bd7ab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:17 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:17.807068 1023 tablet_service.cc:1467] Processing CreateTablet for tablet 157905316fc545b6a7864ae1c4a2a584 (DEFAULT_TABLE table=default.Table_36c8a9314f6a4da4a1c0dbb51369d5bd [id=9f5add219a7c48da9a8afa966e22d1bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:17.810273 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b834f0acf5314bcb9d81c73de1b31842. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:17.806144 743 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d81eec487dd4016b3b804451da31c49 with delete_type TABLET_DATA_DELETED (replaced by tablet 438d1d2300e044f39a15586f335935ba) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:17.811111 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 157905316fc545b6a7864ae1c4a2a584. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:17.815029 753 tablet_service.cc:1467] Processing CreateTablet for tablet 23e9321f08ee4ce7b2252ad5c76c0a08 (DEFAULT_TABLE table=default.Table_93d706c664024860855d1c9d91992817 [id=6e014400813c4402891b32bcbe3555e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:17.815254 1191 tablet_bootstrap.cc:654] T 7f3cbef7d91d4576822d913071a88248 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:17.806479 741 tablet_service.cc:1467] Processing CreateTablet for tablet 4efa63b8be794e1db4b340f4c0c4cd5a (DEFAULT_TABLE table=default.Table_a27963e06eed4ec08989e43969f3bbbb [id=4b2bd999368c4b628c70cb3ee8c2b1f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:17.811062 740 tablet_service.cc:1467] Processing CreateTablet for tablet 4fe54131aafb40ee9d8bc5ea299af193 (DEFAULT_TABLE table=default.Table_b7158f0e7b8a4fd1a65dc21f11365153 [id=70089f757a1c45f6b983f216fb167478]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:17.815887 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ebc5f1ea1e1648d49904b17603d3cd2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:17.816233 1195 raft_consensus.cc:2238] T 23e3cc4d6004439bb95ec3ab2504b1a1 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:17.816326 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be80b631ccd747f1b9355a00787a7b55. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:17.816457 1195 raft_consensus.cc:2267] T 23e3cc4d6004439bb95ec3ab2504b1a1 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:17.819438 1194 tablet_bootstrap.cc:492] T 5ae18cbff90d4fa88f6b9b8746fc2027 P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:17.821302 1194 tablet_bootstrap.cc:654] T 5ae18cbff90d4fa88f6b9b8746fc2027 P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:17.824187 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet 9a323451cdfe4fd69ec62cda230f9afb (table default.Table_2106a7d1900a44df83cf7ec9f377687d [id=a2c08f472dd7494ab8aff87e9aa43028]) successfully deleted
I20250113 02:51:17.825436 450 master_service.cc:946] ReplaceTablet: received request to replace tablet 3455f010e51b4db1a9804a57fd1a4fb8 from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:17.826141 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_c357cad5db934f938c46acd208cc6f15"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:17.826797 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c357cad5db934f938c46acd208cc6f15 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:17.826947 446 master_service.cc:946] ReplaceTablet: received request to replace tablet 107e6e9e2d84496f8fce92b9ba377189 from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:17.827409 899 tablet_service.cc:1467] Processing CreateTablet for tablet 32351d6de1574353a6e8073d333da477 (DEFAULT_TABLE table=default.Table_5d38fa5123014e919f82dafd64c997c1 [id=e5831758ca184c1998f551fe0285d010]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:17.827876 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 32351d6de1574353a6e8073d333da477. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:17.826155 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_e1ae667a93ee48e8a3d233eeb6335364" } new_table_name: "default.Table_13fd4df9c207466cb85cd8ed9062722c" modify_external_catalogs: true
I20250113 02:51:17.828137 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_b5da750f5930453fa3951ffba96c34d4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:17.828615 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b5da750f5930453fa3951ffba96c34d4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:17.829587 889 tablet_service.cc:1467] Processing CreateTablet for tablet 438d1d2300e044f39a15586f335935ba (DEFAULT_TABLE table=default.Table_8fb0d7bc0499400a85e729a9dd749fad [id=2c7925ec0d8d4cd7b8761268d3962f15]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:17.829604 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_b7158f0e7b8a4fd1a65dc21f11365153" } new_table_name: "default.Table_7c37f90c233c4b79b3e2f866b8198d0c" modify_external_catalogs: true
I20250113 02:51:17.829953 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 438d1d2300e044f39a15586f335935ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:17.830590 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_06e860dcf006410ea4258e4ebe7e3104" } new_table_name: "default.Table_53b46860bb68469fa08f3fd02b2408e5" modify_external_catalogs: true
I20250113 02:51:17.830673 453 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_be2bc384adeb441093bc75976bb51c0b" } modify_external_catalogs: true
I20250113 02:51:17.830834 453 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_be2bc384adeb441093bc75976bb51c0b" } modify_external_catalogs: true
I20250113 02:51:17.831748 896 tablet_service.cc:1467] Processing CreateTablet for tablet 157905316fc545b6a7864ae1c4a2a584 (DEFAULT_TABLE table=default.Table_36c8a9314f6a4da4a1c0dbb51369d5bd [id=9f5add219a7c48da9a8afa966e22d1bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:17.832077 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b834f0acf5314bcb9d81c73de1b31842. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:17.832022 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_9cd2b585fc704fb6a2d0da56a774b8bd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:17.832291 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 157905316fc545b6a7864ae1c4a2a584. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:17.832454 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9cd2b585fc704fb6a2d0da56a774b8bd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:17.833746 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_47dd2257e94f494fbc848efdd878d9c8" } new_table_name: "default.Table_c9a2c2b1c62849babefee677197da844" modify_external_catalogs: true
I20250113 02:51:17.837846 751 tablet_service.cc:1467] Processing CreateTablet for tablet b834f0acf5314bcb9d81c73de1b31842 (DEFAULT_TABLE table=default.Table_5a2e4e24ce9b4618813ca6d023d05664 [id=124669ccadaf49e19d2660b1495f9647]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:17.844451 749 tablet_service.cc:1467] Processing CreateTablet for tablet d3b695ac20ba4205b2d8732db4740531 (DEFAULT_TABLE table=default.Table_6d493198af4046609058d1f1a51ab823 [id=f85f0c435ffa4fdca81d86de4aa4fd5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:17.845016 754 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a323451cdfe4fd69ec62cda230f9afb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:16 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:17.845492 745 tablet_service.cc:1467] Processing CreateTablet for tablet f1a02c37cb4f43f8a9de6484f4431958 (DEFAULT_TABLE table=default.Table_f8af5d3e0dc64d3fae054f82a7b33ccf [id=4b739228564344b8b394a4718750dbb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:17.845865 736 tablet_service.cc:1467] Processing CreateTablet for tablet 438d1d2300e044f39a15586f335935ba (DEFAULT_TABLE table=default.Table_8fb0d7bc0499400a85e729a9dd749fad [id=2c7925ec0d8d4cd7b8761268d3962f15]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:17.845924 750 tablet_service.cc:1467] Processing CreateTablet for tablet ebc5f1ea1e1648d49904b17603d3cd2a (DEFAULT_TABLE table=default.Table_3b57de11e75e42fbb2cbe60078f76c8d [id=f4e09f1ea2184aa78abeb307bfc00d37]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:17.857105 1191 tablet_bootstrap.cc:492] T 7f3cbef7d91d4576822d913071a88248 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:17.863476 1195 ts_tablet_manager.cc:1905] T 23e3cc4d6004439bb95ec3ab2504b1a1 P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:17.861763 742 tablet_service.cc:1467] Processing CreateTablet for tablet d7920c0544c0466bad6bd8c09aae1a7d (DEFAULT_TABLE table=default.Table_47dd2257e94f494fbc848efdd878d9c8 [id=05755562cfc24370a3e6fb245a573dd4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:17.867103 746 tablet_service.cc:1467] Processing CreateTablet for tablet d304b65eba8c40c6922333506a60540c (DEFAULT_TABLE table=default.Table_f86569599f95484fb5cf0a5dccb6935c [id=e62f8e4c51b04729903b1943275c5f7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:17.867419 743 tablet_service.cc:1467] Processing CreateTablet for tablet 93a2729c5a604a27a9bbc451ccb4e128 (DEFAULT_TABLE table=default.Table_43bb1aa67eec4cb89272fcec397e08b6 [id=5d1feb17ef344c9287e571496b2b405f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:17.868566 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c26074fdd804bd79cb0c295622d11ff. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:17.868921 995 leader_election.cc:343] T 7666ece05d334bac88fb7b58a4350b63 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Not found: Tablet not found: 7666ece05d334bac88fb7b58a4350b63
I20250113 02:51:17.869076 995 leader_election.cc:304] T 7666ece05d334bac88fb7b58a4350b63 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4216a7f853d642b7bc3e5cff458f3aca; no voters: 1186fa869eb14812832fa6562531f196, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:17.869254 755 tablet_service.cc:1467] Processing CreateTablet for tablet b0361f11d0974ed19e7d4eb1c32a2f47 (DEFAULT_TABLE table=default.Table_1031f897b6e24d3abd972aca00ddb48b [id=f9202f2a27134b4984d824717b8c1232]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:17.870910 754 tablet_service.cc:1467] Processing CreateTablet for tablet 32351d6de1574353a6e8073d333da477 (DEFAULT_TABLE table=default.Table_5d38fa5123014e919f82dafd64c997c1 [id=e5831758ca184c1998f551fe0285d010]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:17.871412 752 tablet_service.cc:1514] Processing DeleteTablet for tablet 076ed7744da54ddebd8fc99686beb6ae with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:17 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:17.872344 752 tablet_service.cc:1467] Processing CreateTablet for tablet 157905316fc545b6a7864ae1c4a2a584 (DEFAULT_TABLE table=default.Table_36c8a9314f6a4da4a1c0dbb51369d5bd [id=9f5add219a7c48da9a8afa966e22d1bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:17.874215 1191 ts_tablet_manager.cc:1397] T 7f3cbef7d91d4576822d913071a88248 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.088s user 0.004s sys 0.000s
I20250113 02:51:17.875300 1191 raft_consensus.cc:357] T 7f3cbef7d91d4576822d913071a88248 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:17.875793 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1a02c37cb4f43f8a9de6484f4431958. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:17.876076 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b0361f11d0974ed19e7d4eb1c32a2f47. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:17.877350 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 93a2729c5a604a27a9bbc451ccb4e128. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:17.877753 995 raft_consensus.cc:2681] T 7666ece05d334bac88fb7b58a4350b63 P 4216a7f853d642b7bc3e5cff458f3aca: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250113 02:51:17.879154 1042 tablet_service.cc:1467] Processing CreateTablet for tablet 06f36d09bbb1443bafce906907795404 (DEFAULT_TABLE table=default.Table_f771ada0bfa5404e819d999f416d30cb [id=423021f304964246bd29e4daaf470154]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:17.880656 1191 raft_consensus.cc:383] T 7f3cbef7d91d4576822d913071a88248 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:17.880777 1191 raft_consensus.cc:738] T 7f3cbef7d91d4576822d913071a88248 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:17.880676 1239 raft_consensus.cc:491] T b262c70712444c06a61918f21736f339 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:17.881397 1191 consensus_queue.cc:260] T 7f3cbef7d91d4576822d913071a88248 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:17.882014 1191 ts_tablet_manager.cc:1428] T 7f3cbef7d91d4576822d913071a88248 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20250113 02:51:17.882925 1239 raft_consensus.cc:513] T b262c70712444c06a61918f21736f339 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:17.883445 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d304b65eba8c40c6922333506a60540c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:17.885068 1191 tablet_bootstrap.cc:492] T 7dbda00269cf4f7095b3504ae87ac243 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:17.887753 1196 ts_tablet_manager.cc:1918] T 89517af3545c47a5a3a247a411190a47 P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:17.888038 1196 log.cc:1198] T 89517af3545c47a5a3a247a411190a47 P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/89517af3545c47a5a3a247a411190a47
I20250113 02:51:17.888454 1196 ts_tablet_manager.cc:1939] T 89517af3545c47a5a3a247a411190a47 P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:17.897145 896 tablet_service.cc:1467] Processing CreateTablet for tablet 1500e4f9c0664a879503afd3edf14c66 (DEFAULT_TABLE table=default.Table_a79eb97979f9403b97fa08ffd00c6513 [id=05cf81a8622149d29f0f4c567a44371c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:17.897547 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1500e4f9c0664a879503afd3edf14c66. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:17.897600 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet 89517af3545c47a5a3a247a411190a47 (table default.Table_5d3da53a6d984a7ea593b340ad3e1368 [id=e66cde9b8d9b4763b33e7641601c47da]) successfully deleted
W20250113 02:51:17.897876 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 1d81eec487dd4016b3b804451da31c49 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1d81eec487dd4016b3b804451da31c49 already in progress: creating tablet
W20250113 02:51:17.898092 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 34ab7ee573284550aaac9f814e8bd7ab because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 34ab7ee573284550aaac9f814e8bd7ab already in progress: creating tablet
W20250113 02:51:17.898300 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 0c57a5add4914835ac366efdcde4ee28 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0c57a5add4914835ac366efdcde4ee28 already in progress: creating tablet
W20250113 02:51:17.898506 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 68023c30a7f644eaa5b4ddd60d6c5ef5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 68023c30a7f644eaa5b4ddd60d6c5ef5 already in progress: creating tablet
I20250113 02:51:17.899581 1042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 06f36d09bbb1443bafce906907795404. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:17.893832 1200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250113 02:51:17.900254 1200 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250113 02:51:17.900406 1200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250113 02:51:17.904156 711 leader_election.cc:343] T b262c70712444c06a61918f21736f339 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Not found: Tablet not found: b262c70712444c06a61918f21736f339
I20250113 02:51:17.904181 1239 leader_election.cc:290] T b262c70712444c06a61918f21736f339 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
W20250113 02:51:17.904340 709 leader_election.cc:343] T b262c70712444c06a61918f21736f339 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Not found: Tablet not found: b262c70712444c06a61918f21736f339
I20250113 02:51:17.906097 889 tablet_service.cc:1467] Processing CreateTablet for tablet 8aac142230e244c0b59c9c2b7a7300ea (DEFAULT_TABLE table=default.Table_37b1d9d091924f8d9a8a2cbf1d06af1a [id=4d85a53a49d642b798af44eef60b3f78]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:17.906500 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8aac142230e244c0b59c9c2b7a7300ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:17.910190 1034 tablet_service.cc:1467] Processing CreateTablet for tablet ba61c86743e24666b30580c36b9228c3 (DEFAULT_TABLE table=default.Table_3f75f23afe2f449fa7a1f9df1c0734fb [id=68a4c317da7143cf841ee2d869d2f754]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:17.910591 1030 tablet_service.cc:1467] Processing CreateTablet for tablet 8aac142230e244c0b59c9c2b7a7300ea (DEFAULT_TABLE table=default.Table_37b1d9d091924f8d9a8a2cbf1d06af1a [id=4d85a53a49d642b798af44eef60b3f78]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:17.913168 895 tablet_service.cc:1467] Processing CreateTablet for tablet 35e0aed5125c4e1fa5dce226eade3b78 (DEFAULT_TABLE table=default.Table_be2bc384adeb441093bc75976bb51c0b [id=74c4c856f172482d955d7f6d0b62608f]), partition=RANGE (key) PARTITION UNBOUNDED
W20250113 02:51:17.913429 1200 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250113 02:51:17.913650 1200 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250113 02:51:17.913784 1200 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
I20250113 02:51:17.913894 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35e0aed5125c4e1fa5dce226eade3b78. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:17.913933 1200 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250113 02:51:17.913882 1033 tablet_service.cc:1467] Processing CreateTablet for tablet 1500e4f9c0664a879503afd3edf14c66 (DEFAULT_TABLE table=default.Table_a79eb97979f9403b97fa08ffd00c6513 [id=05cf81a8622149d29f0f4c567a44371c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:17.916451 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8aac142230e244c0b59c9c2b7a7300ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:17.922652 709 leader_election.cc:304] T b262c70712444c06a61918f21736f339 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:17.923161 1239 raft_consensus.cc:2743] T b262c70712444c06a61918f21736f339 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:17.924081 1029 tablet_service.cc:1467] Processing CreateTablet for tablet 35e0aed5125c4e1fa5dce226eade3b78 (DEFAULT_TABLE table=default.Table_be2bc384adeb441093bc75976bb51c0b [id=74c4c856f172482d955d7f6d0b62608f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:17.924815 893 tablet_service.cc:1467] Processing CreateTablet for tablet ba61c86743e24666b30580c36b9228c3 (DEFAULT_TABLE table=default.Table_3f75f23afe2f449fa7a1f9df1c0734fb [id=68a4c317da7143cf841ee2d869d2f754]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:17.925921 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba61c86743e24666b30580c36b9228c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:17.929319 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 157905316fc545b6a7864ae1c4a2a584. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:17.929941 1200 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:45573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/master-1/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:44217,127.0.104.61:35271,127.0.104.60:42067
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.61:35271
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.104.61
--webserver_port=37629
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 13 Jan 2025 02:43:23 UTC on 24a791456cd2
build id 4151
ASAN enabled
I20250113 02:51:17.930464 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7920c0544c0466bad6bd8c09aae1a7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:17.932018 1200 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250113 02:51:17.846731 1040 tablet_service.cc:1467] Processing CreateTablet for tablet 32351d6de1574353a6e8073d333da477 (DEFAULT_TABLE table=default.Table_5d38fa5123014e919f82dafd64c997c1 [id=e5831758ca184c1998f551fe0285d010]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:17.934234 1200 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250113 02:51:17.934255 738 tablet_service.cc:1467] Processing CreateTablet for tablet 06f36d09bbb1443bafce906907795404 (DEFAULT_TABLE table=default.Table_f771ada0bfa5404e819d999f416d30cb [id=423021f304964246bd29e4daaf470154]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:17.934759 737 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c57a5add4914835ac366efdcde4ee28 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:17 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:17.939604 890 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e61c6a807884a17843955be24de2975 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:16 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:17.940352 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ebc5f1ea1e1648d49904b17603d3cd2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:17.942847 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4fe54131aafb40ee9d8bc5ea299af193. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:17.932627 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba61c86743e24666b30580c36b9228c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:17.944651 1196 tablet_replica.cc:331] T 4e61c6a807884a17843955be24de2975 P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:17.944947 1196 raft_consensus.cc:2238] T 4e61c6a807884a17843955be24de2975 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:17.945223 1196 raft_consensus.cc:2267] T 4e61c6a807884a17843955be24de2975 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:17.946033 1196 ts_tablet_manager.cc:1905] T 4e61c6a807884a17843955be24de2975 P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250113 02:51:17.946251 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 076ed7744da54ddebd8fc99686beb6ae because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 076ed7744da54ddebd8fc99686beb6ae already in progress: creating tablet
I20250113 02:51:17.947575 894 tablet_service.cc:1467] Processing CreateTablet for tablet 06f36d09bbb1443bafce906907795404 (DEFAULT_TABLE table=default.Table_f771ada0bfa5404e819d999f416d30cb [id=423021f304964246bd29e4daaf470154]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:17.948081 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d3b695ac20ba4205b2d8732db4740531. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:17.949196 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 06f36d09bbb1443bafce906907795404. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:17.949771 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23e9321f08ee4ce7b2252ad5c76c0a08. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:17.950078 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4efa63b8be794e1db4b340f4c0c4cd5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:17.951205 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 32351d6de1574353a6e8073d333da477. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:17.952129 1191 tablet_bootstrap.cc:654] T 7dbda00269cf4f7095b3504ae87ac243 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:17.952908 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b834f0acf5314bcb9d81c73de1b31842. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:17.953935 736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 438d1d2300e044f39a15586f335935ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:17.954111 1198 ts_tablet_manager.cc:1918] T 015447488b734f6a93b8c36be975c031 P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:17.954811 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 06f36d09bbb1443bafce906907795404. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:17.954802 1196 ts_tablet_manager.cc:1918] T 4e61c6a807884a17843955be24de2975 P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:17.954916 1196 log.cc:1198] T 4e61c6a807884a17843955be24de2975 P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/4e61c6a807884a17843955be24de2975
I20250113 02:51:17.954998 1194 tablet_bootstrap.cc:492] T 5ae18cbff90d4fa88f6b9b8746fc2027 P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:17.958472 1194 ts_tablet_manager.cc:1397] T 5ae18cbff90d4fa88f6b9b8746fc2027 P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.139s user 0.004s sys 0.000s
I20250113 02:51:17.958583 1196 ts_tablet_manager.cc:1939] T 4e61c6a807884a17843955be24de2975 P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:17.959430 1194 raft_consensus.cc:357] T 5ae18cbff90d4fa88f6b9b8746fc2027 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:17.959853 737 tablet_service.cc:1467] Processing CreateTablet for tablet 1500e4f9c0664a879503afd3edf14c66 (DEFAULT_TABLE table=default.Table_a79eb97979f9403b97fa08ffd00c6513 [id=05cf81a8622149d29f0f4c567a44371c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:17.959995 1194 raft_consensus.cc:383] T 5ae18cbff90d4fa88f6b9b8746fc2027 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:17.960173 1194 raft_consensus.cc:738] T 5ae18cbff90d4fa88f6b9b8746fc2027 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:17.961966 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet 4e61c6a807884a17843955be24de2975 (table default.Table_6a96fb47f7444dcfb90d4463de217db6 [id=5eea6db2313a4b9abc61ea0545091e17]) successfully deleted
I20250113 02:51:17.962154 737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1500e4f9c0664a879503afd3edf14c66. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:17.965225 1040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 32351d6de1574353a6e8073d333da477. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:17.965901 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet 5ae18cbff90d4fa88f6b9b8746fc2027 (table default.Table_6a4e0c489c1944a69d3f46a874357dd6 [id=b6376e7b605c4b6f8d1373018a6c207e]) successfully deleted
I20250113 02:51:17.966069 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35e0aed5125c4e1fa5dce226eade3b78. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:17.966184 439 catalog_manager.cc:4839] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet c1ab705ebcf1430e9e0c376b18841d27 because the tablet was not found. No further retry: Not found: Tablet not found: c1ab705ebcf1430e9e0c376b18841d27
W20250113 02:51:17.966415 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet d6675bc3cb5f4f809320e28026cdf4b0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d6675bc3cb5f4f809320e28026cdf4b0 already in progress: creating tablet
I20250113 02:51:17.967082 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1500e4f9c0664a879503afd3edf14c66. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:17.967610 1198 log.cc:1198] T 015447488b734f6a93b8c36be975c031 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/015447488b734f6a93b8c36be975c031
I20250113 02:51:17.968096 1198 ts_tablet_manager.cc:1939] T 015447488b734f6a93b8c36be975c031 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:17.968885 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 3455f010e51b4db1a9804a57fd1a4fb8
I20250113 02:51:17.969336 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet 015447488b734f6a93b8c36be975c031 (table default.Table_2106a7d1900a44df83cf7ec9f377687d [id=a2c08f472dd7494ab8aff87e9aa43028]) successfully deleted
I20250113 02:51:17.969352 450 catalog_manager.cc:6596] ReplaceTablet: tablet 3455f010e51b4db1a9804a57fd1a4fb8 deleted and replaced by tablet a6055d77d168495d9b756a1a56c1a920
I20250113 02:51:17.970482 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_f11eaf2fe6674de8973c9363a4afca76"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:17.970948 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet 3455f010e51b4db1a9804a57fd1a4fb8 with delete_type TABLET_DATA_DELETED (replaced by tablet a6055d77d168495d9b756a1a56c1a920) from {username='slave'} at 127.0.0.1:60230
W20250113 02:51:17.971123 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f11eaf2fe6674de8973c9363a4afca76 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:17.976146 895 tablet_service.cc:1514] Processing DeleteTablet for tablet 3455f010e51b4db1a9804a57fd1a4fb8 with delete_type TABLET_DATA_DELETED (replaced by tablet a6055d77d168495d9b756a1a56c1a920) from {username='slave'} at 127.0.0.1:37366
W20250113 02:51:17.977149 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 3455f010e51b4db1a9804a57fd1a4fb8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3455f010e51b4db1a9804a57fd1a4fb8 already in progress: creating tablet
I20250113 02:51:17.982442 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 107e6e9e2d84496f8fce92b9ba377189
I20250113 02:51:17.983028 446 catalog_manager.cc:6596] ReplaceTablet: tablet 107e6e9e2d84496f8fce92b9ba377189 deleted and replaced by tablet 7ae2f30bf92742008e53b1ec94ebd0cf
I20250113 02:51:17.983772 895 tablet_service.cc:1514] Processing DeleteTablet for tablet 107e6e9e2d84496f8fce92b9ba377189 with delete_type TABLET_DATA_DELETED (replaced by tablet 7ae2f30bf92742008e53b1ec94ebd0cf) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:17.984459 454 master_service.cc:946] ReplaceTablet: received request to replace tablet d101e98b72544c30a3a4bc60d1576aa3 from {username='slave'} at 127.0.0.1:49258
W20250113 02:51:17.984807 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 107e6e9e2d84496f8fce92b9ba377189 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 107e6e9e2d84496f8fce92b9ba377189 already in progress: creating tablet
W20250113 02:51:17.985800 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 317c6e8013b745c290bf3190dc237ab8 consensus state has no leader
I20250113 02:51:17.986685 448 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_c94783b201764702b6de4f3219090efe" } modify_external_catalogs: true
I20250113 02:51:17.986876 448 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_c94783b201764702b6de4f3219090efe" } modify_external_catalogs: true
I20250113 02:51:17.964404 1194 consensus_queue.cc:260] T 5ae18cbff90d4fa88f6b9b8746fc2027 P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:17.987833 1194 ts_tablet_manager.cc:1428] T 5ae18cbff90d4fa88f6b9b8746fc2027 P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.029s user 0.000s sys 0.002s
W20250113 02:51:17.988217 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4fe54131aafb40ee9d8bc5ea299af193 consensus state has no leader
I20250113 02:51:17.988848 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 35e0aed5125c4e1fa5dce226eade3b78
I20250113 02:51:17.989540 453 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_1031f897b6e24d3abd972aca00ddb48b" } modify_external_catalogs: true
I20250113 02:51:17.989707 453 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_1031f897b6e24d3abd972aca00ddb48b" } modify_external_catalogs: true
I20250113 02:51:17.990276 893 tablet_service.cc:1514] Processing DeleteTablet for tablet 35e0aed5125c4e1fa5dce226eade3b78 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:17 UTC) from {username='slave'} at 127.0.0.1:37366
W20250113 02:51:17.990970 451 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d7920c0544c0466bad6bd8c09aae1a7d consensus state has no leader
I20250113 02:51:17.991204 451 master_service.cc:946] ReplaceTablet: received request to replace tablet 4e61c6a807884a17843955be24de2975 from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:17.991755 451 master_service.cc:946] ReplaceTablet: received request to replace tablet f0e7cc161f354f86b8c2fef303e87c5b from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:17.991907 1194 tablet_bootstrap.cc:492] T 6f1c2ae654bd46c1a8772fa62c8111ab P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:17.993589 1194 tablet_bootstrap.cc:654] T 6f1c2ae654bd46c1a8772fa62c8111ab P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:17.997879 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_c8bea6af909a424c9101c8934d1aee7b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:17.998441 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c8bea6af909a424c9101c8934d1aee7b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:17.998523 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_2e62961b127b4bb3ac9b5d30f408c299"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:17.999126 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2e62961b127b4bb3ac9b5d30f408c299 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:18.000560 447 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_74627d276edf4d75a2fcaa627a7f921b" } modify_external_catalogs: true
I20250113 02:51:18.000752 447 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_74627d276edf4d75a2fcaa627a7f921b" } modify_external_catalogs: true
W20250113 02:51:18.001844 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 35e0aed5125c4e1fa5dce226eade3b78 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 35e0aed5125c4e1fa5dce226eade3b78 already in progress: creating tablet
W20250113 02:51:18.003048 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 89517af3545c47a5a3a247a411190a47 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 89517af3545c47a5a3a247a411190a47 already in progress: creating tablet
W20250113 02:51:18.003311 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 68023c30a7f644eaa5b4ddd60d6c5ef5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 68023c30a7f644eaa5b4ddd60d6c5ef5 already in progress: creating tablet
W20250113 02:51:18.003553 440 catalog_manager.cc:4839] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 34ab7ee573284550aaac9f814e8bd7ab because the tablet was not found. No further retry: Not found: Tablet not found: 34ab7ee573284550aaac9f814e8bd7ab
W20250113 02:51:18.003746 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 1d81eec487dd4016b3b804451da31c49 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1d81eec487dd4016b3b804451da31c49 already in progress: creating tablet
W20250113 02:51:18.006503 1261 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 02:51:18.007210 1259 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 02:51:18.003952 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 0c57a5add4914835ac366efdcde4ee28 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0c57a5add4914835ac366efdcde4ee28 already in progress: creating tablet
W20250113 02:51:18.013667 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 076ed7744da54ddebd8fc99686beb6ae because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 076ed7744da54ddebd8fc99686beb6ae already in progress: creating tablet
I20250113 02:51:18.011757 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet d101e98b72544c30a3a4bc60d1576aa3
I20250113 02:51:18.015619 454 catalog_manager.cc:6596] ReplaceTablet: tablet d101e98b72544c30a3a4bc60d1576aa3 deleted and replaced by tablet 5b3f012b88c04cb68ed7a1848147b70b
I20250113 02:51:18.015810 894 tablet_service.cc:1514] Processing DeleteTablet for tablet d101e98b72544c30a3a4bc60d1576aa3 with delete_type TABLET_DATA_DELETED (replaced by tablet 5b3f012b88c04cb68ed7a1848147b70b) from {username='slave'} at 127.0.0.1:37366
W20250113 02:51:18.016634 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet d101e98b72544c30a3a4bc60d1576aa3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d101e98b72544c30a3a4bc60d1576aa3 already in progress: creating tablet
I20250113 02:51:18.017385 894 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ae18cbff90d4fa88f6b9b8746fc2027 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:16 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:18.017360 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_abea3b2ec5144f1fadd95a623c48917c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:18.017673 1196 tablet_replica.cc:331] T 5ae18cbff90d4fa88f6b9b8746fc2027 P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:18.017849 1196 raft_consensus.cc:2238] T 5ae18cbff90d4fa88f6b9b8746fc2027 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:18.018014 1196 raft_consensus.cc:2267] T 5ae18cbff90d4fa88f6b9b8746fc2027 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
W20250113 02:51:18.018002 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_abea3b2ec5144f1fadd95a623c48917c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:18.012468 446 catalog_manager.cc:5618] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095) does not have the latest schema for tablet 7f3cbef7d91d4576822d913071a88248 (table default.Table_b581b1a063a54551a8535adf01983fb8 [id=54e4e3810b3b412bbec3b5bf5b28c1a2]). Expected version 1 got 0
I20250113 02:51:18.012485 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet fc92f19390fd48a4a45ba8fb839020fd
I20250113 02:51:18.013059 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet b0361f11d0974ed19e7d4eb1c32a2f47
I20250113 02:51:18.021517 894 tablet_service.cc:1514] Processing DeleteTablet for tablet fc92f19390fd48a4a45ba8fb839020fd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:17 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:18.013363 447 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet 7ae2f30bf92742008e53b1ec94ebd0cf
I20250113 02:51:18.022861 894 tablet_service.cc:1514] Processing DeleteTablet for tablet b0361f11d0974ed19e7d4eb1c32a2f47 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:17 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:18.025079 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_266561dfe36245ecb8230fdf1904cb02"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:18.025733 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_266561dfe36245ecb8230fdf1904cb02 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:18.027704 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_6f2972dcacb44ccb9e4f3d828ee536ec"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:18.028332 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6f2972dcacb44ccb9e4f3d828ee536ec in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:18.029968 448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_870a194df20442a1ae84487760f30246" } new_table_name: "default.Table_20efb43987cc47c7a77436e076cc70f6" modify_external_catalogs: true
I20250113 02:51:18.030570 447 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_5727b5d8bf51442e8ae5f7c97f4bfbc8" } modify_external_catalogs: true
I20250113 02:51:18.030913 447 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_5727b5d8bf51442e8ae5f7c97f4bfbc8" } modify_external_catalogs: true
W20250113 02:51:18.033553 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fb183b4588e84212a3b4b096f430d47d consensus state has no leader
I20250113 02:51:18.036512 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet f0e7cc161f354f86b8c2fef303e87c5b
I20250113 02:51:18.037110 451 catalog_manager.cc:6596] ReplaceTablet: tablet f0e7cc161f354f86b8c2fef303e87c5b deleted and replaced by tablet 572e4eed1e924381bada27f8389821a7
I20250113 02:51:18.037815 894 tablet_service.cc:1514] Processing DeleteTablet for tablet f0e7cc161f354f86b8c2fef303e87c5b with delete_type TABLET_DATA_DELETED (replaced by tablet 572e4eed1e924381bada27f8389821a7) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:18.039343 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet d7c4c7b0cfc44ac1952bc73cbe1bced8
I20250113 02:51:18.040122 894 tablet_service.cc:1514] Processing DeleteTablet for tablet d7c4c7b0cfc44ac1952bc73cbe1bced8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:18 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:18.046891 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_7d21501217454370b9e8e931e13e5a8e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:18.047592 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7d21501217454370b9e8e931e13e5a8e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250113 02:51:18.049278 440 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 4fe54131aafb40ee9d8bc5ea299af193 (table default.Table_7c37f90c233c4b79b3e2f866b8198d0c [id=70089f757a1c45f6b983f216fb167478], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 4fe54131aafb40ee9d8bc5ea299af193 consensus state has no leader
W20250113 02:51:18.052083 448 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9b5dc06ba3ea4056822ca3e4b1249587 consensus state has no leader
I20250113 02:51:18.064232 1196 ts_tablet_manager.cc:1905] T 5ae18cbff90d4fa88f6b9b8746fc2027 P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:18.067281 1031 tablet_service.cc:1514] Processing DeleteTablet for tablet 35e0aed5125c4e1fa5dce226eade3b78 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:17 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:18.067582 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet 107e6e9e2d84496f8fce92b9ba377189 with delete_type TABLET_DATA_DELETED (replaced by tablet 7ae2f30bf92742008e53b1ec94ebd0cf) from {username='slave'} at 127.0.0.1:60230
W20250113 02:51:18.069496 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 3455f010e51b4db1a9804a57fd1a4fb8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3455f010e51b4db1a9804a57fd1a4fb8 already in progress: creating tablet
W20250113 02:51:18.070014 440 catalog_manager.cc:4839] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 35e0aed5125c4e1fa5dce226eade3b78 because the tablet was not found. No further retry: Not found: Tablet not found: 35e0aed5125c4e1fa5dce226eade3b78
I20250113 02:51:18.070065 1196 ts_tablet_manager.cc:1918] T 5ae18cbff90d4fa88f6b9b8746fc2027 P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:18.070190 1196 log.cc:1198] T 5ae18cbff90d4fa88f6b9b8746fc2027 P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/5ae18cbff90d4fa88f6b9b8746fc2027
I20250113 02:51:18.070473 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet fc92f19390fd48a4a45ba8fb839020fd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:17 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:18.070529 1196 ts_tablet_manager.cc:1939] T 5ae18cbff90d4fa88f6b9b8746fc2027 P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:18.071424 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet 5ae18cbff90d4fa88f6b9b8746fc2027 (table default.Table_6a4e0c489c1944a69d3f46a874357dd6 [id=b6376e7b605c4b6f8d1373018a6c207e]) successfully deleted
W20250113 02:51:18.072052 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet fc92f19390fd48a4a45ba8fb839020fd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fc92f19390fd48a4a45ba8fb839020fd already in progress: creating tablet
W20250113 02:51:18.072292 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet b0361f11d0974ed19e7d4eb1c32a2f47 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b0361f11d0974ed19e7d4eb1c32a2f47 already in progress: creating tablet
I20250113 02:51:18.072374 1192 tablet_bootstrap.cc:492] T 23e3cc4d6004439bb95ec3ab2504b1a1 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
W20250113 02:51:18.072497 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet f0e7cc161f354f86b8c2fef303e87c5b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f0e7cc161f354f86b8c2fef303e87c5b already in progress: creating tablet
W20250113 02:51:18.072795 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet d7c4c7b0cfc44ac1952bc73cbe1bced8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d7c4c7b0cfc44ac1952bc73cbe1bced8 already in progress: creating tablet
I20250113 02:51:18.076666 1192 ts_tablet_manager.cc:1397] T 23e3cc4d6004439bb95ec3ab2504b1a1 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.462s user 0.009s sys 0.003s
W20250113 02:51:18.077507 440 catalog_manager.cc:4839] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet fc92f19390fd48a4a45ba8fb839020fd because the tablet was not found. No further retry: Not found: Tablet not found: fc92f19390fd48a4a45ba8fb839020fd
I20250113 02:51:18.079077 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet b0361f11d0974ed19e7d4eb1c32a2f47 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:17 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:18.080063 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet f0e7cc161f354f86b8c2fef303e87c5b with delete_type TABLET_DATA_DELETED (replaced by tablet 572e4eed1e924381bada27f8389821a7) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:18.080271 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet d7c4c7b0cfc44ac1952bc73cbe1bced8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:18 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:18.081401 1192 raft_consensus.cc:357] T 23e3cc4d6004439bb95ec3ab2504b1a1 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
W20250113 02:51:18.082232 440 catalog_manager.cc:4839] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet b0361f11d0974ed19e7d4eb1c32a2f47 because the tablet was not found. No further retry: Not found: Tablet not found: b0361f11d0974ed19e7d4eb1c32a2f47
W20250113 02:51:18.082521 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet f0e7cc161f354f86b8c2fef303e87c5b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f0e7cc161f354f86b8c2fef303e87c5b already in progress: creating tablet
I20250113 02:51:18.077783 1031 tablet_service.cc:1514] Processing DeleteTablet for tablet d101e98b72544c30a3a4bc60d1576aa3 with delete_type TABLET_DATA_DELETED (replaced by tablet 5b3f012b88c04cb68ed7a1848147b70b) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:18.083520 1192 raft_consensus.cc:383] T 23e3cc4d6004439bb95ec3ab2504b1a1 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:18.083623 1192 raft_consensus.cc:738] T 23e3cc4d6004439bb95ec3ab2504b1a1 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:18.084180 1192 consensus_queue.cc:260] T 23e3cc4d6004439bb95ec3ab2504b1a1 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:18.084725 1192 ts_tablet_manager.cc:1428] T 23e3cc4d6004439bb95ec3ab2504b1a1 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20250113 02:51:18.087082 1239 raft_consensus.cc:491] T 6f1c2ae654bd46c1a8772fa62c8111ab P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:18.087319 1239 raft_consensus.cc:513] T 6f1c2ae654bd46c1a8772fa62c8111ab P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:18.088757 1194 tablet_bootstrap.cc:492] T 6f1c2ae654bd46c1a8772fa62c8111ab P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:18.089730 1194 ts_tablet_manager.cc:1397] T 6f1c2ae654bd46c1a8772fa62c8111ab P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.098s user 0.008s sys 0.027s
I20250113 02:51:18.090081 1239 leader_election.cc:290] T 6f1c2ae654bd46c1a8772fa62c8111ab P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:18.090657 1194 raft_consensus.cc:357] T 6f1c2ae654bd46c1a8772fa62c8111ab P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:18.090895 1194 raft_consensus.cc:383] T 6f1c2ae654bd46c1a8772fa62c8111ab P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:18.090955 1194 raft_consensus.cc:738] T 6f1c2ae654bd46c1a8772fa62c8111ab P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:18.091252 744 tablet_service.cc:1467] Processing CreateTablet for tablet 8aac142230e244c0b59c9c2b7a7300ea (DEFAULT_TABLE table=default.Table_37b1d9d091924f8d9a8a2cbf1d06af1a [id=4d85a53a49d642b798af44eef60b3f78]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:18.092128 748 tablet_service.cc:1467] Processing CreateTablet for tablet ba61c86743e24666b30580c36b9228c3 (DEFAULT_TABLE table=default.Table_3f75f23afe2f449fa7a1f9df1c0734fb [id=68a4c317da7143cf841ee2d869d2f754]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:18.091176 1194 consensus_queue.cc:260] T 6f1c2ae654bd46c1a8772fa62c8111ab P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:18.092789 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f1c2ae654bd46c1a8772fa62c8111ab" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:18.092993 1194 ts_tablet_manager.cc:1428] T 6f1c2ae654bd46c1a8772fa62c8111ab P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250113 02:51:18.093293 1194 tablet_bootstrap.cc:492] T 7f3cbef7d91d4576822d913071a88248 P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:18.093000 918 raft_consensus.cc:2463] T 6f1c2ae654bd46c1a8772fa62c8111ab P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1186fa869eb14812832fa6562531f196 in term 0.
I20250113 02:51:18.095887 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8aac142230e244c0b59c9c2b7a7300ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:18.097786 1194 tablet_bootstrap.cc:654] T 7f3cbef7d91d4576822d913071a88248 P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:18.100037 894 tablet_service.cc:1467] Processing CreateTablet for tablet a6055d77d168495d9b756a1a56c1a920 (DEFAULT_TABLE table=default.Table_44f1c4e8cc2b4e83adce8d411ae53448 [id=a9e0aae9f5af42b796d29a75140f9c29]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:18.100538 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6055d77d168495d9b756a1a56c1a920. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:18.102069 739 tablet_service.cc:1467] Processing CreateTablet for tablet 35e0aed5125c4e1fa5dce226eade3b78 (DEFAULT_TABLE table=default.Table_be2bc384adeb441093bc75976bb51c0b [id=74c4c856f172482d955d7f6d0b62608f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:18.106598 1194 tablet_bootstrap.cc:492] T 7f3cbef7d91d4576822d913071a88248 P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:18.106788 1194 ts_tablet_manager.cc:1397] T 7f3cbef7d91d4576822d913071a88248 P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.002s
I20250113 02:51:18.107662 1194 raft_consensus.cc:357] T 7f3cbef7d91d4576822d913071a88248 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:18.107930 1194 raft_consensus.cc:383] T 7f3cbef7d91d4576822d913071a88248 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:18.108047 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba61c86743e24666b30580c36b9228c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:18.108151 1194 raft_consensus.cc:738] T 7f3cbef7d91d4576822d913071a88248 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:18.108479 1194 consensus_queue.cc:260] T 7f3cbef7d91d4576822d913071a88248 P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:18.109148 1194 ts_tablet_manager.cc:1428] T 7f3cbef7d91d4576822d913071a88248 P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250113 02:51:18.110426 893 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f1c2ae654bd46c1a8772fa62c8111ab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:17 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:18.111085 1192 tablet_bootstrap.cc:492] T 89517af3545c47a5a3a247a411190a47 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:18.113448 1196 tablet_replica.cc:331] T 6f1c2ae654bd46c1a8772fa62c8111ab P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:18.113628 1196 raft_consensus.cc:2238] T 6f1c2ae654bd46c1a8772fa62c8111ab P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:18.113803 1196 raft_consensus.cc:2267] T 6f1c2ae654bd46c1a8772fa62c8111ab P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:18.114378 1194 tablet_bootstrap.cc:492] T 7dbda00269cf4f7095b3504ae87ac243 P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:18.115057 711 leader_election.cc:304] T 6f1c2ae654bd46c1a8772fa62c8111ab P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1186fa869eb14812832fa6562531f196, df69f48656b44bb890f76587ee69cc0f; no voters:
I20250113 02:51:18.115130 1196 ts_tablet_manager.cc:1905] T 6f1c2ae654bd46c1a8772fa62c8111ab P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:18.116236 1194 tablet_bootstrap.cc:654] T 7dbda00269cf4f7095b3504ae87ac243 P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:18.117452 1062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f1c2ae654bd46c1a8772fa62c8111ab" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
W20250113 02:51:18.118588 709 leader_election.cc:343] T 6f1c2ae654bd46c1a8772fa62c8111ab P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:18.121395 1191 tablet_bootstrap.cc:492] T 7dbda00269cf4f7095b3504ae87ac243 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:18.121611 1191 ts_tablet_manager.cc:1397] T 7dbda00269cf4f7095b3504ae87ac243 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.238s user 0.021s sys 0.021s
I20250113 02:51:18.122483 1196 ts_tablet_manager.cc:1918] T 6f1c2ae654bd46c1a8772fa62c8111ab P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:18.122646 1196 log.cc:1198] T 6f1c2ae654bd46c1a8772fa62c8111ab P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/6f1c2ae654bd46c1a8772fa62c8111ab
I20250113 02:51:18.123553 1196 ts_tablet_manager.cc:1939] T 6f1c2ae654bd46c1a8772fa62c8111ab P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
W20250113 02:51:18.123802 1264 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 02:51:18.124732 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet 6f1c2ae654bd46c1a8772fa62c8111ab (table default.Table_ec4ee7380446448293eaa15db62d34de [id=7dc8c3e55ce843009ade78391e271091]) successfully deleted
I20250113 02:51:18.126489 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35e0aed5125c4e1fa5dce226eade3b78. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:18.127641 1239 raft_consensus.cc:2798] T 6f1c2ae654bd46c1a8772fa62c8111ab P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 02:51:18.128641 1239 raft_consensus.cc:491] T 6f1c2ae654bd46c1a8772fa62c8111ab P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 02:51:18.128801 1239 raft_consensus.cc:3054] T 6f1c2ae654bd46c1a8772fa62c8111ab P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Advancing to term 1
W20250113 02:51:18.129881 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet d7c4c7b0cfc44ac1952bc73cbe1bced8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d7c4c7b0cfc44ac1952bc73cbe1bced8 already in progress: creating tablet
W20250113 02:51:18.130126 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet d101e98b72544c30a3a4bc60d1576aa3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d101e98b72544c30a3a4bc60d1576aa3 already in progress: creating tablet
I20250113 02:51:18.132016 1191 raft_consensus.cc:357] T 7dbda00269cf4f7095b3504ae87ac243 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:18.132241 1191 raft_consensus.cc:383] T 7dbda00269cf4f7095b3504ae87ac243 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:18.134085 1204 raft_consensus.cc:491] T 23e3cc4d6004439bb95ec3ab2504b1a1 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:18.136523 1204 raft_consensus.cc:513] T 23e3cc4d6004439bb95ec3ab2504b1a1 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:18.137197 1204 leader_election.cc:290] T 23e3cc4d6004439bb95ec3ab2504b1a1 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:18.141907 1028 tablet_service.cc:1467] Processing CreateTablet for tablet a6055d77d168495d9b756a1a56c1a920 (DEFAULT_TABLE table=default.Table_44f1c4e8cc2b4e83adce8d411ae53448 [id=a9e0aae9f5af42b796d29a75140f9c29]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:18.142450 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6055d77d168495d9b756a1a56c1a920. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:18.143896 1191 raft_consensus.cc:738] T 7dbda00269cf4f7095b3504ae87ac243 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
W20250113 02:51:18.145682 998 leader_election.cc:343] T 23e3cc4d6004439bb95ec3ab2504b1a1 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Not found: Tablet not found: 23e3cc4d6004439bb95ec3ab2504b1a1
I20250113 02:51:18.150328 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23e3cc4d6004439bb95ec3ab2504b1a1" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
I20250113 02:51:18.157655 752 tablet_service.cc:1514] Processing DeleteTablet for tablet 3455f010e51b4db1a9804a57fd1a4fb8 with delete_type TABLET_DATA_DELETED (replaced by tablet a6055d77d168495d9b756a1a56c1a920) from {username='slave'} at 127.0.0.1:55398
W20250113 02:51:18.161862 995 leader_election.cc:343] T 23e3cc4d6004439bb95ec3ab2504b1a1 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Illegal state: Tablet not RUNNING: STOPPED
I20250113 02:51:18.158085 1191 consensus_queue.cc:260] T 7dbda00269cf4f7095b3504ae87ac243 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:18.164965 1036 tablet_service.cc:1514] Processing DeleteTablet for tablet 23e3cc4d6004439bb95ec3ab2504b1a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:16 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:18.166620 995 leader_election.cc:304] T 23e3cc4d6004439bb95ec3ab2504b1a1 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4216a7f853d642b7bc3e5cff458f3aca; no voters: 1186fa869eb14812832fa6562531f196, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:18.167841 1204 raft_consensus.cc:2743] T 23e3cc4d6004439bb95ec3ab2504b1a1 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:18.173759 1198 tablet_replica.cc:331] T 23e3cc4d6004439bb95ec3ab2504b1a1 P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:18.173941 1198 raft_consensus.cc:2238] T 23e3cc4d6004439bb95ec3ab2504b1a1 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:18.169842 1191 ts_tablet_manager.cc:1428] T 7dbda00269cf4f7095b3504ae87ac243 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.042s user 0.003s sys 0.000s
I20250113 02:51:18.174815 752 tablet_service.cc:1514] Processing DeleteTablet for tablet 107e6e9e2d84496f8fce92b9ba377189 with delete_type TABLET_DATA_DELETED (replaced by tablet 7ae2f30bf92742008e53b1ec94ebd0cf) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:18.177968 747 tablet_service.cc:1514] Processing DeleteTablet for tablet d101e98b72544c30a3a4bc60d1576aa3 with delete_type TABLET_DATA_DELETED (replaced by tablet 5b3f012b88c04cb68ed7a1848147b70b) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:18.182619 448 master_service.cc:946] ReplaceTablet: received request to replace tablet 107e6e9e2d84496f8fce92b9ba377189 from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:18.184870 752 tablet_service.cc:1514] Processing DeleteTablet for tablet 35e0aed5125c4e1fa5dce226eade3b78 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:17 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:18.185915 1192 tablet_bootstrap.cc:654] T 89517af3545c47a5a3a247a411190a47 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:18.186226 1198 raft_consensus.cc:2267] T 23e3cc4d6004439bb95ec3ab2504b1a1 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:18.187986 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_7a9c9cc8678e4f7fa12237d3f8c9c3de"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:18.188469 448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_c46ffb30f29b44eea3c09b33f6697efe" } new_table_name: "default.Table_8aa63512c7bb4bdb86f37c9ee24df209" modify_external_catalogs: true
W20250113 02:51:18.188642 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7a9c9cc8678e4f7fa12237d3f8c9c3de in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:18.189953 747 tablet_service.cc:1514] Processing DeleteTablet for tablet fc92f19390fd48a4a45ba8fb839020fd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:17 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:18.190699 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_5d38fa5123014e919f82dafd64c997c1" } new_table_name: "default.Table_78c0bcf1e2d64abc81253301920cf566" modify_external_catalogs: true
I20250113 02:51:18.193331 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_8a5b45e248e94efb9b16df971fb9755c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:18.193593 447 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_20efb43987cc47c7a77436e076cc70f6" } modify_external_catalogs: true
I20250113 02:51:18.193842 447 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_20efb43987cc47c7a77436e076cc70f6" } modify_external_catalogs: true
W20250113 02:51:18.193903 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8a5b45e248e94efb9b16df971fb9755c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:18.196102 449 master_service.cc:946] ReplaceTablet: received request to replace tablet 8aac142230e244c0b59c9c2b7a7300ea from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:18.198474 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_6167b33d8e4246cab5fbe98cb57e9a3c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:18.198967 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6167b33d8e4246cab5fbe98cb57e9a3c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:18.200938 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_8fb0d7bc0499400a85e729a9dd749fad" } new_table_name: "default.Table_41f119048882478f85a1344410383dac" modify_external_catalogs: true
W20250113 02:51:18.204420 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 32351d6de1574353a6e8073d333da477 consensus state has no leader
I20250113 02:51:18.204941 1209 raft_consensus.cc:491] T bc88676bf738442d82c25e56f30c4ada P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:18.205546 424 master_runner.cc:186] Error getting master registration for 127.0.104.61:35271: Network error: Client connection negotiation failed: client connection to 127.0.104.61:35271: connect: Connection refused (error 111),
I20250113 02:51:18.205703 424 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250113 02:51:18.207999 1198 ts_tablet_manager.cc:1905] T 23e3cc4d6004439bb95ec3ab2504b1a1 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:18.208254 755 tablet_service.cc:1514] Processing DeleteTablet for tablet d7c4c7b0cfc44ac1952bc73cbe1bced8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:18 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:18.208740 755 tablet_service.cc:1467] Processing CreateTablet for tablet a6055d77d168495d9b756a1a56c1a920 (DEFAULT_TABLE table=default.Table_44f1c4e8cc2b4e83adce8d411ae53448 [id=a9e0aae9f5af42b796d29a75140f9c29]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:18.209654 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6055d77d168495d9b756a1a56c1a920. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:18.210521 747 tablet_service.cc:1514] Processing DeleteTablet for tablet f0e7cc161f354f86b8c2fef303e87c5b with delete_type TABLET_DATA_DELETED (replaced by tablet 572e4eed1e924381bada27f8389821a7) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:18.210677 1191 tablet_bootstrap.cc:492] T db266a1848094b7a98aa75f7afabd471 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:18.211477 752 tablet_service.cc:1514] Processing DeleteTablet for tablet b0361f11d0974ed19e7d4eb1c32a2f47 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:17 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:18.212970 1209 raft_consensus.cc:513] T bc88676bf738442d82c25e56f30c4ada P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:18.214613 446 master_service.cc:946] ReplaceTablet: received request to replace tablet d101e98b72544c30a3a4bc60d1576aa3 from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:18.217659 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc88676bf738442d82c25e56f30c4ada" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:18.219111 711 leader_election.cc:343] T bc88676bf738442d82c25e56f30c4ada P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:18.220495 1209 leader_election.cc:290] T bc88676bf738442d82c25e56f30c4ada P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:18.221335 1062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc88676bf738442d82c25e56f30c4ada" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
W20250113 02:51:18.222710 709 leader_election.cc:343] T bc88676bf738442d82c25e56f30c4ada P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:18.222852 709 leader_election.cc:304] T bc88676bf738442d82c25e56f30c4ada P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:18.223881 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_f9aa5df6ba3d43999ebd6570df3108d1" } modify_external_catalogs: true
I20250113 02:51:18.224115 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_f9aa5df6ba3d43999ebd6570df3108d1" } modify_external_catalogs: true
I20250113 02:51:18.224257 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_87cfc114b15047b49fd25446ff7aa441" } modify_external_catalogs: true
I20250113 02:51:18.224398 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_87cfc114b15047b49fd25446ff7aa441" } modify_external_catalogs: true
I20250113 02:51:18.225450 446 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_78c0bcf1e2d64abc81253301920cf566" } modify_external_catalogs: true
I20250113 02:51:18.225641 446 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_78c0bcf1e2d64abc81253301920cf566" } modify_external_catalogs: true
I20250113 02:51:18.230398 1209 raft_consensus.cc:2743] T bc88676bf738442d82c25e56f30c4ada P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:18.231168 1239 raft_consensus.cc:513] T 6f1c2ae654bd46c1a8772fa62c8111ab P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:18.235626 1192 tablet_bootstrap.cc:492] T 89517af3545c47a5a3a247a411190a47 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:18.235860 1192 ts_tablet_manager.cc:1397] T 89517af3545c47a5a3a247a411190a47 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.150s user 0.002s sys 0.003s
W20250113 02:51:18.237028 711 leader_election.cc:343] T 6f1c2ae654bd46c1a8772fa62c8111ab P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Not found: Tablet not found: 6f1c2ae654bd46c1a8772fa62c8111ab
I20250113 02:51:18.237171 1062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f1c2ae654bd46c1a8772fa62c8111ab" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca"
I20250113 02:51:18.237531 1192 raft_consensus.cc:357] T 89517af3545c47a5a3a247a411190a47 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:18.237767 1192 raft_consensus.cc:383] T 89517af3545c47a5a3a247a411190a47 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:18.237913 1192 raft_consensus.cc:738] T 89517af3545c47a5a3a247a411190a47 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:18.238221 1192 consensus_queue.cc:260] T 89517af3545c47a5a3a247a411190a47 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:18.238742 1192 ts_tablet_manager.cc:1428] T 89517af3545c47a5a3a247a411190a47 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250113 02:51:18.239082 1192 tablet_bootstrap.cc:492] T c1ab705ebcf1430e9e0c376b18841d27 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
W20250113 02:51:18.240763 709 leader_election.cc:343] T 6f1c2ae654bd46c1a8772fa62c8111ab P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:18.240913 709 leader_election.cc:304] T 6f1c2ae654bd46c1a8772fa62c8111ab P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:18.242149 1209 raft_consensus.cc:491] T c97a6ad98ddc4680a0f071cb91f05402 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:18.245079 1198 ts_tablet_manager.cc:1918] T 23e3cc4d6004439bb95ec3ab2504b1a1 P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:18.245272 1198 log.cc:1198] T 23e3cc4d6004439bb95ec3ab2504b1a1 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/23e3cc4d6004439bb95ec3ab2504b1a1
I20250113 02:51:18.245671 1192 tablet_bootstrap.cc:654] T c1ab705ebcf1430e9e0c376b18841d27 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:18.245816 1198 ts_tablet_manager.cc:1939] T 23e3cc4d6004439bb95ec3ab2504b1a1 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:18.247113 1209 raft_consensus.cc:513] T c97a6ad98ddc4680a0f071cb91f05402 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:18.247658 1195 ts_tablet_manager.cc:1918] T 23e3cc4d6004439bb95ec3ab2504b1a1 P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:18.247769 1209 leader_election.cc:290] T c97a6ad98ddc4680a0f071cb91f05402 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:18.242599 1239 leader_election.cc:290] T 6f1c2ae654bd46c1a8772fa62c8111ab P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 election: Requested vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:18.249821 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet 23e3cc4d6004439bb95ec3ab2504b1a1 (table default.Table_dfb2da2848a94fe4b5d89023934e8740 [id=b409b6c2b10e471aba03feb404311e3d]) successfully deleted
I20250113 02:51:18.251500 1062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c97a6ad98ddc4680a0f071cb91f05402" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:18.252096 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c97a6ad98ddc4680a0f071cb91f05402" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:18.256301 1209 raft_consensus.cc:2743] T 6f1c2ae654bd46c1a8772fa62c8111ab P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250113 02:51:18.256515 1191 tablet_bootstrap.cc:654] T db266a1848094b7a98aa75f7afabd471 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
W20250113 02:51:18.259385 709 leader_election.cc:343] T c97a6ad98ddc4680a0f071cb91f05402 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:18.261883 1195 log.cc:1198] T 23e3cc4d6004439bb95ec3ab2504b1a1 P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/23e3cc4d6004439bb95ec3ab2504b1a1
W20250113 02:51:18.262879 711 leader_election.cc:343] T c97a6ad98ddc4680a0f071cb91f05402 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:18.269098 1194 tablet_bootstrap.cc:492] T 7dbda00269cf4f7095b3504ae87ac243 P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:18.271067 1194 ts_tablet_manager.cc:1397] T 7dbda00269cf4f7095b3504ae87ac243 P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.157s user 0.037s sys 0.047s
I20250113 02:51:18.271771 1194 raft_consensus.cc:357] T 7dbda00269cf4f7095b3504ae87ac243 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:18.271961 1194 raft_consensus.cc:383] T 7dbda00269cf4f7095b3504ae87ac243 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:18.272053 1194 raft_consensus.cc:738] T 7dbda00269cf4f7095b3504ae87ac243 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:18.272295 1194 consensus_queue.cc:260] T 7dbda00269cf4f7095b3504ae87ac243 P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:18.272605 711 leader_election.cc:304] T c97a6ad98ddc4680a0f071cb91f05402 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:18.272771 1194 ts_tablet_manager.cc:1428] T 7dbda00269cf4f7095b3504ae87ac243 P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250113 02:51:18.273453 1209 raft_consensus.cc:2743] T c97a6ad98ddc4680a0f071cb91f05402 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:18.273816 1194 tablet_bootstrap.cc:492] T 1643b3e1607f42fc82886d0b09e2fe8c P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:18.275396 1194 tablet_bootstrap.cc:654] T 1643b3e1607f42fc82886d0b09e2fe8c P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
W20250113 02:51:18.285135 448 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6c8ca30720664f3a819c4bff2493934d consensus state has no leader
I20250113 02:51:18.286779 448 master_service.cc:946] ReplaceTablet: received request to replace tablet f1a02c37cb4f43f8a9de6484f4431958 from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:18.288782 1042 tablet_service.cc:1514] Processing DeleteTablet for tablet 89517af3545c47a5a3a247a411190a47 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:17 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:18.289116 1198 tablet_replica.cc:331] T 89517af3545c47a5a3a247a411190a47 P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:18.289362 1198 raft_consensus.cc:2238] T 89517af3545c47a5a3a247a411190a47 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:18.289587 1198 raft_consensus.cc:2267] T 89517af3545c47a5a3a247a411190a47 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:18.295051 453 catalog_manager.cc:5618] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095) does not have the latest schema for tablet 7f3cbef7d91d4576822d913071a88248 (table default.Table_b581b1a063a54551a8535adf01983fb8 [id=54e4e3810b3b412bbec3b5bf5b28c1a2]). Expected version 1 got 0
I20250113 02:51:18.299074 454 catalog_manager.cc:5618] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467) does not have the latest schema for tablet 7f3cbef7d91d4576822d913071a88248 (table default.Table_b581b1a063a54551a8535adf01983fb8 [id=54e4e3810b3b412bbec3b5bf5b28c1a2]). Expected version 1 got 0
I20250113 02:51:18.302652 1198 ts_tablet_manager.cc:1905] T 89517af3545c47a5a3a247a411190a47 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:18.308048 1195 ts_tablet_manager.cc:1939] T 23e3cc4d6004439bb95ec3ab2504b1a1 P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:18.309336 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 84) took 1045 ms. Trace:
I20250113 02:51:18.309315 1029 tablet_service.cc:1467] Processing CreateTablet for tablet d5fbf97536a2472fa091168363bf9549 (DEFAULT_TABLE table=default.Table_abea3b2ec5144f1fadd95a623c48917c [id=cdb71df855274f13a405028421724d37]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:18.309772 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d5fbf97536a2472fa091168363bf9549. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:18.309667 1041 tablet_service.cc:1467] Processing CreateTablet for tablet b2a35fd891c645d68d79c61a2c832ba7 (DEFAULT_TABLE table=default.Table_266561dfe36245ecb8230fdf1904cb02 [id=60925cf1a1224988b0d349b621ab5368]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:18.310123 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b2a35fd891c645d68d79c61a2c832ba7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:18.309497 1195 rpcz_store.cc:276] 0113 02:51:17.263564 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:17.264491 (+ 927us) service_pool.cc:225] Handling call
0113 02:51:17.800167 (+535676us) ts_tablet_manager.cc:1145] Deleting tablet 23e3cc4d6004439bb95ec3ab2504b1a1
0113 02:51:17.800178 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:18.129833 (+329655us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:18.247514 (+117681us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:18.309186 (+ 61672us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":535363}
I20250113 02:51:18.311795 1195 tablet_replica.cc:331] T 015447488b734f6a93b8c36be975c031 P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:18.311990 1195 raft_consensus.cc:2238] T 015447488b734f6a93b8c36be975c031 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:18.312346 1034 tablet_service.cc:1467] Processing CreateTablet for tablet bcb59573e82743858bd39bde64627d6d (DEFAULT_TABLE table=default.Table_9cd2b585fc704fb6a2d0da56a774b8bd [id=7b5976daebfb49eeb72f919006fa149b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:18.312789 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bcb59573e82743858bd39bde64627d6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:18.313450 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet 23e3cc4d6004439bb95ec3ab2504b1a1 (table default.Table_dfb2da2848a94fe4b5d89023934e8740 [id=b409b6c2b10e471aba03feb404311e3d]) successfully deleted
I20250113 02:51:18.314081 894 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f1c2ae654bd46c1a8772fa62c8111ab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:17 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:18.314752 893 tablet_service.cc:1467] Processing CreateTablet for tablet 64cda8796dff46ce95c37439395c77ec (DEFAULT_TABLE table=default.Table_7d21501217454370b9e8e931e13e5a8e [id=983d3b296c064375a1f2747ce556f01d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:18.315214 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64cda8796dff46ce95c37439395c77ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:18.315143 895 tablet_service.cc:1467] Processing CreateTablet for tablet b2a35fd891c645d68d79c61a2c832ba7 (DEFAULT_TABLE table=default.Table_266561dfe36245ecb8230fdf1904cb02 [id=60925cf1a1224988b0d349b621ab5368]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:18.315474 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b2a35fd891c645d68d79c61a2c832ba7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:18.315696 894 tablet_service.cc:1467] Processing CreateTablet for tablet d5fbf97536a2472fa091168363bf9549 (DEFAULT_TABLE table=default.Table_abea3b2ec5144f1fadd95a623c48917c [id=cdb71df855274f13a405028421724d37]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:18.316030 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d5fbf97536a2472fa091168363bf9549. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:18.316285 1040 tablet_service.cc:1467] Processing CreateTablet for tablet 7f08b72301dc431396cf8cc7790d56d4 (DEFAULT_TABLE table=default.Table_f11eaf2fe6674de8973c9363a4afca76 [id=bd14ce0e2b0a4a60a63022be2bc80fef]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:18.316745 1040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f08b72301dc431396cf8cc7790d56d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:18.320389 1042 tablet_service.cc:1467] Processing CreateTablet for tablet 64cda8796dff46ce95c37439395c77ec (DEFAULT_TABLE table=default.Table_7d21501217454370b9e8e931e13e5a8e [id=983d3b296c064375a1f2747ce556f01d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:18.320873 1042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64cda8796dff46ce95c37439395c77ec. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:18.321202 442 catalog_manager.cc:4839] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 6f1c2ae654bd46c1a8772fa62c8111ab because the tablet was not found. No further retry: Not found: Tablet not found: 6f1c2ae654bd46c1a8772fa62c8111ab
I20250113 02:51:18.323199 741 tablet_service.cc:1467] Processing CreateTablet for tablet d5fbf97536a2472fa091168363bf9549 (DEFAULT_TABLE table=default.Table_abea3b2ec5144f1fadd95a623c48917c [id=cdb71df855274f13a405028421724d37]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:18.323657 889 tablet_service.cc:1467] Processing CreateTablet for tablet bcb59573e82743858bd39bde64627d6d (DEFAULT_TABLE table=default.Table_9cd2b585fc704fb6a2d0da56a774b8bd [id=7b5976daebfb49eeb72f919006fa149b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:18.323890 736 tablet_service.cc:1467] Processing CreateTablet for tablet bcb59573e82743858bd39bde64627d6d (DEFAULT_TABLE table=default.Table_9cd2b585fc704fb6a2d0da56a774b8bd [id=7b5976daebfb49eeb72f919006fa149b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:18.324038 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bcb59573e82743858bd39bde64627d6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:18.324422 749 tablet_service.cc:1467] Processing CreateTablet for tablet 7f08b72301dc431396cf8cc7790d56d4 (DEFAULT_TABLE table=default.Table_f11eaf2fe6674de8973c9363a4afca76 [id=bd14ce0e2b0a4a60a63022be2bc80fef]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:18.325023 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d5fbf97536a2472fa091168363bf9549. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:18.328877 451 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 438d1d2300e044f39a15586f335935ba consensus state has no leader
I20250113 02:51:18.329922 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_bb624ca10b84446bbaffca5a2173e00c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:18.330636 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bb624ca10b84446bbaffca5a2173e00c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:18.332067 740 tablet_service.cc:1467] Processing CreateTablet for tablet 64cda8796dff46ce95c37439395c77ec (DEFAULT_TABLE table=default.Table_7d21501217454370b9e8e931e13e5a8e [id=983d3b296c064375a1f2747ce556f01d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:18.332538 1195 raft_consensus.cc:2267] T 015447488b734f6a93b8c36be975c031 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:18.333341 1195 ts_tablet_manager.cc:1905] T 015447488b734f6a93b8c36be975c031 P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:18.338631 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_630328ef70a24cc5b8f01094d0abae76"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:18.339313 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_630328ef70a24cc5b8f01094d0abae76 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:18.339303 896 tablet_service.cc:1467] Processing CreateTablet for tablet 7f08b72301dc431396cf8cc7790d56d4 (DEFAULT_TABLE table=default.Table_f11eaf2fe6674de8973c9363a4afca76 [id=bd14ce0e2b0a4a60a63022be2bc80fef]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:18.333967 736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bcb59573e82743858bd39bde64627d6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:18.340065 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f08b72301dc431396cf8cc7790d56d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:18.343245 899 tablet_service.cc:1467] Processing CreateTablet for tablet 5b1076a5a17b4fa495975285616401cf (DEFAULT_TABLE table=default.Table_c357cad5db934f938c46acd208cc6f15 [id=bc2aa39c74b04c7ca71a6d0acd780853]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:18.343627 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b1076a5a17b4fa495975285616401cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:18.347927 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f08b72301dc431396cf8cc7790d56d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:18.348578 454 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_5e955e7a2ca54f1f8ee6012902e3d98c" } new_table_name: "default.Table_936e73f0b76046d285567ac0778e36a9" modify_external_catalogs: true
I20250113 02:51:18.350977 451 catalog_manager.cc:5618] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467) does not have the latest schema for tablet 7f3cbef7d91d4576822d913071a88248 (table default.Table_b581b1a063a54551a8535adf01983fb8 [id=54e4e3810b3b412bbec3b5bf5b28c1a2]). Expected version 1 got 0
I20250113 02:51:18.351245 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64cda8796dff46ce95c37439395c77ec. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:18.356535 711 consensus_peers.cc:487] T 89517af3545c47a5a3a247a411190a47 P 1186fa869eb14812832fa6562531f196 -> Peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Couldn't send request to peer df69f48656b44bb890f76587ee69cc0f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 89517af3545c47a5a3a247a411190a47. This is attempt 1: this message will repeat every 5th retry.
I20250113 02:51:18.336215 754 tablet_service.cc:1467] Processing CreateTablet for tablet b2a35fd891c645d68d79c61a2c832ba7 (DEFAULT_TABLE table=default.Table_266561dfe36245ecb8230fdf1904cb02 [id=60925cf1a1224988b0d349b621ab5368]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:18.357878 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b2a35fd891c645d68d79c61a2c832ba7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:18.370473 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_abea3b2ec5144f1fadd95a623c48917c" } modify_external_catalogs: true
I20250113 02:51:18.370813 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_abea3b2ec5144f1fadd95a623c48917c" } modify_external_catalogs: true
I20250113 02:51:18.372822 895 tablet_service.cc:1514] Processing DeleteTablet for tablet 7dbda00269cf4f7095b3504ae87ac243 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:16 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:18.373136 1196 tablet_replica.cc:331] T 7dbda00269cf4f7095b3504ae87ac243 P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:18.373339 1196 raft_consensus.cc:2238] T 7dbda00269cf4f7095b3504ae87ac243 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:18.373528 1196 raft_consensus.cc:2267] T 7dbda00269cf4f7095b3504ae87ac243 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:18.374447 1196 ts_tablet_manager.cc:1905] T 7dbda00269cf4f7095b3504ae87ac243 P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:18.374708 896 tablet_service.cc:1467] Processing CreateTablet for tablet 572e4eed1e924381bada27f8389821a7 (DEFAULT_TABLE table=default.Table_20ca05c407b14ba0a92f76f4596ae2ef [id=75cb0745bbb84e1c883cb13b546ecb11]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:18.375576 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 572e4eed1e924381bada27f8389821a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:18.377137 1029 tablet_service.cc:1467] Processing CreateTablet for tablet 5b1076a5a17b4fa495975285616401cf (DEFAULT_TABLE table=default.Table_c357cad5db934f938c46acd208cc6f15 [id=bc2aa39c74b04c7ca71a6d0acd780853]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:18.377228 1030 tablet_service.cc:1467] Processing CreateTablet for tablet ee7582ee04494c0db1179359bcb311c8 (DEFAULT_TABLE table=default.Table_c8bea6af909a424c9101c8934d1aee7b [id=bd9d66f2d35446e9bc4ed5e589f23012]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:18.377543 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b1076a5a17b4fa495975285616401cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:18.377584 1028 tablet_service.cc:1467] Processing CreateTablet for tablet 572e4eed1e924381bada27f8389821a7 (DEFAULT_TABLE table=default.Table_20ca05c407b14ba0a92f76f4596ae2ef [id=75cb0745bbb84e1c883cb13b546ecb11]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:18.377784 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee7582ee04494c0db1179359bcb311c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:18.378091 1037 tablet_service.cc:1467] Processing CreateTablet for tablet 5b3f012b88c04cb68ed7a1848147b70b (DEFAULT_TABLE table=default.Table_ce0f77e091564c29bf54791dda458d78 [id=86a46eed92bc4bddad870e392232cb51]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:18.378125 1033 tablet_service.cc:1467] Processing CreateTablet for tablet 0fb75c551b4c424186ef2021db742e8e (DEFAULT_TABLE table=default.Table_2e62961b127b4bb3ac9b5d30f408c299 [id=e2dbf349032c4342818c5f8a891b7f99]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:18.378429 1037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b3f012b88c04cb68ed7a1848147b70b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:18.378516 895 tablet_service.cc:1467] Processing CreateTablet for tablet ee7582ee04494c0db1179359bcb311c8 (DEFAULT_TABLE table=default.Table_c8bea6af909a424c9101c8934d1aee7b [id=bd9d66f2d35446e9bc4ed5e589f23012]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:18.378844 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee7582ee04494c0db1179359bcb311c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:18.382536 1024 tablet_service.cc:1467] Processing CreateTablet for tablet 7d53de6379b84103ba905effb0835676 (DEFAULT_TABLE table=default.Table_6f2972dcacb44ccb9e4f3d828ee536ec [id=d1b0bb18a7b84473803cd1294b0829ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:18.382877 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 572e4eed1e924381bada27f8389821a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:18.384244 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d53de6379b84103ba905effb0835676. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:18.385212 744 tablet_service.cc:1467] Processing CreateTablet for tablet 0fb75c551b4c424186ef2021db742e8e (DEFAULT_TABLE table=default.Table_2e62961b127b4bb3ac9b5d30f408c299 [id=e2dbf349032c4342818c5f8a891b7f99]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:18.385582 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0fb75c551b4c424186ef2021db742e8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:18.388252 753 tablet_service.cc:1467] Processing CreateTablet for tablet 572e4eed1e924381bada27f8389821a7 (DEFAULT_TABLE table=default.Table_20ca05c407b14ba0a92f76f4596ae2ef [id=75cb0745bbb84e1c883cb13b546ecb11]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:18.388653 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 572e4eed1e924381bada27f8389821a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:18.392295 739 tablet_service.cc:1467] Processing CreateTablet for tablet ee7582ee04494c0db1179359bcb311c8 (DEFAULT_TABLE table=default.Table_c8bea6af909a424c9101c8934d1aee7b [id=bd9d66f2d35446e9bc4ed5e589f23012]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:18.392666 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee7582ee04494c0db1179359bcb311c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:18.388262 737 tablet_service.cc:1467] Processing CreateTablet for tablet 5b3f012b88c04cb68ed7a1848147b70b (DEFAULT_TABLE table=default.Table_ce0f77e091564c29bf54791dda458d78 [id=86a46eed92bc4bddad870e392232cb51]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:18.396737 737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b3f012b88c04cb68ed7a1848147b70b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:18.400223 741 tablet_service.cc:1467] Processing CreateTablet for tablet 5b1076a5a17b4fa495975285616401cf (DEFAULT_TABLE table=default.Table_c357cad5db934f938c46acd208cc6f15 [id=bc2aa39c74b04c7ca71a6d0acd780853]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:18.400681 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b1076a5a17b4fa495975285616401cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:18.403795 893 tablet_service.cc:1467] Processing CreateTablet for tablet 7d53de6379b84103ba905effb0835676 (DEFAULT_TABLE table=default.Table_6f2972dcacb44ccb9e4f3d828ee536ec [id=d1b0bb18a7b84473803cd1294b0829ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:18.404222 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d53de6379b84103ba905effb0835676. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:18.410907 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0fb75c551b4c424186ef2021db742e8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:18.411672 899 tablet_service.cc:1467] Processing CreateTablet for tablet 5b3f012b88c04cb68ed7a1848147b70b (DEFAULT_TABLE table=default.Table_ce0f77e091564c29bf54791dda458d78 [id=86a46eed92bc4bddad870e392232cb51]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:18.412045 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b3f012b88c04cb68ed7a1848147b70b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:18.412830 1198 ts_tablet_manager.cc:1918] T 89517af3545c47a5a3a247a411190a47 P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:18.413009 1198 log.cc:1198] T 89517af3545c47a5a3a247a411190a47 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/89517af3545c47a5a3a247a411190a47
I20250113 02:51:18.413476 1198 ts_tablet_manager.cc:1939] T 89517af3545c47a5a3a247a411190a47 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:18.414728 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet 89517af3545c47a5a3a247a411190a47 (table default.Table_5d3da53a6d984a7ea593b340ad3e1368 [id=e66cde9b8d9b4763b33e7641601c47da]) successfully deleted
I20250113 02:51:18.416260 896 tablet_service.cc:1467] Processing CreateTablet for tablet 0fb75c551b4c424186ef2021db742e8e (DEFAULT_TABLE table=default.Table_2e62961b127b4bb3ac9b5d30f408c299 [id=e2dbf349032c4342818c5f8a891b7f99]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:18.416605 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0fb75c551b4c424186ef2021db742e8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:18.421658 743 tablet_service.cc:1467] Processing CreateTablet for tablet 1a26e2b5e1054aa6afe7f44bf8d43108 (DEFAULT_TABLE table=default.Table_b5da750f5930453fa3951ffba96c34d4 [id=fa5c22ac7c024f0aba13091163742391]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:18.422039 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a26e2b5e1054aa6afe7f44bf8d43108. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:18.422458 1029 tablet_service.cc:1467] Processing CreateTablet for tablet 1a26e2b5e1054aa6afe7f44bf8d43108 (DEFAULT_TABLE table=default.Table_b5da750f5930453fa3951ffba96c34d4 [id=fa5c22ac7c024f0aba13091163742391]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:18.422844 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a26e2b5e1054aa6afe7f44bf8d43108. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:18.422789 893 tablet_service.cc:1467] Processing CreateTablet for tablet 1a26e2b5e1054aa6afe7f44bf8d43108 (DEFAULT_TABLE table=default.Table_b5da750f5930453fa3951ffba96c34d4 [id=fa5c22ac7c024f0aba13091163742391]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:18.423187 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a26e2b5e1054aa6afe7f44bf8d43108. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:18.427842 755 tablet_service.cc:1467] Processing CreateTablet for tablet 7d53de6379b84103ba905effb0835676 (DEFAULT_TABLE table=default.Table_6f2972dcacb44ccb9e4f3d828ee536ec [id=d1b0bb18a7b84473803cd1294b0829ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:18.428344 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d53de6379b84103ba905effb0835676. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:18.435936 1191 tablet_bootstrap.cc:492] T db266a1848094b7a98aa75f7afabd471 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:18.436182 1191 ts_tablet_manager.cc:1397] T db266a1848094b7a98aa75f7afabd471 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.255s user 0.008s sys 0.000s
I20250113 02:51:18.437098 1191 raft_consensus.cc:357] T db266a1848094b7a98aa75f7afabd471 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:18.437417 1191 raft_consensus.cc:383] T db266a1848094b7a98aa75f7afabd471 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:18.437515 1191 raft_consensus.cc:738] T db266a1848094b7a98aa75f7afabd471 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:18.437847 1191 consensus_queue.cc:260] T db266a1848094b7a98aa75f7afabd471 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:18.440008 1191 ts_tablet_manager.cc:1428] T db266a1848094b7a98aa75f7afabd471 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250113 02:51:18.441517 1191 tablet_bootstrap.cc:492] T 1643b3e1607f42fc82886d0b09e2fe8c P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
W20250113 02:51:18.443912 709 consensus_peers.cc:487] T 89517af3545c47a5a3a247a411190a47 P 1186fa869eb14812832fa6562531f196 -> Peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Couldn't send request to peer 4216a7f853d642b7bc3e5cff458f3aca. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 89517af3545c47a5a3a247a411190a47. This is attempt 1: this message will repeat every 5th retry.
I20250113 02:51:18.445907 1191 tablet_bootstrap.cc:654] T 1643b3e1607f42fc82886d0b09e2fe8c P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:18.452723 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 18e2d79d731142f89b9089869bc736e5
I20250113 02:51:18.455566 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_2caed919541e43c78a858513a2128f50"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:18.456269 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 32351d6de1574353a6e8073d333da477
I20250113 02:51:18.457118 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_c7093afcc0ae47c78e7c5cf85219481c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:18.457397 895 tablet_service.cc:1514] Processing DeleteTablet for tablet 18e2d79d731142f89b9089869bc736e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:18 UTC) from {username='slave'} at 127.0.0.1:37366
W20250113 02:51:18.457788 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c7093afcc0ae47c78e7c5cf85219481c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:18.458463 895 tablet_service.cc:1514] Processing DeleteTablet for tablet 32351d6de1574353a6e8073d333da477 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:18 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:18.459345 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 8aac142230e244c0b59c9c2b7a7300ea
I20250113 02:51:18.459789 449 catalog_manager.cc:6596] ReplaceTablet: tablet 8aac142230e244c0b59c9c2b7a7300ea deleted and replaced by tablet f31dd916336a4d8fb3ab015af7de8ccb
I20250113 02:51:18.460198 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_0917607e26ce49bfad7e1dc99a894a05"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:18.460785 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0917607e26ce49bfad7e1dc99a894a05 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:18.461323 743 tablet_service.cc:1514] Processing DeleteTablet for tablet 7dbda00269cf4f7095b3504ae87ac243 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:16 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:18.461812 740 tablet_service.cc:1514] Processing DeleteTablet for tablet 18e2d79d731142f89b9089869bc736e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:18 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:18.465205 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet 32351d6de1574353a6e8073d333da477 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:18 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:18.465369 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet 18e2d79d731142f89b9089869bc736e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:18 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:18.465672 1042 tablet_service.cc:1514] Processing DeleteTablet for tablet 8aac142230e244c0b59c9c2b7a7300ea with delete_type TABLET_DATA_DELETED (replaced by tablet f31dd916336a4d8fb3ab015af7de8ccb) from {username='slave'} at 127.0.0.1:60230
W20250113 02:51:18.466835 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 32351d6de1574353a6e8073d333da477 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 32351d6de1574353a6e8073d333da477 already in progress: creating tablet
I20250113 02:51:18.466856 743 tablet_service.cc:1514] Processing DeleteTablet for tablet 8aac142230e244c0b59c9c2b7a7300ea with delete_type TABLET_DATA_DELETED (replaced by tablet f31dd916336a4d8fb3ab015af7de8ccb) from {username='slave'} at 127.0.0.1:55398
W20250113 02:51:18.467146 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 18e2d79d731142f89b9089869bc736e5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 18e2d79d731142f89b9089869bc736e5 already in progress: creating tablet
W20250113 02:51:18.467447 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 8aac142230e244c0b59c9c2b7a7300ea because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8aac142230e244c0b59c9c2b7a7300ea already in progress: creating tablet
I20250113 02:51:18.467635 740 tablet_service.cc:1514] Processing DeleteTablet for tablet 32351d6de1574353a6e8073d333da477 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:18 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:18.468259 1200 server_base.cc:1034] running on GCE node
W20250113 02:51:18.468911 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2caed919541e43c78a858513a2128f50 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:18.469858 1200 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250113 02:51:18.472378 1200 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250113 02:51:18.476306 1200 hybrid_clock.cc:648] HybridClock initialized: now 1736736678476131 us; error 168 us; skew 500 ppm
I20250113 02:51:18.476893 1200 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250113 02:51:18.479082 895 tablet_service.cc:1514] Processing DeleteTablet for tablet 8aac142230e244c0b59c9c2b7a7300ea with delete_type TABLET_DATA_DELETED (replaced by tablet f31dd916336a4d8fb3ab015af7de8ccb) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:18.480254 1196 ts_tablet_manager.cc:1918] T 7dbda00269cf4f7095b3504ae87ac243 P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:18.480480 1196 log.cc:1198] T 7dbda00269cf4f7095b3504ae87ac243 P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/7dbda00269cf4f7095b3504ae87ac243
I20250113 02:51:18.481055 1196 ts_tablet_manager.cc:1939] T 7dbda00269cf4f7095b3504ae87ac243 P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:18.486450 1200 webserver.cc:458] Webserver started at http://127.0.104.61:37629/ using document root <none> and password file <none>
I20250113 02:51:18.488941 1200 fs_manager.cc:362] Metadata directory not provided
I20250113 02:51:18.489157 1200 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 02:51:18.493470 563 master_runner.cc:186] Error getting master registration for 127.0.104.61:35271: Network error: Client connection negotiation failed: client connection to 127.0.104.61:35271: connect: Connection refused (error 111),
I20250113 02:51:18.493705 563 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250113 02:51:18.500708 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet 7dbda00269cf4f7095b3504ae87ac243 (table default.Table_8e22a139d87f407da7668fde4cf6c67a [id=46fce62be9184eea82a2fdc88b9aff21]) successfully deleted
W20250113 02:51:18.501288 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 18e2d79d731142f89b9089869bc736e5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 18e2d79d731142f89b9089869bc736e5 already in progress: creating tablet
W20250113 02:51:18.501642 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 32351d6de1574353a6e8073d333da477 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 32351d6de1574353a6e8073d333da477 already in progress: creating tablet
W20250113 02:51:18.502035 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 8aac142230e244c0b59c9c2b7a7300ea because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8aac142230e244c0b59c9c2b7a7300ea already in progress: creating tablet
I20250113 02:51:18.507334 1200 fs_manager.cc:714] Time spent opening directory manager: real 0.013s user 0.002s sys 0.008s
I20250113 02:51:18.517273 1195 ts_tablet_manager.cc:1918] T 015447488b734f6a93b8c36be975c031 P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:18.517484 1195 log.cc:1198] T 015447488b734f6a93b8c36be975c031 P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/015447488b734f6a93b8c36be975c031
I20250113 02:51:18.517984 1195 ts_tablet_manager.cc:1939] T 015447488b734f6a93b8c36be975c031 P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:18.519169 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 85) took 1237 ms. Trace:
I20250113 02:51:18.519512 1279 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 02:51:18.519402 1195 rpcz_store.cc:276] 0113 02:51:17.281585 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:17.282410 (+ 825us) service_pool.cc:225] Handling call
0113 02:51:18.311674 (+1029264us) ts_tablet_manager.cc:1145] Deleting tablet 015447488b734f6a93b8c36be975c031
0113 02:51:18.311683 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:18.398499 (+ 86816us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:18.517065 (+118566us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:18.519115 (+ 2050us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1027780}
I20250113 02:51:18.520215 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet 015447488b734f6a93b8c36be975c031 (table default.Table_2106a7d1900a44df83cf7ec9f377687d [id=a2c08f472dd7494ab8aff87e9aa43028]) successfully deleted
I20250113 02:51:18.522802 1200 fs_manager.cc:730] Time spent opening block manager: real 0.013s user 0.003s sys 0.007s
I20250113 02:51:18.523233 1200 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/master-1/data,/tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/master-1/wal
uuid: "6bbc3541604a455584bb32dfc2cc4efd"
format_stamp: "Formatted at 2025-01-13 02:51:12 on dist-test-slave-v0t5"
I20250113 02:51:18.524981 1200 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 02:51:18.530284 1192 tablet_bootstrap.cc:492] T c1ab705ebcf1430e9e0c376b18841d27 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:18.530542 1192 ts_tablet_manager.cc:1397] T c1ab705ebcf1430e9e0c376b18841d27 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.292s user 0.079s sys 0.085s
I20250113 02:51:18.531679 1192 raft_consensus.cc:357] T c1ab705ebcf1430e9e0c376b18841d27 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:18.532055 1192 raft_consensus.cc:383] T c1ab705ebcf1430e9e0c376b18841d27 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:18.532162 1192 raft_consensus.cc:738] T c1ab705ebcf1430e9e0c376b18841d27 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:18.532516 1192 consensus_queue.cc:260] T c1ab705ebcf1430e9e0c376b18841d27 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:18.533154 1192 ts_tablet_manager.cc:1428] T c1ab705ebcf1430e9e0c376b18841d27 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20250113 02:51:18.533350 1191 tablet_bootstrap.cc:492] T 1643b3e1607f42fc82886d0b09e2fe8c P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:18.534390 1191 ts_tablet_manager.cc:1397] T 1643b3e1607f42fc82886d0b09e2fe8c P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.093s user 0.002s sys 0.002s
I20250113 02:51:18.535243 1192 tablet_bootstrap.cc:492] T 455b143af8c94f9d8a4111330648cac6 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:18.536883 1191 raft_consensus.cc:357] T 1643b3e1607f42fc82886d0b09e2fe8c P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:18.537268 1191 raft_consensus.cc:383] T 1643b3e1607f42fc82886d0b09e2fe8c P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:18.537432 1191 raft_consensus.cc:738] T 1643b3e1607f42fc82886d0b09e2fe8c P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:18.537521 1194 tablet_bootstrap.cc:492] T 1643b3e1607f42fc82886d0b09e2fe8c P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:18.537751 1194 ts_tablet_manager.cc:1397] T 1643b3e1607f42fc82886d0b09e2fe8c P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.264s user 0.072s sys 0.142s
I20250113 02:51:18.537883 1192 tablet_bootstrap.cc:654] T 455b143af8c94f9d8a4111330648cac6 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:18.537868 1191 consensus_queue.cc:260] T 1643b3e1607f42fc82886d0b09e2fe8c P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:18.538609 1191 ts_tablet_manager.cc:1428] T 1643b3e1607f42fc82886d0b09e2fe8c P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250113 02:51:18.538691 1194 raft_consensus.cc:357] T 1643b3e1607f42fc82886d0b09e2fe8c P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:18.538975 1194 raft_consensus.cc:383] T 1643b3e1607f42fc82886d0b09e2fe8c P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:18.539111 1194 raft_consensus.cc:738] T 1643b3e1607f42fc82886d0b09e2fe8c P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:18.539441 1194 consensus_queue.cc:260] T 1643b3e1607f42fc82886d0b09e2fe8c P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:18.540107 1209 raft_consensus.cc:491] T 9a323451cdfe4fd69ec62cda230f9afb P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:18.540220 1194 ts_tablet_manager.cc:1428] T 1643b3e1607f42fc82886d0b09e2fe8c P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20250113 02:51:18.540280 1209 raft_consensus.cc:513] T 9a323451cdfe4fd69ec62cda230f9afb P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:18.541175 1209 leader_election.cc:290] T 9a323451cdfe4fd69ec62cda230f9afb P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:18.542752 1062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a323451cdfe4fd69ec62cda230f9afb" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
W20250113 02:51:18.543941 709 leader_election.cc:343] T 9a323451cdfe4fd69ec62cda230f9afb P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
W20250113 02:51:18.545159 711 leader_election.cc:343] T 9a323451cdfe4fd69ec62cda230f9afb P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Not found: Tablet not found: 9a323451cdfe4fd69ec62cda230f9afb
I20250113 02:51:18.545310 711 leader_election.cc:304] T 9a323451cdfe4fd69ec62cda230f9afb P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:18.545985 1194 tablet_bootstrap.cc:492] T d6675bc3cb5f4f809320e28026cdf4b0 P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:18.547631 1209 raft_consensus.cc:2743] T 9a323451cdfe4fd69ec62cda230f9afb P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:18.549625 1194 tablet_bootstrap.cc:654] T d6675bc3cb5f4f809320e28026cdf4b0 P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:18.556007 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet db266a1848094b7a98aa75f7afabd471
I20250113 02:51:18.556059 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 9b5dc06ba3ea4056822ca3e4b1249587
I20250113 02:51:18.557417 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet f1a02c37cb4f43f8a9de6484f4431958
I20250113 02:51:18.557905 448 catalog_manager.cc:6596] ReplaceTablet: tablet f1a02c37cb4f43f8a9de6484f4431958 deleted and replaced by tablet 2c399784af46426b8327d175c7a6cce8
I20250113 02:51:18.558282 899 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b5dc06ba3ea4056822ca3e4b1249587 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:18 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:18.559751 739 tablet_service.cc:1514] Processing DeleteTablet for tablet db266a1848094b7a98aa75f7afabd471 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:18 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:18.559782 893 tablet_service.cc:1514] Processing DeleteTablet for tablet f1a02c37cb4f43f8a9de6484f4431958 with delete_type TABLET_DATA_DELETED (replaced by tablet 2c399784af46426b8327d175c7a6cce8) from {username='slave'} at 127.0.0.1:37366
W20250113 02:51:18.560024 454 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ad9b95c5450a4c9c89a0930862348d49 consensus state has no leader
I20250113 02:51:18.560114 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet db266a1848094b7a98aa75f7afabd471 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:18 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:18.560437 749 tablet_service.cc:1514] Processing DeleteTablet for tablet f1a02c37cb4f43f8a9de6484f4431958 with delete_type TABLET_DATA_DELETED (replaced by tablet 2c399784af46426b8327d175c7a6cce8) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:18.560544 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_732e990e6f154d6486889304bbf88cef"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:18.560832 1042 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b5dc06ba3ea4056822ca3e4b1249587 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:18 UTC) from {username='slave'} at 127.0.0.1:60230
W20250113 02:51:18.560853 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 9b5dc06ba3ea4056822ca3e4b1249587 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9b5dc06ba3ea4056822ca3e4b1249587 already in progress: creating tablet
W20250113 02:51:18.561218 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_732e990e6f154d6486889304bbf88cef in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250113 02:51:18.561328 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet db266a1848094b7a98aa75f7afabd471 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet db266a1848094b7a98aa75f7afabd471 already in progress: creating tablet
I20250113 02:51:18.561427 896 tablet_service.cc:1514] Processing DeleteTablet for tablet db266a1848094b7a98aa75f7afabd471 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:18 UTC) from {username='slave'} at 127.0.0.1:37366
W20250113 02:51:18.561610 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 9b5dc06ba3ea4056822ca3e4b1249587 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9b5dc06ba3ea4056822ca3e4b1249587 already in progress: creating tablet
I20250113 02:51:18.561893 1042 tablet_service.cc:1514] Processing DeleteTablet for tablet f1a02c37cb4f43f8a9de6484f4431958 with delete_type TABLET_DATA_DELETED (replaced by tablet 2c399784af46426b8327d175c7a6cce8) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:18.562260 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_c45864b4578e45b59a53d2d913da12f9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:18.561328 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet f1a02c37cb4f43f8a9de6484f4431958 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f1a02c37cb4f43f8a9de6484f4431958 already in progress: creating tablet
W20250113 02:51:18.562774 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c45864b4578e45b59a53d2d913da12f9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:18.563074 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet d5fbf97536a2472fa091168363bf9549
W20250113 02:51:18.563273 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet db266a1848094b7a98aa75f7afabd471 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet db266a1848094b7a98aa75f7afabd471 already in progress: creating tablet
I20250113 02:51:18.563782 896 tablet_service.cc:1514] Processing DeleteTablet for tablet d5fbf97536a2472fa091168363bf9549 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:18 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:18.563833 453 master_service.cc:946] ReplaceTablet: received request to replace tablet bc88676bf738442d82c25e56f30c4ada from {username='slave'} at 127.0.0.1:49258
W20250113 02:51:18.564570 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet f1a02c37cb4f43f8a9de6484f4431958 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f1a02c37cb4f43f8a9de6484f4431958 already in progress: creating tablet
I20250113 02:51:18.564693 448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_a79eb97979f9403b97fa08ffd00c6513" } new_table_name: "default.Table_d05b7517b14c4168bea99385f1a7d8b6" modify_external_catalogs: true
I20250113 02:51:18.565081 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet d5fbf97536a2472fa091168363bf9549 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:18 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:18.565306 1042 tablet_service.cc:1514] Processing DeleteTablet for tablet c1ab705ebcf1430e9e0c376b18841d27 with delete_type TABLET_DATA_DELETED (replaced by tablet ce92c2b45a274eb09e72a96b7ec133be) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:18.565690 1198 tablet_replica.cc:331] T c1ab705ebcf1430e9e0c376b18841d27 P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:18.565855 1198 raft_consensus.cc:2238] T c1ab705ebcf1430e9e0c376b18841d27 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:18.562808 455 master_service.cc:946] ReplaceTablet: received request to replace tablet 9b5dc06ba3ea4056822ca3e4b1249587 from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:18.566066 1198 raft_consensus.cc:2267] T c1ab705ebcf1430e9e0c376b18841d27 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:18.566527 1209 raft_consensus.cc:491] T db266a1848094b7a98aa75f7afabd471 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:18.566572 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_7d21501217454370b9e8e931e13e5a8e" } new_table_name: "default.Table_e45d864b62364107b47a45916574fed5" modify_external_catalogs: true
W20250113 02:51:18.566718 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet d5fbf97536a2472fa091168363bf9549 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d5fbf97536a2472fa091168363bf9549 already in progress: creating tablet
I20250113 02:51:18.566720 749 tablet_service.cc:1514] Processing DeleteTablet for tablet d5fbf97536a2472fa091168363bf9549 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:18 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:18.566721 1209 raft_consensus.cc:513] T db266a1848094b7a98aa75f7afabd471 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:18.567437 1198 ts_tablet_manager.cc:1905] T c1ab705ebcf1430e9e0c376b18841d27 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:18.567504 1209 leader_election.cc:290] T db266a1848094b7a98aa75f7afabd471 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:18.567785 754 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b5dc06ba3ea4056822ca3e4b1249587 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:18 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:18.563908 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_e44b0cdc7f4640ec867d92338d94426b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:18.568761 1062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db266a1848094b7a98aa75f7afabd471" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
W20250113 02:51:18.569679 709 leader_election.cc:343] T db266a1848094b7a98aa75f7afabd471 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
W20250113 02:51:18.570670 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e44b0cdc7f4640ec867d92338d94426b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:18.572196 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db266a1848094b7a98aa75f7afabd471" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:18.572719 1192 tablet_bootstrap.cc:492] T 455b143af8c94f9d8a4111330648cac6 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:18.572873 1192 ts_tablet_manager.cc:1397] T 455b143af8c94f9d8a4111330648cac6 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.038s user 0.011s sys 0.007s
W20250113 02:51:18.572924 711 leader_election.cc:343] T db266a1848094b7a98aa75f7afabd471 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:18.573091 711 leader_election.cc:304] T db266a1848094b7a98aa75f7afabd471 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:18.573472 1209 raft_consensus.cc:2743] T db266a1848094b7a98aa75f7afabd471 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:18.574952 1194 tablet_bootstrap.cc:492] T d6675bc3cb5f4f809320e28026cdf4b0 P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:18.574923 1192 raft_consensus.cc:357] T 455b143af8c94f9d8a4111330648cac6 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:18.575165 1192 raft_consensus.cc:383] T 455b143af8c94f9d8a4111330648cac6 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:18.575273 1194 ts_tablet_manager.cc:1397] T d6675bc3cb5f4f809320e28026cdf4b0 P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.029s user 0.002s sys 0.004s
I20250113 02:51:18.575285 1192 raft_consensus.cc:738] T 455b143af8c94f9d8a4111330648cac6 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:18.575574 1192 consensus_queue.cc:260] T 455b143af8c94f9d8a4111330648cac6 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:18.576187 1192 ts_tablet_manager.cc:1428] T 455b143af8c94f9d8a4111330648cac6 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20250113 02:51:18.577039 1192 tablet_bootstrap.cc:492] T 6f1c2ae654bd46c1a8772fa62c8111ab P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:18.578189 1194 raft_consensus.cc:357] T d6675bc3cb5f4f809320e28026cdf4b0 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:18.578559 1194 raft_consensus.cc:383] T d6675bc3cb5f4f809320e28026cdf4b0 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:18.578682 1194 raft_consensus.cc:738] T d6675bc3cb5f4f809320e28026cdf4b0 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:18.579121 1194 consensus_queue.cc:260] T d6675bc3cb5f4f809320e28026cdf4b0 P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:18.579681 1194 ts_tablet_manager.cc:1428] T d6675bc3cb5f4f809320e28026cdf4b0 P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250113 02:51:18.580070 1194 tablet_bootstrap.cc:492] T a028cc02273644608f046135851da126 P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:18.582093 1194 tablet_bootstrap.cc:654] T a028cc02273644608f046135851da126 P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:18.582118 1192 tablet_bootstrap.cc:654] T 6f1c2ae654bd46c1a8772fa62c8111ab P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
W20250113 02:51:18.586604 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet d5fbf97536a2472fa091168363bf9549 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d5fbf97536a2472fa091168363bf9549 already in progress: creating tablet
I20250113 02:51:18.587082 896 tablet_service.cc:1514] Processing DeleteTablet for tablet 1643b3e1607f42fc82886d0b09e2fe8c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:16 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:18.587460 1196 tablet_replica.cc:331] T 1643b3e1607f42fc82886d0b09e2fe8c P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:18.587674 1196 raft_consensus.cc:2238] T 1643b3e1607f42fc82886d0b09e2fe8c P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:18.587870 1196 raft_consensus.cc:2267] T 1643b3e1607f42fc82886d0b09e2fe8c P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:18.588800 1196 ts_tablet_manager.cc:1905] T 1643b3e1607f42fc82886d0b09e2fe8c P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:18.595053 1192 tablet_bootstrap.cc:492] T 6f1c2ae654bd46c1a8772fa62c8111ab P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:18.595233 1192 ts_tablet_manager.cc:1397] T 6f1c2ae654bd46c1a8772fa62c8111ab P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.018s user 0.000s sys 0.003s
I20250113 02:51:18.596056 1192 raft_consensus.cc:357] T 6f1c2ae654bd46c1a8772fa62c8111ab P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:18.596311 1192 raft_consensus.cc:383] T 6f1c2ae654bd46c1a8772fa62c8111ab P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:18.596387 1192 raft_consensus.cc:738] T 6f1c2ae654bd46c1a8772fa62c8111ab P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:18.596647 1192 consensus_queue.cc:260] T 6f1c2ae654bd46c1a8772fa62c8111ab P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:18.597198 1192 ts_tablet_manager.cc:1428] T 6f1c2ae654bd46c1a8772fa62c8111ab P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250113 02:51:18.597528 1192 tablet_bootstrap.cc:492] T 5ae18cbff90d4fa88f6b9b8746fc2027 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:18.598124 1198 ts_tablet_manager.cc:1918] T c1ab705ebcf1430e9e0c376b18841d27 P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:18.598974 1198 log.cc:1198] T c1ab705ebcf1430e9e0c376b18841d27 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/c1ab705ebcf1430e9e0c376b18841d27
I20250113 02:51:18.599467 1198 ts_tablet_manager.cc:1939] T c1ab705ebcf1430e9e0c376b18841d27 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:18.599722 1192 tablet_bootstrap.cc:654] T 5ae18cbff90d4fa88f6b9b8746fc2027 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:18.600293 1194 tablet_bootstrap.cc:492] T a028cc02273644608f046135851da126 P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:18.601291 1194 ts_tablet_manager.cc:1397] T a028cc02273644608f046135851da126 P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.021s user 0.007s sys 0.003s
I20250113 02:51:18.602521 1196 ts_tablet_manager.cc:1918] T 1643b3e1607f42fc82886d0b09e2fe8c P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:18.602694 1196 log.cc:1198] T 1643b3e1607f42fc82886d0b09e2fe8c P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/1643b3e1607f42fc82886d0b09e2fe8c
I20250113 02:51:18.604324 1194 raft_consensus.cc:357] T a028cc02273644608f046135851da126 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:18.606398 1192 tablet_bootstrap.cc:492] T 5ae18cbff90d4fa88f6b9b8746fc2027 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:18.606606 1192 ts_tablet_manager.cc:1397] T 5ae18cbff90d4fa88f6b9b8746fc2027 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.003s
I20250113 02:51:18.607559 1192 raft_consensus.cc:357] T 5ae18cbff90d4fa88f6b9b8746fc2027 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:18.607865 1192 raft_consensus.cc:383] T 5ae18cbff90d4fa88f6b9b8746fc2027 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:18.608027 1192 raft_consensus.cc:738] T 5ae18cbff90d4fa88f6b9b8746fc2027 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:18.519937 1195 tablet_replica.cc:331] T b262c70712444c06a61918f21736f339 P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:18.608330 1192 consensus_queue.cc:260] T 5ae18cbff90d4fa88f6b9b8746fc2027 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:18.609238 1192 ts_tablet_manager.cc:1428] T 5ae18cbff90d4fa88f6b9b8746fc2027 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250113 02:51:18.609756 1192 tablet_bootstrap.cc:492] T bc88676bf738442d82c25e56f30c4ada P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:18.611711 1192 tablet_bootstrap.cc:654] T bc88676bf738442d82c25e56f30c4ada P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:18.605325 1196 ts_tablet_manager.cc:1939] T 1643b3e1607f42fc82886d0b09e2fe8c P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:18.612769 1194 raft_consensus.cc:383] T a028cc02273644608f046135851da126 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:18.612900 1194 raft_consensus.cc:738] T a028cc02273644608f046135851da126 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:18.613178 1194 consensus_queue.cc:260] T a028cc02273644608f046135851da126 P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:18.613317 1195 raft_consensus.cc:2238] T b262c70712444c06a61918f21736f339 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:18.613533 1195 raft_consensus.cc:2267] T b262c70712444c06a61918f21736f339 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:18.613787 1194 ts_tablet_manager.cc:1428] T a028cc02273644608f046135851da126 P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20250113 02:51:18.614095 1194 tablet_bootstrap.cc:492] T db266a1848094b7a98aa75f7afabd471 P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:18.614387 1191 tablet_bootstrap.cc:492] T d6675bc3cb5f4f809320e28026cdf4b0 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:18.615128 1195 ts_tablet_manager.cc:1905] T b262c70712444c06a61918f21736f339 P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:18.615751 1194 tablet_bootstrap.cc:654] T db266a1848094b7a98aa75f7afabd471 P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:18.620553 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet c1ab705ebcf1430e9e0c376b18841d27 (table default.Table_c5c12ef64b7b438eb52313e87793e601 [id=7c2bc6d4232f4585b84d71e3e1f85794]) successfully deleted
I20250113 02:51:18.620576 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet 1643b3e1607f42fc82886d0b09e2fe8c (table default.Table_fff2447caa364c3381a3ba0abbde1c50 [id=15d9be3deeb84c1494965848dfb804ac]) successfully deleted
W20250113 02:51:18.620854 440 consensus_peers.cc:487] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9 -> Peer 6bbc3541604a455584bb32dfc2cc4efd (127.0.104.61:35271): Couldn't send request to peer 6bbc3541604a455584bb32dfc2cc4efd. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.61:35271: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250113 02:51:18.622526 1191 tablet_bootstrap.cc:654] T d6675bc3cb5f4f809320e28026cdf4b0 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:18.625125 1192 tablet_bootstrap.cc:492] T bc88676bf738442d82c25e56f30c4ada P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:18.626077 1194 tablet_bootstrap.cc:492] T db266a1848094b7a98aa75f7afabd471 P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:18.626245 1194 ts_tablet_manager.cc:1397] T db266a1848094b7a98aa75f7afabd471 P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.003s
I20250113 02:51:18.626993 1194 raft_consensus.cc:357] T db266a1848094b7a98aa75f7afabd471 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:18.627239 1194 raft_consensus.cc:383] T db266a1848094b7a98aa75f7afabd471 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:18.627316 1194 raft_consensus.cc:738] T db266a1848094b7a98aa75f7afabd471 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:18.627570 1194 consensus_queue.cc:260] T db266a1848094b7a98aa75f7afabd471 P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:18.628175 1194 ts_tablet_manager.cc:1428] T db266a1848094b7a98aa75f7afabd471 P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250113 02:51:18.628293 1192 ts_tablet_manager.cc:1397] T bc88676bf738442d82c25e56f30c4ada P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.000s
I20250113 02:51:18.628505 1194 tablet_bootstrap.cc:492] T 78689f7ed2a64405adb75f372465a792 P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:18.629223 1192 raft_consensus.cc:357] T bc88676bf738442d82c25e56f30c4ada P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:18.629518 1192 raft_consensus.cc:383] T bc88676bf738442d82c25e56f30c4ada P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:18.629626 1192 raft_consensus.cc:738] T bc88676bf738442d82c25e56f30c4ada P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:18.629889 1192 consensus_queue.cc:260] T bc88676bf738442d82c25e56f30c4ada P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:18.630790 1192 ts_tablet_manager.cc:1428] T bc88676bf738442d82c25e56f30c4ada P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250113 02:51:18.631110 1194 tablet_bootstrap.cc:654] T 78689f7ed2a64405adb75f372465a792 P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:18.631574 1192 tablet_bootstrap.cc:492] T 7dbda00269cf4f7095b3504ae87ac243 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:18.633114 1192 tablet_bootstrap.cc:654] T 7dbda00269cf4f7095b3504ae87ac243 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:18.636477 1203 raft_consensus.cc:491] T db266a1848094b7a98aa75f7afabd471 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:18.636647 1203 raft_consensus.cc:513] T db266a1848094b7a98aa75f7afabd471 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:18.636914 1192 tablet_bootstrap.cc:492] T 7dbda00269cf4f7095b3504ae87ac243 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:18.637035 1192 ts_tablet_manager.cc:1397] T 7dbda00269cf4f7095b3504ae87ac243 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20250113 02:51:18.637298 1203 leader_election.cc:290] T db266a1848094b7a98aa75f7afabd471 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:18.637802 1192 raft_consensus.cc:357] T 7dbda00269cf4f7095b3504ae87ac243 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:18.638000 1192 raft_consensus.cc:383] T 7dbda00269cf4f7095b3504ae87ac243 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:18.638084 1192 raft_consensus.cc:738] T 7dbda00269cf4f7095b3504ae87ac243 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:18.638120 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db266a1848094b7a98aa75f7afabd471" candidate_uuid: "df69f48656b44bb890f76587ee69cc0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
I20250113 02:51:18.638440 775 raft_consensus.cc:2463] T db266a1848094b7a98aa75f7afabd471 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df69f48656b44bb890f76587ee69cc0f in term 0.
I20250113 02:51:18.638325 1192 consensus_queue.cc:260] T 7dbda00269cf4f7095b3504ae87ac243 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:18.639019 1192 ts_tablet_manager.cc:1428] T 7dbda00269cf4f7095b3504ae87ac243 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250113 02:51:18.639477 1192 tablet_bootstrap.cc:492] T 7f3cbef7d91d4576822d913071a88248 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:18.641525 1192 tablet_bootstrap.cc:654] T 7f3cbef7d91d4576822d913071a88248 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:18.646009 896 tablet_service.cc:1514] Processing DeleteTablet for tablet d6675bc3cb5f4f809320e28026cdf4b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:17 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:18.646327 1196 tablet_replica.cc:331] T d6675bc3cb5f4f809320e28026cdf4b0 P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:18.646561 1196 raft_consensus.cc:2238] T d6675bc3cb5f4f809320e28026cdf4b0 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:18.646785 1196 raft_consensus.cc:2267] T d6675bc3cb5f4f809320e28026cdf4b0 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:18.647589 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_6c21e014853e4ab1996e23f0822d2b85"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:18.647818 755 tablet_service.cc:1514] Processing DeleteTablet for tablet 1643b3e1607f42fc82886d0b09e2fe8c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:16 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:18.649816 1062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db266a1848094b7a98aa75f7afabd471" candidate_uuid: "df69f48656b44bb890f76587ee69cc0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:18.650600 1191 tablet_bootstrap.cc:492] T d6675bc3cb5f4f809320e28026cdf4b0 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:18.650816 1191 ts_tablet_manager.cc:1397] T d6675bc3cb5f4f809320e28026cdf4b0 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.036s user 0.002s sys 0.002s
W20250113 02:51:18.651060 852 leader_election.cc:343] T db266a1848094b7a98aa75f7afabd471 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:18.651413 851 leader_election.cc:304] T db266a1848094b7a98aa75f7afabd471 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1186fa869eb14812832fa6562531f196, df69f48656b44bb890f76587ee69cc0f; no voters: 4216a7f853d642b7bc3e5cff458f3aca
I20250113 02:51:18.651825 1203 raft_consensus.cc:2798] T db266a1848094b7a98aa75f7afabd471 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 02:51:18.651935 1203 raft_consensus.cc:491] T db266a1848094b7a98aa75f7afabd471 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 02:51:18.652070 1203 raft_consensus.cc:3054] T db266a1848094b7a98aa75f7afabd471 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:18.648494 450 master_service.cc:946] ReplaceTablet: received request to replace tablet d101e98b72544c30a3a4bc60d1576aa3 from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:18.655011 1203 raft_consensus.cc:513] T db266a1848094b7a98aa75f7afabd471 P df69f48656b44bb890f76587ee69cc0f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:18.656172 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_6d493198af4046609058d1f1a51ab823" } new_table_name: "default.Table_4cc2e51b05714a449a4476d8d4d628f7" modify_external_catalogs: true
I20250113 02:51:18.656536 1062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db266a1848094b7a98aa75f7afabd471" candidate_uuid: "df69f48656b44bb890f76587ee69cc0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca"
I20250113 02:51:18.656895 1203 leader_election.cc:290] T db266a1848094b7a98aa75f7afabd471 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 election: Requested vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
W20250113 02:51:18.657248 852 leader_election.cc:343] T db266a1848094b7a98aa75f7afabd471 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:18.657785 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db266a1848094b7a98aa75f7afabd471" candidate_uuid: "df69f48656b44bb890f76587ee69cc0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196"
I20250113 02:51:18.657982 775 raft_consensus.cc:3054] T db266a1848094b7a98aa75f7afabd471 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:18.658376 1042 tablet_service.cc:1514] Processing DeleteTablet for tablet c1ab705ebcf1430e9e0c376b18841d27 with delete_type TABLET_DATA_DELETED (replaced by tablet ce92c2b45a274eb09e72a96b7ec133be) from {username='slave'} at 127.0.0.1:60230
W20250113 02:51:18.659090 440 catalog_manager.cc:4839] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet c1ab705ebcf1430e9e0c376b18841d27 because the tablet was not found. No further retry: Not found: Tablet not found: c1ab705ebcf1430e9e0c376b18841d27
I20250113 02:51:18.663053 1196 ts_tablet_manager.cc:1905] T d6675bc3cb5f4f809320e28026cdf4b0 P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250113 02:51:18.663666 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6c21e014853e4ab1996e23f0822d2b85 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:18.663782 775 raft_consensus.cc:2463] T db266a1848094b7a98aa75f7afabd471 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df69f48656b44bb890f76587ee69cc0f in term 1.
I20250113 02:51:18.664251 851 leader_election.cc:304] T db266a1848094b7a98aa75f7afabd471 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1186fa869eb14812832fa6562531f196, df69f48656b44bb890f76587ee69cc0f; no voters: 4216a7f853d642b7bc3e5cff458f3aca
I20250113 02:51:18.665138 1203 raft_consensus.cc:2798] T db266a1848094b7a98aa75f7afabd471 P df69f48656b44bb890f76587ee69cc0f [term 1 FOLLOWER]: Leader election won for term 1
I20250113 02:51:18.665539 1191 raft_consensus.cc:357] T d6675bc3cb5f4f809320e28026cdf4b0 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:18.665803 1191 raft_consensus.cc:383] T d6675bc3cb5f4f809320e28026cdf4b0 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:18.665910 1191 raft_consensus.cc:738] T d6675bc3cb5f4f809320e28026cdf4b0 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:18.666196 1191 consensus_queue.cc:260] T d6675bc3cb5f4f809320e28026cdf4b0 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:18.666771 1191 ts_tablet_manager.cc:1428] T d6675bc3cb5f4f809320e28026cdf4b0 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.016s user 0.001s sys 0.001s
I20250113 02:51:18.667565 1191 tablet_bootstrap.cc:492] T ec0750fd2fa247feb3ea310ffabe05e5 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:18.670516 1191 tablet_bootstrap.cc:654] T ec0750fd2fa247feb3ea310ffabe05e5 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:18.672806 1194 tablet_bootstrap.cc:492] T 78689f7ed2a64405adb75f372465a792 P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:18.673034 1194 ts_tablet_manager.cc:1397] T 78689f7ed2a64405adb75f372465a792 P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.045s user 0.001s sys 0.008s
I20250113 02:51:18.673875 1194 raft_consensus.cc:357] T 78689f7ed2a64405adb75f372465a792 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:18.674142 1194 raft_consensus.cc:383] T 78689f7ed2a64405adb75f372465a792 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:18.674299 1194 raft_consensus.cc:738] T 78689f7ed2a64405adb75f372465a792 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:18.674604 1194 consensus_queue.cc:260] T 78689f7ed2a64405adb75f372465a792 P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:18.684290 1196 ts_tablet_manager.cc:1918] T d6675bc3cb5f4f809320e28026cdf4b0 P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:18.684540 1196 log.cc:1198] T d6675bc3cb5f4f809320e28026cdf4b0 P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/d6675bc3cb5f4f809320e28026cdf4b0
I20250113 02:51:18.685283 1196 ts_tablet_manager.cc:1939] T d6675bc3cb5f4f809320e28026cdf4b0 P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:18.685411 1195 ts_tablet_manager.cc:1918] T b262c70712444c06a61918f21736f339 P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:18.685559 1195 log.cc:1198] T b262c70712444c06a61918f21736f339 P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/b262c70712444c06a61918f21736f339
I20250113 02:51:18.685926 1195 ts_tablet_manager.cc:1939] T b262c70712444c06a61918f21736f339 P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:18.686256 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet d6675bc3cb5f4f809320e28026cdf4b0 (table default.Table_d12d804163994a63a236be44401aa53b [id=f806136f234042af82e9831467a69caa]) successfully deleted
I20250113 02:51:18.686636 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 86) took 1391 ms. Trace:
I20250113 02:51:18.686733 1195 rpcz_store.cc:276] 0113 02:51:17.295220 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:17.295414 (+ 194us) service_pool.cc:225] Handling call
0113 02:51:18.519836 (+1224422us) ts_tablet_manager.cc:1145] Deleting tablet b262c70712444c06a61918f21736f339
0113 02:51:18.519851 (+ 15us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:18.627282 (+107431us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:18.685307 (+ 58025us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:18.686587 (+ 1280us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1222991}
I20250113 02:51:18.687155 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet b262c70712444c06a61918f21736f339 (table default.Table_5e0263e22f1e4d2b983ae6b6ef03fca5 [id=377f041bf95e41f3b7e8ea26f9795b6a]) successfully deleted
I20250113 02:51:18.692162 1209 raft_consensus.cc:491] T 7f3cbef7d91d4576822d913071a88248 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:18.692374 1209 raft_consensus.cc:513] T 7f3cbef7d91d4576822d913071a88248 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:18.693219 1209 leader_election.cc:290] T 7f3cbef7d91d4576822d913071a88248 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:18.693719 1062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f3cbef7d91d4576822d913071a88248" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:18.694649 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f3cbef7d91d4576822d913071a88248" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:18.694929 918 raft_consensus.cc:2463] T 7f3cbef7d91d4576822d913071a88248 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1186fa869eb14812832fa6562531f196 in term 0.
I20250113 02:51:18.695690 711 leader_election.cc:304] T 7f3cbef7d91d4576822d913071a88248 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1186fa869eb14812832fa6562531f196, df69f48656b44bb890f76587ee69cc0f; no voters:
I20250113 02:51:18.696117 1209 raft_consensus.cc:2798] T 7f3cbef7d91d4576822d913071a88248 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 02:51:18.696308 1209 raft_consensus.cc:491] T 7f3cbef7d91d4576822d913071a88248 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 02:51:18.696427 1209 raft_consensus.cc:3054] T 7f3cbef7d91d4576822d913071a88248 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:18.700822 454 catalog_manager.cc:5618] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587) does not have the latest schema for tablet bc88676bf738442d82c25e56f30c4ada (table default.Table_c991f9dbd70d4c4682664b8711f05db4 [id=2c64da64ecdb4bf48e11b523533902e8]). Expected version 1 got 0
I20250113 02:51:18.701771 741 tablet_service.cc:1467] Processing CreateTablet for tablet 501d4e1ae84e4795913035f8a19cc32d (DEFAULT_TABLE table=default.Table_8a5b45e248e94efb9b16df971fb9755c [id=bd54b5a058a44d718071be41b1e29a8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:18.701925 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_af3ebf543c9448b7ae473d721a23d47e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:18.702270 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 501d4e1ae84e4795913035f8a19cc32d. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:18.702459 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_af3ebf543c9448b7ae473d721a23d47e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:18.702402 454 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_3b57de11e75e42fbb2cbe60078f76c8d" } new_table_name: "default.Table_154343289ac5428da04f6f330235d2a0" modify_external_catalogs: true
I20250113 02:51:18.703027 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_ce0f77e091564c29bf54791dda458d78" } modify_external_catalogs: true
I20250113 02:51:18.703189 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_ce0f77e091564c29bf54791dda458d78" } modify_external_catalogs: true
I20250113 02:51:18.704303 755 tablet_service.cc:1467] Processing CreateTablet for tablet 8adfa738c17a4169a5d99f6ceca75100 (DEFAULT_TABLE table=default.Table_6167b33d8e4246cab5fbe98cb57e9a3c [id=ae544735623e4cfd894dc9340aca177c]), partition=RANGE (key) PARTITION UNBOUNDED
W20250113 02:51:18.704524 448 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1500e4f9c0664a879503afd3edf14c66 consensus state has no leader
I20250113 02:51:18.704679 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8adfa738c17a4169a5d99f6ceca75100. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:18.704867 448 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_8aa63512c7bb4bdb86f37c9ee24df209" } modify_external_catalogs: true
I20250113 02:51:18.705032 448 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_8aa63512c7bb4bdb86f37c9ee24df209" } modify_external_catalogs: true
W20250113 02:51:18.706851 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 64cda8796dff46ce95c37439395c77ec consensus state has no leader
I20250113 02:51:18.707298 1042 tablet_service.cc:1467] Processing CreateTablet for tablet 8adfa738c17a4169a5d99f6ceca75100 (DEFAULT_TABLE table=default.Table_6167b33d8e4246cab5fbe98cb57e9a3c [id=ae544735623e4cfd894dc9340aca177c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:18.707681 1042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8adfa738c17a4169a5d99f6ceca75100. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:18.708510 709 leader_election.cc:343] T 7f3cbef7d91d4576822d913071a88248 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:18.709301 754 tablet_service.cc:1514] Processing DeleteTablet for tablet d6675bc3cb5f4f809320e28026cdf4b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:17 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:18.709620 1041 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ae18cbff90d4fa88f6b9b8746fc2027 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:16 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:18.709878 1198 tablet_replica.cc:331] T 5ae18cbff90d4fa88f6b9b8746fc2027 P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:18.710065 1198 raft_consensus.cc:2238] T 5ae18cbff90d4fa88f6b9b8746fc2027 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:18.710212 1198 raft_consensus.cc:2267] T 5ae18cbff90d4fa88f6b9b8746fc2027 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:18.712133 1198 ts_tablet_manager.cc:1905] T 5ae18cbff90d4fa88f6b9b8746fc2027 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:18.699780 1209 raft_consensus.cc:513] T 7f3cbef7d91d4576822d913071a88248 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:18.712261 1029 tablet_service.cc:1467] Processing CreateTablet for tablet 501d4e1ae84e4795913035f8a19cc32d (DEFAULT_TABLE table=default.Table_8a5b45e248e94efb9b16df971fb9755c [id=bd54b5a058a44d718071be41b1e29a8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:18.712577 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 501d4e1ae84e4795913035f8a19cc32d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:18.713253 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f3cbef7d91d4576822d913071a88248" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f"
I20250113 02:51:18.713528 918 raft_consensus.cc:3054] T 7f3cbef7d91d4576822d913071a88248 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:18.716101 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet 455b143af8c94f9d8a4111330648cac6 with delete_type TABLET_DATA_DELETED (replaced by tablet bc88676bf738442d82c25e56f30c4ada) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:18.716315 1030 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f1c2ae654bd46c1a8772fa62c8111ab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:17 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:18.716488 1033 tablet_service.cc:1514] Processing DeleteTablet for tablet 7dbda00269cf4f7095b3504ae87ac243 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:16 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:18.717115 918 raft_consensus.cc:2463] T 7f3cbef7d91d4576822d913071a88248 P df69f48656b44bb890f76587ee69cc0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1186fa869eb14812832fa6562531f196 in term 1.
I20250113 02:51:18.717638 1209 leader_election.cc:290] T 7f3cbef7d91d4576822d913071a88248 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 election: Requested vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:18.718063 711 leader_election.cc:304] T 7f3cbef7d91d4576822d913071a88248 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1186fa869eb14812832fa6562531f196, df69f48656b44bb890f76587ee69cc0f; no voters:
I20250113 02:51:18.718924 754 tablet_service.cc:1467] Processing CreateTablet for tablet 601c71d5f7624fbb87b365d2e775d33e (DEFAULT_TABLE table=default.Table_7a9c9cc8678e4f7fa12237d3f8c9c3de [id=97a27e2641d2468aac5c827fad4e2091]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:18.719261 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 601c71d5f7624fbb87b365d2e775d33e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:18.719580 893 tablet_service.cc:1467] Processing CreateTablet for tablet 501d4e1ae84e4795913035f8a19cc32d (DEFAULT_TABLE table=default.Table_8a5b45e248e94efb9b16df971fb9755c [id=bd54b5a058a44d718071be41b1e29a8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:18.719959 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 501d4e1ae84e4795913035f8a19cc32d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:18.720048 1033 tablet_service.cc:1467] Processing CreateTablet for tablet 601c71d5f7624fbb87b365d2e775d33e (DEFAULT_TABLE table=default.Table_7a9c9cc8678e4f7fa12237d3f8c9c3de [id=97a27e2641d2468aac5c827fad4e2091]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:18.720363 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 601c71d5f7624fbb87b365d2e775d33e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:18.722651 1062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f3cbef7d91d4576822d913071a88248" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca"
W20250113 02:51:18.723380 709 leader_election.cc:343] T 7f3cbef7d91d4576822d913071a88248 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:18.724113 899 tablet_service.cc:1514] Processing DeleteTablet for tablet a028cc02273644608f046135851da126 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:16 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:18.724330 895 tablet_service.cc:1514] Processing DeleteTablet for tablet db266a1848094b7a98aa75f7afabd471 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:18 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:18.724669 889 tablet_service.cc:1467] Processing CreateTablet for tablet 601c71d5f7624fbb87b365d2e775d33e (DEFAULT_TABLE table=default.Table_7a9c9cc8678e4f7fa12237d3f8c9c3de [id=97a27e2641d2468aac5c827fad4e2091]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:18.725001 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 601c71d5f7624fbb87b365d2e775d33e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:18.728291 896 tablet_service.cc:1467] Processing CreateTablet for tablet 8adfa738c17a4169a5d99f6ceca75100 (DEFAULT_TABLE table=default.Table_6167b33d8e4246cab5fbe98cb57e9a3c [id=ae544735623e4cfd894dc9340aca177c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:18.728663 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8adfa738c17a4169a5d99f6ceca75100. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:18.734869 1209 raft_consensus.cc:2798] T 7f3cbef7d91d4576822d913071a88248 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Leader election won for term 1
I20250113 02:51:18.735304 1209 raft_consensus.cc:695] T 7f3cbef7d91d4576822d913071a88248 P 1186fa869eb14812832fa6562531f196 [term 1 LEADER]: Becoming Leader. State: Replica: 1186fa869eb14812832fa6562531f196, State: Running, Role: LEADER
I20250113 02:51:18.735623 1209 consensus_queue.cc:237] T 7f3cbef7d91d4576822d913071a88248 P 1186fa869eb14812832fa6562531f196 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:18.738232 1203 raft_consensus.cc:695] T db266a1848094b7a98aa75f7afabd471 P df69f48656b44bb890f76587ee69cc0f [term 1 LEADER]: Becoming Leader. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Running, Role: LEADER
I20250113 02:51:18.738821 1203 consensus_queue.cc:237] T db266a1848094b7a98aa75f7afabd471 P df69f48656b44bb890f76587ee69cc0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:18.740334 1204 raft_consensus.cc:491] T 7dbda00269cf4f7095b3504ae87ac243 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:18.740494 1204 raft_consensus.cc:513] T 7dbda00269cf4f7095b3504ae87ac243 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:18.741097 1204 leader_election.cc:290] T 7dbda00269cf4f7095b3504ae87ac243 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
W20250113 02:51:18.742645 998 leader_election.cc:343] T 7dbda00269cf4f7095b3504ae87ac243 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Not found: Tablet not found: 7dbda00269cf4f7095b3504ae87ac243
I20250113 02:51:18.743440 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dbda00269cf4f7095b3504ae87ac243" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
I20250113 02:51:18.743773 775 raft_consensus.cc:2463] T 7dbda00269cf4f7095b3504ae87ac243 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 0.
I20250113 02:51:18.744441 995 leader_election.cc:304] T 7dbda00269cf4f7095b3504ae87ac243 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters: df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:18.744490 1200 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 02:51:18.745039 1204 raft_consensus.cc:2798] T 7dbda00269cf4f7095b3504ae87ac243 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 02:51:18.745158 1204 raft_consensus.cc:491] T 7dbda00269cf4f7095b3504ae87ac243 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 02:51:18.745261 1204 raft_consensus.cc:3054] T 7dbda00269cf4f7095b3504ae87ac243 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:18.745625 1200 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250113 02:51:18.745906 1200 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 02:51:18.748173 1196 tablet_replica.cc:331] T a028cc02273644608f046135851da126 P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:18.748603 1196 raft_consensus.cc:2238] T a028cc02273644608f046135851da126 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:18.749096 1196 raft_consensus.cc:2267] T a028cc02273644608f046135851da126 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:18.749537 1204 raft_consensus.cc:513] T 7dbda00269cf4f7095b3504ae87ac243 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:18.750123 1196 ts_tablet_manager.cc:1905] T a028cc02273644608f046135851da126 P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:18.750993 1198 ts_tablet_manager.cc:1918] T 5ae18cbff90d4fa88f6b9b8746fc2027 P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:18.751865 1198 log.cc:1198] T 5ae18cbff90d4fa88f6b9b8746fc2027 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/5ae18cbff90d4fa88f6b9b8746fc2027
W20250113 02:51:18.751982 998 leader_election.cc:343] T 7dbda00269cf4f7095b3504ae87ac243 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Not found: Tablet not found: 7dbda00269cf4f7095b3504ae87ac243
I20250113 02:51:18.752413 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dbda00269cf4f7095b3504ae87ac243" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196"
I20250113 02:51:18.752624 1198 ts_tablet_manager.cc:1939] T 5ae18cbff90d4fa88f6b9b8746fc2027 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:18.752660 775 raft_consensus.cc:3054] T 7dbda00269cf4f7095b3504ae87ac243 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:18.753664 1198 tablet_replica.cc:331] T 455b143af8c94f9d8a4111330648cac6 P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:18.753791 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet 5ae18cbff90d4fa88f6b9b8746fc2027 (table default.Table_6a4e0c489c1944a69d3f46a874357dd6 [id=b6376e7b605c4b6f8d1373018a6c207e]) successfully deleted
I20250113 02:51:18.753863 1198 raft_consensus.cc:2238] T 455b143af8c94f9d8a4111330648cac6 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:18.754024 1198 raft_consensus.cc:2267] T 455b143af8c94f9d8a4111330648cac6 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:18.755108 1198 ts_tablet_manager.cc:1905] T 455b143af8c94f9d8a4111330648cac6 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:18.751423 1204 leader_election.cc:290] T 7dbda00269cf4f7095b3504ae87ac243 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Requested vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:18.756197 775 raft_consensus.cc:2463] T 7dbda00269cf4f7095b3504ae87ac243 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 1.
I20250113 02:51:18.756644 995 leader_election.cc:304] T 7dbda00269cf4f7095b3504ae87ac243 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters: df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:18.757181 1204 raft_consensus.cc:2798] T 7dbda00269cf4f7095b3504ae87ac243 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Leader election won for term 1
I20250113 02:51:18.759603 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_bb624ca10b84446bbaffca5a2173e00c" } modify_external_catalogs: true
I20250113 02:51:18.760231 447 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_8a5b45e248e94efb9b16df971fb9755c" } modify_external_catalogs: true
I20250113 02:51:18.760385 447 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_8a5b45e248e94efb9b16df971fb9755c" } modify_external_catalogs: true
I20250113 02:51:18.759876 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_bb624ca10b84446bbaffca5a2173e00c" } modify_external_catalogs: true
I20250113 02:51:18.763058 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_f5342954ee0548ec90d482ed62198240"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:18.763613 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f5342954ee0548ec90d482ed62198240 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:18.767208 1198 ts_tablet_manager.cc:1918] T 455b143af8c94f9d8a4111330648cac6 P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:18.767355 1198 log.cc:1198] T 455b143af8c94f9d8a4111330648cac6 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/455b143af8c94f9d8a4111330648cac6
I20250113 02:51:18.767745 1198 ts_tablet_manager.cc:1939] T 455b143af8c94f9d8a4111330648cac6 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:18.774289 1191 tablet_bootstrap.cc:492] T ec0750fd2fa247feb3ea310ffabe05e5 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:18.774469 1191 ts_tablet_manager.cc:1397] T ec0750fd2fa247feb3ea310ffabe05e5 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.107s user 0.005s sys 0.000s
I20250113 02:51:18.775440 1191 raft_consensus.cc:357] T ec0750fd2fa247feb3ea310ffabe05e5 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:18.775691 1191 raft_consensus.cc:383] T ec0750fd2fa247feb3ea310ffabe05e5 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:18.775791 1191 raft_consensus.cc:738] T ec0750fd2fa247feb3ea310ffabe05e5 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:18.776099 1204 raft_consensus.cc:695] T 7dbda00269cf4f7095b3504ae87ac243 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: Becoming Leader. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Running, Role: LEADER
I20250113 02:51:18.776614 1204 consensus_queue.cc:237] T 7dbda00269cf4f7095b3504ae87ac243 P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:18.781477 1196 ts_tablet_manager.cc:1918] T a028cc02273644608f046135851da126 P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:18.782579 1196 log.cc:1198] T a028cc02273644608f046135851da126 P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/a028cc02273644608f046135851da126
I20250113 02:51:18.783111 1196 ts_tablet_manager.cc:1939] T a028cc02273644608f046135851da126 P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:18.783994 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet a028cc02273644608f046135851da126 (table default.Table_eec2c35d4a8a4196983a872a0fa4bda5 [id=c7f464a7e8f14a0892d145a69a148e58]) successfully deleted
I20250113 02:51:18.784075 1209 raft_consensus.cc:491] T d6675bc3cb5f4f809320e28026cdf4b0 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:18.784235 1209 raft_consensus.cc:513] T d6675bc3cb5f4f809320e28026cdf4b0 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
W20250113 02:51:18.785959 711 leader_election.cc:343] T d6675bc3cb5f4f809320e28026cdf4b0 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Not found: Tablet not found: d6675bc3cb5f4f809320e28026cdf4b0
I20250113 02:51:18.785998 1062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6675bc3cb5f4f809320e28026cdf4b0" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:18.786240 1196 tablet_replica.cc:331] T db266a1848094b7a98aa75f7afabd471 P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
W20250113 02:51:18.786609 709 leader_election.cc:343] T d6675bc3cb5f4f809320e28026cdf4b0 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:18.786744 709 leader_election.cc:304] T d6675bc3cb5f4f809320e28026cdf4b0 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:18.787086 1270 raft_consensus.cc:2743] T d6675bc3cb5f4f809320e28026cdf4b0 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:18.787295 1209 leader_election.cc:290] T d6675bc3cb5f4f809320e28026cdf4b0 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:18.787636 1304 raft_consensus.cc:491] T 6f1c2ae654bd46c1a8772fa62c8111ab P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:18.787801 1304 raft_consensus.cc:513] T 6f1c2ae654bd46c1a8772fa62c8111ab P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
W20250113 02:51:18.789501 998 leader_election.cc:343] T 6f1c2ae654bd46c1a8772fa62c8111ab P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Not found: Tablet not found: 6f1c2ae654bd46c1a8772fa62c8111ab
I20250113 02:51:18.789600 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f1c2ae654bd46c1a8772fa62c8111ab" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
I20250113 02:51:18.789960 775 raft_consensus.cc:2388] T 6f1c2ae654bd46c1a8772fa62c8111ab P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4216a7f853d642b7bc3e5cff458f3aca in current term 1: Already voted for candidate 1186fa869eb14812832fa6562531f196 in this term.
I20250113 02:51:18.790665 995 leader_election.cc:304] T 6f1c2ae654bd46c1a8772fa62c8111ab P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4216a7f853d642b7bc3e5cff458f3aca; no voters: 1186fa869eb14812832fa6562531f196, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:18.794076 1194 ts_tablet_manager.cc:1428] T 78689f7ed2a64405adb75f372465a792 P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.121s user 0.009s sys 0.032s
I20250113 02:51:18.794502 1194 tablet_bootstrap.cc:492] T ad6b21ed2b824cb0837fc1d8a39d2472 P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:18.795485 1308 raft_consensus.cc:3054] T 6f1c2ae654bd46c1a8772fa62c8111ab P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:18.797243 1194 tablet_bootstrap.cc:654] T ad6b21ed2b824cb0837fc1d8a39d2472 P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:18.797330 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 90) took 1445 ms. Trace:
I20250113 02:51:18.797451 1195 rpcz_store.cc:276] 0113 02:51:17.351762 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:17.351816 (+ 54us) service_pool.cc:225] Handling call
0113 02:51:18.797141 (+1445325us) ts_tablet_manager.cc:1145] Deleting tablet 3d132c29cb2c46108e5877e38068ef17
0113 02:51:18.797154 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:18.797277 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1445075}
I20250113 02:51:18.797919 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 91) took 1440 ms. Trace:
I20250113 02:51:18.798012 1195 rpcz_store.cc:276] 0113 02:51:17.357112 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:17.357504 (+ 392us) service_pool.cc:225] Handling call
0113 02:51:18.797722 (+1440218us) ts_tablet_manager.cc:1145] Deleting tablet 3d3a6d9f687e45218d5875a7c5597648
0113 02:51:18.797734 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:18.797866 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1439970}
W20250113 02:51:18.798131 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 3d132c29cb2c46108e5877e38068ef17 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3d132c29cb2c46108e5877e38068ef17 already in progress: creating tablet
I20250113 02:51:18.798341 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 93) took 1431 ms. Trace:
I20250113 02:51:18.798442 1195 rpcz_store.cc:276] 0113 02:51:17.367254 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:17.367298 (+ 44us) service_pool.cc:225] Handling call
0113 02:51:18.798215 (+1430917us) ts_tablet_manager.cc:1145] Deleting tablet c62a5e9e43d14b5c94eb87260528469d
0113 02:51:18.798223 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:18.798292 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1426126}
I20250113 02:51:18.798744 1195 tablet_replica.cc:331] T 6f1c2ae654bd46c1a8772fa62c8111ab P 1186fa869eb14812832fa6562531f196: stopping tablet replica
W20250113 02:51:18.798719 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 3d3a6d9f687e45218d5875a7c5597648 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3d3a6d9f687e45218d5875a7c5597648 already in progress: creating tablet
I20250113 02:51:18.798939 1195 raft_consensus.cc:2238] T 6f1c2ae654bd46c1a8772fa62c8111ab P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:18.799122 1195 raft_consensus.cc:2267] T 6f1c2ae654bd46c1a8772fa62c8111ab P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250113 02:51:18.799158 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet c62a5e9e43d14b5c94eb87260528469d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c62a5e9e43d14b5c94eb87260528469d already in progress: creating tablet
I20250113 02:51:18.799659 1195 ts_tablet_manager.cc:1905] T 6f1c2ae654bd46c1a8772fa62c8111ab P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250113 02:51:18.800279 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d3b695ac20ba4205b2d8732db4740531 consensus state has no leader
I20250113 02:51:18.800479 1192 tablet_bootstrap.cc:492] T 7f3cbef7d91d4576822d913071a88248 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:18.800750 1192 ts_tablet_manager.cc:1397] T 7f3cbef7d91d4576822d913071a88248 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.161s user 0.024s sys 0.022s
I20250113 02:51:18.801792 1192 raft_consensus.cc:357] T 7f3cbef7d91d4576822d913071a88248 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:18.801486 446 catalog_manager.cc:5526] T 7f3cbef7d91d4576822d913071a88248 P 1186fa869eb14812832fa6562531f196 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1186fa869eb14812832fa6562531f196 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "1186fa869eb14812832fa6562531f196" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } health_report { overall_health: UNKNOWN } } }
I20250113 02:51:18.802374 446 catalog_manager.cc:5618] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095) does not have the latest schema for tablet 7f3cbef7d91d4576822d913071a88248 (table default.Table_b581b1a063a54551a8535adf01983fb8 [id=54e4e3810b3b412bbec3b5bf5b28c1a2]). Expected version 1 got 0
I20250113 02:51:18.803521 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_f1d42e491cb34117a3bed634de503a6d" } new_table_name: "default.Table_4a0258998d4f413b80d701c74526b86d" modify_external_catalogs: true
I20250113 02:51:18.803807 1192 raft_consensus.cc:383] T 7f3cbef7d91d4576822d913071a88248 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:18.804009 1192 raft_consensus.cc:738] T 7f3cbef7d91d4576822d913071a88248 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:18.805528 1192 consensus_queue.cc:260] T 7f3cbef7d91d4576822d913071a88248 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:18.805189 1308 raft_consensus.cc:2743] T 6f1c2ae654bd46c1a8772fa62c8111ab P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:18.806244 1195 ts_tablet_manager.cc:1918] T 6f1c2ae654bd46c1a8772fa62c8111ab P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:18.806368 1195 log.cc:1198] T 6f1c2ae654bd46c1a8772fa62c8111ab P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/6f1c2ae654bd46c1a8772fa62c8111ab
I20250113 02:51:18.806717 1195 ts_tablet_manager.cc:1939] T 6f1c2ae654bd46c1a8772fa62c8111ab P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:18.808193 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 92) took 1441 ms. Trace:
I20250113 02:51:18.808327 1195 rpcz_store.cc:276] 0113 02:51:17.367090 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:17.372177 (+ 5087us) service_pool.cc:225] Handling call
0113 02:51:18.798674 (+1426497us) ts_tablet_manager.cc:1145] Deleting tablet 6f1c2ae654bd46c1a8772fa62c8111ab
0113 02:51:18.798681 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:18.802741 (+ 4060us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:18.806130 (+ 3389us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:18.808145 (+ 2015us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1426348}
I20250113 02:51:18.808828 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet 6f1c2ae654bd46c1a8772fa62c8111ab (table default.Table_ec4ee7380446448293eaa15db62d34de [id=7dc8c3e55ce843009ade78391e271091]) successfully deleted
I20250113 02:51:18.809273 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 103) took 1124 ms. Trace:
I20250113 02:51:18.809389 1195 rpcz_store.cc:276] 0113 02:51:17.684453 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:17.684559 (+ 106us) service_pool.cc:225] Handling call
0113 02:51:18.809092 (+1124533us) ts_tablet_manager.cc:1145] Deleting tablet 8193357427f4491c8c6c1396ceeef2dc
0113 02:51:18.809104 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:18.809236 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1124200}
W20250113 02:51:18.809880 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 8193357427f4491c8c6c1396ceeef2dc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8193357427f4491c8c6c1396ceeef2dc already in progress: creating tablet
I20250113 02:51:18.810159 1195 tablet_replica.cc:331] T 89517af3545c47a5a3a247a411190a47 P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:18.810356 1195 raft_consensus.cc:2238] T 89517af3545c47a5a3a247a411190a47 P 1186fa869eb14812832fa6562531f196 [term 1 LEADER]: Raft consensus shutting down.
I20250113 02:51:18.813411 1196 raft_consensus.cc:2238] T db266a1848094b7a98aa75f7afabd471 P df69f48656b44bb890f76587ee69cc0f [term 1 LEADER]: Raft consensus shutting down.
I20250113 02:51:18.814277 1196 pending_rounds.cc:62] T db266a1848094b7a98aa75f7afabd471 P df69f48656b44bb890f76587ee69cc0f: Trying to abort 1 pending ops.
I20250113 02:51:18.814464 1196 pending_rounds.cc:69] T db266a1848094b7a98aa75f7afabd471 P df69f48656b44bb890f76587ee69cc0f: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113673436227670016 op_type: NO_OP noop_request { }
I20250113 02:51:18.810796 1195 pending_rounds.cc:62] T 89517af3545c47a5a3a247a411190a47 P 1186fa869eb14812832fa6562531f196: Trying to abort 1 pending ops.
I20250113 02:51:18.814649 1196 raft_consensus.cc:2883] T db266a1848094b7a98aa75f7afabd471 P df69f48656b44bb890f76587ee69cc0f [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250113 02:51:18.814785 1196 raft_consensus.cc:2267] T db266a1848094b7a98aa75f7afabd471 P df69f48656b44bb890f76587ee69cc0f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:18.814802 1195 pending_rounds.cc:69] T 89517af3545c47a5a3a247a411190a47 P 1186fa869eb14812832fa6562531f196: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113673430800965632 op_type: NO_OP noop_request { }
I20250113 02:51:18.815047 1195 raft_consensus.cc:2883] T 89517af3545c47a5a3a247a411190a47 P 1186fa869eb14812832fa6562531f196 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250113 02:51:18.815771 1195 raft_consensus.cc:2267] T 89517af3545c47a5a3a247a411190a47 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:18.812754 1191 consensus_queue.cc:260] T ec0750fd2fa247feb3ea310ffabe05e5 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:18.817238 1191 ts_tablet_manager.cc:1428] T ec0750fd2fa247feb3ea310ffabe05e5 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.043s user 0.001s sys 0.001s
I20250113 02:51:18.817593 1191 tablet_bootstrap.cc:492] T a028cc02273644608f046135851da126 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:18.819728 1191 tablet_bootstrap.cc:654] T a028cc02273644608f046135851da126 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:18.820472 1195 ts_tablet_manager.cc:1905] T 89517af3545c47a5a3a247a411190a47 P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:18.824247 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 5b3f012b88c04cb68ed7a1848147b70b
I20250113 02:51:18.824331 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_480e362015cc422a85dffa0b15d1dcae"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:18.824947 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_480e362015cc422a85dffa0b15d1dcae in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:18.824991 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_630328ef70a24cc5b8f01094d0abae76" } modify_external_catalogs: true
I20250113 02:51:18.825137 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_630328ef70a24cc5b8f01094d0abae76" } modify_external_catalogs: true
I20250113 02:51:18.826200 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 6c8ca30720664f3a819c4bff2493934d
I20250113 02:51:18.826944 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet bc88676bf738442d82c25e56f30c4ada
I20250113 02:51:18.827862 453 catalog_manager.cc:6596] ReplaceTablet: tablet bc88676bf738442d82c25e56f30c4ada deleted and replaced by tablet 401c4f5dbd304950877b59a865f417b3
W20250113 02:51:18.828259 454 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ebc5f1ea1e1648d49904b17603d3cd2a consensus state has no leader
I20250113 02:51:18.828620 451 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet d15e173b6b5245bd828fc1246626766d
I20250113 02:51:18.829121 754 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c8ca30720664f3a819c4bff2493934d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:18 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:18.829417 755 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b3f012b88c04cb68ed7a1848147b70b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:18 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:18.830255 1196 ts_tablet_manager.cc:1905] T db266a1848094b7a98aa75f7afabd471 P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:18.830507 893 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b3f012b88c04cb68ed7a1848147b70b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:18 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:18.830739 896 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c8ca30720664f3a819c4bff2493934d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:18 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:18.830973 889 tablet_service.cc:1514] Processing DeleteTablet for tablet bc88676bf738442d82c25e56f30c4ada with delete_type TABLET_DATA_DELETED (replaced by tablet 401c4f5dbd304950877b59a865f417b3) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:18.840340 755 tablet_service.cc:1514] Processing DeleteTablet for tablet bc88676bf738442d82c25e56f30c4ada with delete_type TABLET_DATA_DELETED (replaced by tablet 401c4f5dbd304950877b59a865f417b3) from {username='slave'} at 127.0.0.1:55398
W20250113 02:51:18.843024 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 21c4b419bf2148e0b140301fa80622bb consensus state has no leader
I20250113 02:51:18.843394 889 tablet_service.cc:1514] Processing DeleteTablet for tablet 78689f7ed2a64405adb75f372465a792 with delete_type TABLET_DATA_DELETED (replaced by tablet 0433cfe0cb524511954e1012d4d137e6) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:18.843473 1196 ts_tablet_manager.cc:1918] T db266a1848094b7a98aa75f7afabd471 P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250113 02:51:18.843750 1196 log.cc:1198] T db266a1848094b7a98aa75f7afabd471 P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/db266a1848094b7a98aa75f7afabd471
I20250113 02:51:18.846570 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_b3284070cce54277bed74120ca8f199b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:18.847170 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b3284070cce54277bed74120ca8f199b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:18.847219 1198 tablet_replica.cc:331] T 6f1c2ae654bd46c1a8772fa62c8111ab P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:18.847896 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet 455b143af8c94f9d8a4111330648cac6 (table default.Table_c991f9dbd70d4c4682664b8711f05db4 [id=2c64da64ecdb4bf48e11b523533902e8]) successfully deleted
I20250113 02:51:18.848618 755 tablet_service.cc:1514] Processing DeleteTablet for tablet ec0750fd2fa247feb3ea310ffabe05e5 with delete_type TABLET_DATA_DELETED (replaced by tablet 107e6e9e2d84496f8fce92b9ba377189) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:18.850229 446 master_service.cc:946] ReplaceTablet: received request to replace tablet d35ec7e4a5084a74bea172ef8d9990b5 from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:18.850662 452 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet 95e5a284a66c425a98b10e973ccd54a5
I20250113 02:51:18.852221 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_f08525e2a08e48ec85ef55287d0bac49"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:18.852438 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_47f735660c3a49d991a3593625822367"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:18.852746 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f08525e2a08e48ec85ef55287d0bac49 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250113 02:51:18.852782 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_47f735660c3a49d991a3593625822367 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:18.855410 454 master_service.cc:946] ReplaceTablet: received request to replace tablet 1c26074fdd804bd79cb0c295622d11ff from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:18.858740 1192 ts_tablet_manager.cc:1428] T 7f3cbef7d91d4576822d913071a88248 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.058s user 0.001s sys 0.002s
I20250113 02:51:18.859867 1196 ts_tablet_manager.cc:1939] T db266a1848094b7a98aa75f7afabd471 P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:18.861440 1042 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b3f012b88c04cb68ed7a1848147b70b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:18 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:18.860961 1304 leader_election.cc:290] T 6f1c2ae654bd46c1a8772fa62c8111ab P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
I20250113 02:51:18.861922 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet d35ec7e4a5084a74bea172ef8d9990b5
I20250113 02:51:18.861891 1304 raft_consensus.cc:491] T 7f3cbef7d91d4576822d913071a88248 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:18.862077 1304 raft_consensus.cc:513] T 7f3cbef7d91d4576822d913071a88248 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:18.862334 446 catalog_manager.cc:6596] ReplaceTablet: tablet d35ec7e4a5084a74bea172ef8d9990b5 deleted and replaced by tablet b4806b18a4194f9c95350d89ba62e13a
I20250113 02:51:18.862607 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c8ca30720664f3a819c4bff2493934d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:18 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:18.862815 1033 tablet_service.cc:1514] Processing DeleteTablet for tablet bc88676bf738442d82c25e56f30c4ada with delete_type TABLET_DATA_DELETED (replaced by tablet 401c4f5dbd304950877b59a865f417b3) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:18.862849 1198 raft_consensus.cc:2238] T 6f1c2ae654bd46c1a8772fa62c8111ab P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:18.863032 1198 raft_consensus.cc:2267] T 6f1c2ae654bd46c1a8772fa62c8111ab P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:18.863420 1033 tablet_service.cc:1514] Processing DeleteTablet for tablet d35ec7e4a5084a74bea172ef8d9990b5 with delete_type TABLET_DATA_DELETED (replaced by tablet b4806b18a4194f9c95350d89ba62e13a) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:18.864046 1198 ts_tablet_manager.cc:1905] T 6f1c2ae654bd46c1a8772fa62c8111ab P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:18.869171 755 tablet_service.cc:1514] Processing DeleteTablet for tablet d35ec7e4a5084a74bea172ef8d9990b5 with delete_type TABLET_DATA_DELETED (replaced by tablet b4806b18a4194f9c95350d89ba62e13a) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:18.870285 1198 ts_tablet_manager.cc:1918] T 6f1c2ae654bd46c1a8772fa62c8111ab P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:18.870515 1198 log.cc:1198] T 6f1c2ae654bd46c1a8772fa62c8111ab P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/6f1c2ae654bd46c1a8772fa62c8111ab
I20250113 02:51:18.871003 1198 ts_tablet_manager.cc:1939] T 6f1c2ae654bd46c1a8772fa62c8111ab P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:18.872262 1192 tablet_bootstrap.cc:492] T c97a6ad98ddc4680a0f071cb91f05402 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:18.872658 1198 tablet_replica.cc:331] T 7dbda00269cf4f7095b3504ae87ac243 P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:18.872745 1195 ts_tablet_manager.cc:1918] T 89517af3545c47a5a3a247a411190a47 P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250113 02:51:18.872890 1195 log.cc:1198] T 89517af3545c47a5a3a247a411190a47 P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/89517af3545c47a5a3a247a411190a47
I20250113 02:51:18.872997 1198 raft_consensus.cc:2238] T 7dbda00269cf4f7095b3504ae87ac243 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: Raft consensus shutting down.
I20250113 02:51:18.873307 1195 ts_tablet_manager.cc:1939] T 89517af3545c47a5a3a247a411190a47 P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:18.873668 1198 pending_rounds.cc:62] T 7dbda00269cf4f7095b3504ae87ac243 P 4216a7f853d642b7bc3e5cff458f3aca: Trying to abort 1 pending ops.
I20250113 02:51:18.873929 1198 pending_rounds.cc:69] T 7dbda00269cf4f7095b3504ae87ac243 P 4216a7f853d642b7bc3e5cff458f3aca: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113673436275777536 op_type: NO_OP noop_request { }
I20250113 02:51:18.874181 1198 raft_consensus.cc:2883] T 7dbda00269cf4f7095b3504ae87ac243 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250113 02:51:18.874357 1198 raft_consensus.cc:2267] T 7dbda00269cf4f7095b3504ae87ac243 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:18.874416 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet 6f1c2ae654bd46c1a8772fa62c8111ab (table default.Table_ec4ee7380446448293eaa15db62d34de [id=7dc8c3e55ce843009ade78391e271091]) successfully deleted
I20250113 02:51:18.874480 1192 tablet_bootstrap.cc:654] T c97a6ad98ddc4680a0f071cb91f05402 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:18.875340 452 master_service.cc:946] ReplaceTablet: received request to replace tablet 107e6e9e2d84496f8fce92b9ba377189 from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:18.875737 1198 ts_tablet_manager.cc:1905] T 7dbda00269cf4f7095b3504ae87ac243 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:18.875806 1304 leader_election.cc:290] T 7f3cbef7d91d4576822d913071a88248 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:18.876147 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f3cbef7d91d4576822d913071a88248" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
I20250113 02:51:18.874158 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 104) took 1188 ms. Trace:
I20250113 02:51:18.880338 1195 rpcz_store.cc:276] 0113 02:51:17.685280 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:17.685360 (+ 80us) service_pool.cc:225] Handling call
0113 02:51:18.810102 (+1124742us) ts_tablet_manager.cc:1145] Deleting tablet 89517af3545c47a5a3a247a411190a47
0113 02:51:18.810111 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:18.868051 (+ 57940us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:18.872631 (+ 4580us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:18.874126 (+ 1495us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1124367}
I20250113 02:51:18.880796 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet 89517af3545c47a5a3a247a411190a47 (table default.Table_5d3da53a6d984a7ea593b340ad3e1368 [id=e66cde9b8d9b4763b33e7641601c47da]) successfully deleted
I20250113 02:51:18.881485 1195 tablet_replica.cc:331] T c97a6ad98ddc4680a0f071cb91f05402 P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:18.881711 1195 raft_consensus.cc:2238] T c97a6ad98ddc4680a0f071cb91f05402 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:18.881893 1195 raft_consensus.cc:2267] T c97a6ad98ddc4680a0f071cb91f05402 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:18.863101 889 tablet_service.cc:1514] Processing DeleteTablet for tablet d35ec7e4a5084a74bea172ef8d9990b5 with delete_type TABLET_DATA_DELETED (replaced by tablet b4806b18a4194f9c95350d89ba62e13a) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:18.876886 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f3cbef7d91d4576822d913071a88248" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:18.883121 918 raft_consensus.cc:2388] T 7f3cbef7d91d4576822d913071a88248 P df69f48656b44bb890f76587ee69cc0f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4216a7f853d642b7bc3e5cff458f3aca in current term 1: Already voted for candidate 1186fa869eb14812832fa6562531f196 in this term.
I20250113 02:51:18.883755 998 leader_election.cc:304] T 7f3cbef7d91d4576822d913071a88248 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4216a7f853d642b7bc3e5cff458f3aca; no voters: 1186fa869eb14812832fa6562531f196, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:18.884793 1311 raft_consensus.cc:3054] T 7f3cbef7d91d4576822d913071a88248 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:18.885192 1195 ts_tablet_manager.cc:1905] T c97a6ad98ddc4680a0f071cb91f05402 P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:18.885288 1194 tablet_bootstrap.cc:492] T ad6b21ed2b824cb0837fc1d8a39d2472 P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:18.885466 1194 ts_tablet_manager.cc:1397] T ad6b21ed2b824cb0837fc1d8a39d2472 P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.091s user 0.016s sys 0.030s
I20250113 02:51:18.886207 1194 raft_consensus.cc:357] T ad6b21ed2b824cb0837fc1d8a39d2472 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:18.886466 1194 raft_consensus.cc:383] T ad6b21ed2b824cb0837fc1d8a39d2472 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:18.886548 1194 raft_consensus.cc:738] T ad6b21ed2b824cb0837fc1d8a39d2472 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:18.886785 1194 consensus_queue.cc:260] T ad6b21ed2b824cb0837fc1d8a39d2472 P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:18.887364 1194 ts_tablet_manager.cc:1428] T ad6b21ed2b824cb0837fc1d8a39d2472 P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250113 02:51:18.887722 1194 tablet_bootstrap.cc:492] T c97a6ad98ddc4680a0f071cb91f05402 P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:18.888033 1311 raft_consensus.cc:2743] T 7f3cbef7d91d4576822d913071a88248 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:18.889173 450 catalog_manager.cc:5618] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587) does not have the latest schema for tablet 7f3cbef7d91d4576822d913071a88248 (table default.Table_b581b1a063a54551a8535adf01983fb8 [id=54e4e3810b3b412bbec3b5bf5b28c1a2]). Expected version 1 got 0
I20250113 02:51:18.889633 1194 tablet_bootstrap.cc:654] T c97a6ad98ddc4680a0f071cb91f05402 P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:18.890434 889 tablet_service.cc:1514] Processing DeleteTablet for tablet ad6b21ed2b824cb0837fc1d8a39d2472 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:16 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:18.891588 889 tablet_service.cc:1467] Processing CreateTablet for tablet 258fad818b7b4aa4b337f9949e20aa46 (DEFAULT_TABLE table=default.Table_e44b0cdc7f4640ec867d92338d94426b [id=9538429e42bb48c7b7fbae43644f2b2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:18.892184 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 258fad818b7b4aa4b337f9949e20aa46. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:18.895309 1198 ts_tablet_manager.cc:1918] T 7dbda00269cf4f7095b3504ae87ac243 P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250113 02:51:18.896831 1198 log.cc:1198] T 7dbda00269cf4f7095b3504ae87ac243 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/7dbda00269cf4f7095b3504ae87ac243
I20250113 02:51:18.897403 1198 ts_tablet_manager.cc:1939] T 7dbda00269cf4f7095b3504ae87ac243 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:18.897979 451 catalog_manager.cc:5618] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587) does not have the latest schema for tablet 7f3cbef7d91d4576822d913071a88248 (table default.Table_b581b1a063a54551a8535adf01983fb8 [id=54e4e3810b3b412bbec3b5bf5b28c1a2]). Expected version 1 got 0
I20250113 02:51:18.900434 1194 tablet_bootstrap.cc:492] T c97a6ad98ddc4680a0f071cb91f05402 P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:18.901727 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet 7dbda00269cf4f7095b3504ae87ac243 (table default.Table_8e22a139d87f407da7668fde4cf6c67a [id=46fce62be9184eea82a2fdc88b9aff21]) successfully deleted
I20250113 02:51:18.902101 1198 tablet_replica.cc:331] T bc88676bf738442d82c25e56f30c4ada P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
W20250113 02:51:18.902274 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 5b3f012b88c04cb68ed7a1848147b70b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5b3f012b88c04cb68ed7a1848147b70b already in progress: creating tablet
I20250113 02:51:18.902346 1198 raft_consensus.cc:2238] T bc88676bf738442d82c25e56f30c4ada P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus shutting down.
W20250113 02:51:18.902508 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 6c8ca30720664f3a819c4bff2493934d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6c8ca30720664f3a819c4bff2493934d already in progress: creating tablet
I20250113 02:51:18.902580 1198 raft_consensus.cc:2267] T bc88676bf738442d82c25e56f30c4ada P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:18.903203 1194 ts_tablet_manager.cc:1397] T c97a6ad98ddc4680a0f071cb91f05402 P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.002s
I20250113 02:51:18.904453 1033 tablet_service.cc:1467] Processing CreateTablet for tablet 258fad818b7b4aa4b337f9949e20aa46 (DEFAULT_TABLE table=default.Table_e44b0cdc7f4640ec867d92338d94426b [id=9538429e42bb48c7b7fbae43644f2b2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:18.904973 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 258fad818b7b4aa4b337f9949e20aa46. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:18.905360 1194 raft_consensus.cc:357] T c97a6ad98ddc4680a0f071cb91f05402 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:18.905637 1194 raft_consensus.cc:383] T c97a6ad98ddc4680a0f071cb91f05402 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:18.905740 1194 raft_consensus.cc:738] T c97a6ad98ddc4680a0f071cb91f05402 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:18.906013 1194 consensus_queue.cc:260] T c97a6ad98ddc4680a0f071cb91f05402 P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:18.906735 1194 ts_tablet_manager.cc:1428] T c97a6ad98ddc4680a0f071cb91f05402 P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250113 02:51:18.907074 1194 tablet_bootstrap.cc:492] T ec0750fd2fa247feb3ea310ffabe05e5 P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:18.913028 1194 tablet_bootstrap.cc:654] T ec0750fd2fa247feb3ea310ffabe05e5 P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:18.913520 1198 ts_tablet_manager.cc:1905] T bc88676bf738442d82c25e56f30c4ada P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:18.914338 896 tablet_service.cc:1514] Processing DeleteTablet for tablet c97a6ad98ddc4680a0f071cb91f05402 with delete_type TABLET_DATA_DELETED (replaced by tablet 2fe235d5280944ef92cf15fb458f2f81) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:18.921249 1194 tablet_bootstrap.cc:492] T ec0750fd2fa247feb3ea310ffabe05e5 P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:18.921231 755 tablet_service.cc:1467] Processing CreateTablet for tablet 258fad818b7b4aa4b337f9949e20aa46 (DEFAULT_TABLE table=default.Table_e44b0cdc7f4640ec867d92338d94426b [id=9538429e42bb48c7b7fbae43644f2b2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:18.921469 1194 ts_tablet_manager.cc:1397] T ec0750fd2fa247feb3ea310ffabe05e5 P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.002s
I20250113 02:51:18.921700 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 258fad818b7b4aa4b337f9949e20aa46. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:18.921795 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 1c26074fdd804bd79cb0c295622d11ff
I20250113 02:51:18.922308 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 501d4e1ae84e4795913035f8a19cc32d
I20250113 02:51:18.923136 454 catalog_manager.cc:6596] ReplaceTablet: tablet 1c26074fdd804bd79cb0c295622d11ff deleted and replaced by tablet 9fff52e99c64462a8a73fccc3c22ca63
I20250113 02:51:18.924381 893 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c26074fdd804bd79cb0c295622d11ff with delete_type TABLET_DATA_DELETED (replaced by tablet 9fff52e99c64462a8a73fccc3c22ca63) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:18.924432 1194 raft_consensus.cc:357] T ec0750fd2fa247feb3ea310ffabe05e5 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:18.924635 896 tablet_service.cc:1514] Processing DeleteTablet for tablet 501d4e1ae84e4795913035f8a19cc32d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:18 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:18.924736 1194 raft_consensus.cc:383] T ec0750fd2fa247feb3ea310ffabe05e5 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:18.924840 1194 raft_consensus.cc:738] T ec0750fd2fa247feb3ea310ffabe05e5 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:18.925164 1194 consensus_queue.cc:260] T ec0750fd2fa247feb3ea310ffabe05e5 P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:18.925887 1194 ts_tablet_manager.cc:1428] T ec0750fd2fa247feb3ea310ffabe05e5 P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20250113 02:51:18.927964 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet 501d4e1ae84e4795913035f8a19cc32d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:18 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:18.928229 1042 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c26074fdd804bd79cb0c295622d11ff with delete_type TABLET_DATA_DELETED (replaced by tablet 9fff52e99c64462a8a73fccc3c22ca63) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:18.928301 1194 tablet_bootstrap.cc:492] T 0c57a5add4914835ac366efdcde4ee28 P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:18.930680 1194 tablet_bootstrap.cc:654] T 0c57a5add4914835ac366efdcde4ee28 P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:18.932060 896 tablet_service.cc:1514] Processing DeleteTablet for tablet ec0750fd2fa247feb3ea310ffabe05e5 with delete_type TABLET_DATA_DELETED (replaced by tablet 107e6e9e2d84496f8fce92b9ba377189) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:18.934065 1195 ts_tablet_manager.cc:1918] T c97a6ad98ddc4680a0f071cb91f05402 P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:18.934227 1195 log.cc:1198] T c97a6ad98ddc4680a0f071cb91f05402 P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/c97a6ad98ddc4680a0f071cb91f05402
I20250113 02:51:18.934878 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_d4146353239a45b281b482d6e9537502"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:18.935504 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d4146353239a45b281b482d6e9537502 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:18.936004 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_c79704f27bf14cba8a0afc2dd36a143f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:18.936506 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c79704f27bf14cba8a0afc2dd36a143f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:18.935422 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_a6447976a51a4e2f93d22bd1c657c49d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:18.938117 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a6447976a51a4e2f93d22bd1c657c49d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:18.943928 1195 ts_tablet_manager.cc:1939] T c97a6ad98ddc4680a0f071cb91f05402 P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:18.945173 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 113) took 1165 ms. Trace:
I20250113 02:51:18.948396 455 master_service.cc:946] ReplaceTablet: received request to replace tablet 157905316fc545b6a7864ae1c4a2a584 from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:18.950222 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_36c8a9314f6a4da4a1c0dbb51369d5bd" } new_table_name: "default.Table_22fdcd0b65d14830878ae2d87bca9bb9" modify_external_catalogs: true
I20250113 02:51:18.944036 754 tablet_service.cc:1514] Processing DeleteTablet for tablet 501d4e1ae84e4795913035f8a19cc32d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:18 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:18.950464 1195 rpcz_store.cc:276] 0113 02:51:17.779514 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:17.779761 (+ 247us) service_pool.cc:225] Handling call
0113 02:51:18.881409 (+1101648us) ts_tablet_manager.cc:1145] Deleting tablet c97a6ad98ddc4680a0f071cb91f05402
0113 02:51:18.881421 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:18.900641 (+ 19220us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:18.933929 (+ 33288us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:18.945111 (+ 11182us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1101396}
I20250113 02:51:18.947705 741 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c26074fdd804bd79cb0c295622d11ff with delete_type TABLET_DATA_DELETED (replaced by tablet 9fff52e99c64462a8a73fccc3c22ca63) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:18.951422 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 117) took 1151 ms. Trace:
I20250113 02:51:18.951540 1195 rpcz_store.cc:276] 0113 02:51:17.799645 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:17.799774 (+ 129us) service_pool.cc:225] Handling call
0113 02:51:18.951204 (+1151430us) ts_tablet_manager.cc:1145] Deleting tablet 68023c30a7f644eaa5b4ddd60d6c5ef5
0113 02:51:18.951216 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:18.951359 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1144741}
I20250113 02:51:18.953205 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet c97a6ad98ddc4680a0f071cb91f05402 (table default.Table_351e5f6de7c64dafb0d9ee4d86510d01 [id=a383dc51c39e4c8ba4b421119c752470]) successfully deleted
I20250113 02:51:18.953622 1194 tablet_bootstrap.cc:492] T 0c57a5add4914835ac366efdcde4ee28 P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:18.953815 1194 ts_tablet_manager.cc:1397] T 0c57a5add4914835ac366efdcde4ee28 P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.026s user 0.002s sys 0.005s
I20250113 02:51:18.952803 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_1c9fae2ae0b14e7ca0b0ed24b12c0313" } new_table_name: "default.Table_edd55c813c94406ea65e9215b40632b6" modify_external_catalogs: true
I20250113 02:51:18.955912 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 119) took 1149 ms. Trace:
I20250113 02:51:18.956053 1195 rpcz_store.cc:276] 0113 02:51:17.806043 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:17.806091 (+ 48us) service_pool.cc:225] Handling call
0113 02:51:18.955128 (+1149037us) ts_tablet_manager.cc:1145] Deleting tablet 34ab7ee573284550aaac9f814e8bd7ab
0113 02:51:18.955144 (+ 16us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:18.955854 (+ 710us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1147780}
W20250113 02:51:18.957103 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 34ab7ee573284550aaac9f814e8bd7ab because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 34ab7ee573284550aaac9f814e8bd7ab already in progress: creating tablet
I20250113 02:51:18.959337 896 tablet_service.cc:1467] Processing CreateTablet for tablet fdbf03c89bff438a9a25115eddf60be9 (DEFAULT_TABLE table=default.Table_732e990e6f154d6486889304bbf88cef [id=48c7c3f6b4a840ee9b9e3f3844add13d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:18.959877 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fdbf03c89bff438a9a25115eddf60be9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:18.959895 1198 ts_tablet_manager.cc:1918] T bc88676bf738442d82c25e56f30c4ada P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:18.960505 1194 raft_consensus.cc:357] T 0c57a5add4914835ac366efdcde4ee28 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:18.960763 1194 raft_consensus.cc:383] T 0c57a5add4914835ac366efdcde4ee28 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:18.960870 1194 raft_consensus.cc:738] T 0c57a5add4914835ac366efdcde4ee28 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:18.961126 1194 consensus_queue.cc:260] T 0c57a5add4914835ac366efdcde4ee28 P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:18.962153 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 118) took 1156 ms. Trace:
I20250113 02:51:18.962272 1195 rpcz_store.cc:276] 0113 02:51:17.802495 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:17.802588 (+ 93us) service_pool.cc:225] Handling call
0113 02:51:18.958513 (+1155925us) ts_tablet_manager.cc:1145] Deleting tablet 1d81eec487dd4016b3b804451da31c49
0113 02:51:18.958523 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:18.958646 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1147824}
I20250113 02:51:18.962818 1195 tablet_replica.cc:331] T 9a323451cdfe4fd69ec62cda230f9afb P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:18.963014 1195 raft_consensus.cc:2238] T 9a323451cdfe4fd69ec62cda230f9afb P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:18.963374 1195 raft_consensus.cc:2267] T 9a323451cdfe4fd69ec62cda230f9afb P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:18.964051 1195 ts_tablet_manager.cc:1905] T 9a323451cdfe4fd69ec62cda230f9afb P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:18.965341 1198 log.cc:1198] T bc88676bf738442d82c25e56f30c4ada P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/bc88676bf738442d82c25e56f30c4ada
I20250113 02:51:18.965811 1198 ts_tablet_manager.cc:1939] T bc88676bf738442d82c25e56f30c4ada P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
W20250113 02:51:18.966379 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 1d81eec487dd4016b3b804451da31c49 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1d81eec487dd4016b3b804451da31c49 already in progress: creating tablet
I20250113 02:51:18.967008 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet bc88676bf738442d82c25e56f30c4ada (table default.Table_c991f9dbd70d4c4682664b8711f05db4 [id=2c64da64ecdb4bf48e11b523533902e8]) successfully deleted
I20250113 02:51:18.971769 1194 ts_tablet_manager.cc:1428] T 0c57a5add4914835ac366efdcde4ee28 P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.018s user 0.001s sys 0.002s
I20250113 02:51:18.972024 893 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c57a5add4914835ac366efdcde4ee28 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:17 UTC) from {username='slave'} at 127.0.0.1:37366
W20250113 02:51:18.972896 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet d35ec7e4a5084a74bea172ef8d9990b5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d35ec7e4a5084a74bea172ef8d9990b5 already in progress: creating tablet
W20250113 02:51:18.973191 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 501d4e1ae84e4795913035f8a19cc32d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 501d4e1ae84e4795913035f8a19cc32d already in progress: creating tablet
W20250113 02:51:18.973436 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 1c26074fdd804bd79cb0c295622d11ff because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1c26074fdd804bd79cb0c295622d11ff already in progress: creating tablet
I20250113 02:51:18.975711 1194 tablet_bootstrap.cc:492] T bc88676bf738442d82c25e56f30c4ada P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:18.978084 1033 tablet_service.cc:1467] Processing CreateTablet for tablet fdbf03c89bff438a9a25115eddf60be9 (DEFAULT_TABLE table=default.Table_732e990e6f154d6486889304bbf88cef [id=48c7c3f6b4a840ee9b9e3f3844add13d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:18.978776 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fdbf03c89bff438a9a25115eddf60be9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:18.979624 741 tablet_service.cc:1467] Processing CreateTablet for tablet fdbf03c89bff438a9a25115eddf60be9 (DEFAULT_TABLE table=default.Table_732e990e6f154d6486889304bbf88cef [id=48c7c3f6b4a840ee9b9e3f3844add13d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:18.980139 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fdbf03c89bff438a9a25115eddf60be9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:18.987334 1194 tablet_bootstrap.cc:654] T bc88676bf738442d82c25e56f30c4ada P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:18.990415 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet db266a1848094b7a98aa75f7afabd471 (table default.Table_87cfc114b15047b49fd25446ff7aa441 [id=ca79853d13ee4a76a18d9fa68c1feb31]) successfully deleted
W20250113 02:51:18.991434 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 5b3f012b88c04cb68ed7a1848147b70b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5b3f012b88c04cb68ed7a1848147b70b already in progress: creating tablet
W20250113 02:51:18.992413 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 6c8ca30720664f3a819c4bff2493934d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6c8ca30720664f3a819c4bff2493934d already in progress: creating tablet
I20250113 02:51:18.995163 1042 tablet_service.cc:1467] Processing CreateTablet for tablet 55ebe9c044d944bc813396c699ba9aa1 (DEFAULT_TABLE table=default.Table_2caed919541e43c78a858513a2128f50 [id=26ba87b9bd384b99a016b29fe5b5babd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:18.995630 1042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 55ebe9c044d944bc813396c699ba9aa1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:18.995692 889 tablet_service.cc:1467] Processing CreateTablet for tablet 55ebe9c044d944bc813396c699ba9aa1 (DEFAULT_TABLE table=default.Table_2caed919541e43c78a858513a2128f50 [id=26ba87b9bd384b99a016b29fe5b5babd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:18.996086 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 55ebe9c044d944bc813396c699ba9aa1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:19.003931 1042 tablet_service.cc:1467] Processing CreateTablet for tablet e8ab4516789e4585a101c8766d338af3 (DEFAULT_TABLE table=default.Table_0917607e26ce49bfad7e1dc99a894a05 [id=b112ac4e2e58460db083da1eb788ca07]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:19.004438 1042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e8ab4516789e4585a101c8766d338af3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:19.005072 893 tablet_service.cc:1467] Processing CreateTablet for tablet f760fc1e4bed433dbb987a42c12a1ee6 (DEFAULT_TABLE table=default.Table_c45864b4578e45b59a53d2d913da12f9 [id=f02cfa6472eb42de956f8591417eb7b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:19.006639 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_e45d864b62364107b47a45916574fed5" } new_table_name: "default.Table_45ad8711e23e44f0b7b83b7a6128c427" modify_external_catalogs: true
I20250113 02:51:19.006889 1029 tablet_service.cc:1467] Processing CreateTablet for tablet f760fc1e4bed433dbb987a42c12a1ee6 (DEFAULT_TABLE table=default.Table_c45864b4578e45b59a53d2d913da12f9 [id=f02cfa6472eb42de956f8591417eb7b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:19.007272 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f760fc1e4bed433dbb987a42c12a1ee6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:19.008831 896 tablet_service.cc:1467] Processing CreateTablet for tablet e8ab4516789e4585a101c8766d338af3 (DEFAULT_TABLE table=default.Table_0917607e26ce49bfad7e1dc99a894a05 [id=b112ac4e2e58460db083da1eb788ca07]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:19.010550 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_003cb5a672084be4b29d515fa4485747"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:19.011140 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_003cb5a672084be4b29d515fa4485747 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250113 02:51:19.015648 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet bc88676bf738442d82c25e56f30c4ada because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bc88676bf738442d82c25e56f30c4ada already in progress: creating tablet
I20250113 02:51:19.015739 1196 tablet_replica.cc:331] T 78689f7ed2a64405adb75f372465a792 P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:19.016002 1196 raft_consensus.cc:2238] T 78689f7ed2a64405adb75f372465a792 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:19.016206 1196 raft_consensus.cc:2267] T 78689f7ed2a64405adb75f372465a792 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:19.016858 1194 tablet_bootstrap.cc:492] T bc88676bf738442d82c25e56f30c4ada P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:19.017113 1194 ts_tablet_manager.cc:1397] T bc88676bf738442d82c25e56f30c4ada P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.041s user 0.004s sys 0.007s
I20250113 02:51:19.017357 1196 ts_tablet_manager.cc:1905] T 78689f7ed2a64405adb75f372465a792 P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:19.018514 895 tablet_service.cc:1467] Processing CreateTablet for tablet 2c399784af46426b8327d175c7a6cce8 (DEFAULT_TABLE table=default.Table_f8af5d3e0dc64d3fae054f82a7b33ccf [id=4b739228564344b8b394a4718750dbb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:19.019263 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c399784af46426b8327d175c7a6cce8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:19.019526 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f760fc1e4bed433dbb987a42c12a1ee6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:19.027065 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e8ab4516789e4585a101c8766d338af3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:19.018116 1194 raft_consensus.cc:357] T bc88676bf738442d82c25e56f30c4ada P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:19.031536 1194 raft_consensus.cc:383] T bc88676bf738442d82c25e56f30c4ada P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:19.033056 453 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_44f1c4e8cc2b4e83adce8d411ae53448" } modify_external_catalogs: true
I20250113 02:51:19.033291 453 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_44f1c4e8cc2b4e83adce8d411ae53448" } modify_external_catalogs: true
I20250113 02:51:19.035029 448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_059eaed9949d445198527eb449fd1447" } new_table_name: "default.Table_8b9dee7f78684d9c80977952c0e5a2df" modify_external_catalogs: true
I20250113 02:51:19.038244 1029 tablet_service.cc:1467] Processing CreateTablet for tablet 2c399784af46426b8327d175c7a6cce8 (DEFAULT_TABLE table=default.Table_f8af5d3e0dc64d3fae054f82a7b33ccf [id=4b739228564344b8b394a4718750dbb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:19.045480 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_9cfb6bce41c0468e8d29a6d0a1f32499"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:19.046108 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9cfb6bce41c0468e8d29a6d0a1f32499 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:19.046376 1192 tablet_bootstrap.cc:492] T c97a6ad98ddc4680a0f071cb91f05402 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:19.046558 1192 ts_tablet_manager.cc:1397] T c97a6ad98ddc4680a0f071cb91f05402 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.174s user 0.040s sys 0.075s
I20250113 02:51:19.047693 1192 raft_consensus.cc:357] T c97a6ad98ddc4680a0f071cb91f05402 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:19.047838 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_c45864b4578e45b59a53d2d913da12f9" } modify_external_catalogs: true
I20250113 02:51:19.048012 1192 raft_consensus.cc:383] T c97a6ad98ddc4680a0f071cb91f05402 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:19.048609 1192 raft_consensus.cc:738] T c97a6ad98ddc4680a0f071cb91f05402 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:19.048600 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_c45864b4578e45b59a53d2d913da12f9" } modify_external_catalogs: true
I20250113 02:51:19.048956 1192 consensus_queue.cc:260] T c97a6ad98ddc4680a0f071cb91f05402 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:19.050114 1192 ts_tablet_manager.cc:1428] T c97a6ad98ddc4680a0f071cb91f05402 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.003s user 0.000s sys 0.004s
I20250113 02:51:19.055130 1030 tablet_service.cc:1467] Processing CreateTablet for tablet 19f14b3db0b64847b5422bfd5ed323d0 (DEFAULT_TABLE table=default.Table_c7093afcc0ae47c78e7c5cf85219481c [id=141489c57ce44e96a33a1140a8c2b077]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:19.055251 1024 tablet_service.cc:1467] Processing CreateTablet for tablet f31dd916336a4d8fb3ab015af7de8ccb (DEFAULT_TABLE table=default.Table_37b1d9d091924f8d9a8a2cbf1d06af1a [id=4d85a53a49d642b798af44eef60b3f78]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:19.056764 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 157905316fc545b6a7864ae1c4a2a584
I20250113 02:51:19.057263 455 catalog_manager.cc:6596] ReplaceTablet: tablet 157905316fc545b6a7864ae1c4a2a584 deleted and replaced by tablet de4a58bf63054890808e2e2fe46e43d2
I20250113 02:51:19.057658 455 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_732e990e6f154d6486889304bbf88cef" } modify_external_catalogs: true
I20250113 02:51:19.057837 455 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_732e990e6f154d6486889304bbf88cef" } modify_external_catalogs: true
I20250113 02:51:19.058386 754 tablet_service.cc:1467] Processing CreateTablet for tablet 55ebe9c044d944bc813396c699ba9aa1 (DEFAULT_TABLE table=default.Table_2caed919541e43c78a858513a2128f50 [id=26ba87b9bd384b99a016b29fe5b5babd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:19.058861 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 55ebe9c044d944bc813396c699ba9aa1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:19.063558 1041 tablet_service.cc:1514] Processing DeleteTablet for tablet 157905316fc545b6a7864ae1c4a2a584 with delete_type TABLET_DATA_DELETED (replaced by tablet de4a58bf63054890808e2e2fe46e43d2) from {username='slave'} at 127.0.0.1:60230
W20250113 02:51:19.064451 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 64cda8796dff46ce95c37439395c77ec consensus state has no leader
I20250113 02:51:19.064868 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_668e70d7e5c64c37ab05e7bcbe56d994"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:19.065387 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_668e70d7e5c64c37ab05e7bcbe56d994 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:19.066917 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_41f119048882478f85a1344410383dac" } new_table_name: "default.Table_d7d1f5a34e5f4c05b474b0b61f912cd9" modify_external_catalogs: true
I20250113 02:51:19.069109 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c399784af46426b8327d175c7a6cce8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:19.069523 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19f14b3db0b64847b5422bfd5ed323d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:19.064544 1191 tablet_bootstrap.cc:492] T a028cc02273644608f046135851da126 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:19.079818 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f31dd916336a4d8fb3ab015af7de8ccb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:19.080693 1192 tablet_bootstrap.cc:492] T 9a323451cdfe4fd69ec62cda230f9afb P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:19.076808 739 tablet_service.cc:1467] Processing CreateTablet for tablet f760fc1e4bed433dbb987a42c12a1ee6 (DEFAULT_TABLE table=default.Table_c45864b4578e45b59a53d2d913da12f9 [id=f02cfa6472eb42de956f8591417eb7b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:19.082564 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f760fc1e4bed433dbb987a42c12a1ee6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:19.082749 1192 tablet_bootstrap.cc:654] T 9a323451cdfe4fd69ec62cda230f9afb P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:19.037127 1196 ts_tablet_manager.cc:1918] T 78689f7ed2a64405adb75f372465a792 P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:19.087865 1196 log.cc:1198] T 78689f7ed2a64405adb75f372465a792 P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/78689f7ed2a64405adb75f372465a792
I20250113 02:51:19.078065 743 tablet_service.cc:1467] Processing CreateTablet for tablet 19f14b3db0b64847b5422bfd5ed323d0 (DEFAULT_TABLE table=default.Table_c7093afcc0ae47c78e7c5cf85219481c [id=141489c57ce44e96a33a1140a8c2b077]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:19.088317 1196 ts_tablet_manager.cc:1939] T 78689f7ed2a64405adb75f372465a792 P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:19.088605 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19f14b3db0b64847b5422bfd5ed323d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:19.089653 1196 tablet_replica.cc:331] T ad6b21ed2b824cb0837fc1d8a39d2472 P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:19.089851 1196 raft_consensus.cc:2238] T ad6b21ed2b824cb0837fc1d8a39d2472 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:19.090060 1196 raft_consensus.cc:2267] T ad6b21ed2b824cb0837fc1d8a39d2472 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:19.040526 893 tablet_service.cc:1467] Processing CreateTablet for tablet 19f14b3db0b64847b5422bfd5ed323d0 (DEFAULT_TABLE table=default.Table_c7093afcc0ae47c78e7c5cf85219481c [id=141489c57ce44e96a33a1140a8c2b077]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:19.090653 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19f14b3db0b64847b5422bfd5ed323d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:19.090998 1196 ts_tablet_manager.cc:1905] T ad6b21ed2b824cb0837fc1d8a39d2472 P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:19.078130 753 tablet_service.cc:1514] Processing DeleteTablet for tablet 157905316fc545b6a7864ae1c4a2a584 with delete_type TABLET_DATA_DELETED (replaced by tablet de4a58bf63054890808e2e2fe46e43d2) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:19.061554 899 tablet_service.cc:1514] Processing DeleteTablet for tablet 157905316fc545b6a7864ae1c4a2a584 with delete_type TABLET_DATA_DELETED (replaced by tablet de4a58bf63054890808e2e2fe46e43d2) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:19.078601 737 tablet_service.cc:1467] Processing CreateTablet for tablet f31dd916336a4d8fb3ab015af7de8ccb (DEFAULT_TABLE table=default.Table_37b1d9d091924f8d9a8a2cbf1d06af1a [id=4d85a53a49d642b798af44eef60b3f78]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:19.092690 737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f31dd916336a4d8fb3ab015af7de8ccb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:19.078866 740 tablet_service.cc:1467] Processing CreateTablet for tablet 2c399784af46426b8327d175c7a6cce8 (DEFAULT_TABLE table=default.Table_f8af5d3e0dc64d3fae054f82a7b33ccf [id=4b739228564344b8b394a4718750dbb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:19.096423 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c399784af46426b8327d175c7a6cce8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:19.096925 1191 ts_tablet_manager.cc:1397] T a028cc02273644608f046135851da126 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.262s user 0.017s sys 0.008s
I20250113 02:51:19.097929 1191 raft_consensus.cc:357] T a028cc02273644608f046135851da126 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:19.087881 1194 raft_consensus.cc:738] T bc88676bf738442d82c25e56f30c4ada P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:19.098184 1191 raft_consensus.cc:383] T a028cc02273644608f046135851da126 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:19.098268 1191 raft_consensus.cc:738] T a028cc02273644608f046135851da126 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:19.098414 1194 consensus_queue.cc:260] T bc88676bf738442d82c25e56f30c4ada P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:19.098532 1191 consensus_queue.cc:260] T a028cc02273644608f046135851da126 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:19.100394 1194 ts_tablet_manager.cc:1428] T bc88676bf738442d82c25e56f30c4ada P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.083s user 0.001s sys 0.001s
I20250113 02:51:19.100788 1194 tablet_bootstrap.cc:492] T c135506190f849008ce6542d569b402a P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:19.101339 1203 raft_consensus.cc:491] T 0c57a5add4914835ac366efdcde4ee28 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:19.101500 1203 raft_consensus.cc:513] T 0c57a5add4914835ac366efdcde4ee28 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:19.102193 1203 leader_election.cc:290] T 0c57a5add4914835ac366efdcde4ee28 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:19.077922 749 tablet_service.cc:1467] Processing CreateTablet for tablet e8ab4516789e4585a101c8766d338af3 (DEFAULT_TABLE table=default.Table_0917607e26ce49bfad7e1dc99a894a05 [id=b112ac4e2e58460db083da1eb788ca07]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:19.102948 1194 tablet_bootstrap.cc:654] T c135506190f849008ce6542d569b402a P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:19.102981 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e8ab4516789e4585a101c8766d338af3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:19.104310 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c57a5add4914835ac366efdcde4ee28" candidate_uuid: "df69f48656b44bb890f76587ee69cc0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
W20250113 02:51:19.106058 851 leader_election.cc:343] T 0c57a5add4914835ac366efdcde4ee28 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:19.107362 1062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c57a5add4914835ac366efdcde4ee28" candidate_uuid: "df69f48656b44bb890f76587ee69cc0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:19.102556 1191 ts_tablet_manager.cc:1428] T a028cc02273644608f046135851da126 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
W20250113 02:51:19.108088 441 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 98f038ad31ff4e05bb88ac68ae341cc2 (table default.Table_4fcd958510914007817364e5a8e8ebec [id=6666a614063942c99a78b95d35add037], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 98f038ad31ff4e05bb88ac68ae341cc2 consensus state has no leader
W20250113 02:51:19.108280 852 leader_election.cc:343] T 0c57a5add4914835ac366efdcde4ee28 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:19.108395 852 leader_election.cc:304] T 0c57a5add4914835ac366efdcde4ee28 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df69f48656b44bb890f76587ee69cc0f; no voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca
I20250113 02:51:19.110217 1194 tablet_bootstrap.cc:492] T c135506190f849008ce6542d569b402a P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:19.110888 1194 ts_tablet_manager.cc:1397] T c135506190f849008ce6542d569b402a P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.005s
I20250113 02:51:19.111166 1196 ts_tablet_manager.cc:1918] T ad6b21ed2b824cb0837fc1d8a39d2472 P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:19.111281 1196 log.cc:1198] T ad6b21ed2b824cb0837fc1d8a39d2472 P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/ad6b21ed2b824cb0837fc1d8a39d2472
I20250113 02:51:19.111696 1196 ts_tablet_manager.cc:1939] T ad6b21ed2b824cb0837fc1d8a39d2472 P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:19.112666 1194 raft_consensus.cc:357] T c135506190f849008ce6542d569b402a P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:19.113699 1194 raft_consensus.cc:383] T c135506190f849008ce6542d569b402a P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:19.113981 1194 raft_consensus.cc:738] T c135506190f849008ce6542d569b402a P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:19.112892 1196 tablet_replica.cc:331] T c97a6ad98ddc4680a0f071cb91f05402 P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:19.115445 1191 tablet_bootstrap.cc:492] T 0c57a5add4914835ac366efdcde4ee28 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:19.113183 1203 raft_consensus.cc:2743] T 0c57a5add4914835ac366efdcde4ee28 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:19.115028 1194 consensus_queue.cc:260] T c135506190f849008ce6542d569b402a P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:19.116950 1194 ts_tablet_manager.cc:1428] T c135506190f849008ce6542d569b402a P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20250113 02:51:19.117244 1194 tablet_bootstrap.cc:492] T ce92c2b45a274eb09e72a96b7ec133be P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:19.117475 1196 raft_consensus.cc:2238] T c97a6ad98ddc4680a0f071cb91f05402 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:19.117666 1196 raft_consensus.cc:2267] T c97a6ad98ddc4680a0f071cb91f05402 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:19.118148 1192 tablet_bootstrap.cc:492] T 9a323451cdfe4fd69ec62cda230f9afb P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:19.118353 1192 ts_tablet_manager.cc:1397] T 9a323451cdfe4fd69ec62cda230f9afb P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.038s user 0.001s sys 0.008s
I20250113 02:51:19.118523 1196 ts_tablet_manager.cc:1905] T c97a6ad98ddc4680a0f071cb91f05402 P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:19.119314 1192 raft_consensus.cc:357] T 9a323451cdfe4fd69ec62cda230f9afb P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:19.119813 1192 raft_consensus.cc:383] T 9a323451cdfe4fd69ec62cda230f9afb P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:19.119961 1192 raft_consensus.cc:738] T 9a323451cdfe4fd69ec62cda230f9afb P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:19.120334 1192 consensus_queue.cc:260] T 9a323451cdfe4fd69ec62cda230f9afb P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:19.123683 1191 tablet_bootstrap.cc:654] T 0c57a5add4914835ac366efdcde4ee28 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:19.128229 1192 ts_tablet_manager.cc:1428] T 9a323451cdfe4fd69ec62cda230f9afb P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20250113 02:51:19.133080 1194 tablet_bootstrap.cc:654] T ce92c2b45a274eb09e72a96b7ec133be P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:19.136477 1196 ts_tablet_manager.cc:1918] T c97a6ad98ddc4680a0f071cb91f05402 P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:19.142022 1196 log.cc:1198] T c97a6ad98ddc4680a0f071cb91f05402 P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/c97a6ad98ddc4680a0f071cb91f05402
I20250113 02:51:19.142570 1196 ts_tablet_manager.cc:1939] T c97a6ad98ddc4680a0f071cb91f05402 P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:19.143277 1192 tablet_bootstrap.cc:492] T db266a1848094b7a98aa75f7afabd471 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:19.144380 1196 tablet_replica.cc:331] T ec0750fd2fa247feb3ea310ffabe05e5 P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:19.144603 1196 raft_consensus.cc:2238] T ec0750fd2fa247feb3ea310ffabe05e5 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:19.145982 1192 tablet_bootstrap.cc:654] T db266a1848094b7a98aa75f7afabd471 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:19.148605 1196 raft_consensus.cc:2267] T ec0750fd2fa247feb3ea310ffabe05e5 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:19.149107 1194 tablet_bootstrap.cc:492] T ce92c2b45a274eb09e72a96b7ec133be P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:19.149260 1194 ts_tablet_manager.cc:1397] T ce92c2b45a274eb09e72a96b7ec133be P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.032s user 0.004s sys 0.004s
I20250113 02:51:19.150079 1194 raft_consensus.cc:357] T ce92c2b45a274eb09e72a96b7ec133be P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:19.150282 1194 raft_consensus.cc:383] T ce92c2b45a274eb09e72a96b7ec133be P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:19.150344 1194 raft_consensus.cc:738] T ce92c2b45a274eb09e72a96b7ec133be P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:19.150640 1194 consensus_queue.cc:260] T ce92c2b45a274eb09e72a96b7ec133be P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:19.151857 1196 ts_tablet_manager.cc:1905] T ec0750fd2fa247feb3ea310ffabe05e5 P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:19.152099 1194 ts_tablet_manager.cc:1428] T ce92c2b45a274eb09e72a96b7ec133be P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250113 02:51:19.152427 1194 tablet_bootstrap.cc:492] T 8193357427f4491c8c6c1396ceeef2dc P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:19.154525 1194 tablet_bootstrap.cc:654] T 8193357427f4491c8c6c1396ceeef2dc P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:19.162777 1196 ts_tablet_manager.cc:1918] T ec0750fd2fa247feb3ea310ffabe05e5 P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:19.162914 1196 log.cc:1198] T ec0750fd2fa247feb3ea310ffabe05e5 P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/ec0750fd2fa247feb3ea310ffabe05e5
I20250113 02:51:19.077876 895 tablet_service.cc:1467] Processing CreateTablet for tablet f31dd916336a4d8fb3ab015af7de8ccb (DEFAULT_TABLE table=default.Table_37b1d9d091924f8d9a8a2cbf1d06af1a [id=4d85a53a49d642b798af44eef60b3f78]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:19.163355 1196 ts_tablet_manager.cc:1939] T ec0750fd2fa247feb3ea310ffabe05e5 P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:19.163540 1194 tablet_bootstrap.cc:492] T 8193357427f4491c8c6c1396ceeef2dc P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:19.163808 1194 ts_tablet_manager.cc:1397] T 8193357427f4491c8c6c1396ceeef2dc P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.001s
I20250113 02:51:19.165167 1192 tablet_bootstrap.cc:492] T db266a1848094b7a98aa75f7afabd471 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:19.165164 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f31dd916336a4d8fb3ab015af7de8ccb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:19.165339 1192 ts_tablet_manager.cc:1397] T db266a1848094b7a98aa75f7afabd471 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.033s user 0.003s sys 0.000s
I20250113 02:51:19.165119 1194 raft_consensus.cc:357] T 8193357427f4491c8c6c1396ceeef2dc P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:19.166503 1194 raft_consensus.cc:383] T 8193357427f4491c8c6c1396ceeef2dc P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:19.166837 1194 raft_consensus.cc:738] T 8193357427f4491c8c6c1396ceeef2dc P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:19.166955 1192 raft_consensus.cc:357] T db266a1848094b7a98aa75f7afabd471 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:19.167223 1192 raft_consensus.cc:383] T db266a1848094b7a98aa75f7afabd471 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:19.167328 1192 raft_consensus.cc:738] T db266a1848094b7a98aa75f7afabd471 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:19.167501 1194 consensus_queue.cc:260] T 8193357427f4491c8c6c1396ceeef2dc P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:19.167584 1192 consensus_queue.cc:260] T db266a1848094b7a98aa75f7afabd471 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:19.168555 1194 ts_tablet_manager.cc:1428] T 8193357427f4491c8c6c1396ceeef2dc P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250113 02:51:19.170043 1194 tablet_bootstrap.cc:492] T 0c55662a71444a3683a825bcec6c5b68 P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:19.170115 1192 ts_tablet_manager.cc:1428] T db266a1848094b7a98aa75f7afabd471 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20250113 02:51:19.170630 1192 tablet_bootstrap.cc:492] T 0c57a5add4914835ac366efdcde4ee28 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:19.172418 1194 tablet_bootstrap.cc:654] T 0c55662a71444a3683a825bcec6c5b68 P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:19.174840 1192 tablet_bootstrap.cc:654] T 0c57a5add4914835ac366efdcde4ee28 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:19.176193 1196 tablet_replica.cc:331] T 0c57a5add4914835ac366efdcde4ee28 P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:19.176455 1196 raft_consensus.cc:2238] T 0c57a5add4914835ac366efdcde4ee28 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:19.176651 1196 raft_consensus.cc:2267] T 0c57a5add4914835ac366efdcde4ee28 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:19.182519 1194 tablet_bootstrap.cc:492] T 0c55662a71444a3683a825bcec6c5b68 P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:19.183769 1194 ts_tablet_manager.cc:1397] T 0c55662a71444a3683a825bcec6c5b68 P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.000s
I20250113 02:51:19.185262 1194 raft_consensus.cc:357] T 0c55662a71444a3683a825bcec6c5b68 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:19.186002 1191 tablet_bootstrap.cc:492] T 0c57a5add4914835ac366efdcde4ee28 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
W20250113 02:51:19.186007 451 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet de4a58bf63054890808e2e2fe46e43d2 has no consensus state
I20250113 02:51:19.186162 1191 ts_tablet_manager.cc:1397] T 0c57a5add4914835ac366efdcde4ee28 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.077s user 0.006s sys 0.000s
I20250113 02:51:19.186947 1194 raft_consensus.cc:383] T 0c55662a71444a3683a825bcec6c5b68 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:19.187150 1194 raft_consensus.cc:738] T 0c55662a71444a3683a825bcec6c5b68 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:19.187487 451 master_service.cc:946] ReplaceTablet: received request to replace tablet 53cc528d282c47788d5d558569f074c1 from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:19.187563 1194 consensus_queue.cc:260] T 0c55662a71444a3683a825bcec6c5b68 P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:19.187745 1191 raft_consensus.cc:357] T 0c57a5add4914835ac366efdcde4ee28 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:19.187994 1191 raft_consensus.cc:383] T 0c57a5add4914835ac366efdcde4ee28 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:19.188145 1192 tablet_bootstrap.cc:492] T 0c57a5add4914835ac366efdcde4ee28 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:19.188417 1192 ts_tablet_manager.cc:1397] T 0c57a5add4914835ac366efdcde4ee28 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.008s
I20250113 02:51:19.188629 1194 ts_tablet_manager.cc:1428] T 0c55662a71444a3683a825bcec6c5b68 P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250113 02:51:19.189014 1194 tablet_bootstrap.cc:492] T ad9b95c5450a4c9c89a0930862348d49 P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:19.189075 1191 raft_consensus.cc:738] T 0c57a5add4914835ac366efdcde4ee28 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:19.189339 1191 consensus_queue.cc:260] T 0c57a5add4914835ac366efdcde4ee28 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:19.190001 1192 raft_consensus.cc:357] T 0c57a5add4914835ac366efdcde4ee28 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:19.190304 1192 raft_consensus.cc:383] T 0c57a5add4914835ac366efdcde4ee28 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:19.190388 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet f760fc1e4bed433dbb987a42c12a1ee6
I20250113 02:51:19.190438 1192 raft_consensus.cc:738] T 0c57a5add4914835ac366efdcde4ee28 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:19.190590 1191 ts_tablet_manager.cc:1428] T 0c57a5add4914835ac366efdcde4ee28 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250113 02:51:19.190881 1191 tablet_bootstrap.cc:492] T c135506190f849008ce6542d569b402a P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
W20250113 02:51:19.191222 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet eb4a985261f4452db4de74891b7bf65a consensus state has no leader
I20250113 02:51:19.190922 1192 consensus_queue.cc:260] T 0c57a5add4914835ac366efdcde4ee28 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:19.191828 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet a6055d77d168495d9b756a1a56c1a920
I20250113 02:51:19.192350 1192 ts_tablet_manager.cc:1428] T 0c57a5add4914835ac366efdcde4ee28 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250113 02:51:19.192921 1192 tablet_bootstrap.cc:492] T c135506190f849008ce6542d569b402a P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
W20250113 02:51:19.195009 448 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2fb62cc83f9c4dcfa9aa01ae8b46c025 consensus state has no leader
I20250113 02:51:19.195276 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet 78689f7ed2a64405adb75f372465a792 (table default.Table_f59080d9d91849deafd22506a1af8ff5 [id=d5354129133b446b8a4a5613db8bc411]) successfully deleted
W20250113 02:51:19.195502 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet d35ec7e4a5084a74bea172ef8d9990b5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d35ec7e4a5084a74bea172ef8d9990b5 already in progress: creating tablet
I20250113 02:51:19.196521 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet ad6b21ed2b824cb0837fc1d8a39d2472 (table default.Table_449479ad3dbb4c1aaa5cd954816b9935 [id=61e1ae59b7db4fd0a33fe3f73a712028]) successfully deleted
I20250113 02:51:19.196707 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet c97a6ad98ddc4680a0f071cb91f05402 (table default.Table_351e5f6de7c64dafb0d9ee4d86510d01 [id=a383dc51c39e4c8ba4b421119c752470]) successfully deleted
W20250113 02:51:19.197696 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 157905316fc545b6a7864ae1c4a2a584 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 157905316fc545b6a7864ae1c4a2a584 already in progress: creating tablet
I20250113 02:51:19.198799 1192 tablet_bootstrap.cc:654] T c135506190f849008ce6542d569b402a P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:19.200013 1191 tablet_bootstrap.cc:654] T c135506190f849008ce6542d569b402a P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:19.199909 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet c97a6ad98ddc4680a0f071cb91f05402 with delete_type TABLET_DATA_DELETED (replaced by tablet 2fe235d5280944ef92cf15fb458f2f81) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:19.200489 1198 tablet_replica.cc:331] T c97a6ad98ddc4680a0f071cb91f05402 P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:19.200729 1198 raft_consensus.cc:2238] T c97a6ad98ddc4680a0f071cb91f05402 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:19.200935 1198 raft_consensus.cc:2267] T c97a6ad98ddc4680a0f071cb91f05402 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:19.201319 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_c7093afcc0ae47c78e7c5cf85219481c" } modify_external_catalogs: true
I20250113 02:51:19.201393 1033 tablet_service.cc:1514] Processing DeleteTablet for tablet f760fc1e4bed433dbb987a42c12a1ee6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:19 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:19.201515 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_c7093afcc0ae47c78e7c5cf85219481c" } modify_external_catalogs: true
I20250113 02:51:19.202263 1198 ts_tablet_manager.cc:1905] T c97a6ad98ddc4680a0f071cb91f05402 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250113 02:51:19.204346 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 1c26074fdd804bd79cb0c295622d11ff because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1c26074fdd804bd79cb0c295622d11ff already in progress: creating tablet
I20250113 02:51:19.204566 1033 tablet_service.cc:1514] Processing DeleteTablet for tablet a6055d77d168495d9b756a1a56c1a920 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:19 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:19.203079 1195 ts_tablet_manager.cc:1918] T 9a323451cdfe4fd69ec62cda230f9afb P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:19.205025 1195 log.cc:1198] T 9a323451cdfe4fd69ec62cda230f9afb P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/9a323451cdfe4fd69ec62cda230f9afb
I20250113 02:51:19.205469 1195 ts_tablet_manager.cc:1939] T 9a323451cdfe4fd69ec62cda230f9afb P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
W20250113 02:51:19.205991 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 501d4e1ae84e4795913035f8a19cc32d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 501d4e1ae84e4795913035f8a19cc32d already in progress: creating tablet
I20250113 02:51:19.206852 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 131) took 1364 ms. Trace:
I20250113 02:51:19.208487 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_5aafd46e1cbf4e03a5b87797d5d97bdd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:19.209122 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5aafd46e1cbf4e03a5b87797d5d97bdd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:19.209105 1195 rpcz_store.cc:276] 0113 02:51:17.842634 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:17.843828 (+ 1194us) service_pool.cc:225] Handling call
0113 02:51:18.962662 (+1118834us) ts_tablet_manager.cc:1145] Deleting tablet 9a323451cdfe4fd69ec62cda230f9afb
0113 02:51:18.962673 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:18.983226 (+ 20553us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:19.202938 (+219712us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:19.206814 (+ 3876us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1117524}
I20250113 02:51:19.209244 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_22cf65ed324549aa9fa3c6e305bf0e1c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:19.209703 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_22cf65ed324549aa9fa3c6e305bf0e1c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:19.209898 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet 9a323451cdfe4fd69ec62cda230f9afb (table default.Table_2106a7d1900a44df83cf7ec9f377687d [id=a2c08f472dd7494ab8aff87e9aa43028]) successfully deleted
I20250113 02:51:19.210356 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 130) took 1367 ms. Trace:
I20250113 02:51:19.211283 1198 ts_tablet_manager.cc:1918] T c97a6ad98ddc4680a0f071cb91f05402 P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:19.212049 1195 rpcz_store.cc:276] 0113 02:51:17.842549 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:17.843802 (+ 1253us) service_pool.cc:225] Handling call
0113 02:51:19.210208 (+1366406us) ts_tablet_manager.cc:1145] Deleting tablet 076ed7744da54ddebd8fc99686beb6ae
0113 02:51:19.210220 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:19.210328 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1338663}
I20250113 02:51:19.213281 1195 tablet_replica.cc:331] T 0c57a5add4914835ac366efdcde4ee28 P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:19.213522 1195 raft_consensus.cc:2238] T 0c57a5add4914835ac366efdcde4ee28 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:19.214537 1194 tablet_bootstrap.cc:654] T ad9b95c5450a4c9c89a0930862348d49 P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:19.214798 1198 log.cc:1198] T c97a6ad98ddc4680a0f071cb91f05402 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/c97a6ad98ddc4680a0f071cb91f05402
W20250113 02:51:19.215166 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 076ed7744da54ddebd8fc99686beb6ae because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 076ed7744da54ddebd8fc99686beb6ae already in progress: creating tablet
I20250113 02:51:19.215322 1198 ts_tablet_manager.cc:1939] T c97a6ad98ddc4680a0f071cb91f05402 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:19.215647 1195 raft_consensus.cc:2267] T 0c57a5add4914835ac366efdcde4ee28 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:19.216900 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet ec0750fd2fa247feb3ea310ffabe05e5 (table default.Table_74627d276edf4d75a2fcaa627a7f921b [id=de6b4835f6c04c20b6a5ef8404dc78f7]) successfully deleted
I20250113 02:51:19.218390 896 tablet_service.cc:1514] Processing DeleteTablet for tablet a6055d77d168495d9b756a1a56c1a920 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:19 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:19.218621 893 tablet_service.cc:1514] Processing DeleteTablet for tablet bc88676bf738442d82c25e56f30c4ada with delete_type TABLET_DATA_DELETED (replaced by tablet 401c4f5dbd304950877b59a865f417b3) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:19.219350 1033 tablet_service.cc:1514] Processing DeleteTablet for tablet db266a1848094b7a98aa75f7afabd471 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:18 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:19.219787 895 tablet_service.cc:1514] Processing DeleteTablet for tablet f760fc1e4bed433dbb987a42c12a1ee6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:19 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:19.221280 1198 tablet_replica.cc:331] T db266a1848094b7a98aa75f7afabd471 P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:19.221625 1198 raft_consensus.cc:2238] T db266a1848094b7a98aa75f7afabd471 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:19.221920 1198 raft_consensus.cc:2267] T db266a1848094b7a98aa75f7afabd471 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:19.223129 739 tablet_service.cc:1514] Processing DeleteTablet for tablet a6055d77d168495d9b756a1a56c1a920 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:19 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:19.223239 1198 ts_tablet_manager.cc:1905] T db266a1848094b7a98aa75f7afabd471 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:19.224443 737 tablet_service.cc:1514] Processing DeleteTablet for tablet f760fc1e4bed433dbb987a42c12a1ee6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:19 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:19.224987 737 tablet_service.cc:1514] Processing DeleteTablet for tablet a028cc02273644608f046135851da126 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:16 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:19.225734 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_5b825152ff4d4e20a7600a9ad0dfe0a3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:19.227252 1192 tablet_bootstrap.cc:492] T c135506190f849008ce6542d569b402a P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:19.228060 1192 ts_tablet_manager.cc:1397] T c135506190f849008ce6542d569b402a P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.035s user 0.008s sys 0.006s
I20250113 02:51:19.229326 1194 tablet_bootstrap.cc:492] T ad9b95c5450a4c9c89a0930862348d49 P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:19.229323 1192 raft_consensus.cc:357] T c135506190f849008ce6542d569b402a P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:19.229554 1194 ts_tablet_manager.cc:1397] T ad9b95c5450a4c9c89a0930862348d49 P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.041s user 0.001s sys 0.004s
I20250113 02:51:19.230518 1196 ts_tablet_manager.cc:1905] T 0c57a5add4914835ac366efdcde4ee28 P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:19.230875 1192 raft_consensus.cc:383] T c135506190f849008ce6542d569b402a P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:19.230841 1194 raft_consensus.cc:357] T ad9b95c5450a4c9c89a0930862348d49 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:19.231165 1192 raft_consensus.cc:738] T c135506190f849008ce6542d569b402a P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:19.231170 1194 raft_consensus.cc:383] T ad9b95c5450a4c9c89a0930862348d49 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:19.231294 1194 raft_consensus.cc:738] T ad9b95c5450a4c9c89a0930862348d49 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:19.231614 1192 consensus_queue.cc:260] T c135506190f849008ce6542d569b402a P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:19.231715 1194 consensus_queue.cc:260] T ad9b95c5450a4c9c89a0930862348d49 P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:19.233783 1194 ts_tablet_manager.cc:1428] T ad9b95c5450a4c9c89a0930862348d49 P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250113 02:51:19.235852 1192 ts_tablet_manager.cc:1428] T c135506190f849008ce6542d569b402a P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20250113 02:51:19.236105 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_796801d17a3c458693693af6e39cf0f5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:19.236446 1195 ts_tablet_manager.cc:1905] T 0c57a5add4914835ac366efdcde4ee28 P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:19.238449 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet fdbf03c89bff438a9a25115eddf60be9
I20250113 02:51:19.239495 1192 tablet_bootstrap.cc:492] T ad6b21ed2b824cb0837fc1d8a39d2472 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:19.235035 1298 raft_consensus.cc:491] T bc88676bf738442d82c25e56f30c4ada P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:19.240888 1298 raft_consensus.cc:513] T bc88676bf738442d82c25e56f30c4ada P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:19.242121 1298 leader_election.cc:290] T bc88676bf738442d82c25e56f30c4ada P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:19.242663 1194 tablet_bootstrap.cc:492] T 15a05a32febd443dbb519c186048428a P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:19.242691 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc88676bf738442d82c25e56f30c4ada" candidate_uuid: "df69f48656b44bb890f76587ee69cc0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
I20250113 02:51:19.242928 775 raft_consensus.cc:2463] T bc88676bf738442d82c25e56f30c4ada P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df69f48656b44bb890f76587ee69cc0f in term 0.
W20250113 02:51:19.244841 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5b825152ff4d4e20a7600a9ad0dfe0a3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:19.245944 1192 tablet_bootstrap.cc:654] T ad6b21ed2b824cb0837fc1d8a39d2472 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:19.246008 1191 tablet_bootstrap.cc:492] T c135506190f849008ce6542d569b402a P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:19.246230 1191 ts_tablet_manager.cc:1397] T c135506190f849008ce6542d569b402a P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.055s user 0.001s sys 0.003s
I20250113 02:51:19.240164 754 tablet_service.cc:1514] Processing DeleteTablet for tablet fdbf03c89bff438a9a25115eddf60be9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:19 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:19.246834 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 19f14b3db0b64847b5422bfd5ed323d0
I20250113 02:51:19.247448 1191 raft_consensus.cc:357] T c135506190f849008ce6542d569b402a P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:19.247741 1191 raft_consensus.cc:383] T c135506190f849008ce6542d569b402a P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:19.247875 1191 raft_consensus.cc:738] T c135506190f849008ce6542d569b402a P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:19.248224 1191 consensus_queue.cc:260] T c135506190f849008ce6542d569b402a P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
W20250113 02:51:19.248634 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_796801d17a3c458693693af6e39cf0f5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:19.248876 455 master_service.cc:946] ReplaceTablet: received request to replace tablet 3455f010e51b4db1a9804a57fd1a4fb8 from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:19.249177 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 53cc528d282c47788d5d558569f074c1
I20250113 02:51:19.248944 1191 ts_tablet_manager.cc:1428] T c135506190f849008ce6542d569b402a P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20250113 02:51:19.249637 451 catalog_manager.cc:6596] ReplaceTablet: tablet 53cc528d282c47788d5d558569f074c1 deleted and replaced by tablet 7d3562a48c6e4badbbcf37640e91b5ba
I20250113 02:51:19.250113 1191 tablet_bootstrap.cc:492] T ad6b21ed2b824cb0837fc1d8a39d2472 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:19.250355 1209 raft_consensus.cc:491] T c135506190f849008ce6542d569b402a P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:19.250542 1209 raft_consensus.cc:513] T c135506190f849008ce6542d569b402a P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:19.251690 1209 leader_election.cc:290] T c135506190f849008ce6542d569b402a P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:19.252008 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_3f75f23afe2f449fa7a1f9df1c0734fb" } modify_external_catalogs: true
I20250113 02:51:19.252180 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_3f75f23afe2f449fa7a1f9df1c0734fb" } modify_external_catalogs: true
I20250113 02:51:19.252183 1062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c135506190f849008ce6542d569b402a" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:19.252465 1062 raft_consensus.cc:2463] T c135506190f849008ce6542d569b402a P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1186fa869eb14812832fa6562531f196 in term 0.
I20250113 02:51:19.252938 709 leader_election.cc:304] T c135506190f849008ce6542d569b402a P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters:
I20250113 02:51:19.253309 1209 raft_consensus.cc:2798] T c135506190f849008ce6542d569b402a P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 02:51:19.253422 1209 raft_consensus.cc:491] T c135506190f849008ce6542d569b402a P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 02:51:19.253509 1209 raft_consensus.cc:3054] T c135506190f849008ce6542d569b402a P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Advancing to term 1
W20250113 02:51:19.254278 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 438d1d2300e044f39a15586f335935ba consensus state has no leader
I20250113 02:51:19.254530 446 master_service.cc:946] ReplaceTablet: received request to replace tablet b834f0acf5314bcb9d81c73de1b31842 from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:19.257002 1209 raft_consensus.cc:513] T c135506190f849008ce6542d569b402a P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:19.257408 450 master_service.cc:946] ReplaceTablet: received request to replace tablet 55ebe9c044d944bc813396c699ba9aa1 from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:19.258092 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_7761343ccac947069c26e1e705e0a9b3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:19.258611 737 tablet_service.cc:1514] Processing DeleteTablet for tablet 53cc528d282c47788d5d558569f074c1 with delete_type TABLET_DATA_DELETED (replaced by tablet 7d3562a48c6e4badbbcf37640e91b5ba) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:19.258889 754 tablet_service.cc:1514] Processing DeleteTablet for tablet 19f14b3db0b64847b5422bfd5ed323d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:19 UTC) from {username='slave'} at 127.0.0.1:55398
W20250113 02:51:19.258939 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7761343ccac947069c26e1e705e0a9b3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:19.265205 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_67151d370cfd45ccaac504631aa0f1f9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:19.265862 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_67151d370cfd45ccaac504631aa0f1f9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:19.268842 889 tablet_service.cc:1514] Processing DeleteTablet for tablet c135506190f849008ce6542d569b402a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:16 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:19.268975 1062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c135506190f849008ce6542d569b402a" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca"
I20250113 02:51:19.269004 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_154343289ac5428da04f6f330235d2a0" } new_table_name: "default.Table_3d6c2586a98845c7989a5b715c4d678e" modify_external_catalogs: true
I20250113 02:51:19.269165 894 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c55662a71444a3683a825bcec6c5b68 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:16 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:19.269214 1062 raft_consensus.cc:3054] T c135506190f849008ce6542d569b402a P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:19.269402 891 tablet_service.cc:1514] Processing DeleteTablet for tablet ce92c2b45a274eb09e72a96b7ec133be with delete_type TABLET_DATA_DELETED (replaced by tablet 53cc528d282c47788d5d558569f074c1) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:19.269891 899 tablet_service.cc:1514] Processing DeleteTablet for tablet 8193357427f4491c8c6c1396ceeef2dc with delete_type TABLET_DATA_DELETED (replaced by tablet d3b695ac20ba4205b2d8732db4740531) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:19.273133 453 catalog_manager.cc:5618] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095) does not have the latest schema for tablet 7f3cbef7d91d4576822d913071a88248 (table default.Table_b581b1a063a54551a8535adf01983fb8 [id=54e4e3810b3b412bbec3b5bf5b28c1a2]). Expected version 1 got 0
I20250113 02:51:19.273188 1062 raft_consensus.cc:2463] T c135506190f849008ce6542d569b402a P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1186fa869eb14812832fa6562531f196 in term 1.
I20250113 02:51:19.243064 895 tablet_service.cc:1514] Processing DeleteTablet for tablet fdbf03c89bff438a9a25115eddf60be9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:19 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:19.243392 851 leader_election.cc:304] T bc88676bf738442d82c25e56f30c4ada P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1186fa869eb14812832fa6562531f196, df69f48656b44bb890f76587ee69cc0f; no voters:
W20250113 02:51:19.274761 852 leader_election.cc:343] T bc88676bf738442d82c25e56f30c4ada P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Not found: Tablet not found: bc88676bf738442d82c25e56f30c4ada
I20250113 02:51:19.247855 893 tablet_service.cc:1514] Processing DeleteTablet for tablet 19f14b3db0b64847b5422bfd5ed323d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:19 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:19.252502 896 tablet_service.cc:1514] Processing DeleteTablet for tablet 53cc528d282c47788d5d558569f074c1 with delete_type TABLET_DATA_DELETED (replaced by tablet 7d3562a48c6e4badbbcf37640e91b5ba) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:19.276255 1298 raft_consensus.cc:2798] T bc88676bf738442d82c25e56f30c4ada P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 02:51:19.278151 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_616eda5ca6ce4155b20351b6b8b000f5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:19.278630 1298 raft_consensus.cc:491] T bc88676bf738442d82c25e56f30c4ada P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 02:51:19.278687 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet b834f0acf5314bcb9d81c73de1b31842
I20250113 02:51:19.278796 1298 raft_consensus.cc:3054] T bc88676bf738442d82c25e56f30c4ada P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:19.279242 446 catalog_manager.cc:6596] ReplaceTablet: tablet b834f0acf5314bcb9d81c73de1b31842 deleted and replaced by tablet be4b2a8a3927410eaf12e7592646e4af
I20250113 02:51:19.279537 446 master_service.cc:946] ReplaceTablet: received request to replace tablet 21c4b419bf2148e0b140301fa80622bb from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:19.280081 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet ba61c86743e24666b30580c36b9228c3
W20250113 02:51:19.280160 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_616eda5ca6ce4155b20351b6b8b000f5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:19.283411 896 tablet_service.cc:1514] Processing DeleteTablet for tablet b834f0acf5314bcb9d81c73de1b31842 with delete_type TABLET_DATA_DELETED (replaced by tablet be4b2a8a3927410eaf12e7592646e4af) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:19.283622 893 tablet_service.cc:1514] Processing DeleteTablet for tablet ba61c86743e24666b30580c36b9228c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:19 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:19.285092 1033 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a323451cdfe4fd69ec62cda230f9afb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:16 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:19.285693 1033 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c57a5add4914835ac366efdcde4ee28 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:17 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:19.286624 1033 tablet_service.cc:1514] Processing DeleteTablet for tablet fdbf03c89bff438a9a25115eddf60be9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:19 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:19.287173 1033 tablet_service.cc:1514] Processing DeleteTablet for tablet 19f14b3db0b64847b5422bfd5ed323d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:19 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:19.287768 1033 tablet_service.cc:1514] Processing DeleteTablet for tablet 53cc528d282c47788d5d558569f074c1 with delete_type TABLET_DATA_DELETED (replaced by tablet 7d3562a48c6e4badbbcf37640e91b5ba) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:19.288509 1298 raft_consensus.cc:513] T bc88676bf738442d82c25e56f30c4ada P df69f48656b44bb890f76587ee69cc0f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:19.289443 1298 leader_election.cc:290] T bc88676bf738442d82c25e56f30c4ada P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 election: Requested vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:19.289978 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c135506190f849008ce6542d569b402a" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:19.290196 918 raft_consensus.cc:2463] T c135506190f849008ce6542d569b402a P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1186fa869eb14812832fa6562531f196 in term 0.
I20250113 02:51:19.290170 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c135506190f849008ce6542d569b402a" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f"
I20250113 02:51:19.290374 919 raft_consensus.cc:3054] T c135506190f849008ce6542d569b402a P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Advancing to term 1
W20250113 02:51:19.291996 852 leader_election.cc:343] T bc88676bf738442d82c25e56f30c4ada P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Not found: Tablet not found: bc88676bf738442d82c25e56f30c4ada
I20250113 02:51:19.292338 1298 raft_consensus.cc:491] T ad9b95c5450a4c9c89a0930862348d49 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:19.292532 1298 raft_consensus.cc:513] T ad9b95c5450a4c9c89a0930862348d49 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:19.293955 919 raft_consensus.cc:2463] T c135506190f849008ce6542d569b402a P df69f48656b44bb890f76587ee69cc0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1186fa869eb14812832fa6562531f196 in term 1.
I20250113 02:51:19.294611 1298 leader_election.cc:290] T ad9b95c5450a4c9c89a0930862348d49 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:19.296624 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 55ebe9c044d944bc813396c699ba9aa1
I20250113 02:51:19.297176 450 catalog_manager.cc:6596] ReplaceTablet: tablet 55ebe9c044d944bc813396c699ba9aa1 deleted and replaced by tablet 41fcde988f72496b824271fa36c1a226
I20250113 02:51:19.297394 893 tablet_service.cc:1514] Processing DeleteTablet for tablet 55ebe9c044d944bc813396c699ba9aa1 with delete_type TABLET_DATA_DELETED (replaced by tablet 41fcde988f72496b824271fa36c1a226) from {username='slave'} at 127.0.0.1:37366
W20250113 02:51:19.299382 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ebc5f1ea1e1648d49904b17603d3cd2a consensus state has no leader
I20250113 02:51:19.301443 1192 tablet_bootstrap.cc:492] T ad6b21ed2b824cb0837fc1d8a39d2472 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:19.301623 451 catalog_manager.cc:5618] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467) does not have the latest schema for tablet ad9b95c5450a4c9c89a0930862348d49 (table default.Table_936e73f0b76046d285567ac0778e36a9 [id=219c739889d04a97aafa4fe1b55b0a24]). Expected version 2 got 0
I20250113 02:51:19.302175 1192 ts_tablet_manager.cc:1397] T ad6b21ed2b824cb0837fc1d8a39d2472 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.063s user 0.005s sys 0.024s
I20250113 02:51:19.303393 1192 raft_consensus.cc:357] T ad6b21ed2b824cb0837fc1d8a39d2472 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:19.304751 1192 raft_consensus.cc:383] T ad6b21ed2b824cb0837fc1d8a39d2472 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:19.305058 1192 raft_consensus.cc:738] T ad6b21ed2b824cb0837fc1d8a39d2472 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:19.305366 1062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad9b95c5450a4c9c89a0930862348d49" candidate_uuid: "df69f48656b44bb890f76587ee69cc0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:19.305492 1192 consensus_queue.cc:260] T ad6b21ed2b824cb0837fc1d8a39d2472 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
W20250113 02:51:19.306309 852 leader_election.cc:343] T ad9b95c5450a4c9c89a0930862348d49 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:19.307227 1033 tablet_service.cc:1514] Processing DeleteTablet for tablet b834f0acf5314bcb9d81c73de1b31842 with delete_type TABLET_DATA_DELETED (replaced by tablet be4b2a8a3927410eaf12e7592646e4af) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:19.307943 1033 tablet_service.cc:1514] Processing DeleteTablet for tablet ba61c86743e24666b30580c36b9228c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:19 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:19.308374 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet c97a6ad98ddc4680a0f071cb91f05402 (table default.Table_351e5f6de7c64dafb0d9ee4d86510d01 [id=a383dc51c39e4c8ba4b421119c752470]) successfully deleted
W20250113 02:51:19.308714 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet f760fc1e4bed433dbb987a42c12a1ee6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f760fc1e4bed433dbb987a42c12a1ee6 already in progress: creating tablet
I20250113 02:51:19.273645 709 leader_election.cc:304] T c135506190f849008ce6542d569b402a P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters:
I20250113 02:51:19.274222 1209 leader_election.cc:290] T c135506190f849008ce6542d569b402a P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 election: Requested vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
W20250113 02:51:19.309100 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet a6055d77d168495d9b756a1a56c1a920 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a6055d77d168495d9b756a1a56c1a920 already in progress: creating tablet
I20250113 02:51:19.309180 1192 ts_tablet_manager.cc:1428] T ad6b21ed2b824cb0837fc1d8a39d2472 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20250113 02:51:19.280272 1191 tablet_bootstrap.cc:654] T ad6b21ed2b824cb0837fc1d8a39d2472 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:19.281263 739 tablet_service.cc:1514] Processing DeleteTablet for tablet ba61c86743e24666b30580c36b9228c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:19 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:19.309693 1033 tablet_service.cc:1514] Processing DeleteTablet for tablet 55ebe9c044d944bc813396c699ba9aa1 with delete_type TABLET_DATA_DELETED (replaced by tablet 41fcde988f72496b824271fa36c1a226) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:19.289898 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc88676bf738442d82c25e56f30c4ada" candidate_uuid: "df69f48656b44bb890f76587ee69cc0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196"
I20250113 02:51:19.294505 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad9b95c5450a4c9c89a0930862348d49" candidate_uuid: "df69f48656b44bb890f76587ee69cc0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
I20250113 02:51:19.311683 1192 tablet_bootstrap.cc:492] T 1643b3e1607f42fc82886d0b09e2fe8c P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:19.312438 1033 tablet_service.cc:1514] Processing DeleteTablet for tablet c135506190f849008ce6542d569b402a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:16 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:19.313113 1194 tablet_bootstrap.cc:654] T 15a05a32febd443dbb519c186048428a P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:19.316144 1198 ts_tablet_manager.cc:1918] T db266a1848094b7a98aa75f7afabd471 P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:19.316174 1033 tablet_service.cc:1514] Processing DeleteTablet for tablet ad6b21ed2b824cb0837fc1d8a39d2472 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:16 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:19.316291 1192 tablet_bootstrap.cc:654] T 1643b3e1607f42fc82886d0b09e2fe8c P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:19.317108 1198 log.cc:1198] T db266a1848094b7a98aa75f7afabd471 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/db266a1848094b7a98aa75f7afabd471
I20250113 02:51:19.317945 1198 ts_tablet_manager.cc:1939] T db266a1848094b7a98aa75f7afabd471 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:19.319531 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet db266a1848094b7a98aa75f7afabd471 (table default.Table_87cfc114b15047b49fd25446ff7aa441 [id=ca79853d13ee4a76a18d9fa68c1feb31]) successfully deleted
I20250113 02:51:19.321092 1198 tablet_replica.cc:331] T 9a323451cdfe4fd69ec62cda230f9afb P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:19.321923 1198 raft_consensus.cc:2238] T 9a323451cdfe4fd69ec62cda230f9afb P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:19.323675 1194 tablet_bootstrap.cc:492] T 15a05a32febd443dbb519c186048428a P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:19.323945 1194 ts_tablet_manager.cc:1397] T 15a05a32febd443dbb519c186048428a P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.081s user 0.003s sys 0.003s
W20250113 02:51:19.324579 709 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.3:42587, user_credentials={real_user=slave}} blocked reactor thread for 50586.6us
I20250113 02:51:19.325142 1194 raft_consensus.cc:357] T 15a05a32febd443dbb519c186048428a P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:19.325441 1194 raft_consensus.cc:383] T 15a05a32febd443dbb519c186048428a P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:19.325537 1194 raft_consensus.cc:738] T 15a05a32febd443dbb519c186048428a P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:19.325928 1194 consensus_queue.cc:260] T 15a05a32febd443dbb519c186048428a P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:19.326746 1194 ts_tablet_manager.cc:1428] T 15a05a32febd443dbb519c186048428a P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250113 02:51:19.327057 1194 tablet_bootstrap.cc:492] T 3d132c29cb2c46108e5877e38068ef17 P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:19.322746 1198 raft_consensus.cc:2267] T 9a323451cdfe4fd69ec62cda230f9afb P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:19.329726 1209 raft_consensus.cc:2798] T c135506190f849008ce6542d569b402a P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Leader election won for term 1
I20250113 02:51:19.329916 1209 raft_consensus.cc:695] T c135506190f849008ce6542d569b402a P 1186fa869eb14812832fa6562531f196 [term 1 LEADER]: Becoming Leader. State: Replica: 1186fa869eb14812832fa6562531f196, State: Running, Role: LEADER
I20250113 02:51:19.331171 1209 consensus_queue.cc:237] T c135506190f849008ce6542d569b402a P 1186fa869eb14812832fa6562531f196 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:19.331904 893 tablet_service.cc:1514] Processing DeleteTablet for tablet 15a05a32febd443dbb519c186048428a with delete_type TABLET_DATA_DELETED (replaced by tablet 18e2d79d731142f89b9089869bc736e5) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:19.323695 1192 tablet_bootstrap.cc:492] T 1643b3e1607f42fc82886d0b09e2fe8c P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:19.332300 737 tablet_service.cc:1514] Processing DeleteTablet for tablet b834f0acf5314bcb9d81c73de1b31842 with delete_type TABLET_DATA_DELETED (replaced by tablet be4b2a8a3927410eaf12e7592646e4af) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:19.332399 1192 ts_tablet_manager.cc:1397] T 1643b3e1607f42fc82886d0b09e2fe8c P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.021s user 0.005s sys 0.001s
I20250113 02:51:19.332862 1198 ts_tablet_manager.cc:1905] T 9a323451cdfe4fd69ec62cda230f9afb P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:19.333654 1192 raft_consensus.cc:357] T 1643b3e1607f42fc82886d0b09e2fe8c P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:19.334002 1270 raft_consensus.cc:491] T 1643b3e1607f42fc82886d0b09e2fe8c P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:19.334199 1192 raft_consensus.cc:383] T 1643b3e1607f42fc82886d0b09e2fe8c P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:19.334471 1192 raft_consensus.cc:738] T 1643b3e1607f42fc82886d0b09e2fe8c P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
W20250113 02:51:19.334621 851 leader_election.cc:343] T ad9b95c5450a4c9c89a0930862348d49 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:19.334765 851 leader_election.cc:304] T ad9b95c5450a4c9c89a0930862348d49 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df69f48656b44bb890f76587ee69cc0f; no voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca
I20250113 02:51:19.334918 1192 consensus_queue.cc:260] T 1643b3e1607f42fc82886d0b09e2fe8c P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:19.335215 1298 raft_consensus.cc:2743] T ad9b95c5450a4c9c89a0930862348d49 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:19.336001 775 raft_consensus.cc:3054] T bc88676bf738442d82c25e56f30c4ada P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:19.336725 737 tablet_service.cc:1514] Processing DeleteTablet for tablet 55ebe9c044d944bc813396c699ba9aa1 with delete_type TABLET_DATA_DELETED (replaced by tablet 41fcde988f72496b824271fa36c1a226) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:19.336907 1270 raft_consensus.cc:513] T 1643b3e1607f42fc82886d0b09e2fe8c P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:19.338652 1062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1643b3e1607f42fc82886d0b09e2fe8c" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:19.338747 1192 ts_tablet_manager.cc:1428] T 1643b3e1607f42fc82886d0b09e2fe8c P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250113 02:51:19.338904 1062 raft_consensus.cc:2463] T 1643b3e1607f42fc82886d0b09e2fe8c P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1186fa869eb14812832fa6562531f196 in term 0.
I20250113 02:51:19.337769 1270 leader_election.cc:290] T 1643b3e1607f42fc82886d0b09e2fe8c P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
W20250113 02:51:19.339625 711 leader_election.cc:343] T 1643b3e1607f42fc82886d0b09e2fe8c P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Not found: Tablet not found: 1643b3e1607f42fc82886d0b09e2fe8c
I20250113 02:51:19.340255 1033 tablet_service.cc:1514] Processing DeleteTablet for tablet 1643b3e1607f42fc82886d0b09e2fe8c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:16 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:19.341624 1194 tablet_bootstrap.cc:654] T 3d132c29cb2c46108e5877e38068ef17 P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:19.342574 1192 tablet_bootstrap.cc:492] T ec0750fd2fa247feb3ea310ffabe05e5 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:19.342696 775 raft_consensus.cc:2463] T bc88676bf738442d82c25e56f30c4ada P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df69f48656b44bb890f76587ee69cc0f in term 1.
I20250113 02:51:19.344352 709 leader_election.cc:304] T 1643b3e1607f42fc82886d0b09e2fe8c P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters: df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:19.344990 1270 raft_consensus.cc:2798] T 1643b3e1607f42fc82886d0b09e2fe8c P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 02:51:19.345104 1270 raft_consensus.cc:491] T 1643b3e1607f42fc82886d0b09e2fe8c P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 02:51:19.345499 1270 raft_consensus.cc:3054] T 1643b3e1607f42fc82886d0b09e2fe8c P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:19.344012 851 leader_election.cc:304] T bc88676bf738442d82c25e56f30c4ada P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1186fa869eb14812832fa6562531f196, df69f48656b44bb890f76587ee69cc0f; no voters: 4216a7f853d642b7bc3e5cff458f3aca
I20250113 02:51:19.346566 1192 tablet_bootstrap.cc:654] T ec0750fd2fa247feb3ea310ffabe05e5 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:19.346830 1196 ts_tablet_manager.cc:1918] T 0c57a5add4914835ac366efdcde4ee28 P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:19.346930 1298 raft_consensus.cc:2798] T bc88676bf738442d82c25e56f30c4ada P df69f48656b44bb890f76587ee69cc0f [term 1 FOLLOWER]: Leader election won for term 1
I20250113 02:51:19.346982 1196 log.cc:1198] T 0c57a5add4914835ac366efdcde4ee28 P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/0c57a5add4914835ac366efdcde4ee28
I20250113 02:51:19.347093 1298 raft_consensus.cc:695] T bc88676bf738442d82c25e56f30c4ada P df69f48656b44bb890f76587ee69cc0f [term 1 LEADER]: Becoming Leader. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Running, Role: LEADER
I20250113 02:51:19.347353 1196 ts_tablet_manager.cc:1939] T 0c57a5add4914835ac366efdcde4ee28 P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:19.347465 1298 consensus_queue.cc:237] T bc88676bf738442d82c25e56f30c4ada P df69f48656b44bb890f76587ee69cc0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:19.349500 1270 raft_consensus.cc:513] T 1643b3e1607f42fc82886d0b09e2fe8c P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:19.351575 1192 tablet_bootstrap.cc:492] T ec0750fd2fa247feb3ea310ffabe05e5 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:19.352421 1192 ts_tablet_manager.cc:1397] T ec0750fd2fa247feb3ea310ffabe05e5 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.004s
I20250113 02:51:19.353147 1062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1643b3e1607f42fc82886d0b09e2fe8c" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca"
I20250113 02:51:19.353422 1062 raft_consensus.cc:3054] T 1643b3e1607f42fc82886d0b09e2fe8c P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:19.353605 1192 raft_consensus.cc:357] T ec0750fd2fa247feb3ea310ffabe05e5 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:19.353850 1192 raft_consensus.cc:383] T ec0750fd2fa247feb3ea310ffabe05e5 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:19.353935 1192 raft_consensus.cc:738] T ec0750fd2fa247feb3ea310ffabe05e5 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:19.355572 1198 ts_tablet_manager.cc:1918] T 9a323451cdfe4fd69ec62cda230f9afb P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:19.355721 1198 log.cc:1198] T 9a323451cdfe4fd69ec62cda230f9afb P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/9a323451cdfe4fd69ec62cda230f9afb
I20250113 02:51:19.356107 1198 ts_tablet_manager.cc:1939] T 9a323451cdfe4fd69ec62cda230f9afb P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:19.357824 1192 consensus_queue.cc:260] T ec0750fd2fa247feb3ea310ffabe05e5 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:19.358055 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet 9a323451cdfe4fd69ec62cda230f9afb (table default.Table_2106a7d1900a44df83cf7ec9f377687d [id=a2c08f472dd7494ab8aff87e9aa43028]) successfully deleted
I20250113 02:51:19.358570 1198 tablet_replica.cc:331] T 0c57a5add4914835ac366efdcde4ee28 P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:19.361150 1192 ts_tablet_manager.cc:1428] T ec0750fd2fa247feb3ea310ffabe05e5 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20250113 02:51:19.362944 1033 tablet_service.cc:1514] Processing DeleteTablet for tablet ec0750fd2fa247feb3ea310ffabe05e5 with delete_type TABLET_DATA_DELETED (replaced by tablet 107e6e9e2d84496f8fce92b9ba377189) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:19.363003 1062 raft_consensus.cc:2463] T 1643b3e1607f42fc82886d0b09e2fe8c P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1186fa869eb14812832fa6562531f196 in term 1.
I20250113 02:51:19.363855 1198 raft_consensus.cc:2238] T 0c57a5add4914835ac366efdcde4ee28 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus shutting down.
W20250113 02:51:19.365751 711 leader_election.cc:343] T 1643b3e1607f42fc82886d0b09e2fe8c P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Not found: Tablet not found: 1643b3e1607f42fc82886d0b09e2fe8c
I20250113 02:51:19.367363 709 leader_election.cc:304] T 1643b3e1607f42fc82886d0b09e2fe8c P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters: df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:19.367646 1209 raft_consensus.cc:2798] T 1643b3e1607f42fc82886d0b09e2fe8c P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Leader election won for term 1
I20250113 02:51:19.370357 737 tablet_service.cc:1467] Processing CreateTablet for tablet 63efcb6b2e194c469fb30a197bd6565b (DEFAULT_TABLE table=default.Table_a6447976a51a4e2f93d22bd1c657c49d [id=0485ff4fda424e97bb71b77d657ab5e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:19.370440 893 tablet_service.cc:1467] Processing CreateTablet for tablet 63efcb6b2e194c469fb30a197bd6565b (DEFAULT_TABLE table=default.Table_a6447976a51a4e2f93d22bd1c657c49d [id=0485ff4fda424e97bb71b77d657ab5e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:19.370759 737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 63efcb6b2e194c469fb30a197bd6565b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:19.370998 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 63efcb6b2e194c469fb30a197bd6565b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:19.377384 1209 raft_consensus.cc:695] T 1643b3e1607f42fc82886d0b09e2fe8c P 1186fa869eb14812832fa6562531f196 [term 1 LEADER]: Becoming Leader. State: Replica: 1186fa869eb14812832fa6562531f196, State: Running, Role: LEADER
I20250113 02:51:19.377702 1209 consensus_queue.cc:237] T 1643b3e1607f42fc82886d0b09e2fe8c P 1186fa869eb14812832fa6562531f196 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:19.377733 1033 tablet_service.cc:1467] Processing CreateTablet for tablet 63efcb6b2e194c469fb30a197bd6565b (DEFAULT_TABLE table=default.Table_a6447976a51a4e2f93d22bd1c657c49d [id=0485ff4fda424e97bb71b77d657ab5e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:19.378612 1029 tablet_service.cc:1467] Processing CreateTablet for tablet c215ffe61b9b426286cc5e808463078d (DEFAULT_TABLE table=default.Table_d4146353239a45b281b482d6e9537502 [id=20b69dfe61d841829e1804153c0d0184]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:19.379531 739 tablet_service.cc:1467] Processing CreateTablet for tablet c215ffe61b9b426286cc5e808463078d (DEFAULT_TABLE table=default.Table_d4146353239a45b281b482d6e9537502 [id=20b69dfe61d841829e1804153c0d0184]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:19.379870 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c215ffe61b9b426286cc5e808463078d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:19.381681 896 tablet_service.cc:1467] Processing CreateTablet for tablet c215ffe61b9b426286cc5e808463078d (DEFAULT_TABLE table=default.Table_d4146353239a45b281b482d6e9537502 [id=20b69dfe61d841829e1804153c0d0184]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:19.382495 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet 0c57a5add4914835ac366efdcde4ee28 (table default.Table_3b52330dd7f04b9db3ef41770dc87651 [id=1a69ebe950b846c78ec36ca96a1f9e6c]) successfully deleted
I20250113 02:51:19.384851 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c215ffe61b9b426286cc5e808463078d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:19.385174 1196 tablet_replica.cc:331] T bc88676bf738442d82c25e56f30c4ada P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:19.385391 1196 raft_consensus.cc:2238] T bc88676bf738442d82c25e56f30c4ada P df69f48656b44bb890f76587ee69cc0f [term 1 LEADER]: Raft consensus shutting down.
W20250113 02:51:19.386199 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 157905316fc545b6a7864ae1c4a2a584 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 157905316fc545b6a7864ae1c4a2a584 already in progress: creating tablet
W20250113 02:51:19.386515 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet a6055d77d168495d9b756a1a56c1a920 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a6055d77d168495d9b756a1a56c1a920 already in progress: creating tablet
I20250113 02:51:19.387332 1192 tablet_bootstrap.cc:492] T a028cc02273644608f046135851da126 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:19.387403 1196 pending_rounds.cc:62] T bc88676bf738442d82c25e56f30c4ada P df69f48656b44bb890f76587ee69cc0f: Trying to abort 1 pending ops.
I20250113 02:51:19.387519 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 63efcb6b2e194c469fb30a197bd6565b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:19.387475 1196 pending_rounds.cc:69] T bc88676bf738442d82c25e56f30c4ada P df69f48656b44bb890f76587ee69cc0f: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113673438638141440 op_type: NO_OP noop_request { }
I20250113 02:51:19.388028 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_c604c45a4d3a49f0b6fd1ec85582d495"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:19.389058 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c604c45a4d3a49f0b6fd1ec85582d495 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:19.389480 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_37d63a1b8622483a855edf2ce71c8d71"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:19.389958 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_37d63a1b8622483a855edf2ce71c8d71 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:19.390254 1196 raft_consensus.cc:2883] T bc88676bf738442d82c25e56f30c4ada P df69f48656b44bb890f76587ee69cc0f [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250113 02:51:19.390385 1196 raft_consensus.cc:2267] T bc88676bf738442d82c25e56f30c4ada P df69f48656b44bb890f76587ee69cc0f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:19.392882 1192 tablet_bootstrap.cc:654] T a028cc02273644608f046135851da126 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:19.393054 1194 tablet_bootstrap.cc:492] T 3d132c29cb2c46108e5877e38068ef17 P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:19.393221 1194 ts_tablet_manager.cc:1397] T 3d132c29cb2c46108e5877e38068ef17 P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.066s user 0.001s sys 0.003s
I20250113 02:51:19.394273 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c215ffe61b9b426286cc5e808463078d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:19.394241 1194 raft_consensus.cc:357] T 3d132c29cb2c46108e5877e38068ef17 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:19.394443 1194 raft_consensus.cc:383] T 3d132c29cb2c46108e5877e38068ef17 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:19.394515 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 21c4b419bf2148e0b140301fa80622bb
I20250113 02:51:19.395064 446 catalog_manager.cc:6596] ReplaceTablet: tablet 21c4b419bf2148e0b140301fa80622bb deleted and replaced by tablet 23e04961f072440fa3e3f9485e40f79c
I20250113 02:51:19.397688 754 tablet_service.cc:1514] Processing DeleteTablet for tablet 21c4b419bf2148e0b140301fa80622bb with delete_type TABLET_DATA_DELETED (replaced by tablet 23e04961f072440fa3e3f9485e40f79c) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:19.398193 754 tablet_service.cc:1467] Processing CreateTablet for tablet 32cba8f3efc54e78b4df41c15df82eea (DEFAULT_TABLE table=default.Table_6c21e014853e4ab1996e23f0822d2b85 [id=8aff13d6d3c5483eadb596b705fe5151]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:19.398234 743 tablet_service.cc:1467] Processing CreateTablet for tablet 6d63f0e5835540a78c82020b20a07dac (DEFAULT_TABLE table=default.Table_af3ebf543c9448b7ae473d721a23d47e [id=93c6f37549ce4544bc0cb4a267f44b00]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:19.398653 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 32cba8f3efc54e78b4df41c15df82eea. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:19.400660 749 tablet_service.cc:1467] Processing CreateTablet for tablet a0ca2e3a3ce94f3aa20df64c89c7cf95 (DEFAULT_TABLE table=default.Table_f08525e2a08e48ec85ef55287d0bac49 [id=96ff1ee9327f41a5a1bb2757b188254a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:19.401075 741 tablet_service.cc:1467] Processing CreateTablet for tablet 401c4f5dbd304950877b59a865f417b3 (DEFAULT_TABLE table=default.Table_c991f9dbd70d4c4682664b8711f05db4 [id=2c64da64ecdb4bf48e11b523533902e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:19.401432 753 tablet_service.cc:1467] Processing CreateTablet for tablet b4806b18a4194f9c95350d89ba62e13a (DEFAULT_TABLE table=default.Table_4369a143c72643b084df58d25314c6dd [id=36ff239c08a54be99ca307354810fff3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:19.396550 895 tablet_service.cc:1514] Processing DeleteTablet for tablet 21c4b419bf2148e0b140301fa80622bb with delete_type TABLET_DATA_DELETED (replaced by tablet 23e04961f072440fa3e3f9485e40f79c) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:19.396912 899 tablet_service.cc:1467] Processing CreateTablet for tablet 2778d7dc7d9c4843bbc9eb705cec71cd (DEFAULT_TABLE table=default.Table_b3284070cce54277bed74120ca8f199b [id=4fafe716b4704c0c96514223a04e480a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:19.402705 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2778d7dc7d9c4843bbc9eb705cec71cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:19.397236 891 tablet_service.cc:1467] Processing CreateTablet for tablet 6d63f0e5835540a78c82020b20a07dac (DEFAULT_TABLE table=default.Table_af3ebf543c9448b7ae473d721a23d47e [id=93c6f37549ce4544bc0cb4a267f44b00]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:19.399324 894 tablet_service.cc:1467] Processing CreateTablet for tablet 32cba8f3efc54e78b4df41c15df82eea (DEFAULT_TABLE table=default.Table_6c21e014853e4ab1996e23f0822d2b85 [id=8aff13d6d3c5483eadb596b705fe5151]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:19.403025 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d63f0e5835540a78c82020b20a07dac. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:19.403240 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 32cba8f3efc54e78b4df41c15df82eea. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:19.394536 1194 raft_consensus.cc:738] T 3d132c29cb2c46108e5877e38068ef17 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:19.403502 1194 consensus_queue.cc:260] T 3d132c29cb2c46108e5877e38068ef17 P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:19.404225 1194 ts_tablet_manager.cc:1428] T 3d132c29cb2c46108e5877e38068ef17 P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.011s user 0.001s sys 0.001s
I20250113 02:51:19.404600 1194 tablet_bootstrap.cc:492] T 3455f010e51b4db1a9804a57fd1a4fb8 P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:19.405059 1270 leader_election.cc:290] T 1643b3e1607f42fc82886d0b09e2fe8c P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 election: Requested vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:19.406571 1194 tablet_bootstrap.cc:654] T 3455f010e51b4db1a9804a57fd1a4fb8 P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:19.399551 889 tablet_service.cc:1467] Processing CreateTablet for tablet a0ca2e3a3ce94f3aa20df64c89c7cf95 (DEFAULT_TABLE table=default.Table_f08525e2a08e48ec85ef55287d0bac49 [id=96ff1ee9327f41a5a1bb2757b188254a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:19.407734 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a0ca2e3a3ce94f3aa20df64c89c7cf95. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:19.408273 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a0ca2e3a3ce94f3aa20df64c89c7cf95. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:19.412206 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 401c4f5dbd304950877b59a865f417b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:19.416224 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b4806b18a4194f9c95350d89ba62e13a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:19.419620 895 tablet_service.cc:1467] Processing CreateTablet for tablet 401c4f5dbd304950877b59a865f417b3 (DEFAULT_TABLE table=default.Table_c991f9dbd70d4c4682664b8711f05db4 [id=2c64da64ecdb4bf48e11b523533902e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:19.420050 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 401c4f5dbd304950877b59a865f417b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:19.420257 888 tablet_service.cc:1467] Processing CreateTablet for tablet 378d59d314a84ddb84cf2ab041c51cf1 (DEFAULT_TABLE table=default.Table_47f735660c3a49d991a3593625822367 [id=0c01441cec7043b687d6744e72a0e45d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:19.420688 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 378d59d314a84ddb84cf2ab041c51cf1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:19.421681 897 tablet_service.cc:1467] Processing CreateTablet for tablet b4806b18a4194f9c95350d89ba62e13a (DEFAULT_TABLE table=default.Table_4369a143c72643b084df58d25314c6dd [id=36ff239c08a54be99ca307354810fff3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:19.421989 892 tablet_service.cc:1467] Processing CreateTablet for tablet ddf79eec41214693859f526265a55492 (DEFAULT_TABLE table=default.Table_f5342954ee0548ec90d482ed62198240 [id=888ae0b48950407086e5e2faa7fcf506]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:19.423864 1042 tablet_service.cc:1467] Processing CreateTablet for tablet 6d63f0e5835540a78c82020b20a07dac (DEFAULT_TABLE table=default.Table_af3ebf543c9448b7ae473d721a23d47e [id=93c6f37549ce4544bc0cb4a267f44b00]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:19.424319 1042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d63f0e5835540a78c82020b20a07dac. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:19.425494 1198 raft_consensus.cc:2267] T 0c57a5add4914835ac366efdcde4ee28 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:19.426929 1198 ts_tablet_manager.cc:1905] T 0c57a5add4914835ac366efdcde4ee28 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:19.428397 1024 tablet_service.cc:1467] Processing CreateTablet for tablet 32cba8f3efc54e78b4df41c15df82eea (DEFAULT_TABLE table=default.Table_6c21e014853e4ab1996e23f0822d2b85 [id=8aff13d6d3c5483eadb596b705fe5151]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:19.428853 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 32cba8f3efc54e78b4df41c15df82eea. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:19.432327 1030 tablet_service.cc:1467] Processing CreateTablet for tablet a0ca2e3a3ce94f3aa20df64c89c7cf95 (DEFAULT_TABLE table=default.Table_f08525e2a08e48ec85ef55287d0bac49 [id=96ff1ee9327f41a5a1bb2757b188254a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:19.432762 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a0ca2e3a3ce94f3aa20df64c89c7cf95. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:19.435424 1192 tablet_bootstrap.cc:492] T a028cc02273644608f046135851da126 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:19.435638 1192 ts_tablet_manager.cc:1397] T a028cc02273644608f046135851da126 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.071s user 0.008s sys 0.000s
I20250113 02:51:19.436331 1041 tablet_service.cc:1467] Processing CreateTablet for tablet 401c4f5dbd304950877b59a865f417b3 (DEFAULT_TABLE table=default.Table_c991f9dbd70d4c4682664b8711f05db4 [id=2c64da64ecdb4bf48e11b523533902e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:19.436650 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 401c4f5dbd304950877b59a865f417b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:19.436864 1192 raft_consensus.cc:357] T a028cc02273644608f046135851da126 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:19.437167 1192 raft_consensus.cc:383] T a028cc02273644608f046135851da126 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:19.437299 1192 raft_consensus.cc:738] T a028cc02273644608f046135851da126 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:19.437724 1192 consensus_queue.cc:260] T a028cc02273644608f046135851da126 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:19.439527 1192 ts_tablet_manager.cc:1428] T a028cc02273644608f046135851da126 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20250113 02:51:19.439841 1192 tablet_bootstrap.cc:492] T 78689f7ed2a64405adb75f372465a792 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:19.440811 1040 tablet_service.cc:1467] Processing CreateTablet for tablet b4806b18a4194f9c95350d89ba62e13a (DEFAULT_TABLE table=default.Table_4369a143c72643b084df58d25314c6dd [id=36ff239c08a54be99ca307354810fff3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:19.441300 1040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b4806b18a4194f9c95350d89ba62e13a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:19.441880 1192 tablet_bootstrap.cc:654] T 78689f7ed2a64405adb75f372465a792 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:19.444407 1034 tablet_service.cc:1467] Processing CreateTablet for tablet 9fff52e99c64462a8a73fccc3c22ca63 (DEFAULT_TABLE table=default.Table_b147d4f32fa04b849ff6b75c9382a4e5 [id=bd8bd572ed524f2cae01f820f07e3c36]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:19.444777 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9fff52e99c64462a8a73fccc3c22ca63. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:19.447835 1198 ts_tablet_manager.cc:1918] T 0c57a5add4914835ac366efdcde4ee28 P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:19.448007 1198 log.cc:1198] T 0c57a5add4914835ac366efdcde4ee28 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/0c57a5add4914835ac366efdcde4ee28
I20250113 02:51:19.448457 1028 tablet_service.cc:1467] Processing CreateTablet for tablet 378d59d314a84ddb84cf2ab041c51cf1 (DEFAULT_TABLE table=default.Table_47f735660c3a49d991a3593625822367 [id=0c01441cec7043b687d6744e72a0e45d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:19.448575 1198 ts_tablet_manager.cc:1939] T 0c57a5add4914835ac366efdcde4ee28 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:19.448799 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 378d59d314a84ddb84cf2ab041c51cf1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:19.452239 1192 tablet_bootstrap.cc:492] T 78689f7ed2a64405adb75f372465a792 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:19.452448 1192 ts_tablet_manager.cc:1397] T 78689f7ed2a64405adb75f372465a792 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.000s
I20250113 02:51:19.453871 1037 tablet_service.cc:1467] Processing CreateTablet for tablet ddf79eec41214693859f526265a55492 (DEFAULT_TABLE table=default.Table_f5342954ee0548ec90d482ed62198240 [id=888ae0b48950407086e5e2faa7fcf506]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:19.454015 1192 raft_consensus.cc:357] T 78689f7ed2a64405adb75f372465a792 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:19.454396 1037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ddf79eec41214693859f526265a55492. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:19.455284 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d63f0e5835540a78c82020b20a07dac. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:19.456660 1038 tablet_service.cc:1467] Processing CreateTablet for tablet f68f9f4076d9475d938e795e72bdd468 (DEFAULT_TABLE table=default.Table_480e362015cc422a85dffa0b15d1dcae [id=9e58548d1a184cc0b362f148c44005ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:19.457011 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f68f9f4076d9475d938e795e72bdd468. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:19.460541 737 tablet_service.cc:1467] Processing CreateTablet for tablet 9fff52e99c64462a8a73fccc3c22ca63 (DEFAULT_TABLE table=default.Table_b147d4f32fa04b849ff6b75c9382a4e5 [id=bd8bd572ed524f2cae01f820f07e3c36]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:19.460718 736 tablet_service.cc:1514] Processing DeleteTablet for tablet c135506190f849008ce6542d569b402a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:16 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:19.462138 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet 0c57a5add4914835ac366efdcde4ee28 (table default.Table_3b52330dd7f04b9db3ef41770dc87651 [id=1a69ebe950b846c78ec36ca96a1f9e6c]) successfully deleted
I20250113 02:51:19.462896 739 tablet_service.cc:1467] Processing CreateTablet for tablet ddf79eec41214693859f526265a55492 (DEFAULT_TABLE table=default.Table_f5342954ee0548ec90d482ed62198240 [id=888ae0b48950407086e5e2faa7fcf506]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:19.465214 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_5a2e4e24ce9b4618813ca6d023d05664" } new_table_name: "default.Table_01a7a0bb02fd483ebe043578768a48e6" modify_external_catalogs: true
I20250113 02:51:19.465737 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_f11eaf2fe6674de8973c9363a4afca76" } new_table_name: "default.Table_0ea497aeeb3742fd946679cb72cdf291" modify_external_catalogs: true
I20250113 02:51:19.467348 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_5aafd46e1cbf4e03a5b87797d5d97bdd" } modify_external_catalogs: true
I20250113 02:51:19.467567 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_5aafd46e1cbf4e03a5b87797d5d97bdd" } modify_external_catalogs: true
I20250113 02:51:19.468600 755 tablet_service.cc:1467] Processing CreateTablet for tablet f68f9f4076d9475d938e795e72bdd468 (DEFAULT_TABLE table=default.Table_480e362015cc422a85dffa0b15d1dcae [id=9e58548d1a184cc0b362f148c44005ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:19.469170 448 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_c8bea6af909a424c9101c8934d1aee7b" } modify_external_catalogs: true
I20250113 02:51:19.469364 448 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_c8bea6af909a424c9101c8934d1aee7b" } modify_external_catalogs: true
I20250113 02:51:19.470191 449 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_7a9c9cc8678e4f7fa12237d3f8c9c3de" } modify_external_catalogs: true
I20250113 02:51:19.470396 449 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_7a9c9cc8678e4f7fa12237d3f8c9c3de" } modify_external_catalogs: true
I20250113 02:51:19.471164 740 tablet_service.cc:1467] Processing CreateTablet for tablet 2778d7dc7d9c4843bbc9eb705cec71cd (DEFAULT_TABLE table=default.Table_b3284070cce54277bed74120ca8f199b [id=4fafe716b4704c0c96514223a04e480a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:19.471380 754 tablet_service.cc:1467] Processing CreateTablet for tablet 378d59d314a84ddb84cf2ab041c51cf1 (DEFAULT_TABLE table=default.Table_47f735660c3a49d991a3593625822367 [id=0c01441cec7043b687d6744e72a0e45d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:19.471273 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_dc98b9959d0c42b58d9f47e3c2c2b73d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:19.471532 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ddf79eec41214693859f526265a55492. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:19.472412 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_7761343ccac947069c26e1e705e0a9b3" } new_table_name: "default.Table_649d3c3eaebd4d4da258735b332a4a75" modify_external_catalogs: true
W20250113 02:51:19.471909 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dc98b9959d0c42b58d9f47e3c2c2b73d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:19.475786 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f68f9f4076d9475d938e795e72bdd468. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:19.477802 737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9fff52e99c64462a8a73fccc3c22ca63. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:19.478032 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_43bb1aa67eec4cb89272fcec397e08b6" } new_table_name: "default.Table_4f1f306815b145218e24c93821376721" modify_external_catalogs: true
I20250113 02:51:19.479774 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2778d7dc7d9c4843bbc9eb705cec71cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:19.482479 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 378d59d314a84ddb84cf2ab041c51cf1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:19.482533 1192 raft_consensus.cc:383] T 78689f7ed2a64405adb75f372465a792 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:19.482652 1192 raft_consensus.cc:738] T 78689f7ed2a64405adb75f372465a792 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:19.482991 1192 consensus_queue.cc:260] T 78689f7ed2a64405adb75f372465a792 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:19.402093 1196 ts_tablet_manager.cc:1905] T bc88676bf738442d82c25e56f30c4ada P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:19.486321 1192 ts_tablet_manager.cc:1428] T 78689f7ed2a64405adb75f372465a792 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.034s user 0.003s sys 0.000s
I20250113 02:51:19.488927 453 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_3d6c2586a98845c7989a5b715c4d678e" } modify_external_catalogs: true
I20250113 02:51:19.489128 1192 tablet_bootstrap.cc:492] T 3d132c29cb2c46108e5877e38068ef17 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:19.489130 453 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_3d6c2586a98845c7989a5b715c4d678e" } modify_external_catalogs: true
I20250113 02:51:19.490314 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_1d46c338f1ed42c9a61f4aa130290e22"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:19.490831 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1d46c338f1ed42c9a61f4aa130290e22 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:19.491780 1192 tablet_bootstrap.cc:654] T 3d132c29cb2c46108e5877e38068ef17 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:19.492563 1195 ts_tablet_manager.cc:1918] T 0c57a5add4914835ac366efdcde4ee28 P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:19.492733 1195 log.cc:1198] T 0c57a5add4914835ac366efdcde4ee28 P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/0c57a5add4914835ac366efdcde4ee28
I20250113 02:51:19.492837 1209 raft_consensus.cc:491] T a028cc02273644608f046135851da126 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:19.492986 1209 raft_consensus.cc:513] T a028cc02273644608f046135851da126 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:19.493184 1195 ts_tablet_manager.cc:1939] T 0c57a5add4914835ac366efdcde4ee28 P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:19.493693 1209 leader_election.cc:290] T a028cc02273644608f046135851da126 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:19.504499 736 tablet_service.cc:1467] Processing CreateTablet for tablet c94eedb5dce44e6d896192c93127903a (DEFAULT_TABLE table=default.Table_616eda5ca6ce4155b20351b6b8b000f5 [id=e0cdd94a9fc947fc8a9555150b7272f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:19.508709 1038 tablet_service.cc:1467] Processing CreateTablet for tablet c94eedb5dce44e6d896192c93127903a (DEFAULT_TABLE table=default.Table_616eda5ca6ce4155b20351b6b8b000f5 [id=e0cdd94a9fc947fc8a9555150b7272f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:19.505357 744 tablet_service.cc:1467] Processing CreateTablet for tablet 0810b9671f824d0a84741bc1904df247 (DEFAULT_TABLE table=default.Table_796801d17a3c458693693af6e39cf0f5 [id=df24b379d986402187cec33190501012]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:19.507035 746 tablet_service.cc:1467] Processing CreateTablet for tablet a77c38a394114d6c8a8da99d7eef7669 (DEFAULT_TABLE table=default.Table_7761343ccac947069c26e1e705e0a9b3 [id=d1b33c3e180343f5b88dcb200e7c9dde]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:19.507782 738 tablet_service.cc:1467] Processing CreateTablet for tablet 840773ee80c7447097ea000b106fc89d (DEFAULT_TABLE table=default.Table_5aafd46e1cbf4e03a5b87797d5d97bdd [id=b6d8d0de849a4d3c988b8fd468b14d62]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:19.494179 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 120) took 1683 ms. Trace:
I20250113 02:51:19.510375 1037 tablet_service.cc:1467] Processing CreateTablet for tablet 0810b9671f824d0a84741bc1904df247 (DEFAULT_TABLE table=default.Table_796801d17a3c458693693af6e39cf0f5 [id=df24b379d986402187cec33190501012]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:19.511045 1032 tablet_service.cc:1467] Processing CreateTablet for tablet a77c38a394114d6c8a8da99d7eef7669 (DEFAULT_TABLE table=default.Table_7761343ccac947069c26e1e705e0a9b3 [id=d1b33c3e180343f5b88dcb200e7c9dde]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:19.511270 1039 tablet_service.cc:1467] Processing CreateTablet for tablet efccefd6787943469131492741ee2bd5 (DEFAULT_TABLE table=default.Table_67151d370cfd45ccaac504631aa0f1f9 [id=bcdd761e38934a2e9197b9c1b9165b27]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:19.511416 1025 tablet_service.cc:1467] Processing CreateTablet for tablet 840773ee80c7447097ea000b106fc89d (DEFAULT_TABLE table=default.Table_5aafd46e1cbf4e03a5b87797d5d97bdd [id=b6d8d0de849a4d3c988b8fd468b14d62]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:19.511766 1036 tablet_service.cc:1467] Processing CreateTablet for tablet 092892a36e0e47cb8d2184c2bbde95d7 (DEFAULT_TABLE table=default.Table_22cf65ed324549aa9fa3c6e305bf0e1c [id=d4bf205fb0464358af61bb3fb889c687]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:19.511868 1027 tablet_service.cc:1467] Processing CreateTablet for tablet 4c615b0b83d74fcfae0c51047dbfbbf4 (DEFAULT_TABLE table=default.Table_668e70d7e5c64c37ab05e7bcbe56d994 [id=a956f95f6dd94b578f6bd3cb3b97f9da]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:19.512344 1035 tablet_service.cc:1467] Processing CreateTablet for tablet 11950e996c814590bc9f3f22c0354902 (DEFAULT_TABLE table=default.Table_003cb5a672084be4b29d515fa4485747 [id=2e275f7f92ac440ca3ea6e29c5566f00]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:19.518391 1311 raft_consensus.cc:491] T 78689f7ed2a64405adb75f372465a792 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:19.518589 1311 raft_consensus.cc:513] T 78689f7ed2a64405adb75f372465a792 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:19.519448 1311 leader_election.cc:290] T 78689f7ed2a64405adb75f372465a792 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:19.519536 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b4806b18a4194f9c95350d89ba62e13a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:19.519457 890 tablet_service.cc:1467] Processing CreateTablet for tablet 9fff52e99c64462a8a73fccc3c22ca63 (DEFAULT_TABLE table=default.Table_b147d4f32fa04b849ff6b75c9382a4e5 [id=bd8bd572ed524f2cae01f820f07e3c36]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:19.519495 1023 tablet_service.cc:1467] Processing CreateTablet for tablet 1cfbcd682b944c89847b40fe69f0a75e (DEFAULT_TABLE table=default.Table_9cfb6bce41c0468e8d29a6d0a1f32499 [id=2b84deb07e5c4ab8b550fe42945d7b71]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:19.520165 1031 tablet_service.cc:1467] Processing CreateTablet for tablet 41fcde988f72496b824271fa36c1a226 (DEFAULT_TABLE table=default.Table_2caed919541e43c78a858513a2128f50 [id=26ba87b9bd384b99a016b29fe5b5babd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:19.520747 1026 tablet_service.cc:1467] Processing CreateTablet for tablet ff8ef6290698477391721fd65b9eeb28 (DEFAULT_TABLE table=default.Table_5b825152ff4d4e20a7600a9ad0dfe0a3 [id=7e176c2354964cdd8e6b724f6bb84b99]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:19.521064 1298 raft_consensus.cc:491] T 0c55662a71444a3683a825bcec6c5b68 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:19.523355 1062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a028cc02273644608f046135851da126" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:19.523663 1062 raft_consensus.cc:2463] T a028cc02273644608f046135851da126 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1186fa869eb14812832fa6562531f196 in term 0.
I20250113 02:51:19.524952 1040 tablet_service.cc:1514] Processing DeleteTablet for tablet a028cc02273644608f046135851da126 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:16 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:19.525460 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c94eedb5dce44e6d896192c93127903a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:19.526278 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9fff52e99c64462a8a73fccc3c22ca63. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:19.526557 898 tablet_service.cc:1467] Processing CreateTablet for tablet f68f9f4076d9475d938e795e72bdd468 (DEFAULT_TABLE table=default.Table_480e362015cc422a85dffa0b15d1dcae [id=9e58548d1a184cc0b362f148c44005ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:19.528035 1298 raft_consensus.cc:513] T 0c55662a71444a3683a825bcec6c5b68 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:19.528687 1298 leader_election.cc:290] T 0c55662a71444a3683a825bcec6c5b68 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:19.531087 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ddf79eec41214693859f526265a55492. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:19.532136 750 tablet_service.cc:1467] Processing CreateTablet for tablet efccefd6787943469131492741ee2bd5 (DEFAULT_TABLE table=default.Table_67151d370cfd45ccaac504631aa0f1f9 [id=bcdd761e38934a2e9197b9c1b9165b27]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:19.532409 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c55662a71444a3683a825bcec6c5b68" candidate_uuid: "df69f48656b44bb890f76587ee69cc0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
I20250113 02:51:19.532824 736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c94eedb5dce44e6d896192c93127903a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:19.535687 1195 rpcz_store.cc:276] 0113 02:51:17.810326 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:17.818095 (+ 7769us) service_pool.cc:225] Handling call
0113 02:51:19.213208 (+1395113us) ts_tablet_manager.cc:1145] Deleting tablet 0c57a5add4914835ac366efdcde4ee28
0113 02:51:19.213218 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:19.458472 (+245254us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:19.492435 (+ 33963us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:19.494115 (+ 1680us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1278187}
I20250113 02:51:19.536361 747 tablet_service.cc:1467] Processing CreateTablet for tablet ff8ef6290698477391721fd65b9eeb28 (DEFAULT_TABLE table=default.Table_5b825152ff4d4e20a7600a9ad0dfe0a3 [id=7e176c2354964cdd8e6b724f6bb84b99]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:19.536602 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78689f7ed2a64405adb75f372465a792" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
W20250113 02:51:19.537247 995 leader_election.cc:343] T 78689f7ed2a64405adb75f372465a792 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:19.537227 752 tablet_service.cc:1467] Processing CreateTablet for tablet 41fcde988f72496b824271fa36c1a226 (DEFAULT_TABLE table=default.Table_2caed919541e43c78a858513a2128f50 [id=26ba87b9bd384b99a016b29fe5b5babd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:19.538034 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a77c38a394114d6c8a8da99d7eef7669. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:19.537953 709 leader_election.cc:304] T a028cc02273644608f046135851da126 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters:
W20250113 02:51:19.539566 711 leader_election.cc:343] T a028cc02273644608f046135851da126 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Not found: Tablet not found: a028cc02273644608f046135851da126
I20250113 02:51:19.539615 745 tablet_service.cc:1467] Processing CreateTablet for tablet 1cfbcd682b944c89847b40fe69f0a75e (DEFAULT_TABLE table=default.Table_9cfb6bce41c0468e8d29a6d0a1f32499 [id=2b84deb07e5c4ab8b550fe42945d7b71]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:19.539578 742 tablet_service.cc:1467] Processing CreateTablet for tablet 11950e996c814590bc9f3f22c0354902 (DEFAULT_TABLE table=default.Table_003cb5a672084be4b29d515fa4485747 [id=2e275f7f92ac440ca3ea6e29c5566f00]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:19.540017 748 tablet_service.cc:1467] Processing CreateTablet for tablet 4c615b0b83d74fcfae0c51047dbfbbf4 (DEFAULT_TABLE table=default.Table_668e70d7e5c64c37ab05e7bcbe56d994 [id=a956f95f6dd94b578f6bd3cb3b97f9da]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:19.539149 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet 21c4b419bf2148e0b140301fa80622bb with delete_type TABLET_DATA_DELETED (replaced by tablet 23e04961f072440fa3e3f9485e40f79c) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:19.540904 751 tablet_service.cc:1467] Processing CreateTablet for tablet 092892a36e0e47cb8d2184c2bbde95d7 (DEFAULT_TABLE table=default.Table_22cf65ed324549aa9fa3c6e305bf0e1c [id=d4bf205fb0464358af61bb3fb889c687]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:19.541412 1209 raft_consensus.cc:2798] T a028cc02273644608f046135851da126 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 02:51:19.541922 1033 tablet_service.cc:1467] Processing CreateTablet for tablet 2778d7dc7d9c4843bbc9eb705cec71cd (DEFAULT_TABLE table=default.Table_b3284070cce54277bed74120ca8f199b [id=4fafe716b4704c0c96514223a04e480a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:19.542546 1209 raft_consensus.cc:491] T a028cc02273644608f046135851da126 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 02:51:19.542682 1209 raft_consensus.cc:3054] T a028cc02273644608f046135851da126 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:19.543160 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2778d7dc7d9c4843bbc9eb705cec71cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:19.543509 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 840773ee80c7447097ea000b106fc89d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:19.544504 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0810b9671f824d0a84741bc1904df247. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:19.545444 1198 tablet_replica.cc:331] T c135506190f849008ce6542d569b402a P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:19.545718 1198 raft_consensus.cc:2238] T c135506190f849008ce6542d569b402a P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:19.545878 1198 raft_consensus.cc:2267] T c135506190f849008ce6542d569b402a P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:19.546065 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet efccefd6787943469131492741ee2bd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:19.546296 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11950e996c814590bc9f3f22c0354902. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:19.547003 1198 ts_tablet_manager.cc:1905] T c135506190f849008ce6542d569b402a P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:19.547235 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 092892a36e0e47cb8d2184c2bbde95d7. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:19.548089 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet fdbf03c89bff438a9a25115eddf60be9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fdbf03c89bff438a9a25115eddf60be9 already in progress: creating tablet
I20250113 02:51:19.548297 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 840773ee80c7447097ea000b106fc89d. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:19.549645 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 19f14b3db0b64847b5422bfd5ed323d0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 19f14b3db0b64847b5422bfd5ed323d0 already in progress: creating tablet
W20250113 02:51:19.550046 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 53cc528d282c47788d5d558569f074c1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 53cc528d282c47788d5d558569f074c1 already in progress: creating tablet
W20250113 02:51:19.550935 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet b834f0acf5314bcb9d81c73de1b31842 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b834f0acf5314bcb9d81c73de1b31842 already in progress: creating tablet
W20250113 02:51:19.551273 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet ba61c86743e24666b30580c36b9228c3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ba61c86743e24666b30580c36b9228c3 already in progress: creating tablet
W20250113 02:51:19.552206 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 55ebe9c044d944bc813396c699ba9aa1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 55ebe9c044d944bc813396c699ba9aa1 already in progress: creating tablet
I20250113 02:51:19.552726 1036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 092892a36e0e47cb8d2184c2bbde95d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:19.553182 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1cfbcd682b944c89847b40fe69f0a75e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:19.553602 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet efccefd6787943469131492741ee2bd5. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:19.553704 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet be4b2a8a3927410eaf12e7592646e4af consensus state has no leader
I20250113 02:51:19.553985 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff8ef6290698477391721fd65b9eeb28. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:19.556371 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c615b0b83d74fcfae0c51047dbfbbf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:19.557380 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 41fcde988f72496b824271fa36c1a226. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:19.557017 1062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c55662a71444a3683a825bcec6c5b68" candidate_uuid: "df69f48656b44bb890f76587ee69cc0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:19.560156 1037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0810b9671f824d0a84741bc1904df247. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:19.561595 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_e44b0cdc7f4640ec867d92338d94426b" } modify_external_catalogs: true
I20250113 02:51:19.562063 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_e44b0cdc7f4640ec867d92338d94426b" } modify_external_catalogs: true
I20250113 02:51:19.564113 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11950e996c814590bc9f3f22c0354902. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:19.565619 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet 0c57a5add4914835ac366efdcde4ee28 (table default.Table_3b52330dd7f04b9db3ef41770dc87651 [id=1a69ebe950b846c78ec36ca96a1f9e6c]) successfully deleted
I20250113 02:51:19.571017 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f68f9f4076d9475d938e795e72bdd468. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:19.576683 887 tablet_service.cc:1467] Processing CreateTablet for tablet c94eedb5dce44e6d896192c93127903a (DEFAULT_TABLE table=default.Table_616eda5ca6ce4155b20351b6b8b000f5 [id=e0cdd94a9fc947fc8a9555150b7272f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:19.581333 896 tablet_service.cc:1467] Processing CreateTablet for tablet 0810b9671f824d0a84741bc1904df247 (DEFAULT_TABLE table=default.Table_796801d17a3c458693693af6e39cf0f5 [id=df24b379d986402187cec33190501012]), partition=RANGE (key) PARTITION UNBOUNDED
W20250113 02:51:19.582283 851 leader_election.cc:343] T 0c55662a71444a3683a825bcec6c5b68 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:19.584070 893 tablet_service.cc:1467] Processing CreateTablet for tablet a77c38a394114d6c8a8da99d7eef7669 (DEFAULT_TABLE table=default.Table_7761343ccac947069c26e1e705e0a9b3 [id=d1b33c3e180343f5b88dcb200e7c9dde]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:19.585472 883 tablet_service.cc:1467] Processing CreateTablet for tablet 4c615b0b83d74fcfae0c51047dbfbbf4 (DEFAULT_TABLE table=default.Table_668e70d7e5c64c37ab05e7bcbe56d994 [id=a956f95f6dd94b578f6bd3cb3b97f9da]), partition=RANGE (key) PARTITION UNBOUNDED
W20250113 02:51:19.587121 852 leader_election.cc:343] T 0c55662a71444a3683a825bcec6c5b68 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
W20250113 02:51:19.586752 851 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:40095, user_credentials={real_user=slave}} blocked reactor thread for 48837.9us
I20250113 02:51:19.607945 852 leader_election.cc:304] T 0c55662a71444a3683a825bcec6c5b68 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df69f48656b44bb890f76587ee69cc0f; no voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca
I20250113 02:51:19.611088 1298 raft_consensus.cc:2743] T 0c55662a71444a3683a825bcec6c5b68 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:19.601622 882 tablet_service.cc:1467] Processing CreateTablet for tablet 11950e996c814590bc9f3f22c0354902 (DEFAULT_TABLE table=default.Table_003cb5a672084be4b29d515fa4485747 [id=2e275f7f92ac440ca3ea6e29c5566f00]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:19.613324 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c94eedb5dce44e6d896192c93127903a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:19.613605 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a77c38a394114d6c8a8da99d7eef7669. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:19.613879 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c615b0b83d74fcfae0c51047dbfbbf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:19.607329 881 tablet_service.cc:1467] Processing CreateTablet for tablet 1cfbcd682b944c89847b40fe69f0a75e (DEFAULT_TABLE table=default.Table_9cfb6bce41c0468e8d29a6d0a1f32499 [id=2b84deb07e5c4ab8b550fe42945d7b71]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:19.607447 884 tablet_service.cc:1467] Processing CreateTablet for tablet 092892a36e0e47cb8d2184c2bbde95d7 (DEFAULT_TABLE table=default.Table_22cf65ed324549aa9fa3c6e305bf0e1c [id=d4bf205fb0464358af61bb3fb889c687]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:19.620414 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0810b9671f824d0a84741bc1904df247. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:19.621922 885 tablet_service.cc:1467] Processing CreateTablet for tablet efccefd6787943469131492741ee2bd5 (DEFAULT_TABLE table=default.Table_67151d370cfd45ccaac504631aa0f1f9 [id=bcdd761e38934a2e9197b9c1b9165b27]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:19.625380 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a77c38a394114d6c8a8da99d7eef7669. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:19.629809 998 leader_election.cc:343] T 78689f7ed2a64405adb75f372465a792 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Not found: Tablet not found: 78689f7ed2a64405adb75f372465a792
I20250113 02:51:19.629966 998 leader_election.cc:304] T 78689f7ed2a64405adb75f372465a792 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4216a7f853d642b7bc3e5cff458f3aca; no voters: 1186fa869eb14812832fa6562531f196, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:19.630421 1311 raft_consensus.cc:2743] T 78689f7ed2a64405adb75f372465a792 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:19.634543 753 tablet_service.cc:1467] Processing CreateTablet for tablet 7d3562a48c6e4badbbcf37640e91b5ba (DEFAULT_TABLE table=default.Table_c5c12ef64b7b438eb52313e87793e601 [id=7c2bc6d4232f4585b84d71e3e1f85794]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:19.635134 1191 tablet_bootstrap.cc:492] T ad6b21ed2b824cb0837fc1d8a39d2472 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:19.636205 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1cfbcd682b944c89847b40fe69f0a75e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:19.635119 895 tablet_service.cc:1467] Processing CreateTablet for tablet ff8ef6290698477391721fd65b9eeb28 (DEFAULT_TABLE table=default.Table_5b825152ff4d4e20a7600a9ad0dfe0a3 [id=7e176c2354964cdd8e6b724f6bb84b99]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:19.638379 1191 ts_tablet_manager.cc:1397] T ad6b21ed2b824cb0837fc1d8a39d2472 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.385s user 0.013s sys 0.020s
W20250113 02:51:19.638752 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7f08b72301dc431396cf8cc7790d56d4 consensus state has no leader
I20250113 02:51:19.638852 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 140) took 1638 ms. Trace:
I20250113 02:51:19.638962 1195 rpcz_store.cc:276] 0113 02:51:17.996323 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:18.154319 (+157996us) service_pool.cc:225] Handling call
0113 02:51:19.551223 (+1396904us) ts_tablet_manager.cc:1145] Deleting tablet 3455f010e51b4db1a9804a57fd1a4fb8
0113 02:51:19.551599 (+ 376us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:19.635087 (+ 83488us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1392904}
I20250113 02:51:19.639241 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_f5f150e8da7a4848bfab67568975d66f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:19.639444 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 141) took 1553 ms. Trace:
I20250113 02:51:19.639539 1195 rpcz_store.cc:276] 0113 02:51:18.085820 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:18.157814 (+ 71994us) service_pool.cc:225] Handling call
0113 02:51:19.639248 (+1481434us) ts_tablet_manager.cc:1145] Deleting tablet 107e6e9e2d84496f8fce92b9ba377189
0113 02:51:19.639259 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:19.639386 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1462968}
W20250113 02:51:19.639837 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f5f150e8da7a4848bfab67568975d66f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:19.639997 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 143) took 1536 ms. Trace:
I20250113 02:51:19.640096 1195 rpcz_store.cc:276] 0113 02:51:18.103261 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:18.177889 (+ 74628us) service_pool.cc:225] Handling call
0113 02:51:19.639798 (+1461909us) ts_tablet_manager.cc:1145] Deleting tablet d101e98b72544c30a3a4bc60d1576aa3
0113 02:51:19.639816 (+ 18us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:19.639913 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1461720}
I20250113 02:51:19.640517 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 142) took 1551 ms. Trace:
I20250113 02:51:19.640614 1195 rpcz_store.cc:276] 0113 02:51:18.089160 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:18.176314 (+ 87154us) service_pool.cc:225] Handling call
0113 02:51:19.640341 (+1464027us) ts_tablet_manager.cc:1145] Deleting tablet 35e0aed5125c4e1fa5dce226eade3b78
0113 02:51:19.640350 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:19.640437 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1455339}
I20250113 02:51:19.641037 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 144) took 1537 ms. Trace:
W20250113 02:51:19.641239 451 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a77c38a394114d6c8a8da99d7eef7669 consensus state has no leader
I20250113 02:51:19.641633 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_4deab2d4aa9e4d6f8e165675d04c119a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:19.642102 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4deab2d4aa9e4d6f8e165675d04c119a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:19.642895 1191 raft_consensus.cc:357] T ad6b21ed2b824cb0837fc1d8a39d2472 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:19.643169 1191 raft_consensus.cc:383] T ad6b21ed2b824cb0837fc1d8a39d2472 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:19.643260 1191 raft_consensus.cc:738] T ad6b21ed2b824cb0837fc1d8a39d2472 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:19.643517 454 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_0917607e26ce49bfad7e1dc99a894a05" } new_table_name: "default.Table_243aff1bbfc6454b92f66035f64a7b84" modify_external_catalogs: true
I20250113 02:51:19.643505 1191 consensus_queue.cc:260] T ad6b21ed2b824cb0837fc1d8a39d2472 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:19.644131 1191 ts_tablet_manager.cc:1428] T ad6b21ed2b824cb0837fc1d8a39d2472 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250113 02:51:19.647876 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_4385ce4af1044706afef0ab60f2710a7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:19.648461 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4385ce4af1044706afef0ab60f2710a7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:19.648524 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d3562a48c6e4badbbcf37640e91b5ba. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:19.649045 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 93a2729c5a604a27a9bbc451ccb4e128 consensus state has no leader
I20250113 02:51:19.649389 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_53a85211f49241c4af531e479ecdc0cb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:19.649869 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_53a85211f49241c4af531e479ecdc0cb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:19.635725 886 tablet_service.cc:1467] Processing CreateTablet for tablet 840773ee80c7447097ea000b106fc89d (DEFAULT_TABLE table=default.Table_5aafd46e1cbf4e03a5b87797d5d97bdd [id=b6d8d0de849a4d3c988b8fd468b14d62]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:19.650887 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 258fad818b7b4aa4b337f9949e20aa46
I20250113 02:51:19.651053 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet ebc5f1ea1e1648d49904b17603d3cd2a
I20250113 02:51:19.641135 1195 rpcz_store.cc:276] 0113 02:51:18.103356 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:18.178086 (+ 74730us) service_pool.cc:225] Handling call
0113 02:51:19.640840 (+1462754us) ts_tablet_manager.cc:1145] Deleting tablet fc92f19390fd48a4a45ba8fb839020fd
0113 02:51:19.640850 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:19.640960 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1450767}
I20250113 02:51:19.629230 880 tablet_service.cc:1467] Processing CreateTablet for tablet 41fcde988f72496b824271fa36c1a226 (DEFAULT_TABLE table=default.Table_2caed919541e43c78a858513a2128f50 [id=26ba87b9bd384b99a016b29fe5b5babd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:19.651700 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_03babf194cc448f296512a09b2e3fe3e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:19.651759 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_9f627945c5404e48977cf58b41173f50"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:19.652199 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet ee7582ee04494c0db1179359bcb311c8
W20250113 02:51:19.652312 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_03babf194cc448f296512a09b2e3fe3e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:19.652864 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 840773ee80c7447097ea000b106fc89d
I20250113 02:51:19.653321 891 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d132c29cb2c46108e5877e38068ef17 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:17 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:19.653518 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 601c71d5f7624fbb87b365d2e775d33e
W20250113 02:51:19.655833 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 35e0aed5125c4e1fa5dce226eade3b78 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 35e0aed5125c4e1fa5dce226eade3b78 already in progress: creating tablet
I20250113 02:51:19.656296 1191 tablet_bootstrap.cc:492] T 78689f7ed2a64405adb75f372465a792 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:19.657783 891 tablet_service.cc:1467] Processing CreateTablet for tablet 7d3562a48c6e4badbbcf37640e91b5ba (DEFAULT_TABLE table=default.Table_c5c12ef64b7b438eb52313e87793e601 [id=7c2bc6d4232f4585b84d71e3e1f85794]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:19.658705 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11950e996c814590bc9f3f22c0354902. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:19.648949 1209 raft_consensus.cc:513] T a028cc02273644608f046135851da126 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:19.661514 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 092892a36e0e47cb8d2184c2bbde95d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:19.662668 1270 raft_consensus.cc:491] T ec0750fd2fa247feb3ea310ffabe05e5 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:19.660921 755 tablet_service.cc:1467] Processing CreateTablet for tablet de4a58bf63054890808e2e2fe46e43d2 (DEFAULT_TABLE table=default.Table_22fdcd0b65d14830878ae2d87bca9bb9 [id=9f5add219a7c48da9a8afa966e22d1bd]), partition=RANGE (key) PARTITION UNBOUNDED
W20250113 02:51:19.661397 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9f627945c5404e48977cf58b41173f50 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250113 02:51:19.664875 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet fc92f19390fd48a4a45ba8fb839020fd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fc92f19390fd48a4a45ba8fb839020fd already in progress: creating tablet
I20250113 02:51:19.665881 1062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a028cc02273644608f046135851da126" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca"
I20250113 02:51:19.666127 1062 raft_consensus.cc:3054] T a028cc02273644608f046135851da126 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:19.668934 1062 raft_consensus.cc:2463] T a028cc02273644608f046135851da126 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1186fa869eb14812832fa6562531f196 in term 1.
I20250113 02:51:19.669765 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c615b0b83d74fcfae0c51047dbfbbf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:19.669708 1025 tablet_service.cc:1467] Processing CreateTablet for tablet 7d3562a48c6e4badbbcf37640e91b5ba (DEFAULT_TABLE table=default.Table_c5c12ef64b7b438eb52313e87793e601 [id=7c2bc6d4232f4585b84d71e3e1f85794]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:19.670121 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d3562a48c6e4badbbcf37640e91b5ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:19.670677 709 leader_election.cc:304] T a028cc02273644608f046135851da126 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters:
I20250113 02:51:19.670629 1270 raft_consensus.cc:513] T ec0750fd2fa247feb3ea310ffabe05e5 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:19.671041 1368 raft_consensus.cc:2798] T a028cc02273644608f046135851da126 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Leader election won for term 1
I20250113 02:51:19.670959 894 tablet_service.cc:1467] Processing CreateTablet for tablet de4a58bf63054890808e2e2fe46e43d2 (DEFAULT_TABLE table=default.Table_22fdcd0b65d14830878ae2d87bca9bb9 [id=9f5add219a7c48da9a8afa966e22d1bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:19.672286 1036 tablet_service.cc:1467] Processing CreateTablet for tablet de4a58bf63054890808e2e2fe46e43d2 (DEFAULT_TABLE table=default.Table_22fdcd0b65d14830878ae2d87bca9bb9 [id=9f5add219a7c48da9a8afa966e22d1bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:19.672415 1209 leader_election.cc:290] T a028cc02273644608f046135851da126 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 election: Requested vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:19.672483 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 840773ee80c7447097ea000b106fc89d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:19.672719 1036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de4a58bf63054890808e2e2fe46e43d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:19.673312 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 41fcde988f72496b824271fa36c1a226. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:19.673923 754 tablet_service.cc:1467] Processing CreateTablet for tablet be4b2a8a3927410eaf12e7592646e4af (DEFAULT_TABLE table=default.Table_01a7a0bb02fd483ebe043578768a48e6 [id=124669ccadaf49e19d2660b1495f9647]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:19.674407 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff8ef6290698477391721fd65b9eeb28. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:19.675235 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 147) took 1553 ms. Trace:
I20250113 02:51:19.675781 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1cfbcd682b944c89847b40fe69f0a75e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:19.676282 1037 tablet_service.cc:1467] Processing CreateTablet for tablet bffec36d567544c5a1c87e64a7e9730b (DEFAULT_TABLE table=default.Table_c79704f27bf14cba8a0afc2dd36a143f [id=f26e79c7d9164cc89d184f1ffbd21036]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:19.676447 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet efccefd6787943469131492741ee2bd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:19.676707 1037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bffec36d567544c5a1c87e64a7e9730b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:19.679751 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff8ef6290698477391721fd65b9eeb28. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:19.680251 1035 tablet_service.cc:1467] Processing CreateTablet for tablet 23e04961f072440fa3e3f9485e40f79c (DEFAULT_TABLE table=default.Table_4a0258998d4f413b80d701c74526b86d [id=6fd9215b11274a76b06813227889927b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:19.680635 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23e04961f072440fa3e3f9485e40f79c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:19.681172 1023 tablet_service.cc:1467] Processing CreateTablet for tablet be4b2a8a3927410eaf12e7592646e4af (DEFAULT_TABLE table=default.Table_01a7a0bb02fd483ebe043578768a48e6 [id=124669ccadaf49e19d2660b1495f9647]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:19.681546 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be4b2a8a3927410eaf12e7592646e4af. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:19.682034 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 41fcde988f72496b824271fa36c1a226. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:19.682772 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet 78689f7ed2a64405adb75f372465a792 with delete_type TABLET_DATA_DELETED (replaced by tablet 0433cfe0cb524511954e1012d4d137e6) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:19.683192 1194 tablet_bootstrap.cc:492] T 3455f010e51b4db1a9804a57fd1a4fb8 P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:19.685055 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d3562a48c6e4badbbcf37640e91b5ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:19.686065 751 tablet_service.cc:1467] Processing CreateTablet for tablet 23e04961f072440fa3e3f9485e40f79c (DEFAULT_TABLE table=default.Table_4a0258998d4f413b80d701c74526b86d [id=6fd9215b11274a76b06813227889927b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:19.686502 743 tablet_service.cc:1467] Processing CreateTablet for tablet bffec36d567544c5a1c87e64a7e9730b (DEFAULT_TABLE table=default.Table_c79704f27bf14cba8a0afc2dd36a143f [id=f26e79c7d9164cc89d184f1ffbd21036]), partition=RANGE (key) PARTITION UNBOUNDED
W20250113 02:51:19.687464 711 leader_election.cc:343] T a028cc02273644608f046135851da126 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Not found: Tablet not found: a028cc02273644608f046135851da126
I20250113 02:51:19.687115 1195 rpcz_store.cc:276] 0113 02:51:18.103563 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:18.190325 (+ 86762us) service_pool.cc:225] Handling call
0113 02:51:19.651275 (+1460950us) ts_tablet_manager.cc:1145] Deleting tablet d7c4c7b0cfc44ac1952bc73cbe1bced8
0113 02:51:19.652591 (+ 1316us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:19.657237 (+ 4646us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1442875}
I20250113 02:51:19.688628 1198 ts_tablet_manager.cc:1918] T c135506190f849008ce6542d569b402a P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:19.688795 1198 log.cc:1198] T c135506190f849008ce6542d569b402a P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/c135506190f849008ce6542d569b402a
I20250113 02:51:19.689787 1198 ts_tablet_manager.cc:1939] T c135506190f849008ce6542d569b402a P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:19.690896 1198 tablet_replica.cc:331] T ad6b21ed2b824cb0837fc1d8a39d2472 P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:19.691133 1198 raft_consensus.cc:2238] T ad6b21ed2b824cb0837fc1d8a39d2472 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:19.691524 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de4a58bf63054890808e2e2fe46e43d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:19.692495 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 146) took 1587 ms. Trace:
I20250113 02:51:19.694095 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_93a7b6abed0446a4baf21e5e7c8f0c1b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:19.694067 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet d7c4c7b0cfc44ac1952bc73cbe1bced8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d7c4c7b0cfc44ac1952bc73cbe1bced8 already in progress: creating tablet
W20250113 02:51:19.694622 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_93a7b6abed0446a4baf21e5e7c8f0c1b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:19.695912 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_4b5273b24d914b4eadec5f883245398a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:19.696417 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4b5273b24d914b4eadec5f883245398a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250113 02:51:19.696517 711 leader_election.cc:343] T ec0750fd2fa247feb3ea310ffabe05e5 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Not found: Tablet not found: ec0750fd2fa247feb3ea310ffabe05e5
I20250113 02:51:19.693812 1368 raft_consensus.cc:695] T a028cc02273644608f046135851da126 P 1186fa869eb14812832fa6562531f196 [term 1 LEADER]: Becoming Leader. State: Replica: 1186fa869eb14812832fa6562531f196, State: Running, Role: LEADER
I20250113 02:51:19.699744 1195 rpcz_store.cc:276] 0113 02:51:18.103492 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:18.190088 (+ 86596us) service_pool.cc:225] Handling call
0113 02:51:19.688612 (+1498524us) ts_tablet_manager.cc:1145] Deleting tablet f0e7cc161f354f86b8c2fef303e87c5b
0113 02:51:19.688625 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:19.690867 (+ 2242us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1477952}
I20250113 02:51:19.700467 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 258fad818b7b4aa4b337f9949e20aa46 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:19 UTC) from {username='slave'} at 127.0.0.1:55398
W20250113 02:51:19.701125 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet f0e7cc161f354f86b8c2fef303e87c5b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f0e7cc161f354f86b8c2fef303e87c5b already in progress: creating tablet
I20250113 02:51:19.701395 1270 leader_election.cc:290] T ec0750fd2fa247feb3ea310ffabe05e5 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:19.701411 1194 ts_tablet_manager.cc:1397] T 3455f010e51b4db1a9804a57fd1a4fb8 P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.280s user 0.018s sys 0.029s
I20250113 02:51:19.702067 740 tablet_service.cc:1514] Processing DeleteTablet for tablet ebc5f1ea1e1648d49904b17603d3cd2a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:19 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:19.702283 740 tablet_service.cc:1514] Processing DeleteTablet for tablet 601c71d5f7624fbb87b365d2e775d33e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:19 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:19.703929 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be4b2a8a3927410eaf12e7592646e4af. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:19.704102 1194 raft_consensus.cc:357] T 3455f010e51b4db1a9804a57fd1a4fb8 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:19.705021 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de4a58bf63054890808e2e2fe46e43d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:19.704375 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet c135506190f849008ce6542d569b402a (table default.Table_5e0263e22f1e4d2b983ae6b6ef03fca5 [id=377f041bf95e41f3b7e8ea26f9795b6a]) successfully deleted
I20250113 02:51:19.709506 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bffec36d567544c5a1c87e64a7e9730b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:19.710695 740 tablet_service.cc:1514] Processing DeleteTablet for tablet ad6b21ed2b824cb0837fc1d8a39d2472 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:16 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:19.718292 899 tablet_service.cc:1467] Processing CreateTablet for tablet bffec36d567544c5a1c87e64a7e9730b (DEFAULT_TABLE table=default.Table_c79704f27bf14cba8a0afc2dd36a143f [id=f26e79c7d9164cc89d184f1ffbd21036]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:19.722271 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 145) took 1597 ms. Trace:
I20250113 02:51:19.724159 739 tablet_service.cc:1514] Processing DeleteTablet for tablet ee7582ee04494c0db1179359bcb311c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:19 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:19.726325 1195 rpcz_store.cc:276] 0113 02:51:18.103420 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:18.185049 (+ 81629us) service_pool.cc:225] Handling call
0113 02:51:19.700239 (+1515190us) ts_tablet_manager.cc:1145] Deleting tablet b0361f11d0974ed19e7d4eb1c32a2f47
0113 02:51:19.700251 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:19.700875 (+ 624us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1486874}
I20250113 02:51:19.691162 1031 tablet_service.cc:1514] Processing DeleteTablet for tablet 258fad818b7b4aa4b337f9949e20aa46 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:19 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:19.694590 1032 tablet_service.cc:1514] Processing DeleteTablet for tablet ebc5f1ea1e1648d49904b17603d3cd2a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:19 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:19.695550 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet 601c71d5f7624fbb87b365d2e775d33e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:19 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:19.695770 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet ee7582ee04494c0db1179359bcb311c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:19 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:19.695926 1039 tablet_service.cc:1514] Processing DeleteTablet for tablet 840773ee80c7447097ea000b106fc89d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:19 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:19.699265 1062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec0750fd2fa247feb3ea310ffabe05e5" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:19.727182 1062 raft_consensus.cc:2463] T ec0750fd2fa247feb3ea310ffabe05e5 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1186fa869eb14812832fa6562531f196 in term 0.
I20250113 02:51:19.727800 1198 raft_consensus.cc:2267] T ad6b21ed2b824cb0837fc1d8a39d2472 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:19.728464 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23e04961f072440fa3e3f9485e40f79c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:19.728590 1195 tablet_replica.cc:331] T 7dbda00269cf4f7095b3504ae87ac243 P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:19.729522 1195 raft_consensus.cc:2238] T 7dbda00269cf4f7095b3504ae87ac243 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:19.729970 709 leader_election.cc:304] T ec0750fd2fa247feb3ea310ffabe05e5 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters: df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:19.726610 1368 consensus_queue.cc:237] T a028cc02273644608f046135851da126 P 1186fa869eb14812832fa6562531f196 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:19.731094 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 840773ee80c7447097ea000b106fc89d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:19 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:19.731174 892 tablet_service.cc:1467] Processing CreateTablet for tablet be4b2a8a3927410eaf12e7592646e4af (DEFAULT_TABLE table=default.Table_01a7a0bb02fd483ebe043578768a48e6 [id=124669ccadaf49e19d2660b1495f9647]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:19.731225 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bffec36d567544c5a1c87e64a7e9730b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:19.732671 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be4b2a8a3927410eaf12e7592646e4af. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:19.732697 1195 raft_consensus.cc:2267] T 7dbda00269cf4f7095b3504ae87ac243 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:19.731094 1194 raft_consensus.cc:383] T 3455f010e51b4db1a9804a57fd1a4fb8 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:19.733922 1194 raft_consensus.cc:738] T 3455f010e51b4db1a9804a57fd1a4fb8 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:19.733999 1192 tablet_bootstrap.cc:492] T 3d132c29cb2c46108e5877e38068ef17 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:19.734153 1192 ts_tablet_manager.cc:1397] T 3d132c29cb2c46108e5877e38068ef17 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.245s user 0.008s sys 0.032s
I20250113 02:51:19.735116 1194 consensus_queue.cc:260] T 3455f010e51b4db1a9804a57fd1a4fb8 P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
W20250113 02:51:19.736099 454 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e8ab4516789e4585a101c8766d338af3 consensus state has no leader
I20250113 02:51:19.736198 1194 ts_tablet_manager.cc:1428] T 3455f010e51b4db1a9804a57fd1a4fb8 P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.034s user 0.000s sys 0.003s
I20250113 02:51:19.736516 1194 tablet_bootstrap.cc:492] T f0e7cc161f354f86b8c2fef303e87c5b P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:19.737833 1383 raft_consensus.cc:2798] T ec0750fd2fa247feb3ea310ffabe05e5 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 02:51:19.737952 1383 raft_consensus.cc:491] T ec0750fd2fa247feb3ea310ffabe05e5 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 02:51:19.741904 1192 raft_consensus.cc:357] T 3d132c29cb2c46108e5877e38068ef17 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:19.742143 1192 raft_consensus.cc:383] T 3d132c29cb2c46108e5877e38068ef17 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:19.742240 1192 raft_consensus.cc:738] T 3d132c29cb2c46108e5877e38068ef17 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:19.743294 1192 consensus_queue.cc:260] T 3d132c29cb2c46108e5877e38068ef17 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:19.743929 1192 ts_tablet_manager.cc:1428] T 3d132c29cb2c46108e5877e38068ef17 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20250113 02:51:19.744133 1195 ts_tablet_manager.cc:1905] T 7dbda00269cf4f7095b3504ae87ac243 P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:19.745477 1192 tablet_bootstrap.cc:492] T 3455f010e51b4db1a9804a57fd1a4fb8 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:19.749392 1192 tablet_bootstrap.cc:654] T 3455f010e51b4db1a9804a57fd1a4fb8 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:19.754990 1191 tablet_bootstrap.cc:654] T 78689f7ed2a64405adb75f372465a792 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:19.757624 1198 ts_tablet_manager.cc:1905] T ad6b21ed2b824cb0837fc1d8a39d2472 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:19.769773 1062 raft_consensus.cc:1270] T 7f3cbef7d91d4576822d913071a88248 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Refusing update from remote peer 1186fa869eb14812832fa6562531f196: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250113 02:51:19.775696 1383 raft_consensus.cc:3054] T ec0750fd2fa247feb3ea310ffabe05e5 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Advancing to term 1
W20250113 02:51:19.740057 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet b0361f11d0974ed19e7d4eb1c32a2f47 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b0361f11d0974ed19e7d4eb1c32a2f47 already in progress: creating tablet
I20250113 02:51:19.780644 451 master_service.cc:946] ReplaceTablet: received request to replace tablet d7c4c7b0cfc44ac1952bc73cbe1bced8 from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:19.782932 1192 tablet_bootstrap.cc:492] T 3455f010e51b4db1a9804a57fd1a4fb8 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:19.782996 1198 ts_tablet_manager.cc:1918] T ad6b21ed2b824cb0837fc1d8a39d2472 P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:19.783135 1198 log.cc:1198] T ad6b21ed2b824cb0837fc1d8a39d2472 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/ad6b21ed2b824cb0837fc1d8a39d2472
I20250113 02:51:19.783155 1192 ts_tablet_manager.cc:1397] T 3455f010e51b4db1a9804a57fd1a4fb8 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.038s user 0.005s sys 0.012s
I20250113 02:51:19.783672 1198 ts_tablet_manager.cc:1939] T ad6b21ed2b824cb0837fc1d8a39d2472 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
W20250113 02:51:19.783511 439 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.104.1:40095, user_credentials={real_user=slave}} blocked reactor thread for 43917.9us
I20250113 02:51:19.784227 1192 raft_consensus.cc:357] T 3455f010e51b4db1a9804a57fd1a4fb8 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:19.784468 1192 raft_consensus.cc:383] T 3455f010e51b4db1a9804a57fd1a4fb8 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:19.784571 1192 raft_consensus.cc:738] T 3455f010e51b4db1a9804a57fd1a4fb8 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:19.787869 1192 consensus_queue.cc:260] T 3455f010e51b4db1a9804a57fd1a4fb8 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:19.788035 1270 consensus_queue.cc:1035] T 7f3cbef7d91d4576822d913071a88248 P 1186fa869eb14812832fa6562531f196 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.015s
I20250113 02:51:19.789286 1192 ts_tablet_manager.cc:1428] T 3455f010e51b4db1a9804a57fd1a4fb8 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20250113 02:51:19.789659 1192 tablet_bootstrap.cc:492] T d6675bc3cb5f4f809320e28026cdf4b0 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:19.791604 1192 tablet_bootstrap.cc:654] T d6675bc3cb5f4f809320e28026cdf4b0 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:19.794898 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet ad6b21ed2b824cb0837fc1d8a39d2472 (table default.Table_449479ad3dbb4c1aaa5cd954816b9935 [id=61e1ae59b7db4fd0a33fe3f73a712028]) successfully deleted
I20250113 02:51:19.784571 1198 tablet_replica.cc:331] T 1643b3e1607f42fc82886d0b09e2fe8c P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:19.796820 1198 raft_consensus.cc:2238] T 1643b3e1607f42fc82886d0b09e2fe8c P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:19.797114 1198 raft_consensus.cc:2267] T 1643b3e1607f42fc82886d0b09e2fe8c P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:19.797832 888 tablet_service.cc:1467] Processing CreateTablet for tablet 23e04961f072440fa3e3f9485e40f79c (DEFAULT_TABLE table=default.Table_4a0258998d4f413b80d701c74526b86d [id=6fd9215b11274a76b06813227889927b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:19.799063 1198 ts_tablet_manager.cc:1905] T 1643b3e1607f42fc82886d0b09e2fe8c P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:19.790686 919 raft_consensus.cc:1270] T 7f3cbef7d91d4576822d913071a88248 P df69f48656b44bb890f76587ee69cc0f [term 1 FOLLOWER]: Refusing update from remote peer 1186fa869eb14812832fa6562531f196: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250113 02:51:19.802522 1298 raft_consensus.cc:491] T ce92c2b45a274eb09e72a96b7ec133be P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:19.804642 889 tablet_service.cc:1514] Processing DeleteTablet for tablet ebc5f1ea1e1648d49904b17603d3cd2a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:19 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:19.805591 895 tablet_service.cc:1514] Processing DeleteTablet for tablet 601c71d5f7624fbb87b365d2e775d33e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:19 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:19.814838 1198 ts_tablet_manager.cc:1918] T 1643b3e1607f42fc82886d0b09e2fe8c P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:19.815064 1198 log.cc:1198] T 1643b3e1607f42fc82886d0b09e2fe8c P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/1643b3e1607f42fc82886d0b09e2fe8c
I20250113 02:51:19.815585 1198 ts_tablet_manager.cc:1939] T 1643b3e1607f42fc82886d0b09e2fe8c P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:19.816907 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet 1643b3e1607f42fc82886d0b09e2fe8c (table default.Table_fff2447caa364c3381a3ba0abbde1c50 [id=15d9be3deeb84c1494965848dfb804ac]) successfully deleted
I20250113 02:51:19.817286 1198 tablet_replica.cc:331] T ec0750fd2fa247feb3ea310ffabe05e5 P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:19.817492 1198 raft_consensus.cc:2238] T ec0750fd2fa247feb3ea310ffabe05e5 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:19.817760 1198 raft_consensus.cc:2267] T ec0750fd2fa247feb3ea310ffabe05e5 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:19.818929 883 tablet_service.cc:1514] Processing DeleteTablet for tablet ee7582ee04494c0db1179359bcb311c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:19 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:19.819236 887 tablet_service.cc:1514] Processing DeleteTablet for tablet 840773ee80c7447097ea000b106fc89d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:19 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:19.819486 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23e04961f072440fa3e3f9485e40f79c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:19.819850 1198 ts_tablet_manager.cc:1905] T ec0750fd2fa247feb3ea310ffabe05e5 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:19.820166 1298 raft_consensus.cc:513] T ce92c2b45a274eb09e72a96b7ec133be P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:19.821282 1194 tablet_bootstrap.cc:654] T f0e7cc161f354f86b8c2fef303e87c5b P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:19.824416 1196 ts_tablet_manager.cc:1918] T bc88676bf738442d82c25e56f30c4ada P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250113 02:51:19.828372 1270 consensus_queue.cc:1035] T 7f3cbef7d91d4576822d913071a88248 P 1186fa869eb14812832fa6562531f196 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.016s
I20250113 02:51:19.832806 1298 leader_election.cc:290] T ce92c2b45a274eb09e72a96b7ec133be P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:19.833685 1196 log.cc:1198] T bc88676bf738442d82c25e56f30c4ada P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/bc88676bf738442d82c25e56f30c4ada
I20250113 02:51:19.833832 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce92c2b45a274eb09e72a96b7ec133be" candidate_uuid: "df69f48656b44bb890f76587ee69cc0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:19.834115 1196 ts_tablet_manager.cc:1939] T bc88676bf738442d82c25e56f30c4ada P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:19.837218 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet bc88676bf738442d82c25e56f30c4ada (table default.Table_c991f9dbd70d4c4682664b8711f05db4 [id=2c64da64ecdb4bf48e11b523533902e8]) successfully deleted
W20250113 02:51:19.837863 852 leader_election.cc:343] T ce92c2b45a274eb09e72a96b7ec133be P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:19.839397 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce92c2b45a274eb09e72a96b7ec133be" candidate_uuid: "df69f48656b44bb890f76587ee69cc0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
W20250113 02:51:19.842690 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet f760fc1e4bed433dbb987a42c12a1ee6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f760fc1e4bed433dbb987a42c12a1ee6 already in progress: creating tablet
W20250113 02:51:19.843509 851 leader_election.cc:343] T ce92c2b45a274eb09e72a96b7ec133be P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:19.843703 851 leader_election.cc:304] T ce92c2b45a274eb09e72a96b7ec133be P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df69f48656b44bb890f76587ee69cc0f; no voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca
I20250113 02:51:19.836195 893 tablet_service.cc:1514] Processing DeleteTablet for tablet 258fad818b7b4aa4b337f9949e20aa46 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:19 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:19.847005 1198 ts_tablet_manager.cc:1918] T ec0750fd2fa247feb3ea310ffabe05e5 P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:19.847146 1298 raft_consensus.cc:2743] T ce92c2b45a274eb09e72a96b7ec133be P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:19.847219 1198 log.cc:1198] T ec0750fd2fa247feb3ea310ffabe05e5 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/ec0750fd2fa247feb3ea310ffabe05e5
I20250113 02:51:19.849381 1196 tablet_replica.cc:331] T c135506190f849008ce6542d569b402a P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:19.852718 1198 ts_tablet_manager.cc:1939] T ec0750fd2fa247feb3ea310ffabe05e5 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:19.853459 1196 raft_consensus.cc:2238] T c135506190f849008ce6542d569b402a P df69f48656b44bb890f76587ee69cc0f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:19.854090 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet ec0750fd2fa247feb3ea310ffabe05e5 (table default.Table_74627d276edf4d75a2fcaa627a7f921b [id=de6b4835f6c04c20b6a5ef8404dc78f7]) successfully deleted
I20250113 02:51:19.854480 1196 raft_consensus.cc:2267] T c135506190f849008ce6542d569b402a P df69f48656b44bb890f76587ee69cc0f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:19.855221 1196 ts_tablet_manager.cc:1905] T c135506190f849008ce6542d569b402a P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:19.856312 1198 tablet_replica.cc:331] T a028cc02273644608f046135851da126 P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:19.856865 1298 raft_consensus.cc:491] T 8193357427f4491c8c6c1396ceeef2dc P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:19.857267 1198 raft_consensus.cc:2238] T a028cc02273644608f046135851da126 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:19.857463 1198 raft_consensus.cc:2267] T a028cc02273644608f046135851da126 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:19.858438 1198 ts_tablet_manager.cc:1905] T a028cc02273644608f046135851da126 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:19.858347 1298 raft_consensus.cc:513] T 8193357427f4491c8c6c1396ceeef2dc P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:19.859666 1298 leader_election.cc:290] T 8193357427f4491c8c6c1396ceeef2dc P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:19.860447 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8193357427f4491c8c6c1396ceeef2dc" candidate_uuid: "df69f48656b44bb890f76587ee69cc0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:19.860677 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8193357427f4491c8c6c1396ceeef2dc" candidate_uuid: "df69f48656b44bb890f76587ee69cc0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
W20250113 02:51:19.861642 852 leader_election.cc:343] T 8193357427f4491c8c6c1396ceeef2dc P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
W20250113 02:51:19.861943 851 leader_election.cc:343] T 8193357427f4491c8c6c1396ceeef2dc P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:19.862077 851 leader_election.cc:304] T 8193357427f4491c8c6c1396ceeef2dc P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df69f48656b44bb890f76587ee69cc0f; no voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca
I20250113 02:51:19.862587 1298 raft_consensus.cc:2743] T 8193357427f4491c8c6c1396ceeef2dc P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:19.872192 1195 ts_tablet_manager.cc:1918] T 7dbda00269cf4f7095b3504ae87ac243 P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:19.872395 1195 log.cc:1198] T 7dbda00269cf4f7095b3504ae87ac243 P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/7dbda00269cf4f7095b3504ae87ac243
I20250113 02:51:19.873512 1195 ts_tablet_manager.cc:1939] T 7dbda00269cf4f7095b3504ae87ac243 P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:19.873934 1198 ts_tablet_manager.cc:1918] T a028cc02273644608f046135851da126 P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:19.874424 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 162) took 1413 ms. Trace:
I20250113 02:51:19.875332 1191 tablet_bootstrap.cc:492] T 78689f7ed2a64405adb75f372465a792 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:19.875432 1198 log.cc:1198] T a028cc02273644608f046135851da126 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/a028cc02273644608f046135851da126
I20250113 02:51:19.875866 1198 ts_tablet_manager.cc:1939] T a028cc02273644608f046135851da126 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:19.877166 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet a028cc02273644608f046135851da126 (table default.Table_eec2c35d4a8a4196983a872a0fa4bda5 [id=c7f464a7e8f14a0892d145a69a148e58]) successfully deleted
I20250113 02:51:19.877252 1198 tablet_replica.cc:331] T 78689f7ed2a64405adb75f372465a792 P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:19.877457 1198 raft_consensus.cc:2238] T 78689f7ed2a64405adb75f372465a792 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:19.877681 1198 raft_consensus.cc:2267] T 78689f7ed2a64405adb75f372465a792 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus is shut down!
W20250113 02:51:19.877879 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 21c4b419bf2148e0b140301fa80622bb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 21c4b419bf2148e0b140301fa80622bb already in progress: creating tablet
I20250113 02:51:19.878475 1194 tablet_bootstrap.cc:492] T f0e7cc161f354f86b8c2fef303e87c5b P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:19.878468 1191 ts_tablet_manager.cc:1397] T 78689f7ed2a64405adb75f372465a792 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.227s user 0.006s sys 0.004s
I20250113 02:51:19.878692 1194 ts_tablet_manager.cc:1397] T f0e7cc161f354f86b8c2fef303e87c5b P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.142s user 0.000s sys 0.009s
I20250113 02:51:19.879469 1194 raft_consensus.cc:357] T f0e7cc161f354f86b8c2fef303e87c5b P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:19.880792 1194 raft_consensus.cc:383] T f0e7cc161f354f86b8c2fef303e87c5b P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:19.881793 1194 raft_consensus.cc:738] T f0e7cc161f354f86b8c2fef303e87c5b P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:19.882114 1194 consensus_queue.cc:260] T f0e7cc161f354f86b8c2fef303e87c5b P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:19.882699 1194 ts_tablet_manager.cc:1428] T f0e7cc161f354f86b8c2fef303e87c5b P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20250113 02:51:19.883121 1194 tablet_bootstrap.cc:492] T c62a5e9e43d14b5c94eb87260528469d P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:19.883606 1198 ts_tablet_manager.cc:1905] T 78689f7ed2a64405adb75f372465a792 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:19.883450 1196 ts_tablet_manager.cc:1918] T c135506190f849008ce6542d569b402a P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:19.883957 1196 log.cc:1198] T c135506190f849008ce6542d569b402a P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/c135506190f849008ce6542d569b402a
I20250113 02:51:19.886246 1196 ts_tablet_manager.cc:1939] T c135506190f849008ce6542d569b402a P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:19.887449 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet c135506190f849008ce6542d569b402a (table default.Table_5e0263e22f1e4d2b983ae6b6ef03fca5 [id=377f041bf95e41f3b7e8ea26f9795b6a]) successfully deleted
I20250113 02:51:19.886792 1195 rpcz_store.cc:276] 0113 02:51:18.461148 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:18.461212 (+ 64us) service_pool.cc:225] Handling call
0113 02:51:19.726699 (+1265487us) ts_tablet_manager.cc:1145] Deleting tablet 7dbda00269cf4f7095b3504ae87ac243
0113 02:51:19.726714 (+ 15us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:19.840746 (+114032us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:19.854839 (+ 14093us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:19.874380 (+ 19541us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5911,"tablet-delete.queue_time_us":1265213}
I20250113 02:51:19.891181 1196 tablet_replica.cc:331] T 0c55662a71444a3683a825bcec6c5b68 P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:19.891616 1196 raft_consensus.cc:2238] T 0c55662a71444a3683a825bcec6c5b68 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:19.892071 1196 raft_consensus.cc:2267] T 0c55662a71444a3683a825bcec6c5b68 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:19.885555 1194 tablet_bootstrap.cc:654] T c62a5e9e43d14b5c94eb87260528469d P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:19.894171 1196 ts_tablet_manager.cc:1905] T 0c55662a71444a3683a825bcec6c5b68 P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:19.895146 1383 raft_consensus.cc:513] T ec0750fd2fa247feb3ea310ffabe05e5 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:19.899291 899 tablet_service.cc:1514] Processing DeleteTablet for tablet 3455f010e51b4db1a9804a57fd1a4fb8 with delete_type TABLET_DATA_DELETED (replaced by tablet a6055d77d168495d9b756a1a56c1a920) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:19.900300 448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_4fcd958510914007817364e5a8e8ebec" } new_table_name: "default.Table_5182a1c5c7a2462ba930940ae04b505f" modify_external_catalogs: true
I20250113 02:51:19.901309 454 master_service.cc:946] ReplaceTablet: received request to replace tablet 1d81eec487dd4016b3b804451da31c49 from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:19.901747 454 master_service.cc:946] ReplaceTablet: received request to replace tablet 23e9321f08ee4ce7b2252ad5c76c0a08 from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:19.902282 451 master_service.cc:946] ReplaceTablet: received request to replace tablet 32351d6de1574353a6e8073d333da477 from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:19.902666 451 master_service.cc:946] ReplaceTablet: received request to replace tablet ebc5f1ea1e1648d49904b17603d3cd2a from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:19.903367 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_22cf65ed324549aa9fa3c6e305bf0e1c" } new_table_name: "default.Table_03fafac4f68e4eb3be2ea41493d9e006" modify_external_catalogs: true
I20250113 02:51:19.904107 1036 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d132c29cb2c46108e5877e38068ef17 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:17 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:19.904453 1198 ts_tablet_manager.cc:1918] T 78689f7ed2a64405adb75f372465a792 P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:19.905874 1194 tablet_bootstrap.cc:492] T c62a5e9e43d14b5c94eb87260528469d P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:19.906037 1194 ts_tablet_manager.cc:1397] T c62a5e9e43d14b5c94eb87260528469d P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.023s user 0.000s sys 0.004s
I20250113 02:51:19.906728 1194 raft_consensus.cc:357] T c62a5e9e43d14b5c94eb87260528469d P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:19.906932 1194 raft_consensus.cc:383] T c62a5e9e43d14b5c94eb87260528469d P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:19.907006 1194 raft_consensus.cc:738] T c62a5e9e43d14b5c94eb87260528469d P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:19.907178 1198 log.cc:1198] T 78689f7ed2a64405adb75f372465a792 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/78689f7ed2a64405adb75f372465a792
I20250113 02:51:19.907959 1198 ts_tablet_manager.cc:1939] T 78689f7ed2a64405adb75f372465a792 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:19.910163 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet 78689f7ed2a64405adb75f372465a792 (table default.Table_f59080d9d91849deafd22506a1af8ff5 [id=d5354129133b446b8a4a5613db8bc411]) successfully deleted
W20250113 02:51:19.910799 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet ebc5f1ea1e1648d49904b17603d3cd2a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ebc5f1ea1e1648d49904b17603d3cd2a already in progress: creating tablet
I20250113 02:51:19.910936 1198 tablet_replica.cc:331] T 3d132c29cb2c46108e5877e38068ef17 P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
W20250113 02:51:19.911018 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 258fad818b7b4aa4b337f9949e20aa46 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 258fad818b7b4aa4b337f9949e20aa46 already in progress: creating tablet
I20250113 02:51:19.911137 1198 raft_consensus.cc:2238] T 3d132c29cb2c46108e5877e38068ef17 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus shutting down.
W20250113 02:51:19.911219 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 601c71d5f7624fbb87b365d2e775d33e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 601c71d5f7624fbb87b365d2e775d33e already in progress: creating tablet
I20250113 02:51:19.911311 1198 raft_consensus.cc:2267] T 3d132c29cb2c46108e5877e38068ef17 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:19.907243 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 161) took 1432 ms. Trace:
I20250113 02:51:19.911536 1195 rpcz_store.cc:276] 0113 02:51:18.460820 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:18.461712 (+ 892us) service_pool.cc:225] Handling call
0113 02:51:19.890434 (+1428722us) ts_tablet_manager.cc:1145] Deleting tablet 18e2d79d731142f89b9089869bc736e5
0113 02:51:19.890450 (+ 16us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:19.893372 (+ 2922us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1428501}
W20250113 02:51:19.911888 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet ee7582ee04494c0db1179359bcb311c8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ee7582ee04494c0db1179359bcb311c8 already in progress: creating tablet
W20250113 02:51:19.912120 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 840773ee80c7447097ea000b106fc89d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 840773ee80c7447097ea000b106fc89d already in progress: creating tablet
W20250113 02:51:19.912137 711 leader_election.cc:343] T ec0750fd2fa247feb3ea310ffabe05e5 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Not found: Tablet not found: ec0750fd2fa247feb3ea310ffabe05e5
I20250113 02:51:19.912770 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 164) took 1446 ms. Trace:
I20250113 02:51:19.912925 1195 rpcz_store.cc:276] 0113 02:51:18.466714 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:18.466774 (+ 60us) service_pool.cc:225] Handling call
0113 02:51:19.912595 (+1445821us) ts_tablet_manager.cc:1145] Deleting tablet 8aac142230e244c0b59c9c2b7a7300ea
0113 02:51:19.912611 (+ 16us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:19.912733 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1445587}
I20250113 02:51:19.913480 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 163) took 1447 ms. Trace:
I20250113 02:51:19.913619 1195 rpcz_store.cc:276] 0113 02:51:18.466369 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:18.467546 (+ 1177us) service_pool.cc:225] Handling call
0113 02:51:19.913336 (+1445790us) ts_tablet_manager.cc:1145] Deleting tablet 32351d6de1574353a6e8073d333da477
0113 02:51:19.913349 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:19.913447 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1445565}
I20250113 02:51:19.908037 1191 raft_consensus.cc:357] T 78689f7ed2a64405adb75f372465a792 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:19.914103 1191 raft_consensus.cc:383] T 78689f7ed2a64405adb75f372465a792 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:19.914225 1191 raft_consensus.cc:738] T 78689f7ed2a64405adb75f372465a792 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:19.914222 1194 consensus_queue.cc:260] T c62a5e9e43d14b5c94eb87260528469d P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:19.915495 1194 ts_tablet_manager.cc:1428] T c62a5e9e43d14b5c94eb87260528469d P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.009s user 0.001s sys 0.002s
I20250113 02:51:19.915809 1194 tablet_bootstrap.cc:492] T fb183b4588e84212a3b4b096f430d47d P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:19.917332 1195 tablet_replica.cc:331] T db266a1848094b7a98aa75f7afabd471 P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:19.917508 1194 tablet_bootstrap.cc:654] T fb183b4588e84212a3b4b096f430d47d P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:19.918592 1198 ts_tablet_manager.cc:1905] T 3d132c29cb2c46108e5877e38068ef17 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:19.924582 1195 raft_consensus.cc:2238] T db266a1848094b7a98aa75f7afabd471 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:19.924801 1195 raft_consensus.cc:2267] T db266a1848094b7a98aa75f7afabd471 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:19.925511 1195 ts_tablet_manager.cc:1905] T db266a1848094b7a98aa75f7afabd471 P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:19.925688 1196 ts_tablet_manager.cc:1918] T 0c55662a71444a3683a825bcec6c5b68 P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:19.928968 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet 7dbda00269cf4f7095b3504ae87ac243 (table default.Table_8e22a139d87f407da7668fde4cf6c67a [id=46fce62be9184eea82a2fdc88b9aff21]) successfully deleted
W20250113 02:51:19.929472 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 18e2d79d731142f89b9089869bc736e5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 18e2d79d731142f89b9089869bc736e5 already in progress: creating tablet
I20250113 02:51:19.914721 1191 consensus_queue.cc:260] T 78689f7ed2a64405adb75f372465a792 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:19.930219 1194 tablet_bootstrap.cc:492] T fb183b4588e84212a3b4b096f430d47d P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:19.930325 1196 log.cc:1198] T 0c55662a71444a3683a825bcec6c5b68 P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/0c55662a71444a3683a825bcec6c5b68
I20250113 02:51:19.930373 1194 ts_tablet_manager.cc:1397] T fb183b4588e84212a3b4b096f430d47d P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.001s
I20250113 02:51:19.930720 1196 ts_tablet_manager.cc:1939] T 0c55662a71444a3683a825bcec6c5b68 P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:19.931211 1194 raft_consensus.cc:357] T fb183b4588e84212a3b4b096f430d47d P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:19.931406 1194 raft_consensus.cc:383] T fb183b4588e84212a3b4b096f430d47d P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:19.931761 1196 tablet_replica.cc:331] T ce92c2b45a274eb09e72a96b7ec133be P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:19.932052 1196 raft_consensus.cc:2238] T ce92c2b45a274eb09e72a96b7ec133be P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:19.932318 1196 raft_consensus.cc:2267] T ce92c2b45a274eb09e72a96b7ec133be P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:19.933425 1196 ts_tablet_manager.cc:1905] T ce92c2b45a274eb09e72a96b7ec133be P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:19.933972 1191 ts_tablet_manager.cc:1428] T 78689f7ed2a64405adb75f372465a792 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.055s user 0.001s sys 0.001s
I20250113 02:51:19.931495 1194 raft_consensus.cc:738] T fb183b4588e84212a3b4b096f430d47d P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:19.934322 1194 consensus_queue.cc:260] T fb183b4588e84212a3b4b096f430d47d P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:19.934815 1194 ts_tablet_manager.cc:1428] T fb183b4588e84212a3b4b096f430d47d P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250113 02:51:19.935111 1194 tablet_bootstrap.cc:492] T 9041b28402664a08b8761fa155c9c229 P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:19.937995 1198 ts_tablet_manager.cc:1918] T 3d132c29cb2c46108e5877e38068ef17 P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:19.940850 1196 ts_tablet_manager.cc:1918] T ce92c2b45a274eb09e72a96b7ec133be P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:19.941044 1196 log.cc:1198] T ce92c2b45a274eb09e72a96b7ec133be P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/ce92c2b45a274eb09e72a96b7ec133be
I20250113 02:51:19.941476 1196 ts_tablet_manager.cc:1939] T ce92c2b45a274eb09e72a96b7ec133be P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
W20250113 02:51:19.935540 709 leader_election.cc:343] T ec0750fd2fa247feb3ea310ffabe05e5 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Not found: Tablet not found: ec0750fd2fa247feb3ea310ffabe05e5
I20250113 02:51:19.941797 709 leader_election.cc:304] T ec0750fd2fa247feb3ea310ffabe05e5 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:19.942049 1198 log.cc:1198] T 3d132c29cb2c46108e5877e38068ef17 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/3d132c29cb2c46108e5877e38068ef17
I20250113 02:51:19.942648 1270 raft_consensus.cc:2743] T ec0750fd2fa247feb3ea310ffabe05e5 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250113 02:51:19.942816 1196 tablet_replica.cc:331] T 8193357427f4491c8c6c1396ceeef2dc P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:19.943037 1194 tablet_bootstrap.cc:654] T 9041b28402664a08b8761fa155c9c229 P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:19.943065 1196 raft_consensus.cc:2238] T 8193357427f4491c8c6c1396ceeef2dc P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:19.943210 1196 raft_consensus.cc:2267] T 8193357427f4491c8c6c1396ceeef2dc P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:19.943701 1196 ts_tablet_manager.cc:1905] T 8193357427f4491c8c6c1396ceeef2dc P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:19.944252 1198 ts_tablet_manager.cc:1939] T 3d132c29cb2c46108e5877e38068ef17 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:19.945947 1382 mvcc.cc:204] Tried to move back new op lower bound from 7113673440299130880 to 7113673436106334208. Current Snapshot: MvccSnapshot[applied={T|T < 7113673440299130880}]
I20250113 02:51:19.950866 1196 ts_tablet_manager.cc:1918] T 8193357427f4491c8c6c1396ceeef2dc P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:19.951095 1196 log.cc:1198] T 8193357427f4491c8c6c1396ceeef2dc P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/8193357427f4491c8c6c1396ceeef2dc
I20250113 02:51:19.951578 1196 ts_tablet_manager.cc:1939] T 8193357427f4491c8c6c1396ceeef2dc P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:19.936667 1383 leader_election.cc:290] T ec0750fd2fa247feb3ea310ffabe05e5 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 election: Requested vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:19.954355 1390 mvcc.cc:204] Tried to move back new op lower bound from 7113673440299130880 to 7113673436106334208. Current Snapshot: MvccSnapshot[applied={T|T < 7113673440299130880}]
I20250113 02:51:19.954531 1196 tablet_replica.cc:331] T 15a05a32febd443dbb519c186048428a P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:19.954847 1196 raft_consensus.cc:2238] T 15a05a32febd443dbb519c186048428a P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:19.955145 1196 raft_consensus.cc:2267] T 15a05a32febd443dbb519c186048428a P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:19.955802 1196 ts_tablet_manager.cc:1905] T 15a05a32febd443dbb519c186048428a P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:19.957460 1194 tablet_bootstrap.cc:492] T 9041b28402664a08b8761fa155c9c229 P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:19.957664 1194 ts_tablet_manager.cc:1397] T 9041b28402664a08b8761fa155c9c229 P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.023s user 0.006s sys 0.001s
I20250113 02:51:19.959517 1398 tablet.cc:1721] T 7f3cbef7d91d4576822d913071a88248 P df69f48656b44bb890f76587ee69cc0f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250113 02:51:19.962613 1191 tablet_bootstrap.cc:492] T ce92c2b45a274eb09e72a96b7ec133be P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:19.931638 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet 0c55662a71444a3683a825bcec6c5b68 (table default.Table_3865c1517889457a87be9ea12b3a156f [id=79b4c84e05ce4571b42e2b4e40fae978]) successfully deleted
I20250113 02:51:19.962009 1194 raft_consensus.cc:357] T 9041b28402664a08b8761fa155c9c229 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:19.963069 1194 raft_consensus.cc:383] T 9041b28402664a08b8761fa155c9c229 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:19.963148 1196 ts_tablet_manager.cc:1918] T 15a05a32febd443dbb519c186048428a P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:19.963315 1196 log.cc:1198] T 15a05a32febd443dbb519c186048428a P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/15a05a32febd443dbb519c186048428a
I20250113 02:51:19.963510 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet ce92c2b45a274eb09e72a96b7ec133be (table default.Table_c5c12ef64b7b438eb52313e87793e601 [id=7c2bc6d4232f4585b84d71e3e1f85794]) successfully deleted
I20250113 02:51:19.963704 1196 ts_tablet_manager.cc:1939] T 15a05a32febd443dbb519c186048428a P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:19.963747 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet 8193357427f4491c8c6c1396ceeef2dc (table default.Table_4cc2e51b05714a449a4476d8d4d628f7 [id=f85f0c435ffa4fdca81d86de4aa4fd5f]) successfully deleted
I20250113 02:51:19.963156 1194 raft_consensus.cc:738] T 9041b28402664a08b8761fa155c9c229 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
W20250113 02:51:19.964027 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet fdbf03c89bff438a9a25115eddf60be9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fdbf03c89bff438a9a25115eddf60be9 already in progress: creating tablet
W20250113 02:51:19.964248 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 19f14b3db0b64847b5422bfd5ed323d0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 19f14b3db0b64847b5422bfd5ed323d0 already in progress: creating tablet
W20250113 02:51:19.964465 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 53cc528d282c47788d5d558569f074c1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 53cc528d282c47788d5d558569f074c1 already in progress: creating tablet
W20250113 02:51:19.964676 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet b834f0acf5314bcb9d81c73de1b31842 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b834f0acf5314bcb9d81c73de1b31842 already in progress: creating tablet
W20250113 02:51:19.964913 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet ba61c86743e24666b30580c36b9228c3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ba61c86743e24666b30580c36b9228c3 already in progress: creating tablet
W20250113 02:51:19.965123 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 55ebe9c044d944bc813396c699ba9aa1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 55ebe9c044d944bc813396c699ba9aa1 already in progress: creating tablet
I20250113 02:51:19.965198 1196 tablet_replica.cc:331] T 3d132c29cb2c46108e5877e38068ef17 P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:19.965324 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet 15a05a32febd443dbb519c186048428a (table default.Table_f9aa5df6ba3d43999ebd6570df3108d1 [id=fa5a1cda58f54103ac637c12d99e4e38]) successfully deleted
I20250113 02:51:19.965427 1196 raft_consensus.cc:2238] T 3d132c29cb2c46108e5877e38068ef17 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
W20250113 02:51:19.965528 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 21c4b419bf2148e0b140301fa80622bb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 21c4b419bf2148e0b140301fa80622bb already in progress: creating tablet
I20250113 02:51:19.965572 1196 raft_consensus.cc:2267] T 3d132c29cb2c46108e5877e38068ef17 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:19.945385 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet 3d132c29cb2c46108e5877e38068ef17 (table default.Table_b31190bf4fa4499da09233dbdd1c9139 [id=44f4b85774f644aabd241d6ecd397a37]) successfully deleted
I20250113 02:51:19.966256 1196 ts_tablet_manager.cc:1905] T 3d132c29cb2c46108e5877e38068ef17 P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250113 02:51:19.966972 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 8aac142230e244c0b59c9c2b7a7300ea because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8aac142230e244c0b59c9c2b7a7300ea already in progress: creating tablet
I20250113 02:51:19.966924 1194 consensus_queue.cc:260] T 9041b28402664a08b8761fa155c9c229 P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
W20250113 02:51:19.968770 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 32351d6de1574353a6e8073d333da477 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 32351d6de1574353a6e8073d333da477 already in progress: creating tablet
I20250113 02:51:19.971035 1194 ts_tablet_manager.cc:1428] T 9041b28402664a08b8761fa155c9c229 P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.013s user 0.000s sys 0.002s
I20250113 02:51:19.971698 1194 tablet_bootstrap.cc:492] T 98f038ad31ff4e05bb88ac68ae341cc2 P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:19.973482 1194 tablet_bootstrap.cc:654] T 98f038ad31ff4e05bb88ac68ae341cc2 P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:19.974470 1196 ts_tablet_manager.cc:1918] T 3d132c29cb2c46108e5877e38068ef17 P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:19.974665 1196 log.cc:1198] T 3d132c29cb2c46108e5877e38068ef17 P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/3d132c29cb2c46108e5877e38068ef17
I20250113 02:51:19.975068 1196 ts_tablet_manager.cc:1939] T 3d132c29cb2c46108e5877e38068ef17 P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:19.976198 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet 3d132c29cb2c46108e5877e38068ef17 (table default.Table_b31190bf4fa4499da09233dbdd1c9139 [id=44f4b85774f644aabd241d6ecd397a37]) successfully deleted
W20250113 02:51:19.976606 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet ebc5f1ea1e1648d49904b17603d3cd2a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ebc5f1ea1e1648d49904b17603d3cd2a already in progress: creating tablet
W20250113 02:51:19.976987 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 601c71d5f7624fbb87b365d2e775d33e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 601c71d5f7624fbb87b365d2e775d33e already in progress: creating tablet
W20250113 02:51:19.977411 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet ee7582ee04494c0db1179359bcb311c8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ee7582ee04494c0db1179359bcb311c8 already in progress: creating tablet
I20250113 02:51:19.977646 1196 tablet_replica.cc:331] T 3455f010e51b4db1a9804a57fd1a4fb8 P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
W20250113 02:51:19.977725 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 840773ee80c7447097ea000b106fc89d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 840773ee80c7447097ea000b106fc89d already in progress: creating tablet
I20250113 02:51:19.977926 1196 raft_consensus.cc:2238] T 3455f010e51b4db1a9804a57fd1a4fb8 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
W20250113 02:51:19.978080 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 258fad818b7b4aa4b337f9949e20aa46 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 258fad818b7b4aa4b337f9949e20aa46 already in progress: creating tablet
I20250113 02:51:19.978178 1196 raft_consensus.cc:2267] T 3455f010e51b4db1a9804a57fd1a4fb8 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:19.978844 1196 ts_tablet_manager.cc:1905] T 3455f010e51b4db1a9804a57fd1a4fb8 P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:19.982410 1194 tablet_bootstrap.cc:492] T 98f038ad31ff4e05bb88ac68ae341cc2 P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:19.982547 1194 ts_tablet_manager.cc:1397] T 98f038ad31ff4e05bb88ac68ae341cc2 P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.011s user 0.010s sys 0.000s
I20250113 02:51:19.984028 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_7805d72c367b4ef8aa585de05f3d9b90"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:19.984808 1194 raft_consensus.cc:357] T 98f038ad31ff4e05bb88ac68ae341cc2 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:19.984992 1194 raft_consensus.cc:383] T 98f038ad31ff4e05bb88ac68ae341cc2 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:19.985069 1194 raft_consensus.cc:738] T 98f038ad31ff4e05bb88ac68ae341cc2 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:19.985414 1196 ts_tablet_manager.cc:1918] T 3455f010e51b4db1a9804a57fd1a4fb8 P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:19.985323 1194 consensus_queue.cc:260] T 98f038ad31ff4e05bb88ac68ae341cc2 P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:19.985567 1196 log.cc:1198] T 3455f010e51b4db1a9804a57fd1a4fb8 P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/3455f010e51b4db1a9804a57fd1a4fb8
I20250113 02:51:19.986181 1196 ts_tablet_manager.cc:1939] T 3455f010e51b4db1a9804a57fd1a4fb8 P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:19.986486 1194 ts_tablet_manager.cc:1428] T 98f038ad31ff4e05bb88ac68ae341cc2 P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250113 02:51:19.986655 446 catalog_manager.cc:5618] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467) does not have the latest schema for tablet 7f3cbef7d91d4576822d913071a88248 (table default.Table_b581b1a063a54551a8535adf01983fb8 [id=54e4e3810b3b412bbec3b5bf5b28c1a2]). Expected version 1 got 0
I20250113 02:51:19.987156 1194 tablet_bootstrap.cc:492] T 9b5dc06ba3ea4056822ca3e4b1249587 P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:19.988853 447 catalog_manager.cc:5618] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587) does not have the latest schema for tablet 7f3cbef7d91d4576822d913071a88248 (table default.Table_b581b1a063a54551a8535adf01983fb8 [id=54e4e3810b3b412bbec3b5bf5b28c1a2]). Expected version 1 got 0
W20250113 02:51:19.989871 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7805d72c367b4ef8aa585de05f3d9b90 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:19.991441 1194 tablet_bootstrap.cc:654] T 9b5dc06ba3ea4056822ca3e4b1249587 P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:19.995795 899 tablet_service.cc:1514] Processing DeleteTablet for tablet f0e7cc161f354f86b8c2fef303e87c5b with delete_type TABLET_DATA_DELETED (replaced by tablet 572e4eed1e924381bada27f8389821a7) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:19.996309 1196 tablet_replica.cc:331] T f0e7cc161f354f86b8c2fef303e87c5b P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:19.996635 1196 raft_consensus.cc:2238] T f0e7cc161f354f86b8c2fef303e87c5b P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:19.996922 1196 raft_consensus.cc:2267] T f0e7cc161f354f86b8c2fef303e87c5b P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:19.997349 899 tablet_service.cc:1467] Processing CreateTablet for tablet b0b267b0d9c54ba496c89f6cb287dffd (DEFAULT_TABLE table=default.Table_c604c45a4d3a49f0b6fd1ec85582d495 [id=dae9b614d621428ab168fe6a978799a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:19.997874 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b0b267b0d9c54ba496c89f6cb287dffd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:19.998342 1196 ts_tablet_manager.cc:1905] T f0e7cc161f354f86b8c2fef303e87c5b P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:19.999316 1036 tablet_service.cc:1467] Processing CreateTablet for tablet b0b267b0d9c54ba496c89f6cb287dffd (DEFAULT_TABLE table=default.Table_c604c45a4d3a49f0b6fd1ec85582d495 [id=dae9b614d621428ab168fe6a978799a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:19.999413 1037 tablet_service.cc:1467] Processing CreateTablet for tablet 8337f3c82d654ecdac2cbc1ffa88ef05 (DEFAULT_TABLE table=default.Table_37d63a1b8622483a855edf2ce71c8d71 [id=885fa0a4645f4211bd874149829284c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:19.999718 1036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b0b267b0d9c54ba496c89f6cb287dffd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:20.000993 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d132c29cb2c46108e5877e38068ef17 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:17 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:20.001497 1037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8337f3c82d654ecdac2cbc1ffa88ef05. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:20.002357 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet 3455f010e51b4db1a9804a57fd1a4fb8 with delete_type TABLET_DATA_DELETED (replaced by tablet a6055d77d168495d9b756a1a56c1a920) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:20.003245 1198 tablet_replica.cc:331] T 3455f010e51b4db1a9804a57fd1a4fb8 P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:20.003450 1198 raft_consensus.cc:2238] T 3455f010e51b4db1a9804a57fd1a4fb8 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:20.003661 1198 raft_consensus.cc:2267] T 3455f010e51b4db1a9804a57fd1a4fb8 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus is shut down!
W20250113 02:51:20.004228 440 catalog_manager.cc:4839] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 3d132c29cb2c46108e5877e38068ef17 because the tablet was not found. No further retry: Not found: Tablet not found: 3d132c29cb2c46108e5877e38068ef17
I20250113 02:51:20.004673 1196 ts_tablet_manager.cc:1918] T f0e7cc161f354f86b8c2fef303e87c5b P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:20.004794 1196 log.cc:1198] T f0e7cc161f354f86b8c2fef303e87c5b P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/f0e7cc161f354f86b8c2fef303e87c5b
I20250113 02:51:20.004997 1198 ts_tablet_manager.cc:1905] T 3455f010e51b4db1a9804a57fd1a4fb8 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:20.005174 1196 ts_tablet_manager.cc:1939] T f0e7cc161f354f86b8c2fef303e87c5b P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:20.008800 1195 ts_tablet_manager.cc:1918] T db266a1848094b7a98aa75f7afabd471 P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:20.008991 1195 log.cc:1198] T db266a1848094b7a98aa75f7afabd471 P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/db266a1848094b7a98aa75f7afabd471
I20250113 02:51:20.009406 1195 ts_tablet_manager.cc:1939] T db266a1848094b7a98aa75f7afabd471 P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:20.010244 1194 tablet_bootstrap.cc:492] T 9b5dc06ba3ea4056822ca3e4b1249587 P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:20.010233 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 166) took 1451 ms. Trace:
I20250113 02:51:20.010408 1194 ts_tablet_manager.cc:1397] T 9b5dc06ba3ea4056822ca3e4b1249587 P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.024s user 0.001s sys 0.008s
I20250113 02:51:20.010329 1195 rpcz_store.cc:276] 0113 02:51:18.558610 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:18.559653 (+ 1043us) service_pool.cc:225] Handling call
0113 02:51:19.917263 (+1357610us) ts_tablet_manager.cc:1145] Deleting tablet db266a1848094b7a98aa75f7afabd471
0113 02:51:19.917274 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:19.994082 (+ 76808us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:20.008660 (+ 14578us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:20.010175 (+ 1515us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1357376}
I20250113 02:51:20.010807 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 167) took 1451 ms. Trace:
I20250113 02:51:20.010907 1195 rpcz_store.cc:276] 0113 02:51:18.559205 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:18.560373 (+ 1168us) service_pool.cc:225] Handling call
0113 02:51:20.010649 (+1450276us) ts_tablet_manager.cc:1145] Deleting tablet f1a02c37cb4f43f8a9de6484f4431958
0113 02:51:20.010661 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:20.010762 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1450098}
I20250113 02:51:20.011302 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 168) took 1444 ms. Trace:
I20250113 02:51:20.011399 1195 rpcz_store.cc:276] 0113 02:51:18.566418 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:18.566605 (+ 187us) service_pool.cc:225] Handling call
0113 02:51:20.011138 (+1444533us) ts_tablet_manager.cc:1145] Deleting tablet d5fbf97536a2472fa091168363bf9549
0113 02:51:20.011146 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:20.011249 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1444270}
I20250113 02:51:20.011374 1194 raft_consensus.cc:357] T 9b5dc06ba3ea4056822ca3e4b1249587 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:20.011611 1194 raft_consensus.cc:383] T 9b5dc06ba3ea4056822ca3e4b1249587 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:20.011694 1194 raft_consensus.cc:738] T 9b5dc06ba3ea4056822ca3e4b1249587 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:20.011749 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 165) took 1453 ms. Trace:
I20250113 02:51:20.011960 1194 consensus_queue.cc:260] T 9b5dc06ba3ea4056822ca3e4b1249587 P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:20.012605 1195 rpcz_store.cc:276] 0113 02:51:18.558474 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:18.567691 (+ 9217us) service_pool.cc:225] Handling call
0113 02:51:20.011612 (+1443921us) ts_tablet_manager.cc:1145] Deleting tablet 9b5dc06ba3ea4056822ca3e4b1249587
0113 02:51:20.011622 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:20.011702 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1442426}
I20250113 02:51:20.012921 1195 tablet_replica.cc:331] T 1643b3e1607f42fc82886d0b09e2fe8c P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:20.013089 1195 raft_consensus.cc:2238] T 1643b3e1607f42fc82886d0b09e2fe8c P 1186fa869eb14812832fa6562531f196 [term 1 LEADER]: Raft consensus shutting down.
I20250113 02:51:20.013114 888 tablet_service.cc:1467] Processing CreateTablet for tablet 8337f3c82d654ecdac2cbc1ffa88ef05 (DEFAULT_TABLE table=default.Table_37d63a1b8622483a855edf2ce71c8d71 [id=885fa0a4645f4211bd874149829284c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:20.013583 1195 pending_rounds.cc:62] T 1643b3e1607f42fc82886d0b09e2fe8c P 1186fa869eb14812832fa6562531f196: Trying to abort 1 pending ops.
I20250113 02:51:20.013567 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8337f3c82d654ecdac2cbc1ffa88ef05. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:20.013649 1195 pending_rounds.cc:69] T 1643b3e1607f42fc82886d0b09e2fe8c P 1186fa869eb14812832fa6562531f196: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113673438733324288 op_type: NO_OP noop_request { }
I20250113 02:51:20.013818 1195 raft_consensus.cc:2883] T 1643b3e1607f42fc82886d0b09e2fe8c P 1186fa869eb14812832fa6562531f196 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250113 02:51:20.013929 1195 raft_consensus.cc:2267] T 1643b3e1607f42fc82886d0b09e2fe8c P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:20.014631 1195 ts_tablet_manager.cc:1905] T 1643b3e1607f42fc82886d0b09e2fe8c P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:20.014714 1194 ts_tablet_manager.cc:1428] T 9b5dc06ba3ea4056822ca3e4b1249587 P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20250113 02:51:20.015061 1194 tablet_bootstrap.cc:492] T 2fb62cc83f9c4dcfa9aa01ae8b46c025 P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:20.017463 1194 tablet_bootstrap.cc:654] T 2fb62cc83f9c4dcfa9aa01ae8b46c025 P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:20.019707 1191 tablet_bootstrap.cc:654] T ce92c2b45a274eb09e72a96b7ec133be P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:20.020500 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_7157800e639b45d8bb947e53d4319cfc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:20.021261 447 catalog_manager.cc:5618] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467) does not have the latest schema for tablet fb183b4588e84212a3b4b096f430d47d (table default.Table_53b46860bb68469fa08f3fd02b2408e5 [id=33bf7fd42ab5458db0f14347353c0db4]). Expected version 2 got 0
I20250113 02:51:20.021486 447 catalog_manager.cc:5618] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467) does not have the latest schema for tablet 98f038ad31ff4e05bb88ac68ae341cc2 (table default.Table_4fcd958510914007817364e5a8e8ebec [id=6666a614063942c99a78b95d35add037]). Expected version 1 got 0
I20250113 02:51:20.021759 449 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_b3284070cce54277bed74120ca8f199b" } modify_external_catalogs: true
I20250113 02:51:20.021955 449 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_b3284070cce54277bed74120ca8f199b" } modify_external_catalogs: true
I20250113 02:51:20.022380 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_4cc2e51b05714a449a4476d8d4d628f7" } new_table_name: "default.Table_8b87c8469fbb465e80375aa0446de6c2" modify_external_catalogs: true
I20250113 02:51:20.023465 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_a6447976a51a4e2f93d22bd1c657c49d" } modify_external_catalogs: true
I20250113 02:51:20.023669 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_a6447976a51a4e2f93d22bd1c657c49d" } modify_external_catalogs: true
I20250113 02:51:20.024276 1195 ts_tablet_manager.cc:1918] T 1643b3e1607f42fc82886d0b09e2fe8c P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250113 02:51:20.024412 1195 log.cc:1198] T 1643b3e1607f42fc82886d0b09e2fe8c P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/1643b3e1607f42fc82886d0b09e2fe8c
I20250113 02:51:20.024744 1194 tablet_bootstrap.cc:492] T 2fb62cc83f9c4dcfa9aa01ae8b46c025 P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:20.024825 1195 ts_tablet_manager.cc:1939] T 1643b3e1607f42fc82886d0b09e2fe8c P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:20.024879 1194 ts_tablet_manager.cc:1397] T 2fb62cc83f9c4dcfa9aa01ae8b46c025 P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.002s
I20250113 02:51:20.021507 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_668e70d7e5c64c37ab05e7bcbe56d994" } new_table_name: "default.Table_a55ba304b6a04e2492d6aedba6aad86e" modify_external_catalogs: true
I20250113 02:51:20.025602 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 169) took 1378 ms. Trace:
I20250113 02:51:20.025656 1194 raft_consensus.cc:357] T 2fb62cc83f9c4dcfa9aa01ae8b46c025 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:20.025835 1194 raft_consensus.cc:383] T 2fb62cc83f9c4dcfa9aa01ae8b46c025 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:20.025713 1195 rpcz_store.cc:276] 0113 02:51:18.647433 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:18.647731 (+ 298us) service_pool.cc:225] Handling call
0113 02:51:20.012847 (+1365116us) ts_tablet_manager.cc:1145] Deleting tablet 1643b3e1607f42fc82886d0b09e2fe8c
0113 02:51:20.012856 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:20.017899 (+ 5043us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:20.024171 (+ 6272us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:20.025552 (+ 1381us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1364888}
I20250113 02:51:20.025924 1194 raft_consensus.cc:738] T 2fb62cc83f9c4dcfa9aa01ae8b46c025 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:20.026109 1195 tablet_replica.cc:331] T d6675bc3cb5f4f809320e28026cdf4b0 P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:20.026144 1194 consensus_queue.cc:260] T 2fb62cc83f9c4dcfa9aa01ae8b46c025 P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:20.026288 1195 raft_consensus.cc:2238] T d6675bc3cb5f4f809320e28026cdf4b0 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:20.026614 1195 raft_consensus.cc:2267] T d6675bc3cb5f4f809320e28026cdf4b0 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:20.026732 1400 tablet.cc:1721] T 7f3cbef7d91d4576822d913071a88248 P 4216a7f853d642b7bc3e5cff458f3aca: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250113 02:51:20.027139 1195 ts_tablet_manager.cc:1905] T d6675bc3cb5f4f809320e28026cdf4b0 P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:20.027258 1194 ts_tablet_manager.cc:1428] T 2fb62cc83f9c4dcfa9aa01ae8b46c025 P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250113 02:51:20.027885 1194 tablet_bootstrap.cc:492] T 68023c30a7f644eaa5b4ddd60d6c5ef5 P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:20.026293 1397 tablet.cc:1721] T 7f3cbef7d91d4576822d913071a88248 P 1186fa869eb14812832fa6562531f196: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250113 02:51:20.034682 1194 tablet_bootstrap.cc:654] T 68023c30a7f644eaa5b4ddd60d6c5ef5 P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:20.035274 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet 3455f010e51b4db1a9804a57fd1a4fb8 (table default.Table_44f1c4e8cc2b4e83adce8d411ae53448 [id=a9e0aae9f5af42b796d29a75140f9c29]) successfully deleted
I20250113 02:51:20.035542 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet f0e7cc161f354f86b8c2fef303e87c5b (table default.Table_20ca05c407b14ba0a92f76f4596ae2ef [id=75cb0745bbb84e1c883cb13b546ecb11]) successfully deleted
I20250113 02:51:20.035754 1195 ts_tablet_manager.cc:1918] T d6675bc3cb5f4f809320e28026cdf4b0 P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:20.035916 1195 log.cc:1198] T d6675bc3cb5f4f809320e28026cdf4b0 P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/d6675bc3cb5f4f809320e28026cdf4b0
W20250113 02:51:20.021230 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7157800e639b45d8bb947e53d4319cfc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:20.036348 1195 ts_tablet_manager.cc:1939] T d6675bc3cb5f4f809320e28026cdf4b0 P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:20.037379 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 172) took 1328 ms. Trace:
I20250113 02:51:20.037559 1195 rpcz_store.cc:276] 0113 02:51:18.709145 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:18.709196 (+ 51us) service_pool.cc:225] Handling call
0113 02:51:20.026016 (+1316820us) ts_tablet_manager.cc:1145] Deleting tablet d6675bc3cb5f4f809320e28026cdf4b0
0113 02:51:20.026026 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:20.030753 (+ 4727us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:20.035616 (+ 4863us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:20.037321 (+ 1705us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":26,"tablet-delete.queue_time_us":1316605}
I20250113 02:51:20.038702 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 175) took 1210 ms. Trace:
I20250113 02:51:20.038849 1195 rpcz_store.cc:276] 0113 02:51:18.828143 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:18.829031 (+ 888us) service_pool.cc:225] Handling call
0113 02:51:20.038168 (+1209137us) ts_tablet_manager.cc:1145] Deleting tablet 6c8ca30720664f3a819c4bff2493934d
0113 02:51:20.038181 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:20.038322 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1208913}
I20250113 02:51:20.039259 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 174) took 1211 ms. Trace:
I20250113 02:51:20.039395 1195 rpcz_store.cc:276] 0113 02:51:18.827938 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:18.829342 (+ 1404us) service_pool.cc:225] Handling call
0113 02:51:20.039118 (+1209776us) ts_tablet_manager.cc:1145] Deleting tablet 5b3f012b88c04cb68ed7a1848147b70b
0113 02:51:20.039130 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:20.039227 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1209585}
I20250113 02:51:20.039711 1195 tablet_replica.cc:331] T bc88676bf738442d82c25e56f30c4ada P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:20.040006 1195 raft_consensus.cc:2238] T bc88676bf738442d82c25e56f30c4ada P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:20.039525 1192 tablet_bootstrap.cc:492] T d6675bc3cb5f4f809320e28026cdf4b0 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:20.040202 1195 raft_consensus.cc:2267] T bc88676bf738442d82c25e56f30c4ada P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:20.040879 1195 ts_tablet_manager.cc:1905] T bc88676bf738442d82c25e56f30c4ada P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:20.040642 1192 ts_tablet_manager.cc:1397] T d6675bc3cb5f4f809320e28026cdf4b0 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.251s user 0.014s sys 0.055s
I20250113 02:51:20.042920 1194 tablet_bootstrap.cc:492] T 68023c30a7f644eaa5b4ddd60d6c5ef5 P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:20.043594 1194 ts_tablet_manager.cc:1397] T 68023c30a7f644eaa5b4ddd60d6c5ef5 P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.002s
I20250113 02:51:20.044525 1194 raft_consensus.cc:357] T 68023c30a7f644eaa5b4ddd60d6c5ef5 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:20.044763 1194 raft_consensus.cc:383] T 68023c30a7f644eaa5b4ddd60d6c5ef5 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:20.044855 1194 raft_consensus.cc:738] T 68023c30a7f644eaa5b4ddd60d6c5ef5 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:20.045150 1194 consensus_queue.cc:260] T 68023c30a7f644eaa5b4ddd60d6c5ef5 P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:20.045117 1192 raft_consensus.cc:357] T d6675bc3cb5f4f809320e28026cdf4b0 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:20.045542 1192 raft_consensus.cc:383] T d6675bc3cb5f4f809320e28026cdf4b0 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:20.045663 1194 ts_tablet_manager.cc:1428] T 68023c30a7f644eaa5b4ddd60d6c5ef5 P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250113 02:51:20.045693 1192 raft_consensus.cc:738] T d6675bc3cb5f4f809320e28026cdf4b0 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:20.046085 1192 consensus_queue.cc:260] T d6675bc3cb5f4f809320e28026cdf4b0 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:20.046373 1194 tablet_bootstrap.cc:492] T 53cc528d282c47788d5d558569f074c1 P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:20.048058 1192 ts_tablet_manager.cc:1428] T d6675bc3cb5f4f809320e28026cdf4b0 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250113 02:51:20.049273 1192 tablet_bootstrap.cc:492] T ad9b95c5450a4c9c89a0930862348d49 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:20.051573 1195 ts_tablet_manager.cc:1918] T bc88676bf738442d82c25e56f30c4ada P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:20.052356 1195 log.cc:1198] T bc88676bf738442d82c25e56f30c4ada P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/bc88676bf738442d82c25e56f30c4ada
I20250113 02:51:20.052836 1194 tablet_bootstrap.cc:654] T 53cc528d282c47788d5d558569f074c1 P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:20.053141 1195 ts_tablet_manager.cc:1939] T bc88676bf738442d82c25e56f30c4ada P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:20.053853 1191 tablet_bootstrap.cc:492] T ce92c2b45a274eb09e72a96b7ec133be P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:20.053977 1192 tablet_bootstrap.cc:654] T ad9b95c5450a4c9c89a0930862348d49 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:20.054239 1191 ts_tablet_manager.cc:1397] T ce92c2b45a274eb09e72a96b7ec133be P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.092s user 0.001s sys 0.004s
I20250113 02:51:20.054690 1198 ts_tablet_manager.cc:1918] T 3455f010e51b4db1a9804a57fd1a4fb8 P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:20.054909 1198 log.cc:1198] T 3455f010e51b4db1a9804a57fd1a4fb8 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/3455f010e51b4db1a9804a57fd1a4fb8
I20250113 02:51:20.055469 1198 ts_tablet_manager.cc:1939] T 3455f010e51b4db1a9804a57fd1a4fb8 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
W20250113 02:51:20.056856 439 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.60:42067, user_credentials={real_user=slave}} blocked reactor thread for 71822.2us
I20250113 02:51:20.058305 743 tablet_service.cc:1467] Processing CreateTablet for tablet 8337f3c82d654ecdac2cbc1ffa88ef05 (DEFAULT_TABLE table=default.Table_37d63a1b8622483a855edf2ce71c8d71 [id=885fa0a4645f4211bd874149829284c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:20.058902 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8337f3c82d654ecdac2cbc1ffa88ef05. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:20.060322 751 tablet_service.cc:1467] Processing CreateTablet for tablet b0b267b0d9c54ba496c89f6cb287dffd (DEFAULT_TABLE table=default.Table_c604c45a4d3a49f0b6fd1ec85582d495 [id=dae9b614d621428ab168fe6a978799a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:20.060864 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b0b267b0d9c54ba496c89f6cb287dffd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:20.062639 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 176) took 1234 ms. Trace:
I20250113 02:51:20.062776 1195 rpcz_store.cc:276] 0113 02:51:18.840083 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:18.840262 (+ 179us) service_pool.cc:225] Handling call
0113 02:51:20.039634 (+1199372us) ts_tablet_manager.cc:1145] Deleting tablet bc88676bf738442d82c25e56f30c4ada
0113 02:51:20.039644 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:20.044059 (+ 4415us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:20.051469 (+ 7410us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:20.062558 (+ 11089us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1199145}
I20250113 02:51:20.063279 1195 tablet_replica.cc:331] T ec0750fd2fa247feb3ea310ffabe05e5 P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:20.063521 1195 raft_consensus.cc:2238] T ec0750fd2fa247feb3ea310ffabe05e5 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:20.063755 1195 raft_consensus.cc:2267] T ec0750fd2fa247feb3ea310ffabe05e5 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:20.064472 1195 ts_tablet_manager.cc:1905] T ec0750fd2fa247feb3ea310ffabe05e5 P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:20.068637 753 tablet_service.cc:1514] Processing DeleteTablet for tablet 78689f7ed2a64405adb75f372465a792 with delete_type TABLET_DATA_DELETED (replaced by tablet 0433cfe0cb524511954e1012d4d137e6) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:20.068841 1191 raft_consensus.cc:357] T ce92c2b45a274eb09e72a96b7ec133be P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:20.069128 1191 raft_consensus.cc:383] T ce92c2b45a274eb09e72a96b7ec133be P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:20.069243 1191 raft_consensus.cc:738] T ce92c2b45a274eb09e72a96b7ec133be P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:20.069516 1191 consensus_queue.cc:260] T ce92c2b45a274eb09e72a96b7ec133be P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:20.070438 1191 ts_tablet_manager.cc:1428] T ce92c2b45a274eb09e72a96b7ec133be P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.016s user 0.000s sys 0.002s
I20250113 02:51:20.070775 1191 tablet_bootstrap.cc:492] T f0e7cc161f354f86b8c2fef303e87c5b P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:20.073873 1368 raft_consensus.cc:491] T ad6b21ed2b824cb0837fc1d8a39d2472 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:20.074066 1368 raft_consensus.cc:513] T ad6b21ed2b824cb0837fc1d8a39d2472 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:20.074337 1191 tablet_bootstrap.cc:654] T f0e7cc161f354f86b8c2fef303e87c5b P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:20.074867 1368 leader_election.cc:290] T ad6b21ed2b824cb0837fc1d8a39d2472 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
W20250113 02:51:20.076798 709 leader_election.cc:343] T ad6b21ed2b824cb0837fc1d8a39d2472 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Not found: Tablet not found: ad6b21ed2b824cb0837fc1d8a39d2472
W20250113 02:51:20.077340 711 leader_election.cc:343] T ad6b21ed2b824cb0837fc1d8a39d2472 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Not found: Tablet not found: ad6b21ed2b824cb0837fc1d8a39d2472
I20250113 02:51:20.077492 711 leader_election.cc:304] T ad6b21ed2b824cb0837fc1d8a39d2472 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:20.077991 1368 raft_consensus.cc:2743] T ad6b21ed2b824cb0837fc1d8a39d2472 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:20.079195 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet db266a1848094b7a98aa75f7afabd471 (table default.Table_87cfc114b15047b49fd25446ff7aa441 [id=ca79853d13ee4a76a18d9fa68c1feb31]) successfully deleted
W20250113 02:51:20.079525 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet f1a02c37cb4f43f8a9de6484f4431958 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f1a02c37cb4f43f8a9de6484f4431958 already in progress: creating tablet
W20250113 02:51:20.079778 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet d5fbf97536a2472fa091168363bf9549 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d5fbf97536a2472fa091168363bf9549 already in progress: creating tablet
I20250113 02:51:20.087546 1195 ts_tablet_manager.cc:1918] T ec0750fd2fa247feb3ea310ffabe05e5 P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:20.087756 1195 log.cc:1198] T ec0750fd2fa247feb3ea310ffabe05e5 P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/ec0750fd2fa247feb3ea310ffabe05e5
I20250113 02:51:20.089295 1191 tablet_bootstrap.cc:492] T f0e7cc161f354f86b8c2fef303e87c5b P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:20.089480 1191 ts_tablet_manager.cc:1397] T f0e7cc161f354f86b8c2fef303e87c5b P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.001s
I20250113 02:51:20.090304 1191 raft_consensus.cc:357] T f0e7cc161f354f86b8c2fef303e87c5b P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:20.090548 1191 raft_consensus.cc:383] T f0e7cc161f354f86b8c2fef303e87c5b P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:20.090649 1191 raft_consensus.cc:738] T f0e7cc161f354f86b8c2fef303e87c5b P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:20.090968 1191 consensus_queue.cc:260] T f0e7cc161f354f86b8c2fef303e87c5b P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:20.091624 1191 ts_tablet_manager.cc:1428] T f0e7cc161f354f86b8c2fef303e87c5b P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250113 02:51:20.092929 1191 tablet_bootstrap.cc:492] T 3d132c29cb2c46108e5877e38068ef17 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:20.095044 1191 tablet_bootstrap.cc:654] T 3d132c29cb2c46108e5877e38068ef17 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:20.104343 1195 ts_tablet_manager.cc:1939] T ec0750fd2fa247feb3ea310ffabe05e5 P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:20.105957 1191 tablet_bootstrap.cc:492] T 3d132c29cb2c46108e5877e38068ef17 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:20.106148 1191 ts_tablet_manager.cc:1397] T 3d132c29cb2c46108e5877e38068ef17 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.004s
I20250113 02:51:20.112882 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet 3455f010e51b4db1a9804a57fd1a4fb8 (table default.Table_44f1c4e8cc2b4e83adce8d411ae53448 [id=a9e0aae9f5af42b796d29a75140f9c29]) successfully deleted
W20250113 02:51:20.122331 441 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet e8ab4516789e4585a101c8766d338af3 (table default.Table_243aff1bbfc6454b92f66035f64a7b84 [id=b112ac4e2e58460db083da1eb788ca07], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet e8ab4516789e4585a101c8766d338af3 consensus state has no leader
W20250113 02:51:20.125627 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 9b5dc06ba3ea4056822ca3e4b1249587 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9b5dc06ba3ea4056822ca3e4b1249587 already in progress: creating tablet
W20250113 02:51:20.126013 439 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.104.1:40095, user_credentials={real_user=slave}} blocked reactor thread for 46026.5us
I20250113 02:51:20.126333 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet 1643b3e1607f42fc82886d0b09e2fe8c (table default.Table_fff2447caa364c3381a3ba0abbde1c50 [id=15d9be3deeb84c1494965848dfb804ac]) successfully deleted
I20250113 02:51:20.126600 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet d6675bc3cb5f4f809320e28026cdf4b0 (table default.Table_d12d804163994a63a236be44401aa53b [id=f806136f234042af82e9831467a69caa]) successfully deleted
W20250113 02:51:20.126931 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 6c8ca30720664f3a819c4bff2493934d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6c8ca30720664f3a819c4bff2493934d already in progress: creating tablet
W20250113 02:51:20.127223 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 5b3f012b88c04cb68ed7a1848147b70b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5b3f012b88c04cb68ed7a1848147b70b already in progress: creating tablet
I20250113 02:51:20.127489 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet bc88676bf738442d82c25e56f30c4ada (table default.Table_c991f9dbd70d4c4682664b8711f05db4 [id=2c64da64ecdb4bf48e11b523533902e8]) successfully deleted
I20250113 02:51:20.130753 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 23e9321f08ee4ce7b2252ad5c76c0a08
I20250113 02:51:20.131016 1191 raft_consensus.cc:357] T 3d132c29cb2c46108e5877e38068ef17 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:20.131254 1191 raft_consensus.cc:383] T 3d132c29cb2c46108e5877e38068ef17 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:20.131301 454 catalog_manager.cc:6596] ReplaceTablet: tablet 23e9321f08ee4ce7b2252ad5c76c0a08 deleted and replaced by tablet 6666922d677a4c678f80d7c6c47fd2a5
I20250113 02:51:20.131354 1191 raft_consensus.cc:738] T 3d132c29cb2c46108e5877e38068ef17 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:20.131583 1191 consensus_queue.cc:260] T 3d132c29cb2c46108e5877e38068ef17 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:20.132177 1191 ts_tablet_manager.cc:1428] T 3d132c29cb2c46108e5877e38068ef17 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.026s user 0.001s sys 0.000s
I20250113 02:51:20.132364 454 catalog_manager.cc:5618] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587) does not have the latest schema for tablet 7f3cbef7d91d4576822d913071a88248 (table default.Table_b581b1a063a54551a8535adf01983fb8 [id=54e4e3810b3b412bbec3b5bf5b28c1a2]). Expected version 1 got 0
I20250113 02:51:20.132496 1191 tablet_bootstrap.cc:492] T ad9b95c5450a4c9c89a0930862348d49 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
W20250113 02:51:20.134466 448 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 98f038ad31ff4e05bb88ac68ae341cc2 consensus state has no leader
I20250113 02:51:20.134688 1191 tablet_bootstrap.cc:654] T ad9b95c5450a4c9c89a0930862348d49 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:20.134876 448 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_4f1f306815b145218e24c93821376721" } modify_external_catalogs: true
I20250113 02:51:20.135073 448 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_4f1f306815b145218e24c93821376721" } modify_external_catalogs: true
W20250113 02:51:20.136651 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 092892a36e0e47cb8d2184c2bbde95d7 consensus state has no leader
I20250113 02:51:20.137035 455 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_649d3c3eaebd4d4da258735b332a4a75" } modify_external_catalogs: true
I20250113 02:51:20.137241 455 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_649d3c3eaebd4d4da258735b332a4a75" } modify_external_catalogs: true
I20250113 02:51:20.137446 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 177) took 1290 ms. Trace:
I20250113 02:51:20.137550 1195 rpcz_store.cc:276] 0113 02:51:18.846896 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:18.848477 (+ 1581us) service_pool.cc:225] Handling call
0113 02:51:20.063174 (+1214697us) ts_tablet_manager.cc:1145] Deleting tablet ec0750fd2fa247feb3ea310ffabe05e5
0113 02:51:20.063188 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:20.081038 (+ 17850us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:20.087361 (+ 6323us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:20.137403 (+ 50042us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5366,"tablet-delete.queue_time_us":1214430}
I20250113 02:51:20.138031 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 178) took 1269 ms. Trace:
I20250113 02:51:20.138528 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 2778d7dc7d9c4843bbc9eb705cec71cd
I20250113 02:51:20.138135 1195 rpcz_store.cc:276] 0113 02:51:18.869021 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:18.869092 (+ 71us) service_pool.cc:225] Handling call
0113 02:51:20.137866 (+1268774us) ts_tablet_manager.cc:1145] Deleting tablet d35ec7e4a5084a74bea172ef8d9990b5
0113 02:51:20.137875 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:20.137993 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1268542}
I20250113 02:51:20.139155 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 180) took 1195 ms. Trace:
I20250113 02:51:20.139181 449 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_4a0258998d4f413b80d701c74526b86d" } modify_external_catalogs: true
I20250113 02:51:20.139261 1195 rpcz_store.cc:276] 0113 02:51:18.943846 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:18.943925 (+ 79us) service_pool.cc:225] Handling call
0113 02:51:20.139030 (+1195105us) ts_tablet_manager.cc:1145] Deleting tablet 501d4e1ae84e4795913035f8a19cc32d
0113 02:51:20.139038 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:20.139127 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1188455}
I20250113 02:51:20.139351 449 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_4a0258998d4f413b80d701c74526b86d" } modify_external_catalogs: true
I20250113 02:51:20.139662 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 181) took 1192 ms. Trace:
I20250113 02:51:20.139768 1195 rpcz_store.cc:276] 0113 02:51:18.947548 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:18.947620 (+ 72us) service_pool.cc:225] Handling call
0113 02:51:20.139468 (+1191848us) ts_tablet_manager.cc:1145] Deleting tablet 1c26074fdd804bd79cb0c295622d11ff
0113 02:51:20.139494 (+ 26us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:20.139631 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1188499}
W20250113 02:51:20.139915 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d3b695ac20ba4205b2d8732db4740531 consensus state has no leader
I20250113 02:51:20.140107 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 189) took 1062 ms. Trace:
I20250113 02:51:20.140208 1195 rpcz_store.cc:276] 0113 02:51:19.077415 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:19.077769 (+ 354us) service_pool.cc:225] Handling call
0113 02:51:20.139949 (+1062180us) ts_tablet_manager.cc:1145] Deleting tablet 157905316fc545b6a7864ae1c4a2a584
0113 02:51:20.139957 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:20.140080 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1047692}
I20250113 02:51:20.140354 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_b77c072e465c474db9349f1a3b31c323"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:20.140784 1195 tablet_replica.cc:331] T a028cc02273644608f046135851da126 P 1186fa869eb14812832fa6562531f196: stopping tablet replica
W20250113 02:51:20.140849 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b77c072e465c474db9349f1a3b31c323 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:20.141798 1195 raft_consensus.cc:2238] T a028cc02273644608f046135851da126 P 1186fa869eb14812832fa6562531f196 [term 1 LEADER]: Raft consensus shutting down.
W20250113 02:51:20.142172 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4c615b0b83d74fcfae0c51047dbfbbf4 consensus state has no leader
I20250113 02:51:20.142699 1195 pending_rounds.cc:62] T a028cc02273644608f046135851da126 P 1186fa869eb14812832fa6562531f196: Trying to abort 1 pending ops.
I20250113 02:51:20.142566 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_8a2c850672684fb0aef23bd1edfc8cd8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:20.142781 1195 pending_rounds.cc:69] T a028cc02273644608f046135851da126 P 1186fa869eb14812832fa6562531f196: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113673440210853888 op_type: NO_OP noop_request { }
I20250113 02:51:20.142896 1195 raft_consensus.cc:2883] T a028cc02273644608f046135851da126 P 1186fa869eb14812832fa6562531f196 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250113 02:51:20.143002 1195 raft_consensus.cc:2267] T a028cc02273644608f046135851da126 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250113 02:51:20.143112 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8a2c850672684fb0aef23bd1edfc8cd8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:20.145198 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_c5c12ef64b7b438eb52313e87793e601" } new_table_name: "default.Table_06f81a97cb014a3bad594e88c8cf4733" modify_external_catalogs: true
I20250113 02:51:20.146095 888 tablet_service.cc:1514] Processing DeleteTablet for tablet 2778d7dc7d9c4843bbc9eb705cec71cd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:20 UTC) from {username='slave'} at 127.0.0.1:37366
W20250113 02:51:20.146848 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 2778d7dc7d9c4843bbc9eb705cec71cd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2778d7dc7d9c4843bbc9eb705cec71cd already in progress: creating tablet
I20250113 02:51:20.151729 446 master_service.cc:946] ReplaceTablet: received request to replace tablet 98f038ad31ff4e05bb88ac68ae341cc2 from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:20.153359 899 tablet_service.cc:1514] Processing DeleteTablet for tablet 23e9321f08ee4ce7b2252ad5c76c0a08 with delete_type TABLET_DATA_DELETED (replaced by tablet 6666922d677a4c678f80d7c6c47fd2a5) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:20.153826 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 63efcb6b2e194c469fb30a197bd6565b
I20250113 02:51:20.156073 1195 ts_tablet_manager.cc:1905] T a028cc02273644608f046135851da126 P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250113 02:51:20.156874 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 23e9321f08ee4ce7b2252ad5c76c0a08 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 23e9321f08ee4ce7b2252ad5c76c0a08 already in progress: creating tablet
I20250113 02:51:20.157656 451 master_service.cc:946] ReplaceTablet: received request to replace tablet f760fc1e4bed433dbb987a42c12a1ee6 from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:20.158308 1191 tablet_bootstrap.cc:492] T ad9b95c5450a4c9c89a0930862348d49 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:20.158522 1191 ts_tablet_manager.cc:1397] T ad9b95c5450a4c9c89a0930862348d49 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.026s user 0.003s sys 0.001s
I20250113 02:51:20.158972 899 tablet_service.cc:1514] Processing DeleteTablet for tablet 63efcb6b2e194c469fb30a197bd6565b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:20 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:20.159451 1191 raft_consensus.cc:357] T ad9b95c5450a4c9c89a0930862348d49 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
W20250113 02:51:20.159654 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 63efcb6b2e194c469fb30a197bd6565b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 63efcb6b2e194c469fb30a197bd6565b already in progress: creating tablet
I20250113 02:51:20.159756 1191 raft_consensus.cc:383] T ad9b95c5450a4c9c89a0930862348d49 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:20.159842 1191 raft_consensus.cc:738] T ad9b95c5450a4c9c89a0930862348d49 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:20.160761 753 tablet_service.cc:1514] Processing DeleteTablet for tablet 23e9321f08ee4ce7b2252ad5c76c0a08 with delete_type TABLET_DATA_DELETED (replaced by tablet 6666922d677a4c678f80d7c6c47fd2a5) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:20.161013 1191 consensus_queue.cc:260] T ad9b95c5450a4c9c89a0930862348d49 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:20.161463 753 tablet_service.cc:1514] Processing DeleteTablet for tablet 2778d7dc7d9c4843bbc9eb705cec71cd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:20 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:20.161826 1191 ts_tablet_manager.cc:1428] T ad9b95c5450a4c9c89a0930862348d49 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250113 02:51:20.162153 1191 tablet_bootstrap.cc:492] T 15a05a32febd443dbb519c186048428a P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:20.164084 1191 tablet_bootstrap.cc:654] T 15a05a32febd443dbb519c186048428a P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:20.167567 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_2f3720d8d897466388c3f9dde157d701"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:20.168188 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2f3720d8d897466388c3f9dde157d701 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:20.172278 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet ec0750fd2fa247feb3ea310ffabe05e5 (table default.Table_74627d276edf4d75a2fcaa627a7f921b [id=de6b4835f6c04c20b6a5ef8404dc78f7]) successfully deleted
W20250113 02:51:20.172667 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet d35ec7e4a5084a74bea172ef8d9990b5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d35ec7e4a5084a74bea172ef8d9990b5 already in progress: creating tablet
W20250113 02:51:20.173942 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 501d4e1ae84e4795913035f8a19cc32d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 501d4e1ae84e4795913035f8a19cc32d already in progress: creating tablet
W20250113 02:51:20.174273 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 1c26074fdd804bd79cb0c295622d11ff because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1c26074fdd804bd79cb0c295622d11ff already in progress: creating tablet
W20250113 02:51:20.174546 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 157905316fc545b6a7864ae1c4a2a584 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 157905316fc545b6a7864ae1c4a2a584 already in progress: creating tablet
W20250113 02:51:20.174827 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet a6055d77d168495d9b756a1a56c1a920 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a6055d77d168495d9b756a1a56c1a920 already in progress: creating tablet
W20250113 02:51:20.175101 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet f760fc1e4bed433dbb987a42c12a1ee6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f760fc1e4bed433dbb987a42c12a1ee6 already in progress: creating tablet
I20250113 02:51:20.191577 1192 tablet_bootstrap.cc:492] T ad9b95c5450a4c9c89a0930862348d49 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:20.191776 1192 ts_tablet_manager.cc:1397] T ad9b95c5450a4c9c89a0930862348d49 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.143s user 0.026s sys 0.031s
I20250113 02:51:20.192723 1192 raft_consensus.cc:357] T ad9b95c5450a4c9c89a0930862348d49 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:20.192984 1192 raft_consensus.cc:383] T ad9b95c5450a4c9c89a0930862348d49 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:20.193095 1192 raft_consensus.cc:738] T ad9b95c5450a4c9c89a0930862348d49 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:20.193351 1192 consensus_queue.cc:260] T ad9b95c5450a4c9c89a0930862348d49 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:20.193883 1192 ts_tablet_manager.cc:1428] T ad9b95c5450a4c9c89a0930862348d49 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250113 02:51:20.194201 1192 tablet_bootstrap.cc:492] T ce92c2b45a274eb09e72a96b7ec133be P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:20.195818 1192 tablet_bootstrap.cc:654] T ce92c2b45a274eb09e72a96b7ec133be P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:20.200500 1414 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.61:35271 every 8 connection(s)
I20250113 02:51:20.202205 1192 tablet_bootstrap.cc:492] T ce92c2b45a274eb09e72a96b7ec133be P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:20.202344 1192 ts_tablet_manager.cc:1397] T ce92c2b45a274eb09e72a96b7ec133be P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20250113 02:51:20.203296 1037 tablet_service.cc:1514] Processing DeleteTablet for tablet 23e9321f08ee4ce7b2252ad5c76c0a08 with delete_type TABLET_DATA_DELETED (replaced by tablet 6666922d677a4c678f80d7c6c47fd2a5) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:20.203652 1192 raft_consensus.cc:357] T ce92c2b45a274eb09e72a96b7ec133be P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:20.201095 1200 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.61:35271
I20250113 02:51:20.203858 1298 raft_consensus.cc:491] T 9041b28402664a08b8761fa155c9c229 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:20.204077 1298 raft_consensus.cc:513] T 9041b28402664a08b8761fa155c9c229 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:20.204845 1298 leader_election.cc:290] T 9041b28402664a08b8761fa155c9c229 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:20.205958 1200 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/master-1/data/info.pb
I20250113 02:51:20.206269 1062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9041b28402664a08b8761fa155c9c229" candidate_uuid: "df69f48656b44bb890f76587ee69cc0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
W20250113 02:51:20.206949 852 leader_election.cc:343] T 9041b28402664a08b8761fa155c9c229 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:20.203862 1192 raft_consensus.cc:383] T ce92c2b45a274eb09e72a96b7ec133be P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:20.207998 1192 raft_consensus.cc:738] T ce92c2b45a274eb09e72a96b7ec133be P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:20.208606 1192 consensus_queue.cc:260] T ce92c2b45a274eb09e72a96b7ec133be P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:20.209300 454 catalog_manager.cc:6071] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: default.Table_b581b1a063a54551a8535adf01983fb8 [id=54e4e3810b3b412bbec3b5bf5b28c1a2] alter complete (version 1)
I20250113 02:51:20.209523 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskXJrxug/build/asan/bin/kudu as pid 1200
I20250113 02:51:20.209905 1192 ts_tablet_manager.cc:1428] T ce92c2b45a274eb09e72a96b7ec133be P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20250113 02:51:20.209929 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_5175e117e3f34778a5ef456bed6b5e28"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:20.210619 1192 tablet_bootstrap.cc:492] T 0c55662a71444a3683a825bcec6c5b68 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
W20250113 02:51:20.210587 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5175e117e3f34778a5ef456bed6b5e28 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:20.212203 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet a77c38a394114d6c8a8da99d7eef7669
I20250113 02:51:20.213025 455 master_service.cc:946] ReplaceTablet: received request to replace tablet 0fb75c551b4c424186ef2021db742e8e from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:20.213825 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_49892af4efbf4bdb8230eef4f9a2056e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:20.214316 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_49892af4efbf4bdb8230eef4f9a2056e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:20.216102 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_c52b68d0d3d74003bf10bc8ff205d25b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:20.216552 1192 tablet_bootstrap.cc:654] T 0c55662a71444a3683a825bcec6c5b68 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
W20250113 02:51:20.216609 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c52b68d0d3d74003bf10bc8ff205d25b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:20.217921 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 98f038ad31ff4e05bb88ac68ae341cc2
I20250113 02:51:20.218397 446 catalog_manager.cc:6596] ReplaceTablet: tablet 98f038ad31ff4e05bb88ac68ae341cc2 deleted and replaced by tablet f129bb3fbbfd492d98822be0a6626e44
I20250113 02:51:20.218752 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_d1b93410e6504e71a32397040f50b464"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:20.219199 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d1b93410e6504e71a32397040f50b464 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:20.221158 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_a5e21e7ff75b4d70a6944712fb3d1392"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:20.221598 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a5e21e7ff75b4d70a6944712fb3d1392 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:20.223167 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 93a2729c5a604a27a9bbc451ccb4e128
I20250113 02:51:20.225765 891 tablet_service.cc:1514] Processing DeleteTablet for tablet 93a2729c5a604a27a9bbc451ccb4e128 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:20 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:20.225991 888 tablet_service.cc:1514] Processing DeleteTablet for tablet 98f038ad31ff4e05bb88ac68ae341cc2 with delete_type TABLET_DATA_DELETED (replaced by tablet f129bb3fbbfd492d98822be0a6626e44) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:20.226202 892 tablet_service.cc:1514] Processing DeleteTablet for tablet c62a5e9e43d14b5c94eb87260528469d with delete_type TABLET_DATA_DELETED (replaced by tablet 6c8ca30720664f3a819c4bff2493934d) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:20.226367 1196 tablet_replica.cc:331] T 98f038ad31ff4e05bb88ac68ae341cc2 P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:20.226536 448 master_service.cc:946] ReplaceTablet: received request to replace tablet 32351d6de1574353a6e8073d333da477 from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:20.226560 1196 raft_consensus.cc:2238] T 98f038ad31ff4e05bb88ac68ae341cc2 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:20.226781 1196 raft_consensus.cc:2267] T 98f038ad31ff4e05bb88ac68ae341cc2 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:20.225755 899 tablet_service.cc:1514] Processing DeleteTablet for tablet a77c38a394114d6c8a8da99d7eef7669 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:20 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:20.227299 1298 raft_consensus.cc:491] T 9b5dc06ba3ea4056822ca3e4b1249587 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:20.227452 1298 raft_consensus.cc:513] T 9b5dc06ba3ea4056822ca3e4b1249587 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:20.228015 1196 ts_tablet_manager.cc:1905] T 98f038ad31ff4e05bb88ac68ae341cc2 P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:20.228194 1298 leader_election.cc:290] T 9b5dc06ba3ea4056822ca3e4b1249587 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
W20250113 02:51:20.228652 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7d3562a48c6e4badbbcf37640e91b5ba consensus state has no leader
I20250113 02:51:20.229053 1062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b5dc06ba3ea4056822ca3e4b1249587" candidate_uuid: "df69f48656b44bb890f76587ee69cc0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
W20250113 02:51:20.229647 852 leader_election.cc:343] T 9b5dc06ba3ea4056822ca3e4b1249587 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
W20250113 02:51:20.230252 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 93a2729c5a604a27a9bbc451ccb4e128 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 93a2729c5a604a27a9bbc451ccb4e128 already in progress: creating tablet
I20250113 02:51:20.233426 1192 tablet_bootstrap.cc:492] T 0c55662a71444a3683a825bcec6c5b68 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:20.233610 1192 ts_tablet_manager.cc:1397] T 0c55662a71444a3683a825bcec6c5b68 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.023s user 0.003s sys 0.001s
I20250113 02:51:20.234463 1192 raft_consensus.cc:357] T 0c55662a71444a3683a825bcec6c5b68 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:20.234689 1192 raft_consensus.cc:383] T 0c55662a71444a3683a825bcec6c5b68 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:20.234757 1192 raft_consensus.cc:738] T 0c55662a71444a3683a825bcec6c5b68 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:20.235003 1192 consensus_queue.cc:260] T 0c55662a71444a3683a825bcec6c5b68 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:20.235167 1196 ts_tablet_manager.cc:1918] T 98f038ad31ff4e05bb88ac68ae341cc2 P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:20.235373 1196 log.cc:1198] T 98f038ad31ff4e05bb88ac68ae341cc2 P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/98f038ad31ff4e05bb88ac68ae341cc2
I20250113 02:51:20.235909 1196 ts_tablet_manager.cc:1939] T 98f038ad31ff4e05bb88ac68ae341cc2 P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:20.236987 1192 ts_tablet_manager.cc:1428] T 0c55662a71444a3683a825bcec6c5b68 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250113 02:51:20.237334 1192 tablet_bootstrap.cc:492] T f0e7cc161f354f86b8c2fef303e87c5b P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:20.236610 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet 98f038ad31ff4e05bb88ac68ae341cc2 (table default.Table_5182a1c5c7a2462ba930940ae04b505f [id=6666a614063942c99a78b95d35add037]) successfully deleted
I20250113 02:51:20.239126 1192 tablet_bootstrap.cc:654] T f0e7cc161f354f86b8c2fef303e87c5b P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:20.245390 1194 tablet_bootstrap.cc:492] T 53cc528d282c47788d5d558569f074c1 P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:20.246125 1194 ts_tablet_manager.cc:1397] T 53cc528d282c47788d5d558569f074c1 P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.200s user 0.033s sys 0.019s
I20250113 02:51:20.246978 1194 raft_consensus.cc:357] T 53cc528d282c47788d5d558569f074c1 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:20.247186 1194 raft_consensus.cc:383] T 53cc528d282c47788d5d558569f074c1 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:20.247164 1037 tablet_service.cc:1514] Processing DeleteTablet for tablet 2778d7dc7d9c4843bbc9eb705cec71cd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:20 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:20.247283 1194 raft_consensus.cc:738] T 53cc528d282c47788d5d558569f074c1 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:20.247506 1194 consensus_queue.cc:260] T 53cc528d282c47788d5d558569f074c1 P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:20.248137 1194 ts_tablet_manager.cc:1428] T 53cc528d282c47788d5d558569f074c1 P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250113 02:51:20.248283 1037 tablet_service.cc:1514] Processing DeleteTablet for tablet 63efcb6b2e194c469fb30a197bd6565b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:20 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:20.248462 1194 tablet_bootstrap.cc:492] T 107e6e9e2d84496f8fce92b9ba377189 P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:20.249204 1037 tablet_service.cc:1514] Processing DeleteTablet for tablet a77c38a394114d6c8a8da99d7eef7669 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:20 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:20.250277 1037 tablet_service.cc:1514] Processing DeleteTablet for tablet 98f038ad31ff4e05bb88ac68ae341cc2 with delete_type TABLET_DATA_DELETED (replaced by tablet f129bb3fbbfd492d98822be0a6626e44) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:20.251178 1037 tablet_service.cc:1514] Processing DeleteTablet for tablet 93a2729c5a604a27a9bbc451ccb4e128 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:20 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:20.252454 1192 tablet_bootstrap.cc:492] T f0e7cc161f354f86b8c2fef303e87c5b P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:20.252599 1192 ts_tablet_manager.cc:1397] T f0e7cc161f354f86b8c2fef303e87c5b P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.005s
I20250113 02:51:20.253340 1192 raft_consensus.cc:357] T f0e7cc161f354f86b8c2fef303e87c5b P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:20.253554 1192 raft_consensus.cc:383] T f0e7cc161f354f86b8c2fef303e87c5b P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:20.253654 1192 raft_consensus.cc:738] T f0e7cc161f354f86b8c2fef303e87c5b P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:20.253921 1192 consensus_queue.cc:260] T f0e7cc161f354f86b8c2fef303e87c5b P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:20.254225 1196 tablet_replica.cc:331] T c62a5e9e43d14b5c94eb87260528469d P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:20.254426 1192 ts_tablet_manager.cc:1428] T f0e7cc161f354f86b8c2fef303e87c5b P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250113 02:51:20.254573 1196 raft_consensus.cc:2238] T c62a5e9e43d14b5c94eb87260528469d P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:20.254695 1192 tablet_bootstrap.cc:492] T fb183b4588e84212a3b4b096f430d47d P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:20.254860 1196 raft_consensus.cc:2267] T c62a5e9e43d14b5c94eb87260528469d P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:20.255720 1196 ts_tablet_manager.cc:1905] T c62a5e9e43d14b5c94eb87260528469d P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:20.256385 1192 tablet_bootstrap.cc:654] T fb183b4588e84212a3b4b096f430d47d P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
W20250113 02:51:20.260897 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 23e9321f08ee4ce7b2252ad5c76c0a08 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 23e9321f08ee4ce7b2252ad5c76c0a08 already in progress: creating tablet
W20250113 02:51:20.261294 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 2778d7dc7d9c4843bbc9eb705cec71cd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2778d7dc7d9c4843bbc9eb705cec71cd already in progress: creating tablet
W20250113 02:51:20.261610 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 63efcb6b2e194c469fb30a197bd6565b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 63efcb6b2e194c469fb30a197bd6565b already in progress: creating tablet
W20250113 02:51:20.261883 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet a77c38a394114d6c8a8da99d7eef7669 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a77c38a394114d6c8a8da99d7eef7669 already in progress: creating tablet
I20250113 02:51:20.261862 751 tablet_service.cc:1514] Processing DeleteTablet for tablet 63efcb6b2e194c469fb30a197bd6565b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:20 UTC) from {username='slave'} at 127.0.0.1:55398
W20250113 02:51:20.262173 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 98f038ad31ff4e05bb88ac68ae341cc2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 98f038ad31ff4e05bb88ac68ae341cc2 already in progress: creating tablet
W20250113 02:51:20.262468 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 93a2729c5a604a27a9bbc451ccb4e128 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 93a2729c5a604a27a9bbc451ccb4e128 already in progress: creating tablet
I20250113 02:51:20.263550 753 tablet_service.cc:1514] Processing DeleteTablet for tablet 93a2729c5a604a27a9bbc451ccb4e128 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:20 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:20.264784 1192 tablet_bootstrap.cc:492] T fb183b4588e84212a3b4b096f430d47d P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:20.264957 1192 ts_tablet_manager.cc:1397] T fb183b4588e84212a3b4b096f430d47d P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20250113 02:51:20.265659 1192 raft_consensus.cc:357] T fb183b4588e84212a3b4b096f430d47d P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:20.265887 1192 raft_consensus.cc:383] T fb183b4588e84212a3b4b096f430d47d P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:20.265960 1192 raft_consensus.cc:738] T fb183b4588e84212a3b4b096f430d47d P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:20.266219 1192 consensus_queue.cc:260] T fb183b4588e84212a3b4b096f430d47d P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:20.266738 1192 ts_tablet_manager.cc:1428] T fb183b4588e84212a3b4b096f430d47d P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250113 02:51:20.267038 1192 tablet_bootstrap.cc:492] T 9041b28402664a08b8761fa155c9c229 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:20.268802 1192 tablet_bootstrap.cc:654] T 9041b28402664a08b8761fa155c9c229 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:20.268770 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b5dc06ba3ea4056822ca3e4b1249587" candidate_uuid: "df69f48656b44bb890f76587ee69cc0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
I20250113 02:51:20.271672 1421 sys_catalog.cc:263] Verifying existing consensus state
I20250113 02:51:20.271795 1192 tablet_bootstrap.cc:492] T 9041b28402664a08b8761fa155c9c229 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:20.271935 1192 ts_tablet_manager.cc:1397] T 9041b28402664a08b8761fa155c9c229 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20250113 02:51:20.272774 1192 raft_consensus.cc:357] T 9041b28402664a08b8761fa155c9c229 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:20.273012 1192 raft_consensus.cc:383] T 9041b28402664a08b8761fa155c9c229 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:20.273104 1192 raft_consensus.cc:738] T 9041b28402664a08b8761fa155c9c229 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:20.273346 1192 consensus_queue.cc:260] T 9041b28402664a08b8761fa155c9c229 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:20.273861 1192 ts_tablet_manager.cc:1428] T 9041b28402664a08b8761fa155c9c229 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250113 02:51:20.274168 1192 tablet_bootstrap.cc:492] T 15a05a32febd443dbb519c186048428a P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:20.275419 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9041b28402664a08b8761fa155c9c229" candidate_uuid: "df69f48656b44bb890f76587ee69cc0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
I20250113 02:51:20.275900 1192 tablet_bootstrap.cc:654] T 15a05a32febd443dbb519c186048428a P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:20.284354 1192 tablet_bootstrap.cc:492] T 15a05a32febd443dbb519c186048428a P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:20.284516 1192 ts_tablet_manager.cc:1397] T 15a05a32febd443dbb519c186048428a P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.004s
I20250113 02:51:20.285019 751 tablet_service.cc:1514] Processing DeleteTablet for tablet a77c38a394114d6c8a8da99d7eef7669 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:20 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:20.285226 1195 ts_tablet_manager.cc:1918] T a028cc02273644608f046135851da126 P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250113 02:51:20.285349 1195 log.cc:1198] T a028cc02273644608f046135851da126 P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/a028cc02273644608f046135851da126
I20250113 02:51:20.285270 1192 raft_consensus.cc:357] T 15a05a32febd443dbb519c186048428a P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:20.285485 1192 raft_consensus.cc:383] T 15a05a32febd443dbb519c186048428a P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:20.285555 1192 raft_consensus.cc:738] T 15a05a32febd443dbb519c186048428a P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:20.285823 1195 ts_tablet_manager.cc:1939] T a028cc02273644608f046135851da126 P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:20.285810 1192 consensus_queue.cc:260] T 15a05a32febd443dbb519c186048428a P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:20.286324 1192 ts_tablet_manager.cc:1428] T 15a05a32febd443dbb519c186048428a P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250113 02:51:20.286581 1192 tablet_bootstrap.cc:492] T 8193357427f4491c8c6c1396ceeef2dc P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:20.286625 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 192) took 1061 ms. Trace:
I20250113 02:51:20.286718 1195 rpcz_store.cc:276] 0113 02:51:19.224825 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:19.224885 (+ 60us) service_pool.cc:225] Handling call
0113 02:51:20.140731 (+915846us) ts_tablet_manager.cc:1145] Deleting tablet a028cc02273644608f046135851da126
0113 02:51:20.140739 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:20.192396 (+ 51657us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:20.279424 (+ 87028us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:20.286575 (+ 7151us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":783,"tablet-delete.queue_time_us":915396}
I20250113 02:51:20.287158 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 193) took 1047 ms. Trace:
I20250113 02:51:20.287252 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet a028cc02273644608f046135851da126 (table default.Table_eec2c35d4a8a4196983a872a0fa4bda5 [id=c7f464a7e8f14a0892d145a69a148e58]) successfully deleted
I20250113 02:51:20.287267 1195 rpcz_store.cc:276] 0113 02:51:19.239656 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:19.240065 (+ 409us) service_pool.cc:225] Handling call
0113 02:51:20.287002 (+1046937us) ts_tablet_manager.cc:1145] Deleting tablet fdbf03c89bff438a9a25115eddf60be9
0113 02:51:20.287012 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:20.287127 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1040281}
I20250113 02:51:20.287652 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 195) took 1031 ms. Trace:
I20250113 02:51:20.287746 1195 rpcz_store.cc:276] 0113 02:51:19.256343 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:19.258469 (+ 2126us) service_pool.cc:225] Handling call
0113 02:51:20.287516 (+1029047us) ts_tablet_manager.cc:1145] Deleting tablet 53cc528d282c47788d5d558569f074c1
0113 02:51:20.287526 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:20.287622 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1028743}
W20250113 02:51:20.287744 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet fdbf03c89bff438a9a25115eddf60be9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fdbf03c89bff438a9a25115eddf60be9 already in progress: creating tablet
I20250113 02:51:20.288268 1192 tablet_bootstrap.cc:654] T 8193357427f4491c8c6c1396ceeef2dc P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
W20250113 02:51:20.288319 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 53cc528d282c47788d5d558569f074c1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 53cc528d282c47788d5d558569f074c1 already in progress: creating tablet
I20250113 02:51:20.288646 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 194) took 1040 ms. Trace:
I20250113 02:51:20.288769 1195 rpcz_store.cc:276] 0113 02:51:19.248585 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:19.258821 (+ 10236us) service_pool.cc:225] Handling call
0113 02:51:20.287930 (+1029109us) ts_tablet_manager.cc:1145] Deleting tablet 19f14b3db0b64847b5422bfd5ed323d0
0113 02:51:20.287938 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:20.288611 (+ 673us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":981164}
I20250113 02:51:20.289120 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 197) took 1008 ms. Trace:
W20250113 02:51:20.289264 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 19f14b3db0b64847b5422bfd5ed323d0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 19f14b3db0b64847b5422bfd5ed323d0 already in progress: creating tablet
I20250113 02:51:20.289247 1195 rpcz_store.cc:276] 0113 02:51:19.281123 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:19.281189 (+ 66us) service_pool.cc:225] Handling call
0113 02:51:20.288984 (+1007795us) ts_tablet_manager.cc:1145] Deleting tablet ba61c86743e24666b30580c36b9228c3
0113 02:51:20.288994 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:20.289081 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":979222}
I20250113 02:51:20.289675 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 196) took 1010 ms. Trace:
W20250113 02:51:20.289712 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet ba61c86743e24666b30580c36b9228c3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ba61c86743e24666b30580c36b9228c3 already in progress: creating tablet
I20250113 02:51:20.289793 1195 rpcz_store.cc:276] 0113 02:51:19.279640 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:19.304318 (+ 24678us) service_pool.cc:225] Handling call
0113 02:51:20.289517 (+985199us) ts_tablet_manager.cc:1145] Deleting tablet b834f0acf5314bcb9d81c73de1b31842
0113 02:51:20.289527 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:20.289641 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":957098}
W20250113 02:51:20.290226 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet b834f0acf5314bcb9d81c73de1b31842 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b834f0acf5314bcb9d81c73de1b31842 already in progress: creating tablet
W20250113 02:51:20.290685 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 55ebe9c044d944bc813396c699ba9aa1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 55ebe9c044d944bc813396c699ba9aa1 already in progress: creating tablet
W20250113 02:51:20.290967 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 21c4b419bf2148e0b140301fa80622bb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 21c4b419bf2148e0b140301fa80622bb already in progress: creating tablet
I20250113 02:51:20.291245 743 tablet_service.cc:1514] Processing DeleteTablet for tablet 98f038ad31ff4e05bb88ac68ae341cc2 with delete_type TABLET_DATA_DELETED (replaced by tablet f129bb3fbbfd492d98822be0a6626e44) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:20.294607 1195 tablet_replica.cc:331] T c135506190f849008ce6542d569b402a P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:20.294853 1195 raft_consensus.cc:2238] T c135506190f849008ce6542d569b402a P 1186fa869eb14812832fa6562531f196 [term 1 LEADER]: Raft consensus shutting down.
I20250113 02:51:20.295092 1195 pending_rounds.cc:62] T c135506190f849008ce6542d569b402a P 1186fa869eb14812832fa6562531f196: Trying to abort 1 pending ops.
I20250113 02:51:20.295181 1195 pending_rounds.cc:69] T c135506190f849008ce6542d569b402a P 1186fa869eb14812832fa6562531f196: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113673438542823424 op_type: NO_OP noop_request { }
I20250113 02:51:20.295321 1195 raft_consensus.cc:2883] T c135506190f849008ce6542d569b402a P 1186fa869eb14812832fa6562531f196 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250113 02:51:20.295435 1195 raft_consensus.cc:2267] T c135506190f849008ce6542d569b402a P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:20.295957 1192 tablet_bootstrap.cc:492] T 8193357427f4491c8c6c1396ceeef2dc P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:20.296150 1192 ts_tablet_manager.cc:1397] T 8193357427f4491c8c6c1396ceeef2dc P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.004s
I20250113 02:51:20.296185 1195 ts_tablet_manager.cc:1905] T c135506190f849008ce6542d569b402a P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:20.297034 1192 raft_consensus.cc:357] T 8193357427f4491c8c6c1396ceeef2dc P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:20.297247 1192 raft_consensus.cc:383] T 8193357427f4491c8c6c1396ceeef2dc P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:20.297343 1192 raft_consensus.cc:738] T 8193357427f4491c8c6c1396ceeef2dc P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:20.297585 1192 consensus_queue.cc:260] T 8193357427f4491c8c6c1396ceeef2dc P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:20.298322 1192 ts_tablet_manager.cc:1428] T 8193357427f4491c8c6c1396ceeef2dc P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250113 02:51:20.298573 1191 tablet_bootstrap.cc:492] T 15a05a32febd443dbb519c186048428a P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:20.298628 1192 tablet_bootstrap.cc:492] T c62a5e9e43d14b5c94eb87260528469d P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:20.298725 1191 ts_tablet_manager.cc:1397] T 15a05a32febd443dbb519c186048428a P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.137s user 0.002s sys 0.005s
I20250113 02:51:20.299494 1191 raft_consensus.cc:357] T 15a05a32febd443dbb519c186048428a P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:20.299731 1191 raft_consensus.cc:383] T 15a05a32febd443dbb519c186048428a P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:20.299822 1191 raft_consensus.cc:738] T 15a05a32febd443dbb519c186048428a P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:20.301066 1192 tablet_bootstrap.cc:654] T c62a5e9e43d14b5c94eb87260528469d P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:20.307272 452 catalog_manager.cc:5618] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587) does not have the latest schema for tablet ad9b95c5450a4c9c89a0930862348d49 (table default.Table_936e73f0b76046d285567ac0778e36a9 [id=219c739889d04a97aafa4fe1b55b0a24]). Expected version 2 got 0
I20250113 02:51:20.309051 1421 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6bbc3541604a455584bb32dfc2cc4efd: Bootstrap starting.
I20250113 02:51:20.309098 448 catalog_manager.cc:5618] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467) does not have the latest schema for tablet 2fb62cc83f9c4dcfa9aa01ae8b46c025 (table default.Table_8b9dee7f78684d9c80977952c0e5a2df [id=dd19d0450abb4862a408f2f5f5922a4f]). Expected version 2 got 0
I20250113 02:51:20.309800 454 catalog_manager.cc:5618] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095) does not have the latest schema for tablet ad9b95c5450a4c9c89a0930862348d49 (table default.Table_936e73f0b76046d285567ac0778e36a9 [id=219c739889d04a97aafa4fe1b55b0a24]). Expected version 2 got 0
I20250113 02:51:20.311782 753 tablet_service.cc:1467] Processing CreateTablet for tablet c1fb2b36c0504cb4ba32671ae4c620d1 (DEFAULT_TABLE table=default.Table_53a85211f49241c4af531e479ecdc0cb [id=ff8b884b04954ff48a7c8a72df232e8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:20.312214 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c1fb2b36c0504cb4ba32671ae4c620d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:20.312644 1037 tablet_service.cc:1467] Processing CreateTablet for tablet 0a7e7713bc8648378b668d8b769ac927 (DEFAULT_TABLE table=default.Table_93a7b6abed0446a4baf21e5e7c8f0c1b [id=9fd4d65e7b9d443a82f06a1f660a7275]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:20.313087 1037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0a7e7713bc8648378b668d8b769ac927. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:20.316246 750 tablet_service.cc:1467] Processing CreateTablet for tablet 1a63f485caf54068abd6b50d7de3d14a (DEFAULT_TABLE table=default.Table_4deab2d4aa9e4d6f8e165675d04c119a [id=71508871e2d5469a91aaf0e19b02b638]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:20.316605 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a63f485caf54068abd6b50d7de3d14a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:20.320816 1036 tablet_service.cc:1467] Processing CreateTablet for tablet c826a0b8327f44f9a00e5afe3a36cd8a (DEFAULT_TABLE table=default.Table_03babf194cc448f296512a09b2e3fe3e [id=64b42698782f41a589577af20b33501e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:20.321230 1036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c826a0b8327f44f9a00e5afe3a36cd8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:20.321579 1421 log.cc:826] T 00000000000000000000000000000000 P 6bbc3541604a455584bb32dfc2cc4efd: Log is configured to *not* fsync() on all Append() calls
I20250113 02:51:20.321604 751 tablet_service.cc:1467] Processing CreateTablet for tablet c826a0b8327f44f9a00e5afe3a36cd8a (DEFAULT_TABLE table=default.Table_03babf194cc448f296512a09b2e3fe3e [id=64b42698782f41a589577af20b33501e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:20.321938 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c826a0b8327f44f9a00e5afe3a36cd8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:20.324288 743 tablet_service.cc:1467] Processing CreateTablet for tablet 0a7e7713bc8648378b668d8b769ac927 (DEFAULT_TABLE table=default.Table_93a7b6abed0446a4baf21e5e7c8f0c1b [id=9fd4d65e7b9d443a82f06a1f660a7275]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:20.324743 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0a7e7713bc8648378b668d8b769ac927. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:20.329092 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet d6675bc3cb5f4f809320e28026cdf4b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:17 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:20.329418 1198 tablet_replica.cc:331] T d6675bc3cb5f4f809320e28026cdf4b0 P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:20.330143 1198 raft_consensus.cc:2238] T d6675bc3cb5f4f809320e28026cdf4b0 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:20.330338 1198 raft_consensus.cc:2267] T d6675bc3cb5f4f809320e28026cdf4b0 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:20.331113 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c55662a71444a3683a825bcec6c5b68 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:16 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:20.331885 1198 ts_tablet_manager.cc:1905] T d6675bc3cb5f4f809320e28026cdf4b0 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:20.332343 746 tablet_service.cc:1514] Processing DeleteTablet for tablet f0e7cc161f354f86b8c2fef303e87c5b with delete_type TABLET_DATA_DELETED (replaced by tablet 572e4eed1e924381bada27f8389821a7) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:20.332625 752 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d132c29cb2c46108e5877e38068ef17 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:17 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:20.332828 754 tablet_service.cc:1514] Processing DeleteTablet for tablet ce92c2b45a274eb09e72a96b7ec133be with delete_type TABLET_DATA_DELETED (replaced by tablet 53cc528d282c47788d5d558569f074c1) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:20.333276 747 tablet_service.cc:1467] Processing CreateTablet for tablet 6d4d8b5cc8ad4c9e935d8afaf5766e66 (DEFAULT_TABLE table=default.Table_1d46c338f1ed42c9a61f4aa130290e22 [id=2ab34074f7624b419c1cd36e9a09b4c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:20.333740 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d4d8b5cc8ad4c9e935d8afaf5766e66. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:20.336330 738 tablet_service.cc:1467] Processing CreateTablet for tablet 42fad91fe1b64a21a2c8412a2585af47 (DEFAULT_TABLE table=default.Table_dc98b9959d0c42b58d9f47e3c2c2b73d [id=290b815241d648b6b62b8a70d2efd1c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:20.336796 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 42fad91fe1b64a21a2c8412a2585af47. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:20.340350 744 tablet_service.cc:1467] Processing CreateTablet for tablet 77cf8b0c824f401aba1880eb7fd1807b (DEFAULT_TABLE table=default.Table_9f627945c5404e48977cf58b41173f50 [id=6069bb344e624c3bb8ca97e6c7fe23cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:20.340354 454 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_47f735660c3a49d991a3593625822367" } new_table_name: "default.Table_2794f9d2a3cc49f78de33a775c1619f1" modify_external_catalogs: true
I20250113 02:51:20.340840 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 77cf8b0c824f401aba1880eb7fd1807b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:20.343079 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet ce92c2b45a274eb09e72a96b7ec133be with delete_type TABLET_DATA_DELETED (replaced by tablet 53cc528d282c47788d5d558569f074c1) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:20.344038 1035 tablet_service.cc:1467] Processing CreateTablet for tablet c1fb2b36c0504cb4ba32671ae4c620d1 (DEFAULT_TABLE table=default.Table_53a85211f49241c4af531e479ecdc0cb [id=ff8b884b04954ff48a7c8a72df232e8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:20.344318 755 tablet_service.cc:1467] Processing CreateTablet for tablet 056bf1e67be04d179eace3d93282b145 (DEFAULT_TABLE table=default.Table_f5f150e8da7a4848bfab67568975d66f [id=5a295e63ad43499f90b2993ac92a16cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:20.344550 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c1fb2b36c0504cb4ba32671ae4c620d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:20.344744 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 056bf1e67be04d179eace3d93282b145. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:20.348366 748 tablet_service.cc:1467] Processing CreateTablet for tablet c508724edea147e5bf6e20bca2f50397 (DEFAULT_TABLE table=default.Table_4b5273b24d914b4eadec5f883245398a [id=08db71e435a24e36bdb20f9a5f1cc096]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:20.348883 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c508724edea147e5bf6e20bca2f50397. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:20.350230 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_e2eebe9051194e27864922e074e6adbf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:20.351058 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e2eebe9051194e27864922e074e6adbf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:20.352295 745 tablet_service.cc:1467] Processing CreateTablet for tablet 0752fde3a30a4947a2a688c56ceadab5 (DEFAULT_TABLE table=default.Table_4385ce4af1044706afef0ab60f2710a7 [id=94704429b49b46e38a48258a450d2705]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:20.352770 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0752fde3a30a4947a2a688c56ceadab5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:20.356350 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_2e62961b127b4bb3ac9b5d30f408c299" } modify_external_catalogs: true
I20250113 02:51:20.356614 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_2e62961b127b4bb3ac9b5d30f408c299" } modify_external_catalogs: true
I20250113 02:51:20.359246 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_22fdcd0b65d14830878ae2d87bca9bb9" } new_table_name: "default.Table_5b0e60572cb54fcca8b3b5b94326ae3a" modify_external_catalogs: true
I20250113 02:51:20.359730 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_384366cb310c4e248cdc81ecede00d18"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:20.360337 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_384366cb310c4e248cdc81ecede00d18 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:20.360668 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_b147d4f32fa04b849ff6b75c9382a4e5" } new_table_name: "default.Table_38394a92f7d84b02998b70e0ab85e35f" modify_external_catalogs: true
I20250113 02:51:20.362097 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_796801d17a3c458693693af6e39cf0f5" } modify_external_catalogs: true
I20250113 02:51:20.362331 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_796801d17a3c458693693af6e39cf0f5" } modify_external_catalogs: true
I20250113 02:51:20.362798 448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_9cfb6bce41c0468e8d29a6d0a1f32499" } new_table_name: "default.Table_8b93113e8f8f489eb5030ebde0889e72" modify_external_catalogs: true
I20250113 02:51:20.366530 1025 tablet_service.cc:1467] Processing CreateTablet for tablet 6d4d8b5cc8ad4c9e935d8afaf5766e66 (DEFAULT_TABLE table=default.Table_1d46c338f1ed42c9a61f4aa130290e22 [id=2ab34074f7624b419c1cd36e9a09b4c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:20.366979 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d4d8b5cc8ad4c9e935d8afaf5766e66. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:20.368618 1023 tablet_service.cc:1467] Processing CreateTablet for tablet 77cf8b0c824f401aba1880eb7fd1807b (DEFAULT_TABLE table=default.Table_9f627945c5404e48977cf58b41173f50 [id=6069bb344e624c3bb8ca97e6c7fe23cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:20.369004 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 77cf8b0c824f401aba1880eb7fd1807b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:20.369818 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 23e04961f072440fa3e3f9485e40f79c
I20250113 02:51:20.370085 1033 tablet_service.cc:1467] Processing CreateTablet for tablet 42fad91fe1b64a21a2c8412a2585af47 (DEFAULT_TABLE table=default.Table_dc98b9959d0c42b58d9f47e3c2c2b73d [id=290b815241d648b6b62b8a70d2efd1c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:20.371227 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 42fad91fe1b64a21a2c8412a2585af47. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:20.371409 754 tablet_service.cc:1514] Processing DeleteTablet for tablet 23e04961f072440fa3e3f9485e40f79c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:20 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:20.372936 1039 tablet_service.cc:1467] Processing CreateTablet for tablet 056bf1e67be04d179eace3d93282b145 (DEFAULT_TABLE table=default.Table_f5f150e8da7a4848bfab67568975d66f [id=5a295e63ad43499f90b2993ac92a16cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:20.373323 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 056bf1e67be04d179eace3d93282b145. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:20.373616 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 0fb75c551b4c424186ef2021db742e8e
I20250113 02:51:20.374176 455 catalog_manager.cc:6596] ReplaceTablet: tablet 0fb75c551b4c424186ef2021db742e8e deleted and replaced by tablet 73c31c56a82e4fb089e9644eeb0c7bd5
I20250113 02:51:20.376411 754 tablet_service.cc:1514] Processing DeleteTablet for tablet 0fb75c551b4c424186ef2021db742e8e with delete_type TABLET_DATA_DELETED (replaced by tablet 73c31c56a82e4fb089e9644eeb0c7bd5) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:20.377348 1027 tablet_service.cc:1467] Processing CreateTablet for tablet c508724edea147e5bf6e20bca2f50397 (DEFAULT_TABLE table=default.Table_4b5273b24d914b4eadec5f883245398a [id=08db71e435a24e36bdb20f9a5f1cc096]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:20.379078 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c508724edea147e5bf6e20bca2f50397. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:20.379787 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet de4a58bf63054890808e2e2fe46e43d2 consensus state has no leader
W20250113 02:51:20.380518 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9fff52e99c64462a8a73fccc3c22ca63 consensus state has no leader
I20250113 02:51:20.384192 1026 tablet_service.cc:1467] Processing CreateTablet for tablet 0752fde3a30a4947a2a688c56ceadab5 (DEFAULT_TABLE table=default.Table_4385ce4af1044706afef0ab60f2710a7 [id=94704429b49b46e38a48258a450d2705]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:20.384671 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0752fde3a30a4947a2a688c56ceadab5. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:20.388331 448 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1cfbcd682b944c89847b40fe69f0a75e consensus state has no leader
I20250113 02:51:20.389300 449 catalog_manager.cc:5618] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587) does not have the latest schema for tablet fb183b4588e84212a3b4b096f430d47d (table default.Table_53b46860bb68469fa08f3fd02b2408e5 [id=33bf7fd42ab5458db0f14347353c0db4]). Expected version 2 got 0
I20250113 02:51:20.389631 1383 raft_consensus.cc:491] T 78689f7ed2a64405adb75f372465a792 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:20.389798 1383 raft_consensus.cc:513] T 78689f7ed2a64405adb75f372465a792 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:20.390594 1383 leader_election.cc:290] T 78689f7ed2a64405adb75f372465a792 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
W20250113 02:51:20.391537 709 leader_election.cc:343] T 78689f7ed2a64405adb75f372465a792 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Not found: Tablet not found: 78689f7ed2a64405adb75f372465a792
I20250113 02:51:20.403273 1383 raft_consensus.cc:491] T f0e7cc161f354f86b8c2fef303e87c5b P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:20.403440 1383 raft_consensus.cc:513] T f0e7cc161f354f86b8c2fef303e87c5b P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:20.404222 1383 leader_election.cc:290] T f0e7cc161f354f86b8c2fef303e87c5b P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:20.404862 1062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0e7cc161f354f86b8c2fef303e87c5b" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:20.405124 1062 raft_consensus.cc:2463] T f0e7cc161f354f86b8c2fef303e87c5b P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1186fa869eb14812832fa6562531f196 in term 0.
I20250113 02:51:20.405556 709 leader_election.cc:304] T f0e7cc161f354f86b8c2fef303e87c5b P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters:
I20250113 02:51:20.405812 1383 raft_consensus.cc:2798] T f0e7cc161f354f86b8c2fef303e87c5b P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 02:51:20.405931 1383 raft_consensus.cc:491] T f0e7cc161f354f86b8c2fef303e87c5b P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 02:51:20.406009 1383 raft_consensus.cc:3054] T f0e7cc161f354f86b8c2fef303e87c5b P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:20.408111 754 tablet_service.cc:1467] Processing CreateTablet for tablet 05bf5ab0df874dd5b6559f6d82069184 (DEFAULT_TABLE table=default.Table_5175e117e3f34778a5ef456bed6b5e28 [id=5c9d31a6ab2445ad949f45abffdf4cc7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:20.408532 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05bf5ab0df874dd5b6559f6d82069184. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:20.408411 1191 consensus_queue.cc:260] T 15a05a32febd443dbb519c186048428a P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:20.409749 752 tablet_service.cc:1467] Processing CreateTablet for tablet a97efc70946c4af487e33a85abf6b90c (DEFAULT_TABLE table=default.Table_a5e21e7ff75b4d70a6944712fb3d1392 [id=d29a78ceccaa42808de9c2736d3eb4ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:20.410076 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a97efc70946c4af487e33a85abf6b90c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:20.413388 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 0810b9671f824d0a84741bc1904df247
I20250113 02:51:20.414013 1368 raft_consensus.cc:491] T 15a05a32febd443dbb519c186048428a P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:20.414964 1311 raft_consensus.cc:491] T 15a05a32febd443dbb519c186048428a P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:20.415122 1311 raft_consensus.cc:513] T 15a05a32febd443dbb519c186048428a P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:20.415889 1311 leader_election.cc:290] T 15a05a32febd443dbb519c186048428a P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:20.416386 1368 raft_consensus.cc:513] T 15a05a32febd443dbb519c186048428a P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:20.416747 1109 heartbeater.cc:346] Connected to a master server at 127.0.104.61:35271
I20250113 02:51:20.416980 1109 heartbeater.cc:502] Master 127.0.104.61:35271 was elected leader, sending a full tablet report...
I20250113 02:51:20.423720 452 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet 73c31c56a82e4fb089e9644eeb0c7bd5
W20250113 02:51:20.426302 454 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 378d59d314a84ddb84cf2ab041c51cf1 consensus state has no leader
I20250113 02:51:20.426924 746 tablet_service.cc:1467] Processing CreateTablet for tablet 6e1284c8aaba44b5810b0d06d36757eb (DEFAULT_TABLE table=default.Table_49892af4efbf4bdb8230eef4f9a2056e [id=aa4679fc0d7148e3bef21342b3a0ea22]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:20.428485 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_fb08fe0ed5cd4aaebce3169c264441cb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:20.428560 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e1284c8aaba44b5810b0d06d36757eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:20.428591 738 tablet_service.cc:1514] Processing DeleteTablet for tablet 0810b9671f824d0a84741bc1904df247 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:20 UTC) from {username='slave'} at 127.0.0.1:55398
W20250113 02:51:20.429077 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fb08fe0ed5cd4aaebce3169c264441cb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:20.429767 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15a05a32febd443dbb519c186048428a" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
I20250113 02:51:20.430239 738 tablet_service.cc:1467] Processing CreateTablet for tablet 51d9483ecacb413796d0fd9e7560eac1 (DEFAULT_TABLE table=default.Table_8a2c850672684fb0aef23bd1edfc8cd8 [id=b49a91d3b86d44e1824a987db3de13b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:20.431315 454 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_1b75b69f143d4fb2a97f31fd75657b96" } modify_external_catalogs: true
I20250113 02:51:20.431607 454 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_1b75b69f143d4fb2a97f31fd75657b96" } modify_external_catalogs: true
I20250113 02:51:20.433168 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_d4146353239a45b281b482d6e9537502" } modify_external_catalogs: true
I20250113 02:51:20.433403 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_d4146353239a45b281b482d6e9537502" } modify_external_catalogs: true
I20250113 02:51:20.436856 736 tablet_service.cc:1467] Processing CreateTablet for tablet 99c8046f5d124ca49899df365114c69f (DEFAULT_TABLE table=default.Table_7805d72c367b4ef8aa585de05f3d9b90 [id=d4ce999a9a7b47ada946f259228771bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:20.437284 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 51d9483ecacb413796d0fd9e7560eac1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:20.437908 1198 ts_tablet_manager.cc:1918] T d6675bc3cb5f4f809320e28026cdf4b0 P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:20.438143 1198 log.cc:1198] T d6675bc3cb5f4f809320e28026cdf4b0 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/d6675bc3cb5f4f809320e28026cdf4b0
I20250113 02:51:20.438736 1198 ts_tablet_manager.cc:1939] T d6675bc3cb5f4f809320e28026cdf4b0 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:20.440142 448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_c357cad5db934f938c46acd208cc6f15" } new_table_name: "default.Table_769b925f82bb49ab81662917c3011720" modify_external_catalogs: true
I20250113 02:51:20.442292 1037 tablet_service.cc:1467] Processing CreateTablet for tablet 1a63f485caf54068abd6b50d7de3d14a (DEFAULT_TABLE table=default.Table_4deab2d4aa9e4d6f8e165675d04c119a [id=71508871e2d5469a91aaf0e19b02b638]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:20.442831 1037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a63f485caf54068abd6b50d7de3d14a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:20.448329 775 raft_consensus.cc:2463] T 15a05a32febd443dbb519c186048428a P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 0.
I20250113 02:51:20.449777 1368 leader_election.cc:290] T 15a05a32febd443dbb519c186048428a P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:20.450911 822 heartbeater.cc:346] Connected to a master server at 127.0.104.61:35271
I20250113 02:51:20.451802 1062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15a05a32febd443dbb519c186048428a" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:20.452075 1062 raft_consensus.cc:2463] T 15a05a32febd443dbb519c186048428a P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1186fa869eb14812832fa6562531f196 in term 0.
I20250113 02:51:20.454177 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet c215ffe61b9b426286cc5e808463078d
I20250113 02:51:20.456223 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 9041b28402664a08b8761fa155c9c229
I20250113 02:51:20.460183 736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 99c8046f5d124ca49899df365114c69f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:20.460886 742 tablet_service.cc:1467] Processing CreateTablet for tablet b8b6ffe690fc459db78b2b3bc12671dd (DEFAULT_TABLE table=default.Table_b77c072e465c474db9349f1a3b31c323 [id=d7fe00879d3143c1bf0ff603f9284750]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:20.460844 1383 raft_consensus.cc:513] T f0e7cc161f354f86b8c2fef303e87c5b P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:20.461324 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8b6ffe690fc459db78b2b3bc12671dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:20.461755 1383 leader_election.cc:290] T f0e7cc161f354f86b8c2fef303e87c5b P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 election: Requested vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:20.462006 1191 ts_tablet_manager.cc:1428] T 15a05a32febd443dbb519c186048428a P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.145s user 0.001s sys 0.001s
I20250113 02:51:20.462606 1191 tablet_bootstrap.cc:492] T 8193357427f4491c8c6c1396ceeef2dc P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:20.465684 1191 tablet_bootstrap.cc:654] T 8193357427f4491c8c6c1396ceeef2dc P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:20.468801 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_991d8c72ce924153ad119a440b54fb1c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:20.470266 448 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5b1076a5a17b4fa495975285616401cf consensus state has no leader
I20250113 02:51:20.471344 709 leader_election.cc:304] T 15a05a32febd443dbb519c186048428a P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters:
I20250113 02:51:20.471889 749 tablet_service.cc:1467] Processing CreateTablet for tablet 0539cbbe47e64c039ad39c4b52edcc4b (DEFAULT_TABLE table=default.Table_d1b93410e6504e71a32397040f50b464 [id=817479ce0ac04442a0f89e6115d24ba4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:20.473021 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_8be7ebd7e1984b979fce6e7aac1f576e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:20.473342 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_54963a1a7ff24d5a99644441a755f5de"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:20.473737 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8be7ebd7e1984b979fce6e7aac1f576e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250113 02:51:20.474048 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_54963a1a7ff24d5a99644441a755f5de in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:20.475517 1383 raft_consensus.cc:2798] T 15a05a32febd443dbb519c186048428a P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 02:51:20.475665 1383 raft_consensus.cc:491] T 15a05a32febd443dbb519c186048428a P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 02:51:20.476250 1198 tablet_replica.cc:331] T 0c55662a71444a3683a825bcec6c5b68 P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:20.476480 1198 raft_consensus.cc:2238] T 0c55662a71444a3683a825bcec6c5b68 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:20.476706 1198 raft_consensus.cc:2267] T 0c55662a71444a3683a825bcec6c5b68 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:20.476897 1062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0e7cc161f354f86b8c2fef303e87c5b" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca"
I20250113 02:51:20.477139 1062 raft_consensus.cc:3054] T f0e7cc161f354f86b8c2fef303e87c5b P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:20.472698 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_34ae6a8cce9548ee87be497afcc96718"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:20.481119 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_34ae6a8cce9548ee87be497afcc96718 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:20.481185 1198 ts_tablet_manager.cc:1905] T 0c55662a71444a3683a825bcec6c5b68 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:20.482398 1311 raft_consensus.cc:491] T fb183b4588e84212a3b4b096f430d47d P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:20.482641 1311 raft_consensus.cc:513] T fb183b4588e84212a3b4b096f430d47d P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:20.483528 1311 leader_election.cc:290] T fb183b4588e84212a3b4b096f430d47d P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:20.472990 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_ffb4a25d44a04760b8306ea2678f4b42"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:20.471835 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_90fe60d176ce484680307442a262e8d5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:20.485144 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ffb4a25d44a04760b8306ea2678f4b42 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250113 02:51:20.485240 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_90fe60d176ce484680307442a262e8d5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:20.469579 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_6a9cc0903b674ac49fd6ffaa542a5595"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:20.486871 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6a9cc0903b674ac49fd6ffaa542a5595 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250113 02:51:20.470475 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_991d8c72ce924153ad119a440b54fb1c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:20.488338 1062 raft_consensus.cc:2463] T f0e7cc161f354f86b8c2fef303e87c5b P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1186fa869eb14812832fa6562531f196 in term 1.
I20250113 02:51:20.498442 709 leader_election.cc:304] T f0e7cc161f354f86b8c2fef303e87c5b P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters:
I20250113 02:51:20.501788 995 leader_election.cc:304] T 15a05a32febd443dbb519c186048428a P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters:
I20250113 02:51:20.502467 1311 raft_consensus.cc:2798] T 15a05a32febd443dbb519c186048428a P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 02:51:20.502621 1311 raft_consensus.cc:491] T 15a05a32febd443dbb519c186048428a P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 02:51:20.502751 1311 raft_consensus.cc:3054] T 15a05a32febd443dbb519c186048428a P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:20.506319 1311 raft_consensus.cc:513] T 15a05a32febd443dbb519c186048428a P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:20.508935 1311 leader_election.cc:290] T 15a05a32febd443dbb519c186048428a P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Requested vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:20.510018 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet 0fb75c551b4c424186ef2021db742e8e with delete_type TABLET_DATA_DELETED (replaced by tablet 73c31c56a82e4fb089e9644eeb0c7bd5) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:20.510568 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet 8193357427f4491c8c6c1396ceeef2dc with delete_type TABLET_DATA_DELETED (replaced by tablet d3b695ac20ba4205b2d8732db4740531) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:20.511116 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet f0e7cc161f354f86b8c2fef303e87c5b with delete_type TABLET_DATA_DELETED (replaced by tablet 572e4eed1e924381bada27f8389821a7) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:20.511643 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet 15a05a32febd443dbb519c186048428a with delete_type TABLET_DATA_DELETED (replaced by tablet 18e2d79d731142f89b9089869bc736e5) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:20.514411 1037 tablet_service.cc:1514] Processing DeleteTablet for tablet 23e04961f072440fa3e3f9485e40f79c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:20 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:20.515043 1035 tablet_service.cc:1467] Processing CreateTablet for tablet 05bf5ab0df874dd5b6559f6d82069184 (DEFAULT_TABLE table=default.Table_5175e117e3f34778a5ef456bed6b5e28 [id=5c9d31a6ab2445ad949f45abffdf4cc7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:20.515442 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05bf5ab0df874dd5b6559f6d82069184. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:20.525457 1198 ts_tablet_manager.cc:1918] T 0c55662a71444a3683a825bcec6c5b68 P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:20.525668 1198 log.cc:1198] T 0c55662a71444a3683a825bcec6c5b68 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/0c55662a71444a3683a825bcec6c5b68
I20250113 02:51:20.526090 1198 ts_tablet_manager.cc:1939] T 0c55662a71444a3683a825bcec6c5b68 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:20.527252 1198 tablet_replica.cc:331] T ce92c2b45a274eb09e72a96b7ec133be P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:20.527508 1198 raft_consensus.cc:2238] T ce92c2b45a274eb09e72a96b7ec133be P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:20.527724 1198 raft_consensus.cc:2267] T ce92c2b45a274eb09e72a96b7ec133be P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:20.531405 1037 tablet_service.cc:1467] Processing CreateTablet for tablet a97efc70946c4af487e33a85abf6b90c (DEFAULT_TABLE table=default.Table_a5e21e7ff75b4d70a6944712fb3d1392 [id=d29a78ceccaa42808de9c2736d3eb4ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:20.531877 1037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a97efc70946c4af487e33a85abf6b90c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:20.531998 1109 heartbeater.cc:502] Master 127.0.104.61:35271 was elected leader, sending a full tablet report...
I20250113 02:51:20.538097 1198 ts_tablet_manager.cc:1905] T ce92c2b45a274eb09e72a96b7ec133be P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:20.538908 1033 tablet_service.cc:1514] Processing DeleteTablet for tablet 0810b9671f824d0a84741bc1904df247 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:20 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:20.540453 741 tablet_service.cc:1467] Processing CreateTablet for tablet 6666922d677a4c678f80d7c6c47fd2a5 (DEFAULT_TABLE table=default.Table_93d706c664024860855d1c9d91992817 [id=6e014400813c4402891b32bcbe3555e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:20.541777 1035 tablet_service.cc:1467] Processing CreateTablet for tablet 6e1284c8aaba44b5810b0d06d36757eb (DEFAULT_TABLE table=default.Table_49892af4efbf4bdb8230eef4f9a2056e [id=aa4679fc0d7148e3bef21342b3a0ea22]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:20.542165 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e1284c8aaba44b5810b0d06d36757eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:20.542860 1033 tablet_service.cc:1467] Processing CreateTablet for tablet 51d9483ecacb413796d0fd9e7560eac1 (DEFAULT_TABLE table=default.Table_8a2c850672684fb0aef23bd1edfc8cd8 [id=b49a91d3b86d44e1824a987db3de13b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:20.543476 748 tablet_service.cc:1514] Processing DeleteTablet for tablet c215ffe61b9b426286cc5e808463078d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:20 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:20.544097 1383 raft_consensus.cc:3054] T 15a05a32febd443dbb519c186048428a P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Advancing to term 1
W20250113 02:51:20.544493 851 leader_election.cc:343] T 9b5dc06ba3ea4056822ca3e4b1249587 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:20.546097 1298 raft_consensus.cc:491] T 68023c30a7f644eaa5b4ddd60d6c5ef5 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:20.547708 1298 raft_consensus.cc:513] T 68023c30a7f644eaa5b4ddd60d6c5ef5 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:20.548381 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 51d9483ecacb413796d0fd9e7560eac1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:20.548799 851 leader_election.cc:304] T 9b5dc06ba3ea4056822ca3e4b1249587 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df69f48656b44bb890f76587ee69cc0f; no voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca
I20250113 02:51:20.551337 892 tablet_service.cc:1467] Processing CreateTablet for tablet c826a0b8327f44f9a00e5afe3a36cd8a (DEFAULT_TABLE table=default.Table_03babf194cc448f296512a09b2e3fe3e [id=64b42698782f41a589577af20b33501e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:20.541400 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15a05a32febd443dbb519c186048428a" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196"
I20250113 02:51:20.553575 1036 tablet_service.cc:1467] Processing CreateTablet for tablet 99c8046f5d124ca49899df365114c69f (DEFAULT_TABLE table=default.Table_7805d72c367b4ef8aa585de05f3d9b90 [id=d4ce999a9a7b47ada946f259228771bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:20.553951 1036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 99c8046f5d124ca49899df365114c69f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:20.560556 1023 tablet_service.cc:1467] Processing CreateTablet for tablet 0539cbbe47e64c039ad39c4b52edcc4b (DEFAULT_TABLE table=default.Table_d1b93410e6504e71a32397040f50b464 [id=817479ce0ac04442a0f89e6115d24ba4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:20.560943 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0539cbbe47e64c039ad39c4b52edcc4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:20.564972 1026 tablet_service.cc:1467] Processing CreateTablet for tablet c7243d27a1c642269a3d0e67632e458c (DEFAULT_TABLE table=default.Table_7157800e639b45d8bb947e53d4319cfc [id=39997cddb4684b6c984eeeb48d715002]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:20.565297 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c7243d27a1c642269a3d0e67632e458c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:20.566421 1025 tablet_service.cc:1467] Processing CreateTablet for tablet f129bb3fbbfd492d98822be0a6626e44 (DEFAULT_TABLE table=default.Table_5182a1c5c7a2462ba930940ae04b505f [id=6666a614063942c99a78b95d35add037]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:20.567540 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f129bb3fbbfd492d98822be0a6626e44. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:20.570400 1027 tablet_service.cc:1467] Processing CreateTablet for tablet 3b1910496e0849b193e390a5ef1a6b2b (DEFAULT_TABLE table=default.Table_2f3720d8d897466388c3f9dde157d701 [id=8af6fc6e393d4cd3a5eb0671346df592]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:20.570887 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b1910496e0849b193e390a5ef1a6b2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:20.572137 1031 tablet_service.cc:1514] Processing DeleteTablet for tablet c215ffe61b9b426286cc5e808463078d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:20 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:20.572391 1032 tablet_service.cc:1514] Processing DeleteTablet for tablet 9041b28402664a08b8761fa155c9c229 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:20 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:20.572441 888 tablet_service.cc:1467] Processing CreateTablet for tablet c1fb2b36c0504cb4ba32671ae4c620d1 (DEFAULT_TABLE table=default.Table_53a85211f49241c4af531e479ecdc0cb [id=ff8b884b04954ff48a7c8a72df232e8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:20.575295 899 tablet_service.cc:1467] Processing CreateTablet for tablet 0a7e7713bc8648378b668d8b769ac927 (DEFAULT_TABLE table=default.Table_93a7b6abed0446a4baf21e5e7c8f0c1b [id=9fd4d65e7b9d443a82f06a1f660a7275]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:20.576334 1039 tablet_service.cc:1467] Processing CreateTablet for tablet 6666922d677a4c678f80d7c6c47fd2a5 (DEFAULT_TABLE table=default.Table_93d706c664024860855d1c9d91992817 [id=6e014400813c4402891b32bcbe3555e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:20.576726 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6666922d677a4c678f80d7c6c47fd2a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:20.577541 737 tablet_service.cc:1467] Processing CreateTablet for tablet e468b652f71f44e3945f8d919ac0e1a8 (DEFAULT_TABLE table=default.Table_c52b68d0d3d74003bf10bc8ff205d25b [id=caf9972c69f14619be22fb3a9e563b05]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:20.578831 1298 leader_election.cc:290] T 68023c30a7f644eaa5b4ddd60d6c5ef5 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587), 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
I20250113 02:51:20.579365 891 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b5dc06ba3ea4056822ca3e4b1249587 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:18 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:20.579073 1037 tablet_service.cc:1467] Processing CreateTablet for tablet b8b6ffe690fc459db78b2b3bc12671dd (DEFAULT_TABLE table=default.Table_b77c072e465c474db9349f1a3b31c323 [id=d7fe00879d3143c1bf0ff603f9284750]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:20.579622 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c826a0b8327f44f9a00e5afe3a36cd8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:20.580261 740 tablet_service.cc:1467] Processing CreateTablet for tablet 3b1910496e0849b193e390a5ef1a6b2b (DEFAULT_TABLE table=default.Table_2f3720d8d897466388c3f9dde157d701 [id=8af6fc6e393d4cd3a5eb0671346df592]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:20.581107 890 tablet_service.cc:1467] Processing CreateTablet for tablet 6d4d8b5cc8ad4c9e935d8afaf5766e66 (DEFAULT_TABLE table=default.Table_1d46c338f1ed42c9a61f4aa130290e22 [id=2ab34074f7624b419c1cd36e9a09b4c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:20.581506 884 tablet_service.cc:1514] Processing DeleteTablet for tablet 68023c30a7f644eaa5b4ddd60d6c5ef5 with delete_type TABLET_DATA_DELETED (replaced by tablet 32351d6de1574353a6e8073d333da477) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:20.581961 449 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_8606b4caf08b43b18b25ee8849eebd07" } modify_external_catalogs: true
I20250113 02:51:20.582185 449 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_8606b4caf08b43b18b25ee8849eebd07" } modify_external_catalogs: true
I20250113 02:51:20.583119 1037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8b6ffe690fc459db78b2b3bc12671dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:20.583683 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_5b825152ff4d4e20a7600a9ad0dfe0a3" } new_table_name: "default.Table_1be9cb7e8848435fa27f8b8eb7a77d9c" modify_external_catalogs: true
I20250113 02:51:20.585295 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_abd20980e42242719c1529668d68128b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:20.586508 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_9f627945c5404e48977cf58b41173f50" } new_table_name: "default.Table_ea18ad8d07b34ee4b3ef07b8f0431cfd" modify_external_catalogs: true
I20250113 02:51:20.580426 739 tablet_service.cc:1467] Processing CreateTablet for tablet f129bb3fbbfd492d98822be0a6626e44 (DEFAULT_TABLE table=default.Table_5182a1c5c7a2462ba930940ae04b505f [id=6666a614063942c99a78b95d35add037]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:20.588651 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_016e5998c5ef4112908e6917ccd57d15"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:20.589329 1270 raft_consensus.cc:491] T ce92c2b45a274eb09e72a96b7ec133be P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:20.588835 1035 tablet_service.cc:1467] Processing CreateTablet for tablet e468b652f71f44e3945f8d919ac0e1a8 (DEFAULT_TABLE table=default.Table_c52b68d0d3d74003bf10bc8ff205d25b [id=caf9972c69f14619be22fb3a9e563b05]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:20.589989 1298 raft_consensus.cc:491] T fb183b4588e84212a3b4b096f430d47d P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:20.590190 1298 raft_consensus.cc:513] T fb183b4588e84212a3b4b096f430d47d P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:20.590983 894 tablet_service.cc:1467] Processing CreateTablet for tablet 1a63f485caf54068abd6b50d7de3d14a (DEFAULT_TABLE table=default.Table_4deab2d4aa9e4d6f8e165675d04c119a [id=71508871e2d5469a91aaf0e19b02b638]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:20.588152 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb183b4588e84212a3b4b096f430d47d" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
I20250113 02:51:20.592263 1298 leader_election.cc:290] T fb183b4588e84212a3b4b096f430d47d P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:20.592459 1298 raft_consensus.cc:2743] T 9b5dc06ba3ea4056822ca3e4b1249587 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250113 02:51:20.593648 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ff8ef6290698477391721fd65b9eeb28 consensus state has no leader
I20250113 02:51:20.594859 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c1fb2b36c0504cb4ba32671ae4c620d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:20.594907 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_8b9dee7f78684d9c80977952c0e5a2df" } new_table_name: "default.Table_37b69086d7374e7ea7b83b40e8f202db" modify_external_catalogs: true
I20250113 02:51:20.596099 884 tablet_service.cc:1467] Processing CreateTablet for tablet 42fad91fe1b64a21a2c8412a2585af47 (DEFAULT_TABLE table=default.Table_dc98b9959d0c42b58d9f47e3c2c2b73d [id=290b815241d648b6b62b8a70d2efd1c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:20.596701 1194 tablet_bootstrap.cc:654] T 107e6e9e2d84496f8fce92b9ba377189 P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
W20250113 02:51:20.598090 451 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 77cf8b0c824f401aba1880eb7fd1807b consensus state has no leader
I20250113 02:51:20.598299 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0a7e7713bc8648378b668d8b769ac927. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:20.599473 880 tablet_service.cc:1467] Processing CreateTablet for tablet 056bf1e67be04d179eace3d93282b145 (DEFAULT_TABLE table=default.Table_f5f150e8da7a4848bfab67568975d66f [id=5a295e63ad43499f90b2993ac92a16cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:20.599517 886 tablet_service.cc:1467] Processing CreateTablet for tablet c508724edea147e5bf6e20bca2f50397 (DEFAULT_TABLE table=default.Table_4b5273b24d914b4eadec5f883245398a [id=08db71e435a24e36bdb20f9a5f1cc096]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:20.601559 881 tablet_service.cc:1467] Processing CreateTablet for tablet 0752fde3a30a4947a2a688c56ceadab5 (DEFAULT_TABLE table=default.Table_4385ce4af1044706afef0ab60f2710a7 [id=94704429b49b46e38a48258a450d2705]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:20.601583 751 tablet_service.cc:1467] Processing CreateTablet for tablet 377ebf66bbef40988043f6200ae0893c (DEFAULT_TABLE table=default.Table_384366cb310c4e248cdc81ecede00d18 [id=8b35e389de9f4917904e87b43618737e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:20.601745 891 tablet_service.cc:1467] Processing CreateTablet for tablet 77cf8b0c824f401aba1880eb7fd1807b (DEFAULT_TABLE table=default.Table_9f627945c5404e48977cf58b41173f50 [id=6069bb344e624c3bb8ca97e6c7fe23cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:20.603731 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0539cbbe47e64c039ad39c4b52edcc4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:20.604461 747 tablet_service.cc:1514] Processing DeleteTablet for tablet 9041b28402664a08b8761fa155c9c229 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:20 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:20.605027 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a63f485caf54068abd6b50d7de3d14a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:20.605062 1368 raft_consensus.cc:2798] T f0e7cc161f354f86b8c2fef303e87c5b P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Leader election won for term 1
I20250113 02:51:20.605373 885 tablet_service.cc:1514] Processing DeleteTablet for tablet 0fb75c551b4c424186ef2021db742e8e with delete_type TABLET_DATA_DELETED (replaced by tablet 73c31c56a82e4fb089e9644eeb0c7bd5) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:20.605458 753 tablet_service.cc:1467] Processing CreateTablet for tablet c7243d27a1c642269a3d0e67632e458c (DEFAULT_TABLE table=default.Table_7157800e639b45d8bb947e53d4319cfc [id=39997cddb4684b6c984eeeb48d715002]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:20.605644 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0752fde3a30a4947a2a688c56ceadab5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:20.605993 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 77cf8b0c824f401aba1880eb7fd1807b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:20.610229 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c508724edea147e5bf6e20bca2f50397. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:20.611501 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2fb62cc83f9c4dcfa9aa01ae8b46c025 consensus state has no leader
I20250113 02:51:20.614063 743 tablet_service.cc:1467] Processing CreateTablet for tablet 0fe5421dc3ea4be684cab084f13d8df0 (DEFAULT_TABLE table=default.Table_e2eebe9051194e27864922e074e6adbf [id=beb83fe520f94241949a46d436e6374f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:20.615772 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_5b0e60572cb54fcca8b3b5b94326ae3a" } modify_external_catalogs: true
I20250113 02:51:20.616010 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_5b0e60572cb54fcca8b3b5b94326ae3a" } modify_external_catalogs: true
I20250113 02:51:20.617515 748 tablet_service.cc:1514] Processing DeleteTablet for tablet 15a05a32febd443dbb519c186048428a with delete_type TABLET_DATA_DELETED (replaced by tablet 18e2d79d731142f89b9089869bc736e5) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:20.619125 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 056bf1e67be04d179eace3d93282b145. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:20.621433 744 tablet_service.cc:1467] Processing CreateTablet for tablet f56230a2c75d4763a54671c79358f4d9 (DEFAULT_TABLE table=default.Table_fb08fe0ed5cd4aaebce3169c264441cb [id=34b84bee300d4b30b1d426b9f26e6e4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:20.620159 1270 raft_consensus.cc:513] T ce92c2b45a274eb09e72a96b7ec133be P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:20.624356 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f129bb3fbbfd492d98822be0a6626e44. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:20.625337 893 tablet_service.cc:1514] Processing DeleteTablet for tablet 23e04961f072440fa3e3f9485e40f79c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:20 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:20.625710 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_a5e21e7ff75b4d70a6944712fb3d1392" } modify_external_catalogs: true
I20250113 02:51:20.625903 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_a5e21e7ff75b4d70a6944712fb3d1392" } modify_external_catalogs: true
I20250113 02:51:20.626024 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 42fad91fe1b64a21a2c8412a2585af47. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:20.626686 893 tablet_service.cc:1467] Processing CreateTablet for tablet a97efc70946c4af487e33a85abf6b90c (DEFAULT_TABLE table=default.Table_a5e21e7ff75b4d70a6944712fb3d1392 [id=d29a78ceccaa42808de9c2736d3eb4ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:20.627113 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a97efc70946c4af487e33a85abf6b90c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:20.632898 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 377ebf66bbef40988043f6200ae0893c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:20.633139 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6666922d677a4c678f80d7c6c47fd2a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:20.633471 448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_34ae6a8cce9548ee87be497afcc96718" } new_table_name: "default.Table_d866aff056e246d88df242feb6d4e627" modify_external_catalogs: true
I20250113 02:51:20.635663 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b1910496e0849b193e390a5ef1a6b2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:20.636157 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f56230a2c75d4763a54671c79358f4d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:20.637212 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c7243d27a1c642269a3d0e67632e458c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:20.637391 885 tablet_service.cc:1467] Processing CreateTablet for tablet 6e1284c8aaba44b5810b0d06d36757eb (DEFAULT_TABLE table=default.Table_49892af4efbf4bdb8230eef4f9a2056e [id=aa4679fc0d7148e3bef21342b3a0ea22]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:20.637775 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d4d8b5cc8ad4c9e935d8afaf5766e66. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:20.638998 737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e468b652f71f44e3945f8d919ac0e1a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:20.641965 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e1284c8aaba44b5810b0d06d36757eb. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:20.642575 448 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4b07a985897641c3ac48021c2a4daedb has no consensus state
I20250113 02:51:20.551736 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet d6675bc3cb5f4f809320e28026cdf4b0 (table default.Table_d12d804163994a63a236be44401aa53b [id=f806136f234042af82e9831467a69caa]) successfully deleted
W20250113 02:51:20.642827 440 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.104.3:42587, user_credentials={real_user=slave}} blocked reactor thread for 91262.2us
I20250113 02:51:20.643496 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet 0c55662a71444a3683a825bcec6c5b68 (table default.Table_3865c1517889457a87be9ea12b3a156f [id=79b4c84e05ce4571b42e2b4e40fae978]) successfully deleted
I20250113 02:51:20.646631 1195 ts_tablet_manager.cc:1918] T c135506190f849008ce6542d569b402a P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250113 02:51:20.648484 1270 leader_election.cc:290] T ce92c2b45a274eb09e72a96b7ec133be P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:20.649644 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_ec91552e358547e99d908cef4bcb7c36"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:20.656102 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_016e5998c5ef4112908e6917ccd57d15 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:20.656773 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0fe5421dc3ea4be684cab084f13d8df0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:20.658535 447 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_9cd2b585fc704fb6a2d0da56a774b8bd" } modify_external_catalogs: true
I20250113 02:51:20.658744 447 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_9cd2b585fc704fb6a2d0da56a774b8bd" } modify_external_catalogs: true
I20250113 02:51:20.661592 453 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_53b46860bb68469fa08f3fd02b2408e5" } modify_external_catalogs: true
I20250113 02:51:20.661798 1191 tablet_bootstrap.cc:492] T 8193357427f4491c8c6c1396ceeef2dc P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:20.661777 453 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_53b46860bb68469fa08f3fd02b2408e5" } modify_external_catalogs: true
W20250113 02:51:20.662168 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_abd20980e42242719c1529668d68128b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:20.662228 1195 log.cc:1198] T c135506190f849008ce6542d569b402a P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/c135506190f849008ce6542d569b402a
I20250113 02:51:20.662899 446 master_service.cc:946] ReplaceTablet: received request to replace tablet be80b631ccd747f1b9355a00787a7b55 from {username='slave'} at 127.0.0.1:49258
W20250113 02:51:20.663244 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ec91552e358547e99d908cef4bcb7c36 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:20.664058 1194 tablet_bootstrap.cc:492] T 107e6e9e2d84496f8fce92b9ba377189 P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:20.664243 1194 ts_tablet_manager.cc:1397] T 107e6e9e2d84496f8fce92b9ba377189 P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.416s user 0.003s sys 0.001s
I20250113 02:51:20.667914 1383 raft_consensus.cc:513] T 15a05a32febd443dbb519c186048428a P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:20.668116 881 tablet_service.cc:1514] Processing DeleteTablet for tablet 0810b9671f824d0a84741bc1904df247 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:20 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:20.672328 454 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_a27963e06eed4ec08989e43969f3bbbb" } new_table_name: "default.Table_6a0195b054a043269e133f6ed8dd5979" modify_external_catalogs: true
I20250113 02:51:20.672912 886 tablet_service.cc:1467] Processing CreateTablet for tablet 05bf5ab0df874dd5b6559f6d82069184 (DEFAULT_TABLE table=default.Table_5175e117e3f34778a5ef456bed6b5e28 [id=5c9d31a6ab2445ad949f45abffdf4cc7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:20.673345 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05bf5ab0df874dd5b6559f6d82069184. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:20.674046 884 tablet_service.cc:1467] Processing CreateTablet for tablet 51d9483ecacb413796d0fd9e7560eac1 (DEFAULT_TABLE table=default.Table_8a2c850672684fb0aef23bd1edfc8cd8 [id=b49a91d3b86d44e1824a987db3de13b6]), partition=RANGE (key) PARTITION UNBOUNDED
W20250113 02:51:20.674916 995 leader_election.cc:343] T fb183b4588e84212a3b4b096f430d47d P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:20.675177 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e468b652f71f44e3945f8d919ac0e1a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:20.676942 1025 tablet_service.cc:1467] Processing CreateTablet for tablet f56230a2c75d4763a54671c79358f4d9 (DEFAULT_TABLE table=default.Table_fb08fe0ed5cd4aaebce3169c264441cb [id=34b84bee300d4b30b1d426b9f26e6e4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:20.677340 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f56230a2c75d4763a54671c79358f4d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:20.677485 1192 tablet_bootstrap.cc:492] T c62a5e9e43d14b5c94eb87260528469d P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:20.685032 881 tablet_service.cc:1467] Processing CreateTablet for tablet 99c8046f5d124ca49899df365114c69f (DEFAULT_TABLE table=default.Table_7805d72c367b4ef8aa585de05f3d9b90 [id=d4ce999a9a7b47ada946f259228771bb]), partition=RANGE (key) PARTITION UNBOUNDED
W20250113 02:51:20.691438 454 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4efa63b8be794e1db4b340f4c0c4cd5a consensus state has no leader
I20250113 02:51:20.694813 885 tablet_service.cc:1467] Processing CreateTablet for tablet b8b6ffe690fc459db78b2b3bc12671dd (DEFAULT_TABLE table=default.Table_b77c072e465c474db9349f1a3b31c323 [id=d7fe00879d3143c1bf0ff603f9284750]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:20.696635 748 tablet_service.cc:1467] Processing CreateTablet for tablet aa5c172a69bf4ca79999f4b58df08de8 (DEFAULT_TABLE table=default.Table_90fe60d176ce484680307442a262e8d5 [id=5dceb76e1dbe422ab98d4da5bf6c45a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:20.697595 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet a97efc70946c4af487e33a85abf6b90c
I20250113 02:51:20.697525 891 tablet_service.cc:1467] Processing CreateTablet for tablet c7243d27a1c642269a3d0e67632e458c (DEFAULT_TABLE table=default.Table_7157800e639b45d8bb947e53d4319cfc [id=39997cddb4684b6c984eeeb48d715002]), partition=RANGE (key) PARTITION UNBOUNDED
W20250113 02:51:20.698086 851 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:40095, user_credentials={real_user=slave}} blocked reactor thread for 427788us
I20250113 02:51:20.698168 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet be80b631ccd747f1b9355a00787a7b55
W20250113 02:51:20.698611 851 leader_election.cc:343] T 9041b28402664a08b8761fa155c9c229 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:20.698686 446 catalog_manager.cc:6596] ReplaceTablet: tablet be80b631ccd747f1b9355a00787a7b55 deleted and replaced by tablet e70658c90555463498d3fc2e2f13ea7f
I20250113 02:51:20.698724 747 tablet_service.cc:1514] Processing DeleteTablet for tablet a97efc70946c4af487e33a85abf6b90c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:20 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:20.698765 851 leader_election.cc:304] T 9041b28402664a08b8761fa155c9c229 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df69f48656b44bb890f76587ee69cc0f; no voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca
I20250113 02:51:20.698880 880 tablet_service.cc:1467] Processing CreateTablet for tablet 0539cbbe47e64c039ad39c4b52edcc4b (DEFAULT_TABLE table=default.Table_d1b93410e6504e71a32397040f50b464 [id=817479ce0ac04442a0f89e6115d24ba4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:20.699144 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet bcb59573e82743858bd39bde64627d6d
I20250113 02:51:20.701061 745 tablet_service.cc:1514] Processing DeleteTablet for tablet bcb59573e82743858bd39bde64627d6d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:20 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:20.706382 1192 ts_tablet_manager.cc:1397] T c62a5e9e43d14b5c94eb87260528469d P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.381s user 0.035s sys 0.050s
I20250113 02:51:20.706328 1023 tablet_service.cc:1467] Processing CreateTablet for tablet 377ebf66bbef40988043f6200ae0893c (DEFAULT_TABLE table=default.Table_384366cb310c4e248cdc81ecede00d18 [id=8b35e389de9f4917904e87b43618737e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:20.707633 1438 raft_consensus.cc:2743] T 9041b28402664a08b8761fa155c9c229 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:20.708163 1194 raft_consensus.cc:357] T 107e6e9e2d84496f8fce92b9ba377189 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:20.709254 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 51d9483ecacb413796d0fd9e7560eac1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:20.709910 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet fb183b4588e84212a3b4b096f430d47d
I20250113 02:51:20.710529 1194 raft_consensus.cc:383] T 107e6e9e2d84496f8fce92b9ba377189 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:20.710635 1194 raft_consensus.cc:738] T 107e6e9e2d84496f8fce92b9ba377189 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:20.710845 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet c71511a100f14322b5091a9804720fa0
I20250113 02:51:20.711597 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet de4a58bf63054890808e2e2fe46e43d2
I20250113 02:51:20.714197 1062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce92c2b45a274eb09e72a96b7ec133be" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:20.714607 1033 tablet_service.cc:1467] Processing CreateTablet for tablet 0fe5421dc3ea4be684cab084f13d8df0 (DEFAULT_TABLE table=default.Table_e2eebe9051194e27864922e074e6adbf [id=beb83fe520f94241949a46d436e6374f]), partition=RANGE (key) PARTITION UNBOUNDED
W20250113 02:51:20.715906 709 leader_election.cc:343] T ce92c2b45a274eb09e72a96b7ec133be P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: Tablet not RUNNING: STOPPED
I20250113 02:51:20.716977 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_ccebddd68b0448a49d408b9922a78a4d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:20.717547 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ccebddd68b0448a49d408b9922a78a4d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:20.718900 1194 consensus_queue.cc:260] T 107e6e9e2d84496f8fce92b9ba377189 P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:20.719177 887 tablet_service.cc:1467] Processing CreateTablet for tablet 6666922d677a4c678f80d7c6c47fd2a5 (DEFAULT_TABLE table=default.Table_93d706c664024860855d1c9d91992817 [id=6e014400813c4402891b32bcbe3555e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:20.719504 1194 ts_tablet_manager.cc:1428] T 107e6e9e2d84496f8fce92b9ba377189 P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.055s user 0.003s sys 0.000s
I20250113 02:51:20.718905 882 tablet_service.cc:1467] Processing CreateTablet for tablet e468b652f71f44e3945f8d919ac0e1a8 (DEFAULT_TABLE table=default.Table_c52b68d0d3d74003bf10bc8ff205d25b [id=caf9972c69f14619be22fb3a9e563b05]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:20.720657 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_547022f75adf480fb21bc92b5b1c0e2d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:20.720741 897 tablet_service.cc:1467] Processing CreateTablet for tablet 3b1910496e0849b193e390a5ef1a6b2b (DEFAULT_TABLE table=default.Table_2f3720d8d897466388c3f9dde157d701 [id=8af6fc6e393d4cd3a5eb0671346df592]), partition=RANGE (key) PARTITION UNBOUNDED
W20250113 02:51:20.721156 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_547022f75adf480fb21bc92b5b1c0e2d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:20.724232 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15a05a32febd443dbb519c186048428a" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca"
I20250113 02:51:20.724800 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_6185ed1a90a44576bc65ed60294c2537"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:20.725353 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6185ed1a90a44576bc65ed60294c2537 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:20.725495 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c7243d27a1c642269a3d0e67632e458c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:20.725323 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_1065dceda1dd4999a868ec77b48f6ab3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:20.726050 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1065dceda1dd4999a868ec77b48f6ab3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:20.727391 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_5aa8a7a4cb734b219053709e9191daad"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:20.727924 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5aa8a7a4cb734b219053709e9191daad in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:20.729008 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 377ebf66bbef40988043f6200ae0893c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:20.729698 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0fe5421dc3ea4be684cab084f13d8df0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:20.730276 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_6a3d6ef326e74c4ab0a5ed1e1bec2ecc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:20.730844 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6a3d6ef326e74c4ab0a5ed1e1bec2ecc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250113 02:51:20.732311 711 leader_election.cc:343] T 78689f7ed2a64405adb75f372465a792 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Not found: Tablet not found: 78689f7ed2a64405adb75f372465a792
I20250113 02:51:20.732391 1192 raft_consensus.cc:357] T c62a5e9e43d14b5c94eb87260528469d P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:20.733570 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_707144e534a1463986168c22b636e208"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:20.734050 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6666922d677a4c678f80d7c6c47fd2a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:20.734086 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet a97efc70946c4af487e33a85abf6b90c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:20 UTC) from {username='slave'} at 127.0.0.1:60230
W20250113 02:51:20.734149 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_707144e534a1463986168c22b636e208 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:20.734695 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0539cbbe47e64c039ad39c4b52edcc4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:20.734910 966 heartbeater.cc:346] Connected to a master server at 127.0.104.61:35271
I20250113 02:51:20.737635 1036 tablet_service.cc:1514] Processing DeleteTablet for tablet bcb59573e82743858bd39bde64627d6d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:20 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:20.738461 1061 raft_consensus.cc:2388] T 15a05a32febd443dbb519c186048428a P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 1186fa869eb14812832fa6562531f196 in current term 1: Already voted for candidate 4216a7f853d642b7bc3e5cff458f3aca in this term.
I20250113 02:51:20.735824 1062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68023c30a7f644eaa5b4ddd60d6c5ef5" candidate_uuid: "df69f48656b44bb890f76587ee69cc0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:20.741883 883 tablet_service.cc:1514] Processing DeleteTablet for tablet c215ffe61b9b426286cc5e808463078d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:20 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:20.742532 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b1910496e0849b193e390a5ef1a6b2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:20.745112 888 tablet_service.cc:1467] Processing CreateTablet for tablet f129bb3fbbfd492d98822be0a6626e44 (DEFAULT_TABLE table=default.Table_5182a1c5c7a2462ba930940ae04b505f [id=6666a614063942c99a78b95d35add037]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:20.747267 1196 ts_tablet_manager.cc:1918] T c62a5e9e43d14b5c94eb87260528469d P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250113 02:51:20.747769 852 leader_election.cc:343] T 68023c30a7f644eaa5b4ddd60d6c5ef5 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:20.749084 1027 tablet_service.cc:1467] Processing CreateTablet for tablet aa5c172a69bf4ca79999f4b58df08de8 (DEFAULT_TABLE table=default.Table_90fe60d176ce484680307442a262e8d5 [id=5dceb76e1dbe422ab98d4da5bf6c45a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:20.750056 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 99c8046f5d124ca49899df365114c69f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:20.750515 1194 tablet_bootstrap.cc:492] T 18e2d79d731142f89b9089869bc736e5 P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:20.752197 1192 raft_consensus.cc:383] T c62a5e9e43d14b5c94eb87260528469d P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:20.752458 1032 tablet_service.cc:1514] Processing DeleteTablet for tablet c71511a100f14322b5091a9804720fa0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:20 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:20.752619 1039 tablet_service.cc:1514] Processing DeleteTablet for tablet be80b631ccd747f1b9355a00787a7b55 with delete_type TABLET_DATA_DELETED (replaced by tablet e70658c90555463498d3fc2e2f13ea7f) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:20.753078 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa5c172a69bf4ca79999f4b58df08de8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:20.753576 1060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb183b4588e84212a3b4b096f430d47d" candidate_uuid: "df69f48656b44bb890f76587ee69cc0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:20.753815 1036 tablet_service.cc:1467] Processing CreateTablet for tablet 1d15c7a01f854f0aaabcd30ec3166e31 (DEFAULT_TABLE table=default.Table_ffb4a25d44a04760b8306ea2678f4b42 [id=fe31ebf2077b43a285155bef4c983d75]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:20.754324 1192 raft_consensus.cc:738] T c62a5e9e43d14b5c94eb87260528469d P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:20.755007 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e468b652f71f44e3945f8d919ac0e1a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:20.755415 1039 tablet_service.cc:1514] Processing DeleteTablet for tablet de4a58bf63054890808e2e2fe46e43d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:20 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:20.757443 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8b6ffe690fc459db78b2b3bc12671dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:20.765450 452 master_service.cc:946] ReplaceTablet: received request to replace tablet d5fbf97536a2472fa091168363bf9549 from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:20.768788 1060 raft_consensus.cc:2463] T fb183b4588e84212a3b4b096f430d47d P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df69f48656b44bb890f76587ee69cc0f in term 0.
I20250113 02:51:20.768808 1036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d15c7a01f854f0aaabcd30ec3166e31. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:20.770017 750 tablet_service.cc:1467] Processing CreateTablet for tablet 1d15c7a01f854f0aaabcd30ec3166e31 (DEFAULT_TABLE table=default.Table_ffb4a25d44a04760b8306ea2678f4b42 [id=fe31ebf2077b43a285155bef4c983d75]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:20.770160 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet fb183b4588e84212a3b4b096f430d47d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:20 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:20.772562 746 tablet_service.cc:1467] Processing CreateTablet for tablet 4b07a985897641c3ac48021c2a4daedb (DEFAULT_TABLE table=default.Table_d866aff056e246d88df242feb6d4e627 [id=fa29f9bbea034fafad86ad86aa5775eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:20.773049 883 tablet_service.cc:1467] Processing CreateTablet for tablet f56230a2c75d4763a54671c79358f4d9 (DEFAULT_TABLE table=default.Table_fb08fe0ed5cd4aaebce3169c264441cb [id=34b84bee300d4b30b1d426b9f26e6e4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:20.777493 1421 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6bbc3541604a455584bb32dfc2cc4efd: Bootstrap replayed 1/1 log segments. Stats: ops{read=18 overwritten=0 applied=17 ignored=0} inserts{seen=23 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20250113 02:51:20.778298 1421 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6bbc3541604a455584bb32dfc2cc4efd: Bootstrap complete.
I20250113 02:51:20.782511 452 master_service.cc:946] ReplaceTablet: received request to replace tablet ebc5f1ea1e1648d49904b17603d3cd2a from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:20.785198 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f56230a2c75d4763a54671c79358f4d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:20.786199 1311 raft_consensus.cc:491] T 8193357427f4491c8c6c1396ceeef2dc P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:20.788964 898 tablet_service.cc:1467] Processing CreateTablet for tablet 0fe5421dc3ea4be684cab084f13d8df0 (DEFAULT_TABLE table=default.Table_e2eebe9051194e27864922e074e6adbf [id=beb83fe520f94241949a46d436e6374f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:20.792438 1026 tablet_service.cc:1467] Processing CreateTablet for tablet 999fecb3fc65466cad21cc1ff5ef2d44 (DEFAULT_TABLE table=default.Table_991d8c72ce924153ad119a440b54fb1c [id=71e76dffaa6f4b0682422afed1749a64]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:20.792657 1039 tablet_service.cc:1467] Processing CreateTablet for tablet 0916b41c171a419791797559fbc2807d (DEFAULT_TABLE table=default.Table_8be7ebd7e1984b979fce6e7aac1f576e [id=8231428f86ba4a6388c55c3c06f6b194]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:20.788502 1192 consensus_queue.cc:260] T c62a5e9e43d14b5c94eb87260528469d P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:20.792006 1311 raft_consensus.cc:513] T 8193357427f4491c8c6c1396ceeef2dc P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:20.794418 1032 tablet_service.cc:1467] Processing CreateTablet for tablet 56b022c94847411785a7e9db3757a2d9 (DEFAULT_TABLE table=default.Table_6a9cc0903b674ac49fd6ffaa542a5595 [id=40ca872c045b4349b63ba9046e31ed13]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:20.796738 1031 tablet_service.cc:1467] Processing CreateTablet for tablet 42cf1e708644427f9d4dcd41ae2bcb37 (DEFAULT_TABLE table=default.Table_54963a1a7ff24d5a99644441a755f5de [id=acf16114671d4a3eb7ed4a0ffc984fef]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:20.797130 1029 tablet_service.cc:1467] Processing CreateTablet for tablet 4b07a985897641c3ac48021c2a4daedb (DEFAULT_TABLE table=default.Table_d866aff056e246d88df242feb6d4e627 [id=fa29f9bbea034fafad86ad86aa5775eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:20.797191 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_1be9cb7e8848435fa27f8b8eb7a77d9c" } new_table_name: "default.Table_6c7330704479422a817543c4cbbd70d3" modify_external_catalogs: true
I20250113 02:51:20.799178 1198 ts_tablet_manager.cc:1918] T ce92c2b45a274eb09e72a96b7ec133be P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:20.800074 454 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_5aa8a7a4cb734b219053709e9191daad" } new_table_name: "default.Table_ba72b5bfcbe6447196e4d8640ab0ff65" modify_external_catalogs: true
I20250113 02:51:20.800374 1311 leader_election.cc:290] T 8193357427f4491c8c6c1396ceeef2dc P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:20.801513 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 999fecb3fc65466cad21cc1ff5ef2d44. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:20.801745 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 42cf1e708644427f9d4dcd41ae2bcb37. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:20.805063 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b07a985897641c3ac48021c2a4daedb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:20.808442 1311 raft_consensus.cc:491] T c62a5e9e43d14b5c94eb87260528469d P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:20.808636 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 56b022c94847411785a7e9db3757a2d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:20.813673 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_67151d370cfd45ccaac504631aa0f1f9" } new_table_name: "default.Table_faf2fafcef874bf6a5798ee6b19ef704" modify_external_catalogs: true
I20250113 02:51:20.814031 774 raft_consensus.cc:2388] T 15a05a32febd443dbb519c186048428a P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 4216a7f853d642b7bc3e5cff458f3aca in current term 1: Already voted for candidate 1186fa869eb14812832fa6562531f196 in this term.
I20250113 02:51:20.813977 1383 leader_election.cc:290] T 15a05a32febd443dbb519c186048428a P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 election: Requested vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:20.814431 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa5c172a69bf4ca79999f4b58df08de8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:20.814711 1109 heartbeater.cc:346] Connected to a master server at 127.0.104.61:35271
I20250113 02:51:20.814815 747 tablet_service.cc:1514] Processing DeleteTablet for tablet be80b631ccd747f1b9355a00787a7b55 with delete_type TABLET_DATA_DELETED (replaced by tablet e70658c90555463498d3fc2e2f13ea7f) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:20.814883 1109 heartbeater.cc:502] Master 127.0.104.61:35271 was elected leader, sending a full tablet report...
I20250113 02:51:20.815012 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68023c30a7f644eaa5b4ddd60d6c5ef5" candidate_uuid: "df69f48656b44bb890f76587ee69cc0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
I20250113 02:51:20.815435 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb183b4588e84212a3b4b096f430d47d" candidate_uuid: "df69f48656b44bb890f76587ee69cc0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
I20250113 02:51:20.815654 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b07a985897641c3ac48021c2a4daedb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:20.816370 745 tablet_service.cc:1514] Processing DeleteTablet for tablet fb183b4588e84212a3b4b096f430d47d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:20 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:20.816582 711 leader_election.cc:304] T 78689f7ed2a64405adb75f372465a792 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:20.816951 1195 ts_tablet_manager.cc:1939] T c135506190f849008ce6542d569b402a P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:20.817052 1270 raft_consensus.cc:491] T 3d132c29cb2c46108e5877e38068ef17 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:20.817212 1270 raft_consensus.cc:513] T 3d132c29cb2c46108e5877e38068ef17 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:20.817827 1270 leader_election.cc:290] T 3d132c29cb2c46108e5877e38068ef17 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:20.818001 1270 raft_consensus.cc:2743] T 78689f7ed2a64405adb75f372465a792 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250113 02:51:20.816911 711 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:38467, user_credentials={real_user=slave}} blocked reactor thread for 94288.6us
W20250113 02:51:20.818675 711 leader_election.cc:343] T f0e7cc161f354f86b8c2fef303e87c5b P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Not found: Tablet not found: f0e7cc161f354f86b8c2fef303e87c5b
W20250113 02:51:20.818984 711 leader_election.cc:343] T 15a05a32febd443dbb519c186048428a P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Not found: Tablet not found: 15a05a32febd443dbb519c186048428a
W20250113 02:51:20.819257 711 leader_election.cc:343] T f0e7cc161f354f86b8c2fef303e87c5b P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Not found: Tablet not found: f0e7cc161f354f86b8c2fef303e87c5b
W20250113 02:51:20.819540 711 leader_election.cc:343] T ce92c2b45a274eb09e72a96b7ec133be P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Not found: Tablet not found: ce92c2b45a274eb09e72a96b7ec133be
I20250113 02:51:20.819674 711 leader_election.cc:304] T ce92c2b45a274eb09e72a96b7ec133be P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
W20250113 02:51:20.820025 711 leader_election.cc:343] T 15a05a32febd443dbb519c186048428a P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Not found: Tablet not found: 15a05a32febd443dbb519c186048428a
I20250113 02:51:20.820147 711 leader_election.cc:304] T 15a05a32febd443dbb519c186048428a P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:20.820309 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 212) took 1359 ms. Trace:
I20250113 02:51:20.820418 1195 rpcz_store.cc:276] 0113 02:51:19.460582 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:19.460637 (+ 55us) service_pool.cc:225] Handling call
0113 02:51:20.290406 (+829769us) ts_tablet_manager.cc:1145] Deleting tablet c135506190f849008ce6542d569b402a
0113 02:51:20.290415 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:20.299332 (+ 8917us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:20.617579 (+318247us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:20.820270 (+202691us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":829569}
I20250113 02:51:20.820859 1270 raft_consensus.cc:491] T ad9b95c5450a4c9c89a0930862348d49 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:20.820981 1270 raft_consensus.cc:513] T ad9b95c5450a4c9c89a0930862348d49 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:20.821297 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 230) took 1156 ms. Trace:
I20250113 02:51:20.821413 1195 rpcz_store.cc:276] 0113 02:51:19.665187 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:19.689838 (+ 24651us) service_pool.cc:225] Handling call
0113 02:51:20.821163 (+1131325us) ts_tablet_manager.cc:1145] Deleting tablet 258fad818b7b4aa4b337f9949e20aa46
0113 02:51:20.821173 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:20.821259 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1120316}
I20250113 02:51:20.821561 1270 leader_election.cc:290] T ad9b95c5450a4c9c89a0930862348d49 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:20.821696 1270 raft_consensus.cc:2743] T ce92c2b45a274eb09e72a96b7ec133be P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:20.821869 1270 raft_consensus.cc:2743] T 15a05a32febd443dbb519c186048428a P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250113 02:51:20.822058 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 231) took 1155 ms. Trace:
I20250113 02:51:20.822170 1195 rpcz_store.cc:276] 0113 02:51:19.666592 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:19.691793 (+ 25201us) service_pool.cc:225] Handling call
0113 02:51:20.821899 (+1130106us) ts_tablet_manager.cc:1145] Deleting tablet ebc5f1ea1e1648d49904b17603d3cd2a
0113 02:51:20.821908 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:20.822026 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1119702}
I20250113 02:51:20.822418 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet c135506190f849008ce6542d569b402a (table default.Table_5e0263e22f1e4d2b983ae6b6ef03fca5 [id=377f041bf95e41f3b7e8ea26f9795b6a]) successfully deleted
I20250113 02:51:20.822454 1198 log.cc:1198] T ce92c2b45a274eb09e72a96b7ec133be P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/ce92c2b45a274eb09e72a96b7ec133be
W20250113 02:51:20.822664 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 258fad818b7b4aa4b337f9949e20aa46 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 258fad818b7b4aa4b337f9949e20aa46 already in progress: creating tablet
I20250113 02:51:20.822852 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 234) took 1146 ms. Trace:
W20250113 02:51:20.822925 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet ebc5f1ea1e1648d49904b17603d3cd2a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ebc5f1ea1e1648d49904b17603d3cd2a already in progress: creating tablet
I20250113 02:51:20.823181 754 tablet_service.cc:1467] Processing CreateTablet for tablet 0916b41c171a419791797559fbc2807d (DEFAULT_TABLE table=default.Table_8be7ebd7e1984b979fce6e7aac1f576e [id=8231428f86ba4a6388c55c3c06f6b194]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:20.823537 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0916b41c171a419791797559fbc2807d. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:20.824373 451 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ff8ef6290698477391721fd65b9eeb28 consensus state has no leader
I20250113 02:51:20.824569 752 tablet_service.cc:1514] Processing DeleteTablet for tablet c71511a100f14322b5091a9804720fa0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:20 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:20.825925 738 tablet_service.cc:1514] Processing DeleteTablet for tablet de4a58bf63054890808e2e2fe46e43d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:20 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:20.826097 742 tablet_service.cc:1467] Processing CreateTablet for tablet 56b022c94847411785a7e9db3757a2d9 (DEFAULT_TABLE table=default.Table_6a9cc0903b674ac49fd6ffaa542a5595 [id=40ca872c045b4349b63ba9046e31ed13]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:20.826452 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 56b022c94847411785a7e9db3757a2d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:20.835469 755 tablet_service.cc:1467] Processing CreateTablet for tablet 999fecb3fc65466cad21cc1ff5ef2d44 (DEFAULT_TABLE table=default.Table_991d8c72ce924153ad119a440b54fb1c [id=71e76dffaa6f4b0682422afed1749a64]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:20.835915 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 999fecb3fc65466cad21cc1ff5ef2d44. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:20.836766 454 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d8bb60fdd06946739b75b255303d9d24 has no consensus state
I20250113 02:51:20.836930 741 tablet_service.cc:1467] Processing CreateTablet for tablet 42cf1e708644427f9d4dcd41ae2bcb37 (DEFAULT_TABLE table=default.Table_54963a1a7ff24d5a99644441a755f5de [id=acf16114671d4a3eb7ed4a0ffc984fef]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:20.837030 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8193357427f4491c8c6c1396ceeef2dc" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
I20250113 02:51:20.837299 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 42cf1e708644427f9d4dcd41ae2bcb37. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:20.844408 454 master_service.cc:946] ReplaceTablet: received request to replace tablet 2fb62cc83f9c4dcfa9aa01ae8b46c025 from {username='slave'} at 127.0.0.1:49258
W20250113 02:51:20.845062 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet efccefd6787943469131492741ee2bd5 consensus state has no leader
I20250113 02:51:20.848027 896 tablet_service.cc:1467] Processing CreateTablet for tablet 377ebf66bbef40988043f6200ae0893c (DEFAULT_TABLE table=default.Table_384366cb310c4e248cdc81ecede00d18 [id=8b35e389de9f4917904e87b43618737e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:20.848147 889 tablet_service.cc:1514] Processing DeleteTablet for tablet a97efc70946c4af487e33a85abf6b90c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:20 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:20.848110 895 tablet_service.cc:1467] Processing CreateTablet for tablet aa5c172a69bf4ca79999f4b58df08de8 (DEFAULT_TABLE table=default.Table_90fe60d176ce484680307442a262e8d5 [id=5dceb76e1dbe422ab98d4da5bf6c45a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:20.848379 889 tablet_service.cc:1514] Processing DeleteTablet for tablet fb183b4588e84212a3b4b096f430d47d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:20 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:20.848640 889 tablet_service.cc:1514] Processing DeleteTablet for tablet c71511a100f14322b5091a9804720fa0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:20 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:20.848853 889 tablet_service.cc:1514] Processing DeleteTablet for tablet de4a58bf63054890808e2e2fe46e43d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:20 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:20.849072 889 tablet_service.cc:1514] Processing DeleteTablet for tablet 53cc528d282c47788d5d558569f074c1 with delete_type TABLET_DATA_DELETED (replaced by tablet 7d3562a48c6e4badbbcf37640e91b5ba) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:20.849534 889 tablet_service.cc:1467] Processing CreateTablet for tablet 56b022c94847411785a7e9db3757a2d9 (DEFAULT_TABLE table=default.Table_6a9cc0903b674ac49fd6ffaa542a5595 [id=40ca872c045b4349b63ba9046e31ed13]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:20.850442 1439 raft_consensus.cc:491] T 2fb62cc83f9c4dcfa9aa01ae8b46c025 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:20.850898 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb183b4588e84212a3b4b096f430d47d" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:20.851122 919 raft_consensus.cc:2463] T fb183b4588e84212a3b4b096f430d47d P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 0.
I20250113 02:51:20.850632 1439 raft_consensus.cc:513] T 2fb62cc83f9c4dcfa9aa01ae8b46c025 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:20.851864 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d15c7a01f854f0aaabcd30ec3166e31. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:20.853142 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_e2eebe9051194e27864922e074e6adbf" } new_table_name: "default.Table_ca04ad8d0d8b40d1ba6b147731579526" modify_external_catalogs: true
I20250113 02:51:20.853228 892 tablet_service.cc:1467] Processing CreateTablet for tablet 0916b41c171a419791797559fbc2807d (DEFAULT_TABLE table=default.Table_8be7ebd7e1984b979fce6e7aac1f576e [id=8231428f86ba4a6388c55c3c06f6b194]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:20.854166 1439 leader_election.cc:290] T 2fb62cc83f9c4dcfa9aa01ae8b46c025 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:20.847967 852 leader_election.cc:304] T fb183b4588e84212a3b4b096f430d47d P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f; no voters:
W20250113 02:51:20.854594 851 leader_election.cc:343] T fb183b4588e84212a3b4b096f430d47d P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Illegal state: must be running to vote when last-logged opid is not known
W20250113 02:51:20.854635 852 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.3:42587, user_credentials={real_user=slave}} blocked reactor thread for 80119.3us
I20250113 02:51:20.823031 1195 rpcz_store.cc:276] 0113 02:51:19.675929 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:19.702214 (+ 26285us) service_pool.cc:225] Handling call
0113 02:51:20.822686 (+1120472us) ts_tablet_manager.cc:1145] Deleting tablet 601c71d5f7624fbb87b365d2e775d33e
0113 02:51:20.822695 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:20.822820 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1120271}
I20250113 02:51:20.856456 1195 tablet_replica.cc:331] T ad6b21ed2b824cb0837fc1d8a39d2472 P 1186fa869eb14812832fa6562531f196: stopping tablet replica
W20250113 02:51:20.856781 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 601c71d5f7624fbb87b365d2e775d33e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 601c71d5f7624fbb87b365d2e775d33e already in progress: creating tablet
W20250113 02:51:20.857407 851 leader_election.cc:343] T 68023c30a7f644eaa5b4ddd60d6c5ef5 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:20.857547 851 leader_election.cc:304] T 68023c30a7f644eaa5b4ddd60d6c5ef5 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df69f48656b44bb890f76587ee69cc0f; no voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca
I20250113 02:51:20.857829 1438 raft_consensus.cc:2743] T 68023c30a7f644eaa5b4ddd60d6c5ef5 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:20.857901 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0916b41c171a419791797559fbc2807d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:20.858177 1439 raft_consensus.cc:2798] T fb183b4588e84212a3b4b096f430d47d P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 02:51:20.858673 894 tablet_service.cc:1514] Processing DeleteTablet for tablet bcb59573e82743858bd39bde64627d6d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:20 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:20.859428 1439 raft_consensus.cc:491] T fb183b4588e84212a3b4b096f430d47d P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 02:51:20.859544 1439 raft_consensus.cc:3054] T fb183b4588e84212a3b4b096f430d47d P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:20.860688 1198 ts_tablet_manager.cc:1939] T ce92c2b45a274eb09e72a96b7ec133be P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
W20250113 02:51:20.862504 995 leader_election.cc:343] T 8193357427f4491c8c6c1396ceeef2dc P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:20.862478 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet ce92c2b45a274eb09e72a96b7ec133be (table default.Table_06f81a97cb014a3bad594e88c8cf4733 [id=7c2bc6d4232f4585b84d71e3e1f85794]) successfully deleted
I20250113 02:51:20.863499 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fb62cc83f9c4dcfa9aa01ae8b46c025" candidate_uuid: "df69f48656b44bb890f76587ee69cc0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
I20250113 02:51:20.864001 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad9b95c5450a4c9c89a0930862348d49" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:20.864396 1196 log.cc:1198] T c62a5e9e43d14b5c94eb87260528469d P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/c62a5e9e43d14b5c94eb87260528469d
I20250113 02:51:20.861229 1421 raft_consensus.cc:357] T 00000000000000000000000000000000 P 6bbc3541604a455584bb32dfc2cc4efd [term 3 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e13f5bcdcee34225a46c6d2651e5ede9" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44217 } } peers { permanent_uuid: "6bbc3541604a455584bb32dfc2cc4efd" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 35271 } } peers { permanent_uuid: "45e9d43ecac0401fb2c18fe5eaf089b5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42067 } }
I20250113 02:51:20.864595 1195 raft_consensus.cc:2238] T ad6b21ed2b824cb0837fc1d8a39d2472 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:20.864848 1195 raft_consensus.cc:2267] T ad6b21ed2b824cb0837fc1d8a39d2472 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:20.821702 1311 raft_consensus.cc:513] T c62a5e9e43d14b5c94eb87260528469d P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
W20250113 02:51:20.871518 998 leader_election.cc:343] T 15a05a32febd443dbb519c186048428a P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Not found: Tablet not found: 15a05a32febd443dbb519c186048428a
I20250113 02:51:20.871913 1368 raft_consensus.cc:695] T f0e7cc161f354f86b8c2fef303e87c5b P 1186fa869eb14812832fa6562531f196 [term 1 LEADER]: Becoming Leader. State: Replica: 1186fa869eb14812832fa6562531f196, State: Running, Role: LEADER
I20250113 02:51:20.873224 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_5cb0c9e84e72464c895d03166a1d18d7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:20.873904 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5cb0c9e84e72464c895d03166a1d18d7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:20.874235 1061 raft_consensus.cc:2463] T ad9b95c5450a4c9c89a0930862348d49 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1186fa869eb14812832fa6562531f196 in term 0.
I20250113 02:51:20.874900 1194 tablet_bootstrap.cc:654] T 18e2d79d731142f89b9089869bc736e5 P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:20.874933 1195 ts_tablet_manager.cc:1905] T ad6b21ed2b824cb0837fc1d8a39d2472 P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:20.875375 1191 ts_tablet_manager.cc:1397] T 8193357427f4491c8c6c1396ceeef2dc P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.199s user 0.010s sys 0.019s
I20250113 02:51:20.875571 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c62a5e9e43d14b5c94eb87260528469d" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
I20250113 02:51:20.875865 894 tablet_service.cc:1467] Processing CreateTablet for tablet 999fecb3fc65466cad21cc1ff5ef2d44 (DEFAULT_TABLE table=default.Table_991d8c72ce924153ad119a440b54fb1c [id=71e76dffaa6f4b0682422afed1749a64]), partition=RANGE (key) PARTITION UNBOUNDED
W20250113 02:51:20.876336 998 leader_election.cc:343] T 15a05a32febd443dbb519c186048428a P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Not found: Tablet not found: 15a05a32febd443dbb519c186048428a
W20250113 02:51:20.878036 711 leader_election.cc:343] T 3d132c29cb2c46108e5877e38068ef17 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Not found: Tablet not found: 3d132c29cb2c46108e5877e38068ef17
I20250113 02:51:20.878592 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad9b95c5450a4c9c89a0930862348d49" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:20.878839 919 raft_consensus.cc:2463] T ad9b95c5450a4c9c89a0930862348d49 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1186fa869eb14812832fa6562531f196 in term 0.
I20250113 02:51:20.879678 1060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fb62cc83f9c4dcfa9aa01ae8b46c025" candidate_uuid: "df69f48656b44bb890f76587ee69cc0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:20.880565 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_ff6df4afcf4d4f9a964d079ee24f02b5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:20.881350 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ff6df4afcf4d4f9a964d079ee24f02b5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:20.882712 893 tablet_service.cc:1514] Processing DeleteTablet for tablet 9041b28402664a08b8761fa155c9c229 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:20 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:20.883719 883 tablet_service.cc:1514] Processing DeleteTablet for tablet 107e6e9e2d84496f8fce92b9ba377189 with delete_type TABLET_DATA_DELETED (replaced by tablet 7ae2f30bf92742008e53b1ec94ebd0cf) from {username='slave'} at 127.0.0.1:37366
W20250113 02:51:20.884224 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0fe5421dc3ea4be684cab084f13d8df0 consensus state has no leader
I20250113 02:51:20.887463 1311 leader_election.cc:290] T c62a5e9e43d14b5c94eb87260528469d P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:20.887658 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_ac80d162769b4fc183613eb52f1bc4b7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:20.888275 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ac80d162769b4fc183613eb52f1bc4b7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250113 02:51:20.890894 995 leader_election.cc:343] T c62a5e9e43d14b5c94eb87260528469d P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:20.891964 1198 tablet_replica.cc:331] T 8193357427f4491c8c6c1396ceeef2dc P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:20.893201 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_f86569599f95484fb5cf0a5dccb6935c" } new_table_name: "default.Table_e03d954cc20048e4b0d598282689643d" modify_external_catalogs: true
I20250113 02:51:20.893451 998 leader_election.cc:304] T 15a05a32febd443dbb519c186048428a P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4216a7f853d642b7bc3e5cff458f3aca; no voters: 1186fa869eb14812832fa6562531f196, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:20.894148 1196 ts_tablet_manager.cc:1939] T c62a5e9e43d14b5c94eb87260528469d P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:20.895120 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet c62a5e9e43d14b5c94eb87260528469d (table default.Table_8aa63512c7bb4bdb86f37c9ee24df209 [id=aadd55280b654ceb96c290786a0bfce0]) successfully deleted
W20250113 02:51:20.896745 851 leader_election.cc:343] T 2fb62cc83f9c4dcfa9aa01ae8b46c025 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:20.898234 1421 raft_consensus.cc:738] T 00000000000000000000000000000000 P 6bbc3541604a455584bb32dfc2cc4efd [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bbc3541604a455584bb32dfc2cc4efd, State: Initialized, Role: FOLLOWER
W20250113 02:51:20.898474 998 leader_election.cc:343] T 8193357427f4491c8c6c1396ceeef2dc P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Not found: Tablet not found: 8193357427f4491c8c6c1396ceeef2dc
I20250113 02:51:20.899372 1421 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6bbc3541604a455584bb32dfc2cc4efd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 17, Last appended: 3.18, Last appended by leader: 18, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e13f5bcdcee34225a46c6d2651e5ede9" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44217 } } peers { permanent_uuid: "6bbc3541604a455584bb32dfc2cc4efd" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 35271 } } peers { permanent_uuid: "45e9d43ecac0401fb2c18fe5eaf089b5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42067 } }
I20250113 02:51:20.900374 446 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_45ad8711e23e44f0b7b83b7a6128c427" } modify_external_catalogs: true
I20250113 02:51:20.900563 446 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_45ad8711e23e44f0b7b83b7a6128c427" } modify_external_catalogs: true
I20250113 02:51:20.901882 449 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_6c7330704479422a817543c4cbbd70d3" } modify_external_catalogs: true
I20250113 02:51:20.902083 449 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_6c7330704479422a817543c4cbbd70d3" } modify_external_catalogs: true
W20250113 02:51:20.902475 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet a77c38a394114d6c8a8da99d7eef7669 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a77c38a394114d6c8a8da99d7eef7669 already in progress: creating tablet
I20250113 02:51:20.903342 899 tablet_service.cc:1467] Processing CreateTablet for tablet 42cf1e708644427f9d4dcd41ae2bcb37 (DEFAULT_TABLE table=default.Table_54963a1a7ff24d5a99644441a755f5de [id=acf16114671d4a3eb7ed4a0ffc984fef]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:20.903939 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f129bb3fbbfd492d98822be0a6626e44. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:20.903913 1198 raft_consensus.cc:2238] T 8193357427f4491c8c6c1396ceeef2dc P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:20.904639 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 377ebf66bbef40988043f6200ae0893c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:20.905718 1311 raft_consensus.cc:2743] T 15a05a32febd443dbb519c186048428a P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250113 02:51:20.910629 1439 raft_consensus.cc:513] T fb183b4588e84212a3b4b096f430d47d P df69f48656b44bb890f76587ee69cc0f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:20.912102 1439 leader_election.cc:290] T fb183b4588e84212a3b4b096f430d47d P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 election: Requested vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:20.912318 448 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_266561dfe36245ecb8230fdf1904cb02" } modify_external_catalogs: true
I20250113 02:51:20.912544 448 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_266561dfe36245ecb8230fdf1904cb02" } modify_external_catalogs: true
I20250113 02:51:20.913213 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_616eda5ca6ce4155b20351b6b8b000f5" } modify_external_catalogs: true
I20250113 02:51:20.913422 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_616eda5ca6ce4155b20351b6b8b000f5" } modify_external_catalogs: true
I20250113 02:51:20.913985 455 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_ec91552e358547e99d908cef4bcb7c36" } modify_external_catalogs: true
I20250113 02:51:20.914191 455 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_ec91552e358547e99d908cef4bcb7c36" } modify_external_catalogs: true
I20250113 02:51:20.920740 1368 consensus_queue.cc:237] T f0e7cc161f354f86b8c2fef303e87c5b P 1186fa869eb14812832fa6562531f196 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:20.921648 887 tablet_service.cc:1467] Processing CreateTablet for tablet 4b07a985897641c3ac48021c2a4daedb (DEFAULT_TABLE table=default.Table_d866aff056e246d88df242feb6d4e627 [id=fa29f9bbea034fafad86ad86aa5775eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:20.921869 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb183b4588e84212a3b4b096f430d47d" candidate_uuid: "df69f48656b44bb890f76587ee69cc0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196"
I20250113 02:51:20.920070 711 leader_election.cc:304] T ad9b95c5450a4c9c89a0930862348d49 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1186fa869eb14812832fa6562531f196, df69f48656b44bb890f76587ee69cc0f; no voters:
W20250113 02:51:20.923297 711 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:38467, user_credentials={real_user=slave}} blocked reactor thread for 43580.9us
I20250113 02:51:20.923542 1441 raft_consensus.cc:2798] T ad9b95c5450a4c9c89a0930862348d49 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 02:51:20.923635 1441 raft_consensus.cc:491] T ad9b95c5450a4c9c89a0930862348d49 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 02:51:20.923741 1441 raft_consensus.cc:3054] T ad9b95c5450a4c9c89a0930862348d49 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Advancing to term 1
W20250113 02:51:20.923915 851 leader_election.cc:343] T fb183b4588e84212a3b4b096f430d47d P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:20.924290 897 tablet_service.cc:1514] Processing DeleteTablet for tablet be80b631ccd747f1b9355a00787a7b55 with delete_type TABLET_DATA_DELETED (replaced by tablet e70658c90555463498d3fc2e2f13ea7f) from {username='slave'} at 127.0.0.1:37366
W20250113 02:51:20.928498 852 leader_election.cc:343] T 2fb62cc83f9c4dcfa9aa01ae8b46c025 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:20.928546 1441 raft_consensus.cc:513] T ad9b95c5450a4c9c89a0930862348d49 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:20.929491 1191 raft_consensus.cc:357] T 8193357427f4491c8c6c1396ceeef2dc P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:20.930128 1191 raft_consensus.cc:383] T 8193357427f4491c8c6c1396ceeef2dc P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:20.930229 1191 raft_consensus.cc:738] T 8193357427f4491c8c6c1396ceeef2dc P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:20.932838 1191 consensus_queue.cc:260] T 8193357427f4491c8c6c1396ceeef2dc P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:20.933411 1196 tablet_replica.cc:331] T 9b5dc06ba3ea4056822ca3e4b1249587 P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:20.933528 1191 ts_tablet_manager.cc:1428] T 8193357427f4491c8c6c1396ceeef2dc P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.058s user 0.001s sys 0.001s
I20250113 02:51:20.934088 1191 tablet_bootstrap.cc:492] T 0c55662a71444a3683a825bcec6c5b68 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:20.934660 852 leader_election.cc:304] T 2fb62cc83f9c4dcfa9aa01ae8b46c025 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df69f48656b44bb890f76587ee69cc0f; no voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca
I20250113 02:51:20.934667 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0fe5421dc3ea4be684cab084f13d8df0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:20.936971 1441 leader_election.cc:290] T ad9b95c5450a4c9c89a0930862348d49 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 election: Requested vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:20.939059 1191 tablet_bootstrap.cc:654] T 0c55662a71444a3683a825bcec6c5b68 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:20.934253 886 tablet_service.cc:1467] Processing CreateTablet for tablet 1d15c7a01f854f0aaabcd30ec3166e31 (DEFAULT_TABLE table=default.Table_ffb4a25d44a04760b8306ea2678f4b42 [id=fe31ebf2077b43a285155bef4c983d75]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:20.916030 998 leader_election.cc:304] T 8193357427f4491c8c6c1396ceeef2dc P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4216a7f853d642b7bc3e5cff458f3aca; no voters: 1186fa869eb14812832fa6562531f196, df69f48656b44bb890f76587ee69cc0f
W20250113 02:51:20.943542 852 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.3:42587, user_credentials={real_user=slave}} blocked reactor thread for 44002.5us
I20250113 02:51:20.944025 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa5c172a69bf4ca79999f4b58df08de8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:20.947263 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet c62a5e9e43d14b5c94eb87260528469d with delete_type TABLET_DATA_DELETED (replaced by tablet 6c8ca30720664f3a819c4bff2493934d) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:20.947451 1192 ts_tablet_manager.cc:1428] T c62a5e9e43d14b5c94eb87260528469d P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.236s user 0.024s sys 0.009s
I20250113 02:51:20.947576 1060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb183b4588e84212a3b4b096f430d47d" candidate_uuid: "df69f48656b44bb890f76587ee69cc0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca"
W20250113 02:51:20.943261 998 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:38467, user_credentials={real_user=slave}} blocked reactor thread for 47420.5us
I20250113 02:51:20.949607 1439 raft_consensus.cc:2743] T 2fb62cc83f9c4dcfa9aa01ae8b46c025 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:20.949944 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 56b022c94847411785a7e9db3757a2d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:20.950919 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 42cf1e708644427f9d4dcd41ae2bcb37. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:20.951730 1195 ts_tablet_manager.cc:1918] T ad6b21ed2b824cb0837fc1d8a39d2472 P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:20.951903 1195 log.cc:1198] T ad6b21ed2b824cb0837fc1d8a39d2472 P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/ad6b21ed2b824cb0837fc1d8a39d2472
I20250113 02:51:20.952316 1195 ts_tablet_manager.cc:1939] T ad6b21ed2b824cb0837fc1d8a39d2472 P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:20.952710 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 235) took 1266 ms. Trace:
I20250113 02:51:20.952834 1195 rpcz_store.cc:276] 0113 02:51:19.686604 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:19.702412 (+ 15808us) service_pool.cc:225] Handling call
0113 02:51:20.856388 (+1153976us) ts_tablet_manager.cc:1145] Deleting tablet ad6b21ed2b824cb0837fc1d8a39d2472
0113 02:51:20.856401 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:20.928780 (+ 72379us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:20.951568 (+ 22788us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:20.952651 (+ 1083us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1145555}
I20250113 02:51:20.953415 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet ad6b21ed2b824cb0837fc1d8a39d2472 (table default.Table_449479ad3dbb4c1aaa5cd954816b9935 [id=61e1ae59b7db4fd0a33fe3f73a712028]) successfully deleted
I20250113 02:51:20.958984 1196 raft_consensus.cc:2238] T 9b5dc06ba3ea4056822ca3e4b1249587 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:20.959383 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad9b95c5450a4c9c89a0930862348d49" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f"
I20250113 02:51:20.959604 919 raft_consensus.cc:3054] T ad9b95c5450a4c9c89a0930862348d49 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:20.960117 1446 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bbc3541604a455584bb32dfc2cc4efd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e13f5bcdcee34225a46c6d2651e5ede9" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44217 } } peers { permanent_uuid: "6bbc3541604a455584bb32dfc2cc4efd" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 35271 } } peers { permanent_uuid: "45e9d43ecac0401fb2c18fe5eaf089b5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42067 } } }
I20250113 02:51:20.961032 1446 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bbc3541604a455584bb32dfc2cc4efd [sys.catalog]: This master's current role is: FOLLOWER
I20250113 02:51:20.962154 1192 tablet_bootstrap.cc:492] T 9b5dc06ba3ea4056822ca3e4b1249587 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:20.962863 1421 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6bbc3541604a455584bb32dfc2cc4efd [sys.catalog]: configured and running, proceeding with master startup.
I20250113 02:51:20.968181 1060 raft_consensus.cc:3054] T fb183b4588e84212a3b4b096f430d47d P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:20.969676 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 999fecb3fc65466cad21cc1ff5ef2d44. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:20.969678 1198 raft_consensus.cc:2267] T 8193357427f4491c8c6c1396ceeef2dc P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:20.970795 1191 tablet_bootstrap.cc:492] T 0c55662a71444a3683a825bcec6c5b68 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:20.971011 1191 ts_tablet_manager.cc:1397] T 0c55662a71444a3683a825bcec6c5b68 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.037s user 0.001s sys 0.006s
I20250113 02:51:20.972254 1191 raft_consensus.cc:357] T 0c55662a71444a3683a825bcec6c5b68 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:20.972579 1191 raft_consensus.cc:383] T 0c55662a71444a3683a825bcec6c5b68 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:20.972736 1191 raft_consensus.cc:738] T 0c55662a71444a3683a825bcec6c5b68 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:20.973098 1191 consensus_queue.cc:260] T 0c55662a71444a3683a825bcec6c5b68 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:20.973927 1191 ts_tablet_manager.cc:1428] T 0c55662a71444a3683a825bcec6c5b68 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250113 02:51:20.974766 1191 tablet_bootstrap.cc:492] T 3455f010e51b4db1a9804a57fd1a4fb8 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:20.976133 1442 raft_consensus.cc:2701] T 8193357427f4491c8c6c1396ceeef2dc P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Received pre-election callback for term 1 while not running: Illegal state: RaftConsensus is not running: State = Stopped
I20250113 02:51:20.977578 998 leader_election.cc:304] T fb183b4588e84212a3b4b096f430d47d P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f; no voters: 1186fa869eb14812832fa6562531f196
I20250113 02:51:20.978008 1191 tablet_bootstrap.cc:654] T 3455f010e51b4db1a9804a57fd1a4fb8 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:20.978554 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d15c7a01f854f0aaabcd30ec3166e31. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:20.981582 709 leader_election.cc:343] T 3d132c29cb2c46108e5877e38068ef17 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Not found: Tablet not found: 3d132c29cb2c46108e5877e38068ef17
I20250113 02:51:20.981735 709 leader_election.cc:304] T 3d132c29cb2c46108e5877e38068ef17 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:20.982023 1368 raft_consensus.cc:2743] T 3d132c29cb2c46108e5877e38068ef17 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:20.984802 1198 ts_tablet_manager.cc:1905] T 8193357427f4491c8c6c1396ceeef2dc P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250113 02:51:20.987555 998 leader_election.cc:343] T c62a5e9e43d14b5c94eb87260528469d P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Not found: Tablet not found: c62a5e9e43d14b5c94eb87260528469d
I20250113 02:51:20.987740 998 leader_election.cc:304] T c62a5e9e43d14b5c94eb87260528469d P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4216a7f853d642b7bc3e5cff458f3aca; no voters: 1186fa869eb14812832fa6562531f196, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:20.988242 1311 raft_consensus.cc:2743] T c62a5e9e43d14b5c94eb87260528469d P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:20.988190 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad9b95c5450a4c9c89a0930862348d49" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca"
I20250113 02:51:20.988469 1061 raft_consensus.cc:3054] T ad9b95c5450a4c9c89a0930862348d49 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:20.991604 1196 raft_consensus.cc:2267] T 9b5dc06ba3ea4056822ca3e4b1249587 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:20.992239 1192 tablet_bootstrap.cc:654] T 9b5dc06ba3ea4056822ca3e4b1249587 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:20.993252 1191 tablet_bootstrap.cc:492] T 3455f010e51b4db1a9804a57fd1a4fb8 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:20.993491 1191 ts_tablet_manager.cc:1397] T 3455f010e51b4db1a9804a57fd1a4fb8 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.019s user 0.000s sys 0.005s
I20250113 02:51:20.994892 1191 raft_consensus.cc:357] T 3455f010e51b4db1a9804a57fd1a4fb8 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:20.995193 1191 raft_consensus.cc:383] T 3455f010e51b4db1a9804a57fd1a4fb8 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:20.995306 1191 raft_consensus.cc:738] T 3455f010e51b4db1a9804a57fd1a4fb8 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:20.995648 1191 consensus_queue.cc:260] T 3455f010e51b4db1a9804a57fd1a4fb8 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:20.996132 1439 raft_consensus.cc:491] T 53cc528d282c47788d5d558569f074c1 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:20.996580 1191 ts_tablet_manager.cc:1428] T 3455f010e51b4db1a9804a57fd1a4fb8 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250113 02:51:20.996815 1439 raft_consensus.cc:513] T 53cc528d282c47788d5d558569f074c1 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:20.997043 1191 tablet_bootstrap.cc:492] T c62a5e9e43d14b5c94eb87260528469d P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:20.998353 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b07a985897641c3ac48021c2a4daedb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:21.000375 1062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53cc528d282c47788d5d558569f074c1" candidate_uuid: "df69f48656b44bb890f76587ee69cc0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:21.005077 1345 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 6bbc3541604a455584bb32dfc2cc4efd [term 3 FOLLOWER]: Advancing to term 4
I20250113 02:51:21.014549 1191 tablet_bootstrap.cc:654] T c62a5e9e43d14b5c94eb87260528469d P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:21.016212 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_3bb52edb283b4667b984b8e0370430a5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:21.016703 742 tablet_service.cc:1467] Processing CreateTablet for tablet fb18c16fb3ba44ec8a190b88d3bbd1b1 (DEFAULT_TABLE table=default.Table_547022f75adf480fb21bc92b5b1c0e2d [id=75ca7cbd7b9548d784f9a77d0f0cb64f]), partition=RANGE (key) PARTITION UNBOUNDED
W20250113 02:51:21.016960 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3bb52edb283b4667b984b8e0370430a5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:21.017203 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb18c16fb3ba44ec8a190b88d3bbd1b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:21.019958 1025 tablet_service.cc:1467] Processing CreateTablet for tablet a92f9b1c9c1a4cbdbf0cc7b2a3f2275b (DEFAULT_TABLE table=default.Table_6185ed1a90a44576bc65ed60294c2537 [id=5634c185da26460f93cde9c65fbdb24a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:21.021842 755 tablet_service.cc:1467] Processing CreateTablet for tablet eebad045e7bb43fa9ca9c9af8b3b7ea5 (DEFAULT_TABLE table=default.Table_abd20980e42242719c1529668d68128b [id=043d63161d5e4454af0782a50d64d256]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:21.022325 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eebad045e7bb43fa9ca9c9af8b3b7ea5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:21.022719 1191 tablet_bootstrap.cc:492] T c62a5e9e43d14b5c94eb87260528469d P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:21.022873 1191 ts_tablet_manager.cc:1397] T c62a5e9e43d14b5c94eb87260528469d P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.026s user 0.002s sys 0.002s
I20250113 02:51:21.024276 1191 raft_consensus.cc:357] T c62a5e9e43d14b5c94eb87260528469d P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:21.024497 1191 raft_consensus.cc:383] T c62a5e9e43d14b5c94eb87260528469d P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:21.024379 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_deb7c4bd8a954e5081cdbee12274dbcf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:21.024591 754 tablet_service.cc:1467] Processing CreateTablet for tablet a92f9b1c9c1a4cbdbf0cc7b2a3f2275b (DEFAULT_TABLE table=default.Table_6185ed1a90a44576bc65ed60294c2537 [id=5634c185da26460f93cde9c65fbdb24a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:21.025019 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a92f9b1c9c1a4cbdbf0cc7b2a3f2275b. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:21.024993 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_deb7c4bd8a954e5081cdbee12274dbcf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:21.026608 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_dd19b1ec1af841a7a475c3ace96712a6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:21.027122 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dd19b1ec1af841a7a475c3ace96712a6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:21.028317 741 tablet_service.cc:1467] Processing CreateTablet for tablet e70658c90555463498d3fc2e2f13ea7f (DEFAULT_TABLE table=default.Table_2565c0551f9943a084ace8f7e463e59d [id=fe6f8d302c3e489d8bb832503fd05642]), partition=RANGE (key) PARTITION UNBOUNDED
W20250113 02:51:21.028671 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d304b65eba8c40c6922333506a60540c consensus state has no leader
I20250113 02:51:21.028770 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e70658c90555463498d3fc2e2f13ea7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:21.029006 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_d5c2b805dfbf4227bb1ac190525b7dee"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:21.029521 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d5c2b805dfbf4227bb1ac190525b7dee in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:21.037088 1109 heartbeater.cc:502] Master 127.0.104.61:35271 was elected leader, sending a full tablet report...
I20250113 02:51:21.039052 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a92f9b1c9c1a4cbdbf0cc7b2a3f2275b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:21.042520 750 tablet_service.cc:1467] Processing CreateTablet for tablet 3025935f9743442eb3e6f2db22eb52ad (DEFAULT_TABLE table=default.Table_ccebddd68b0448a49d408b9922a78a4d [id=c0b628d230fd4eae83a20530cd998993]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:21.043503 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3025935f9743442eb3e6f2db22eb52ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:21.054168 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 232) took 1380 ms. Trace:
I20250113 02:51:21.054340 1195 rpcz_store.cc:276] 0113 02:51:19.673953 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:19.698561 (+ 24608us) service_pool.cc:225] Handling call
0113 02:51:20.954266 (+1255705us) ts_tablet_manager.cc:1145] Deleting tablet ee7582ee04494c0db1179359bcb311c8
0113 02:51:20.954282 (+ 16us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:21.054092 (+ 99810us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1229917}
W20250113 02:51:21.054977 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet ee7582ee04494c0db1179359bcb311c8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ee7582ee04494c0db1179359bcb311c8 already in progress: creating tablet
I20250113 02:51:21.055637 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 233) took 1379 ms. Trace:
I20250113 02:51:21.057042 1368 raft_consensus.cc:491] T 8193357427f4491c8c6c1396ceeef2dc P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:21.057207 1368 raft_consensus.cc:513] T 8193357427f4491c8c6c1396ceeef2dc P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:21.057987 1368 leader_election.cc:290] T 8193357427f4491c8c6c1396ceeef2dc P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:21.059463 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53cc528d282c47788d5d558569f074c1" candidate_uuid: "df69f48656b44bb890f76587ee69cc0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
I20250113 02:51:21.060384 1060 raft_consensus.cc:2463] T fb183b4588e84212a3b4b096f430d47d P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df69f48656b44bb890f76587ee69cc0f in term 1.
W20250113 02:51:21.062139 852 leader_election.cc:343] T 53cc528d282c47788d5d558569f074c1 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:21.063498 748 tablet_service.cc:1467] Processing CreateTablet for tablet 33c665e149dd48e8868acd935de581bf (DEFAULT_TABLE table=default.Table_ec91552e358547e99d908cef4bcb7c36 [id=0ebda43313f249d6b35f5beda3d5fc0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:21.063876 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 33c665e149dd48e8868acd935de581bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:21.067472 1196 ts_tablet_manager.cc:1905] T 9b5dc06ba3ea4056822ca3e4b1249587 P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:21.060621 1035 tablet_service.cc:1467] Processing CreateTablet for tablet eebad045e7bb43fa9ca9c9af8b3b7ea5 (DEFAULT_TABLE table=default.Table_abd20980e42242719c1529668d68128b [id=043d63161d5e4454af0782a50d64d256]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:21.062620 1192 tablet_bootstrap.cc:492] T 9b5dc06ba3ea4056822ca3e4b1249587 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:21.066087 1061 raft_consensus.cc:2463] T ad9b95c5450a4c9c89a0930862348d49 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1186fa869eb14812832fa6562531f196 in term 1.
I20250113 02:51:21.066895 1031 tablet_service.cc:1467] Processing CreateTablet for tablet 3025935f9743442eb3e6f2db22eb52ad (DEFAULT_TABLE table=default.Table_ccebddd68b0448a49d408b9922a78a4d [id=c0b628d230fd4eae83a20530cd998993]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:21.067109 1062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8193357427f4491c8c6c1396ceeef2dc" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:21.070048 1033 tablet_service.cc:1467] Processing CreateTablet for tablet fb18c16fb3ba44ec8a190b88d3bbd1b1 (DEFAULT_TABLE table=default.Table_547022f75adf480fb21bc92b5b1c0e2d [id=75ca7cbd7b9548d784f9a77d0f0cb64f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:21.063372 1023 tablet_service.cc:1467] Processing CreateTablet for tablet e70658c90555463498d3fc2e2f13ea7f (DEFAULT_TABLE table=default.Table_2565c0551f9943a084ace8f7e463e59d [id=fe6f8d302c3e489d8bb832503fd05642]), partition=RANGE (key) PARTITION UNBOUNDED
W20250113 02:51:21.071759 709 leader_election.cc:343] T 8193357427f4491c8c6c1396ceeef2dc P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: Tablet not RUNNING: STOPPED
I20250113 02:51:21.072074 1027 tablet_service.cc:1467] Processing CreateTablet for tablet 33c665e149dd48e8868acd935de581bf (DEFAULT_TABLE table=default.Table_ec91552e358547e99d908cef4bcb7c36 [id=0ebda43313f249d6b35f5beda3d5fc0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:21.073201 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0916b41c171a419791797559fbc2807d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:21.073426 1439 leader_election.cc:290] T 53cc528d282c47788d5d558569f074c1 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587), 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
I20250113 02:51:21.079851 746 tablet_service.cc:1467] Processing CreateTablet for tablet 7527e6e0b65e4a43b7f48899a34cdf7f (DEFAULT_TABLE table=default.Table_016e5998c5ef4112908e6917ccd57d15 [id=b7846a8ebee84005b1b5d259560a26bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:21.080232 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7527e6e0b65e4a43b7f48899a34cdf7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:21.080960 1192 ts_tablet_manager.cc:1397] T 9b5dc06ba3ea4056822ca3e4b1249587 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.124s user 0.001s sys 0.016s
I20250113 02:51:21.081225 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eebad045e7bb43fa9ca9c9af8b3b7ea5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:21.083107 709 leader_election.cc:304] T ad9b95c5450a4c9c89a0930862348d49 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters:
I20250113 02:51:21.083402 1368 raft_consensus.cc:2798] T ad9b95c5450a4c9c89a0930862348d49 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Leader election won for term 1
I20250113 02:51:21.083564 1368 raft_consensus.cc:695] T ad9b95c5450a4c9c89a0930862348d49 P 1186fa869eb14812832fa6562531f196 [term 1 LEADER]: Becoming Leader. State: Replica: 1186fa869eb14812832fa6562531f196, State: Running, Role: LEADER
W20250113 02:51:21.083570 851 leader_election.cc:343] T 53cc528d282c47788d5d558569f074c1 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:21.083810 851 leader_election.cc:304] T 53cc528d282c47788d5d558569f074c1 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df69f48656b44bb890f76587ee69cc0f; no voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca
I20250113 02:51:21.083859 1368 consensus_queue.cc:237] T ad9b95c5450a4c9c89a0930862348d49 P 1186fa869eb14812832fa6562531f196 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:21.084964 895 tablet_service.cc:1467] Processing CreateTablet for tablet fb18c16fb3ba44ec8a190b88d3bbd1b1 (DEFAULT_TABLE table=default.Table_547022f75adf480fb21bc92b5b1c0e2d [id=75ca7cbd7b9548d784f9a77d0f0cb64f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:21.085863 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 33c665e149dd48e8868acd935de581bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:21.089035 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 33c665e149dd48e8868acd935de581bf
I20250113 02:51:21.090675 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet c94eedb5dce44e6d896192c93127903a
I20250113 02:51:21.091666 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 64cda8796dff46ce95c37439395c77ec
I20250113 02:51:21.092537 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet b2a35fd891c645d68d79c61a2c832ba7
I20250113 02:51:21.095269 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e70658c90555463498d3fc2e2f13ea7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:21.095165 1192 raft_consensus.cc:357] T 9b5dc06ba3ea4056822ca3e4b1249587 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:21.095409 1192 raft_consensus.cc:383] T 9b5dc06ba3ea4056822ca3e4b1249587 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:21.096431 1192 raft_consensus.cc:738] T 9b5dc06ba3ea4056822ca3e4b1249587 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:21.096704 1192 consensus_queue.cc:260] T 9b5dc06ba3ea4056822ca3e4b1249587 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:21.097904 1192 ts_tablet_manager.cc:1428] T 9b5dc06ba3ea4056822ca3e4b1249587 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.016s user 0.000s sys 0.001s
I20250113 02:51:21.098232 754 tablet_service.cc:1514] Processing DeleteTablet for tablet 33c665e149dd48e8868acd935de581bf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:20 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:21.098479 746 tablet_service.cc:1514] Processing DeleteTablet for tablet c94eedb5dce44e6d896192c93127903a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:20 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:21.101280 884 tablet_service.cc:1467] Processing CreateTablet for tablet a92f9b1c9c1a4cbdbf0cc7b2a3f2275b (DEFAULT_TABLE table=default.Table_6185ed1a90a44576bc65ed60294c2537 [id=5634c185da26460f93cde9c65fbdb24a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:21.102530 1192 tablet_bootstrap.cc:492] T 2fb62cc83f9c4dcfa9aa01ae8b46c025 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:21.102574 1042 tablet_service.cc:1514] Processing DeleteTablet for tablet 64cda8796dff46ce95c37439395c77ec with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:20 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:21.055835 1195 rpcz_store.cc:276] 0113 02:51:19.675820 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:19.700754 (+ 24934us) service_pool.cc:225] Handling call
0113 02:51:21.055386 (+1354632us) ts_tablet_manager.cc:1145] Deleting tablet 840773ee80c7447097ea000b106fc89d
0113 02:51:21.055398 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:21.055536 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1324110}
W20250113 02:51:21.110942 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 840773ee80c7447097ea000b106fc89d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 840773ee80c7447097ea000b106fc89d already in progress: creating tablet
I20250113 02:51:21.112100 852 leader_election.cc:304] T fb183b4588e84212a3b4b096f430d47d P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f; no voters: 1186fa869eb14812832fa6562531f196
I20250113 02:51:21.112100 882 tablet_service.cc:1467] Processing CreateTablet for tablet eebad045e7bb43fa9ca9c9af8b3b7ea5 (DEFAULT_TABLE table=default.Table_abd20980e42242719c1529668d68128b [id=043d63161d5e4454af0782a50d64d256]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:21.112534 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb18c16fb3ba44ec8a190b88d3bbd1b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:21.112152 919 raft_consensus.cc:2463] T ad9b95c5450a4c9c89a0930862348d49 P df69f48656b44bb890f76587ee69cc0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1186fa869eb14812832fa6562531f196 in term 1.
I20250113 02:51:21.114118 1442 raft_consensus.cc:2758] T fb183b4588e84212a3b4b096f430d47d P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20250113 02:51:21.114173 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet b2a35fd891c645d68d79c61a2c832ba7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:20 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:21.114540 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3025935f9743442eb3e6f2db22eb52ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:21.116281 898 tablet_service.cc:1467] Processing CreateTablet for tablet 7527e6e0b65e4a43b7f48899a34cdf7f (DEFAULT_TABLE table=default.Table_016e5998c5ef4112908e6917ccd57d15 [id=b7846a8ebee84005b1b5d259560a26bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:21.117053 1037 tablet_service.cc:1514] Processing DeleteTablet for tablet c94eedb5dce44e6d896192c93127903a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:20 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:21.113766 896 tablet_service.cc:1467] Processing CreateTablet for tablet 3025935f9743442eb3e6f2db22eb52ad (DEFAULT_TABLE table=default.Table_ccebddd68b0448a49d408b9922a78a4d [id=c0b628d230fd4eae83a20530cd998993]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:21.120158 1438 raft_consensus.cc:2798] T fb183b4588e84212a3b4b096f430d47d P df69f48656b44bb890f76587ee69cc0f [term 1 FOLLOWER]: Leader election won for term 1
I20250113 02:51:21.122116 1194 tablet_bootstrap.cc:492] T 18e2d79d731142f89b9089869bc736e5 P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:21.123749 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a92f9b1c9c1a4cbdbf0cc7b2a3f2275b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:21.124748 897 tablet_service.cc:1514] Processing DeleteTablet for tablet 33c665e149dd48e8868acd935de581bf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:20 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:21.125029 1439 raft_consensus.cc:2743] T 53cc528d282c47788d5d558569f074c1 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:21.125336 1194 ts_tablet_manager.cc:1397] T 18e2d79d731142f89b9089869bc736e5 P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.375s user 0.018s sys 0.027s
I20250113 02:51:21.126039 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3025935f9743442eb3e6f2db22eb52ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:21.126438 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet 33c665e149dd48e8868acd935de581bf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:20 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:21.126631 1441 raft_consensus.cc:491] T 3455f010e51b4db1a9804a57fd1a4fb8 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:21.127139 881 tablet_service.cc:1467] Processing CreateTablet for tablet 33c665e149dd48e8868acd935de581bf (DEFAULT_TABLE table=default.Table_ec91552e358547e99d908cef4bcb7c36 [id=0ebda43313f249d6b35f5beda3d5fc0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:21.131057 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eebad045e7bb43fa9ca9c9af8b3b7ea5. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:21.131840 440 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 7d3562a48c6e4badbbcf37640e91b5ba (table default.Table_06f81a97cb014a3bad594e88c8cf4733 [id=7c2bc6d4232f4585b84d71e3e1f85794], current schema version=3) failed: Not found: failed to reset TS proxy: No leader found: Tablet 7d3562a48c6e4badbbcf37640e91b5ba consensus state has no leader
I20250113 02:51:21.137588 891 tablet_service.cc:1514] Processing DeleteTablet for tablet 64cda8796dff46ce95c37439395c77ec with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:20 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:21.137713 885 tablet_service.cc:1514] Processing DeleteTablet for tablet b2a35fd891c645d68d79c61a2c832ba7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:20 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:21.140806 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb18c16fb3ba44ec8a190b88d3bbd1b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:21.141218 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 33c665e149dd48e8868acd935de581bf. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:21.141187 711 leader_election.cc:343] T 8193357427f4491c8c6c1396ceeef2dc P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Not found: Tablet not found: 8193357427f4491c8c6c1396ceeef2dc
I20250113 02:51:21.141331 711 leader_election.cc:304] T 8193357427f4491c8c6c1396ceeef2dc P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:21.141642 1383 raft_consensus.cc:2743] T 8193357427f4491c8c6c1396ceeef2dc P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:21.143726 1441 raft_consensus.cc:513] T 3455f010e51b4db1a9804a57fd1a4fb8 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:21.143893 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7527e6e0b65e4a43b7f48899a34cdf7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:21.143862 1194 raft_consensus.cc:357] T 18e2d79d731142f89b9089869bc736e5 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:21.144111 1194 raft_consensus.cc:383] T 18e2d79d731142f89b9089869bc736e5 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:21.144234 1194 raft_consensus.cc:738] T 18e2d79d731142f89b9089869bc736e5 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:21.144466 1441 leader_election.cc:290] T 3455f010e51b4db1a9804a57fd1a4fb8 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:21.143697 755 tablet_service.cc:1514] Processing DeleteTablet for tablet 64cda8796dff46ce95c37439395c77ec with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:20 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:21.144739 1195 tablet_replica.cc:331] T 78689f7ed2a64405adb75f372465a792 P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:21.145840 741 tablet_service.cc:1514] Processing DeleteTablet for tablet b2a35fd891c645d68d79c61a2c832ba7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:20 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:21.146456 1195 raft_consensus.cc:2238] T 78689f7ed2a64405adb75f372465a792 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:21.146670 1195 raft_consensus.cc:2267] T 78689f7ed2a64405adb75f372465a792 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:21.141889 897 tablet_service.cc:1514] Processing DeleteTablet for tablet c94eedb5dce44e6d896192c93127903a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:20 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:21.148443 1195 ts_tablet_manager.cc:1905] T 78689f7ed2a64405adb75f372465a792 P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250113 02:51:21.155189 709 leader_election.cc:343] T 3455f010e51b4db1a9804a57fd1a4fb8 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Not found: Tablet not found: 3455f010e51b4db1a9804a57fd1a4fb8
I20250113 02:51:21.158195 1194 consensus_queue.cc:260] T 18e2d79d731142f89b9089869bc736e5 P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
W20250113 02:51:21.160261 711 leader_election.cc:343] T 3455f010e51b4db1a9804a57fd1a4fb8 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Not found: Tablet not found: 3455f010e51b4db1a9804a57fd1a4fb8
I20250113 02:51:21.160391 711 leader_election.cc:304] T 3455f010e51b4db1a9804a57fd1a4fb8 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:21.160854 1441 raft_consensus.cc:2743] T 3455f010e51b4db1a9804a57fd1a4fb8 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:21.182081 1194 ts_tablet_manager.cc:1428] T 18e2d79d731142f89b9089869bc736e5 P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.040s user 0.002s sys 0.000s
I20250113 02:51:21.182513 1194 tablet_bootstrap.cc:492] T 076ed7744da54ddebd8fc99686beb6ae P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:21.174793 880 tablet_service.cc:1467] Processing CreateTablet for tablet e70658c90555463498d3fc2e2f13ea7f (DEFAULT_TABLE table=default.Table_2565c0551f9943a084ace8f7e463e59d [id=fe6f8d302c3e489d8bb832503fd05642]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:21.190819 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e70658c90555463498d3fc2e2f13ea7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:21.210345 1194 tablet_bootstrap.cc:654] T 076ed7744da54ddebd8fc99686beb6ae P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:21.227593 1442 raft_consensus.cc:491] T 9041b28402664a08b8761fa155c9c229 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:21.227861 1192 tablet_bootstrap.cc:654] T 2fb62cc83f9c4dcfa9aa01ae8b46c025 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:21.231348 1029 tablet_service.cc:1467] Processing CreateTablet for tablet 7527e6e0b65e4a43b7f48899a34cdf7f (DEFAULT_TABLE table=default.Table_016e5998c5ef4112908e6917ccd57d15 [id=b7846a8ebee84005b1b5d259560a26bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:21.231889 1442 raft_consensus.cc:513] T 9041b28402664a08b8761fa155c9c229 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:21.246965 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9041b28402664a08b8761fa155c9c229" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
I20250113 02:51:21.247638 1196 ts_tablet_manager.cc:1918] T 9b5dc06ba3ea4056822ca3e4b1249587 P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:21.247855 1196 log.cc:1198] T 9b5dc06ba3ea4056822ca3e4b1249587 P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/9b5dc06ba3ea4056822ca3e4b1249587
I20250113 02:51:21.246905 1442 leader_election.cc:290] T 9041b28402664a08b8761fa155c9c229 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
W20250113 02:51:21.248443 995 leader_election.cc:343] T 9041b28402664a08b8761fa155c9c229 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:21.249161 1196 ts_tablet_manager.cc:1939] T 9b5dc06ba3ea4056822ca3e4b1249587 P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:21.249773 1196 tablet_replica.cc:331] T 68023c30a7f644eaa5b4ddd60d6c5ef5 P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:21.249949 1192 tablet_bootstrap.cc:492] T 2fb62cc83f9c4dcfa9aa01ae8b46c025 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:21.250515 1192 ts_tablet_manager.cc:1397] T 2fb62cc83f9c4dcfa9aa01ae8b46c025 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.152s user 0.005s sys 0.003s
I20250113 02:51:21.251443 1192 raft_consensus.cc:357] T 2fb62cc83f9c4dcfa9aa01ae8b46c025 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:21.251803 1192 raft_consensus.cc:383] T 2fb62cc83f9c4dcfa9aa01ae8b46c025 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:21.251919 1192 raft_consensus.cc:738] T 2fb62cc83f9c4dcfa9aa01ae8b46c025 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:21.252342 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9041b28402664a08b8761fa155c9c229" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:21.252928 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet 9b5dc06ba3ea4056822ca3e4b1249587 (table default.Table_20efb43987cc47c7a77436e076cc70f6 [id=8d889764fb3046369d87cef8998a5a87]) successfully deleted
I20250113 02:51:21.255425 919 raft_consensus.cc:2463] T 9041b28402664a08b8761fa155c9c229 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 0.
I20250113 02:51:21.256122 998 leader_election.cc:304] T 9041b28402664a08b8761fa155c9c229 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f; no voters: 1186fa869eb14812832fa6562531f196
I20250113 02:51:21.256453 1442 raft_consensus.cc:2798] T 9041b28402664a08b8761fa155c9c229 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 02:51:21.256623 1196 raft_consensus.cc:2238] T 68023c30a7f644eaa5b4ddd60d6c5ef5 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:21.259373 1442 raft_consensus.cc:491] T 9041b28402664a08b8761fa155c9c229 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 02:51:21.259531 1442 raft_consensus.cc:3054] T 9041b28402664a08b8761fa155c9c229 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:21.260516 1192 consensus_queue.cc:260] T 2fb62cc83f9c4dcfa9aa01ae8b46c025 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:21.261570 1192 ts_tablet_manager.cc:1428] T 2fb62cc83f9c4dcfa9aa01ae8b46c025 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20250113 02:51:21.265825 1442 raft_consensus.cc:513] T 9041b28402664a08b8761fa155c9c229 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:21.266855 1442 leader_election.cc:290] T 9041b28402664a08b8761fa155c9c229 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Requested vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:21.267521 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9041b28402664a08b8761fa155c9c229" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196"
W20250113 02:51:21.268972 995 leader_election.cc:343] T 9041b28402664a08b8761fa155c9c229 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:21.270299 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9041b28402664a08b8761fa155c9c229" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f"
I20250113 02:51:21.270746 1196 raft_consensus.cc:2267] T 68023c30a7f644eaa5b4ddd60d6c5ef5 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:21.271257 919 raft_consensus.cc:3054] T 9041b28402664a08b8761fa155c9c229 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:21.273867 919 raft_consensus.cc:2463] T 9041b28402664a08b8761fa155c9c229 P df69f48656b44bb890f76587ee69cc0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 1.
I20250113 02:51:21.278532 1196 ts_tablet_manager.cc:1905] T 68023c30a7f644eaa5b4ddd60d6c5ef5 P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:21.282651 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 2fb62cc83f9c4dcfa9aa01ae8b46c025
I20250113 02:51:21.283185 454 catalog_manager.cc:6596] ReplaceTablet: tablet 2fb62cc83f9c4dcfa9aa01ae8b46c025 deleted and replaced by tablet 234dab42f15f4043a7c74af4f267546b
I20250113 02:51:21.284046 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet ff8ef6290698477391721fd65b9eeb28
I20250113 02:51:21.284538 741 tablet_service.cc:1514] Processing DeleteTablet for tablet 2fb62cc83f9c4dcfa9aa01ae8b46c025 with delete_type TABLET_DATA_DELETED (replaced by tablet 234dab42f15f4043a7c74af4f267546b) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:21.284662 449 master_service.cc:946] ReplaceTablet: received request to replace tablet 9041b28402664a08b8761fa155c9c229 from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:21.285195 741 tablet_service.cc:1514] Processing DeleteTablet for tablet ff8ef6290698477391721fd65b9eeb28 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:20 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:21.290278 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_2d289859f3a2422b9f1c5bd6b3083088"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:21.290930 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2d289859f3a2422b9f1c5bd6b3083088 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:21.292801 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_f771ada0bfa5404e819d999f416d30cb" } new_table_name: "default.Table_f8ea53b156c944ebaff83a83c3b75118" modify_external_catalogs: true
I20250113 02:51:21.293678 1195 ts_tablet_manager.cc:1918] T 78689f7ed2a64405adb75f372465a792 P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:21.295326 454 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_4b5273b24d914b4eadec5f883245398a" } new_table_name: "default.Table_8f20c6ca51ab41a58970945c7946a768" modify_external_catalogs: true
I20250113 02:51:21.296337 1195 log.cc:1198] T 78689f7ed2a64405adb75f372465a792 P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/78689f7ed2a64405adb75f372465a792
I20250113 02:51:21.296756 1195 ts_tablet_manager.cc:1939] T 78689f7ed2a64405adb75f372465a792 P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:21.297552 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 238) took 1229 ms. Trace:
I20250113 02:51:21.297668 1195 rpcz_store.cc:276] 0113 02:51:20.068431 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:20.068517 (+ 86us) service_pool.cc:225] Handling call
0113 02:51:21.111281 (+1042764us) ts_tablet_manager.cc:1145] Deleting tablet 78689f7ed2a64405adb75f372465a792
0113 02:51:21.111351 (+ 70us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:21.285242 (+173891us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:21.293534 (+ 8292us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:21.297515 (+ 3981us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1042316}
I20250113 02:51:21.297921 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_c6c64517b49e4843ba63567a871d22bd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:21.298128 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet 78689f7ed2a64405adb75f372465a792 (table default.Table_f59080d9d91849deafd22506a1af8ff5 [id=d5354129133b446b8a4a5613db8bc411]) successfully deleted
W20250113 02:51:21.298375 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c6c64517b49e4843ba63567a871d22bd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:21.300715 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_0a251c1b92404b18a7dbce48963cdede"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:21.301194 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0a251c1b92404b18a7dbce48963cdede in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:21.303954 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 239) took 1143 ms. Trace:
I20250113 02:51:21.304123 1195 rpcz_store.cc:276] 0113 02:51:20.160344 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:20.160681 (+ 337us) service_pool.cc:225] Handling call
0113 02:51:21.303767 (+1143086us) ts_tablet_manager.cc:1145] Deleting tablet 23e9321f08ee4ce7b2252ad5c76c0a08
0113 02:51:21.303781 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:21.303913 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1142862}
W20250113 02:51:21.304546 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 23e9321f08ee4ce7b2252ad5c76c0a08 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 23e9321f08ee4ce7b2252ad5c76c0a08 already in progress: creating tablet
I20250113 02:51:21.304699 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 240) took 1143 ms. Trace:
I20250113 02:51:21.304827 1195 rpcz_store.cc:276] 0113 02:51:20.161308 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:20.161362 (+ 54us) service_pool.cc:225] Handling call
0113 02:51:21.304529 (+1143167us) ts_tablet_manager.cc:1145] Deleting tablet 2778d7dc7d9c4843bbc9eb705cec71cd
0113 02:51:21.304541 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:21.304666 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1142904}
W20250113 02:51:21.305318 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 2778d7dc7d9c4843bbc9eb705cec71cd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2778d7dc7d9c4843bbc9eb705cec71cd already in progress: creating tablet
I20250113 02:51:21.306281 741 tablet_service.cc:1514] Processing DeleteTablet for tablet 3455f010e51b4db1a9804a57fd1a4fb8 with delete_type TABLET_DATA_DELETED (replaced by tablet a6055d77d168495d9b756a1a56c1a920) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:21.306506 750 tablet_service.cc:1514] Processing DeleteTablet for tablet 8193357427f4491c8c6c1396ceeef2dc with delete_type TABLET_DATA_DELETED (replaced by tablet d3b695ac20ba4205b2d8732db4740531) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:21.306825 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 241) took 1135 ms. Trace:
I20250113 02:51:21.306995 1195 rpcz_store.cc:276] 0113 02:51:20.171697 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:20.195836 (+ 24139us) service_pool.cc:225] Handling call
0113 02:51:21.306513 (+1110677us) ts_tablet_manager.cc:1145] Deleting tablet 63efcb6b2e194c469fb30a197bd6565b
0113 02:51:21.306688 (+ 175us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:21.306786 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":896,"tablet-delete.queue_time_us":1044498}
I20250113 02:51:21.307511 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 244) took 1044 ms. Trace:
W20250113 02:51:21.307498 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 63efcb6b2e194c469fb30a197bd6565b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 63efcb6b2e194c469fb30a197bd6565b already in progress: creating tablet
I20250113 02:51:21.305737 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_480e362015cc422a85dffa0b15d1dcae" } new_table_name: "default.Table_df5d17a7c90c4a5a9e0d75558749d6a8" modify_external_catalogs: true
I20250113 02:51:21.307814 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7527e6e0b65e4a43b7f48899a34cdf7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:21.307688 1195 rpcz_store.cc:276] 0113 02:51:20.263423 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:20.263479 (+ 56us) service_pool.cc:225] Handling call
0113 02:51:21.307395 (+1043916us) ts_tablet_manager.cc:1145] Deleting tablet 93a2729c5a604a27a9bbc451ccb4e128
0113 02:51:21.307404 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:21.307481 (+ 77us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1043710}
W20250113 02:51:21.308126 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 93a2729c5a604a27a9bbc451ccb4e128 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 93a2729c5a604a27a9bbc451ccb4e128 already in progress: creating tablet
I20250113 02:51:21.308267 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 242) took 1045 ms. Trace:
I20250113 02:51:21.308246 998 leader_election.cc:304] T 9041b28402664a08b8761fa155c9c229 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f; no voters: 1186fa869eb14812832fa6562531f196
I20250113 02:51:21.308390 1195 rpcz_store.cc:276] 0113 02:51:20.262543 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:20.263245 (+ 702us) service_pool.cc:225] Handling call
0113 02:51:21.308123 (+1044878us) ts_tablet_manager.cc:1145] Deleting tablet a77c38a394114d6c8a8da99d7eef7669
0113 02:51:21.308131 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:21.308237 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1022954}
W20250113 02:51:21.308794 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet a77c38a394114d6c8a8da99d7eef7669 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a77c38a394114d6c8a8da99d7eef7669 already in progress: creating tablet
I20250113 02:51:21.309193 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 243) took 1046 ms. Trace:
I20250113 02:51:21.309340 1195 rpcz_store.cc:276] 0113 02:51:20.262706 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:20.263081 (+ 375us) service_pool.cc:225] Handling call
0113 02:51:21.309071 (+1045990us) ts_tablet_manager.cc:1145] Deleting tablet 98f038ad31ff4e05bb88ac68ae341cc2
0113 02:51:21.309080 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:21.309161 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1017677}
I20250113 02:51:21.309617 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_d866aff056e246d88df242feb6d4e627" } new_table_name: "default.Table_c6b6ad0e811844518a1f5802b983d3d5" modify_external_catalogs: true
W20250113 02:51:21.309837 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 98f038ad31ff4e05bb88ac68ae341cc2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 98f038ad31ff4e05bb88ac68ae341cc2 already in progress: creating tablet
I20250113 02:51:21.310124 1195 tablet_replica.cc:331] T f0e7cc161f354f86b8c2fef303e87c5b P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:21.310319 1195 raft_consensus.cc:2238] T f0e7cc161f354f86b8c2fef303e87c5b P 1186fa869eb14812832fa6562531f196 [term 1 LEADER]: Raft consensus shutting down.
I20250113 02:51:21.310542 1195 pending_rounds.cc:62] T f0e7cc161f354f86b8c2fef303e87c5b P 1186fa869eb14812832fa6562531f196: Trying to abort 1 pending ops.
I20250113 02:51:21.310621 1195 pending_rounds.cc:69] T f0e7cc161f354f86b8c2fef303e87c5b P 1186fa869eb14812832fa6562531f196: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113673445054443520 op_type: NO_OP noop_request { }
I20250113 02:51:21.310820 1195 raft_consensus.cc:2883] T f0e7cc161f354f86b8c2fef303e87c5b P 1186fa869eb14812832fa6562531f196 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250113 02:51:21.311026 1195 raft_consensus.cc:2267] T f0e7cc161f354f86b8c2fef303e87c5b P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:21.311091 755 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c55662a71444a3683a825bcec6c5b68 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:16 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:21.311900 1195 ts_tablet_manager.cc:1905] T f0e7cc161f354f86b8c2fef303e87c5b P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:21.313431 1198 ts_tablet_manager.cc:1918] T 8193357427f4491c8c6c1396ceeef2dc P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:21.313700 1442 raft_consensus.cc:2798] T 9041b28402664a08b8761fa155c9c229 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Leader election won for term 1
I20250113 02:51:21.314653 1442 raft_consensus.cc:695] T 9041b28402664a08b8761fa155c9c229 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: Becoming Leader. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Running, Role: LEADER
I20250113 02:51:21.317121 1031 tablet_service.cc:1514] Processing DeleteTablet for tablet ff8ef6290698477391721fd65b9eeb28 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:20 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:21.317407 1192 tablet_bootstrap.cc:492] T 98f038ad31ff4e05bb88ac68ae341cc2 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:21.316401 1442 consensus_queue.cc:237] T 9041b28402664a08b8761fa155c9c229 P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:21.319897 1368 raft_consensus.cc:491] T 0c55662a71444a3683a825bcec6c5b68 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:21.320060 1368 raft_consensus.cc:513] T 0c55662a71444a3683a825bcec6c5b68 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:21.320688 1368 leader_election.cc:290] T 0c55662a71444a3683a825bcec6c5b68 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
W20250113 02:51:21.321578 711 leader_election.cc:343] T 0c55662a71444a3683a825bcec6c5b68 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Not found: Tablet not found: 0c55662a71444a3683a825bcec6c5b68
I20250113 02:51:21.320732 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet 2fb62cc83f9c4dcfa9aa01ae8b46c025 with delete_type TABLET_DATA_DELETED (replaced by tablet 234dab42f15f4043a7c74af4f267546b) from {username='slave'} at 127.0.0.1:60230
W20250113 02:51:21.323220 709 leader_election.cc:343] T 0c55662a71444a3683a825bcec6c5b68 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Not found: Tablet not found: 0c55662a71444a3683a825bcec6c5b68
I20250113 02:51:21.323397 709 leader_election.cc:304] T 0c55662a71444a3683a825bcec6c5b68 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:21.323737 1368 raft_consensus.cc:2743] T 0c55662a71444a3683a825bcec6c5b68 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:21.322911 1198 log.cc:1198] T 8193357427f4491c8c6c1396ceeef2dc P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/8193357427f4491c8c6c1396ceeef2dc
I20250113 02:51:21.324698 1198 ts_tablet_manager.cc:1939] T 8193357427f4491c8c6c1396ceeef2dc P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:21.328953 1195 ts_tablet_manager.cc:1918] T f0e7cc161f354f86b8c2fef303e87c5b P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250113 02:51:21.329480 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet 8193357427f4491c8c6c1396ceeef2dc (table default.Table_8b87c8469fbb465e80375aa0446de6c2 [id=f85f0c435ffa4fdca81d86de4aa4fd5f]) successfully deleted
I20250113 02:51:21.329761 892 tablet_service.cc:1514] Processing DeleteTablet for tablet ff8ef6290698477391721fd65b9eeb28 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:20 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:21.334194 881 tablet_service.cc:1514] Processing DeleteTablet for tablet 2fb62cc83f9c4dcfa9aa01ae8b46c025 with delete_type TABLET_DATA_DELETED (replaced by tablet 234dab42f15f4043a7c74af4f267546b) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:21.334306 1439 raft_consensus.cc:491] T 18e2d79d731142f89b9089869bc736e5 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:21.334920 1195 log.cc:1198] T f0e7cc161f354f86b8c2fef303e87c5b P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/f0e7cc161f354f86b8c2fef303e87c5b
I20250113 02:51:21.335397 1195 ts_tablet_manager.cc:1939] T f0e7cc161f354f86b8c2fef303e87c5b P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:21.336299 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 249) took 1005 ms. Trace:
I20250113 02:51:21.336468 1195 rpcz_store.cc:276] 0113 02:51:20.331240 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:20.332235 (+ 995us) service_pool.cc:225] Handling call
0113 02:51:21.310069 (+977834us) ts_tablet_manager.cc:1145] Deleting tablet f0e7cc161f354f86b8c2fef303e87c5b
0113 02:51:21.310078 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:21.322553 (+ 12475us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:21.328816 (+ 6263us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:21.336264 (+ 7448us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":977566}
I20250113 02:51:21.337025 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet f0e7cc161f354f86b8c2fef303e87c5b (table default.Table_20ca05c407b14ba0a92f76f4596ae2ef [id=75cb0745bbb84e1c883cb13b546ecb11]) successfully deleted
I20250113 02:51:21.337028 1033 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b5dc06ba3ea4056822ca3e4b1249587 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:18 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:21.337455 1195 tablet_replica.cc:331] T 3d132c29cb2c46108e5877e38068ef17 P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:21.337718 1195 raft_consensus.cc:2238] T 3d132c29cb2c46108e5877e38068ef17 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:21.337934 1195 raft_consensus.cc:2267] T 3d132c29cb2c46108e5877e38068ef17 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:21.337983 1198 tablet_replica.cc:331] T f0e7cc161f354f86b8c2fef303e87c5b P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:21.338251 1198 raft_consensus.cc:2238] T f0e7cc161f354f86b8c2fef303e87c5b P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:21.338435 1198 raft_consensus.cc:2267] T f0e7cc161f354f86b8c2fef303e87c5b P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:21.338970 1195 ts_tablet_manager.cc:1905] T 3d132c29cb2c46108e5877e38068ef17 P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:21.339818 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_59401ddb940e4e789a5bd54f96548ba5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:21.340483 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_59401ddb940e4e789a5bd54f96548ba5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:21.343613 1439 raft_consensus.cc:513] T 18e2d79d731142f89b9089869bc736e5 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:21.344476 1439 leader_election.cc:290] T 18e2d79d731142f89b9089869bc736e5 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:21.344863 1192 tablet_bootstrap.cc:654] T 98f038ad31ff4e05bb88ac68ae341cc2 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:21.345229 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18e2d79d731142f89b9089869bc736e5" candidate_uuid: "df69f48656b44bb890f76587ee69cc0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
W20250113 02:51:21.345898 851 leader_election.cc:343] T 18e2d79d731142f89b9089869bc736e5 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:21.346627 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18e2d79d731142f89b9089869bc736e5" candidate_uuid: "df69f48656b44bb890f76587ee69cc0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
W20250113 02:51:21.347211 852 leader_election.cc:343] T 18e2d79d731142f89b9089869bc736e5 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:21.347354 852 leader_election.cc:304] T 18e2d79d731142f89b9089869bc736e5 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df69f48656b44bb890f76587ee69cc0f; no voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca
I20250113 02:51:21.347781 1439 raft_consensus.cc:2743] T 18e2d79d731142f89b9089869bc736e5 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:21.353134 1195 ts_tablet_manager.cc:1918] T 3d132c29cb2c46108e5877e38068ef17 P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:21.353371 1438 raft_consensus.cc:695] T fb183b4588e84212a3b4b096f430d47d P df69f48656b44bb890f76587ee69cc0f [term 1 LEADER]: Becoming Leader. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Running, Role: LEADER
I20250113 02:51:21.024590 1191 raft_consensus.cc:738] T c62a5e9e43d14b5c94eb87260528469d P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:21.355424 1195 log.cc:1198] T 3d132c29cb2c46108e5877e38068ef17 P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/3d132c29cb2c46108e5877e38068ef17
I20250113 02:51:21.355496 1191 consensus_queue.cc:260] T c62a5e9e43d14b5c94eb87260528469d P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:21.355705 1438 consensus_queue.cc:237] T fb183b4588e84212a3b4b096f430d47d P df69f48656b44bb890f76587ee69cc0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:21.357411 1195 ts_tablet_manager.cc:1939] T 3d132c29cb2c46108e5877e38068ef17 P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:21.362123 1194 tablet_bootstrap.cc:492] T 076ed7744da54ddebd8fc99686beb6ae P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:21.362596 1194 ts_tablet_manager.cc:1397] T 076ed7744da54ddebd8fc99686beb6ae P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.180s user 0.005s sys 0.005s
I20250113 02:51:21.364612 1198 ts_tablet_manager.cc:1905] T f0e7cc161f354f86b8c2fef303e87c5b P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:21.376817 1194 raft_consensus.cc:357] T 076ed7744da54ddebd8fc99686beb6ae P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:21.377282 1194 raft_consensus.cc:383] T 076ed7744da54ddebd8fc99686beb6ae P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:21.377503 1194 raft_consensus.cc:738] T 076ed7744da54ddebd8fc99686beb6ae P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:21.383754 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 250) took 1030 ms. Trace:
I20250113 02:51:21.383899 1195 rpcz_store.cc:276] 0113 02:51:20.331358 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:20.332554 (+ 1196us) service_pool.cc:225] Handling call
0113 02:51:21.337385 (+1004831us) ts_tablet_manager.cc:1145] Deleting tablet 3d132c29cb2c46108e5877e38068ef17
0113 02:51:21.337396 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:21.342127 (+ 4731us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:21.352972 (+ 10845us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:21.362263 (+ 9291us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1004656}
I20250113 02:51:21.384308 1195 tablet_replica.cc:331] T ce92c2b45a274eb09e72a96b7ec133be P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:21.384503 1195 raft_consensus.cc:2238] T ce92c2b45a274eb09e72a96b7ec133be P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:21.384550 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet 3d132c29cb2c46108e5877e38068ef17 (table default.Table_b31190bf4fa4499da09233dbdd1c9139 [id=44f4b85774f644aabd241d6ecd397a37]) successfully deleted
I20250113 02:51:21.384685 1195 raft_consensus.cc:2267] T ce92c2b45a274eb09e72a96b7ec133be P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:21.385257 1195 ts_tablet_manager.cc:1905] T ce92c2b45a274eb09e72a96b7ec133be P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:21.383733 1191 ts_tablet_manager.cc:1428] T c62a5e9e43d14b5c94eb87260528469d P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.333s user 0.000s sys 0.003s
I20250113 02:51:21.392418 1191 tablet_bootstrap.cc:492] T fb183b4588e84212a3b4b096f430d47d P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:21.395087 1195 ts_tablet_manager.cc:1918] T ce92c2b45a274eb09e72a96b7ec133be P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:21.395303 1195 log.cc:1198] T ce92c2b45a274eb09e72a96b7ec133be P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/ce92c2b45a274eb09e72a96b7ec133be
I20250113 02:51:21.395735 1195 ts_tablet_manager.cc:1939] T ce92c2b45a274eb09e72a96b7ec133be P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:21.396536 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 251) took 1065 ms. Trace:
I20250113 02:51:21.396726 1195 rpcz_store.cc:276] 0113 02:51:20.331458 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:20.332763 (+ 1305us) service_pool.cc:225] Handling call
0113 02:51:21.384220 (+1051457us) ts_tablet_manager.cc:1145] Deleting tablet ce92c2b45a274eb09e72a96b7ec133be
0113 02:51:21.384234 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:21.388682 (+ 4448us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:21.394955 (+ 6273us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:21.396496 (+ 1541us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1051295}
I20250113 02:51:21.397310 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet ce92c2b45a274eb09e72a96b7ec133be (table default.Table_06f81a97cb014a3bad594e88c8cf4733 [id=7c2bc6d4232f4585b84d71e3e1f85794]) successfully deleted
I20250113 02:51:21.397773 1194 consensus_queue.cc:260] T 076ed7744da54ddebd8fc99686beb6ae P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:21.397929 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 258) took 1026 ms. Trace:
I20250113 02:51:21.398051 1195 rpcz_store.cc:276] 0113 02:51:20.371210 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:20.371281 (+ 71us) service_pool.cc:225] Handling call
0113 02:51:21.397768 (+1026487us) ts_tablet_manager.cc:1145] Deleting tablet 23e04961f072440fa3e3f9485e40f79c
0113 02:51:21.397779 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:21.397890 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1026217}
W20250113 02:51:21.398571 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 23e04961f072440fa3e3f9485e40f79c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 23e04961f072440fa3e3f9485e40f79c already in progress: creating tablet
I20250113 02:51:21.399190 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 259) took 1022 ms. Trace:
I20250113 02:51:21.399344 1195 rpcz_store.cc:276] 0113 02:51:20.376264 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:20.376327 (+ 63us) service_pool.cc:225] Handling call
0113 02:51:21.399048 (+1022721us) ts_tablet_manager.cc:1145] Deleting tablet 0fb75c551b4c424186ef2021db742e8e
0113 02:51:21.399057 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:21.399147 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1022500}
I20250113 02:51:21.403157 1109 heartbeater.cc:502] Master 127.0.104.61:35271 was elected leader, sending a full tablet report...
I20250113 02:51:21.404666 1194 ts_tablet_manager.cc:1428] T 076ed7744da54ddebd8fc99686beb6ae P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.042s user 0.001s sys 0.001s
I20250113 02:51:21.410341 1194 tablet_bootstrap.cc:492] T 8547b63c40ec4ccebd962b5a3b2c5e0f P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:21.413626 1194 tablet_bootstrap.cc:654] T 8547b63c40ec4ccebd962b5a3b2c5e0f P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:21.414059 1196 ts_tablet_manager.cc:1918] T 68023c30a7f644eaa5b4ddd60d6c5ef5 P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:21.415397 1196 log.cc:1198] T 68023c30a7f644eaa5b4ddd60d6c5ef5 P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/68023c30a7f644eaa5b4ddd60d6c5ef5
I20250113 02:51:21.415779 1196 ts_tablet_manager.cc:1939] T 68023c30a7f644eaa5b4ddd60d6c5ef5 P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:21.416954 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet 68023c30a7f644eaa5b4ddd60d6c5ef5 (table default.Table_78c0bcf1e2d64abc81253301920cf566 [id=e5831758ca184c1998f551fe0285d010]) successfully deleted
W20250113 02:51:21.418001 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 23e04961f072440fa3e3f9485e40f79c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 23e04961f072440fa3e3f9485e40f79c already in progress: creating tablet
W20250113 02:51:21.418632 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 0810b9671f824d0a84741bc1904df247 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0810b9671f824d0a84741bc1904df247 already in progress: creating tablet
W20250113 02:51:21.421116 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 0810b9671f824d0a84741bc1904df247 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0810b9671f824d0a84741bc1904df247 already in progress: creating tablet
W20250113 02:51:21.421233 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet c215ffe61b9b426286cc5e808463078d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c215ffe61b9b426286cc5e808463078d already in progress: creating tablet
W20250113 02:51:21.421768 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet a97efc70946c4af487e33a85abf6b90c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a97efc70946c4af487e33a85abf6b90c already in progress: creating tablet
W20250113 02:51:21.421886 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet c215ffe61b9b426286cc5e808463078d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c215ffe61b9b426286cc5e808463078d already in progress: creating tablet
I20250113 02:51:21.422096 1196 tablet_replica.cc:331] T fb183b4588e84212a3b4b096f430d47d P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:21.422310 1196 raft_consensus.cc:2238] T fb183b4588e84212a3b4b096f430d47d P df69f48656b44bb890f76587ee69cc0f [term 1 LEADER]: Raft consensus shutting down.
I20250113 02:51:21.422605 1196 pending_rounds.cc:62] T fb183b4588e84212a3b4b096f430d47d P df69f48656b44bb890f76587ee69cc0f: Trying to abort 1 pending ops.
W20250113 02:51:21.422614 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 9041b28402664a08b8761fa155c9c229 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9041b28402664a08b8761fa155c9c229 already in progress: creating tablet
I20250113 02:51:21.422708 1196 pending_rounds.cc:69] T fb183b4588e84212a3b4b096f430d47d P df69f48656b44bb890f76587ee69cc0f: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113673446853029888 op_type: NO_OP noop_request { }
I20250113 02:51:21.422829 1196 raft_consensus.cc:2883] T fb183b4588e84212a3b4b096f430d47d P df69f48656b44bb890f76587ee69cc0f [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250113 02:51:21.422981 1195 tablet_replica.cc:331] T 15a05a32febd443dbb519c186048428a P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:21.423007 1196 raft_consensus.cc:2267] T fb183b4588e84212a3b4b096f430d47d P df69f48656b44bb890f76587ee69cc0f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:21.424005 1194 tablet_bootstrap.cc:492] T 8547b63c40ec4ccebd962b5a3b2c5e0f P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:21.429960 1194 ts_tablet_manager.cc:1397] T 8547b63c40ec4ccebd962b5a3b2c5e0f P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.020s user 0.001s sys 0.002s
I20250113 02:51:21.430823 1194 raft_consensus.cc:357] T 8547b63c40ec4ccebd962b5a3b2c5e0f P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:21.431082 1194 raft_consensus.cc:383] T 8547b63c40ec4ccebd962b5a3b2c5e0f P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:21.431177 1194 raft_consensus.cc:738] T 8547b63c40ec4ccebd962b5a3b2c5e0f P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:21.431402 1194 consensus_queue.cc:260] T 8547b63c40ec4ccebd962b5a3b2c5e0f P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:21.432567 1194 ts_tablet_manager.cc:1428] T 8547b63c40ec4ccebd962b5a3b2c5e0f P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250113 02:51:21.432880 1194 tablet_bootstrap.cc:492] T d5bd09a1ede644c59e400e49c8125793 P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:21.434587 1194 tablet_bootstrap.cc:654] T d5bd09a1ede644c59e400e49c8125793 P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:21.441576 1191 tablet_bootstrap.cc:654] T fb183b4588e84212a3b4b096f430d47d P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:21.442750 1194 tablet_bootstrap.cc:492] T d5bd09a1ede644c59e400e49c8125793 P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:21.442775 1196 ts_tablet_manager.cc:1905] T fb183b4588e84212a3b4b096f430d47d P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:21.442955 1194 ts_tablet_manager.cc:1397] T d5bd09a1ede644c59e400e49c8125793 P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.009s
I20250113 02:51:21.443392 1192 tablet_bootstrap.cc:492] T 98f038ad31ff4e05bb88ac68ae341cc2 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:21.443625 1192 ts_tablet_manager.cc:1397] T 98f038ad31ff4e05bb88ac68ae341cc2 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.146s user 0.005s sys 0.004s
I20250113 02:51:21.443770 1194 raft_consensus.cc:357] T d5bd09a1ede644c59e400e49c8125793 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:21.444089 1194 raft_consensus.cc:383] T d5bd09a1ede644c59e400e49c8125793 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:21.444190 1194 raft_consensus.cc:738] T d5bd09a1ede644c59e400e49c8125793 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:21.444588 1192 raft_consensus.cc:357] T 98f038ad31ff4e05bb88ac68ae341cc2 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:21.444731 1194 consensus_queue.cc:260] T d5bd09a1ede644c59e400e49c8125793 P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:21.444864 1192 raft_consensus.cc:383] T 98f038ad31ff4e05bb88ac68ae341cc2 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:21.444967 1192 raft_consensus.cc:738] T 98f038ad31ff4e05bb88ac68ae341cc2 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:21.445276 1192 consensus_queue.cc:260] T 98f038ad31ff4e05bb88ac68ae341cc2 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:21.445935 1192 ts_tablet_manager.cc:1428] T 98f038ad31ff4e05bb88ac68ae341cc2 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250113 02:51:21.451049 1196 ts_tablet_manager.cc:1918] T fb183b4588e84212a3b4b096f430d47d P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250113 02:51:21.452418 1196 log.cc:1198] T fb183b4588e84212a3b4b096f430d47d P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/fb183b4588e84212a3b4b096f430d47d
I20250113 02:51:21.456629 1196 ts_tablet_manager.cc:1939] T fb183b4588e84212a3b4b096f430d47d P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:21.457901 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet fb183b4588e84212a3b4b096f430d47d (table default.Table_53b46860bb68469fa08f3fd02b2408e5 [id=33bf7fd42ab5458db0f14347353c0db4]) successfully deleted
I20250113 02:51:21.460247 1195 raft_consensus.cc:2238] T 15a05a32febd443dbb519c186048428a P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Raft consensus shutting down.
W20250113 02:51:21.460326 454 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c508724edea147e5bf6e20bca2f50397 consensus state has no leader
I20250113 02:51:21.460744 1195 raft_consensus.cc:2267] T 15a05a32febd443dbb519c186048428a P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:21.460815 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_4757a3729a5c41d2a28c09572ef23181"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:21.461445 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4757a3729a5c41d2a28c09572ef23181 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:21.461516 1195 ts_tablet_manager.cc:1905] T 15a05a32febd443dbb519c186048428a P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:21.461586 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_cd499f4a8487400690a9ab957c606b7f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:21.462098 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cd499f4a8487400690a9ab957c606b7f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:21.463069 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_93d706c664024860855d1c9d91992817" } new_table_name: "default.Table_03b7491a47674f7187f4952feb9b44e2" modify_external_catalogs: true
I20250113 02:51:21.456169 1194 ts_tablet_manager.cc:1428] T d5bd09a1ede644c59e400e49c8125793 P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20250113 02:51:21.464329 966 heartbeater.cc:346] Connected to a master server at 127.0.104.61:35271
I20250113 02:51:21.464536 448 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_c79704f27bf14cba8a0afc2dd36a143f" } modify_external_catalogs: true
I20250113 02:51:21.464715 448 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_c79704f27bf14cba8a0afc2dd36a143f" } modify_external_catalogs: true
I20250113 02:51:21.465366 1192 tablet_bootstrap.cc:492] T d101e98b72544c30a3a4bc60d1576aa3 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
W20250113 02:51:21.466032 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 06f36d09bbb1443bafce906907795404 consensus state has no leader
I20250113 02:51:21.466351 449 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_faf2fafcef874bf6a5798ee6b19ef704" } modify_external_catalogs: true
I20250113 02:51:21.466495 449 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_faf2fafcef874bf6a5798ee6b19ef704" } modify_external_catalogs: true
I20250113 02:51:21.468787 1194 tablet_bootstrap.cc:492] T d35ec7e4a5084a74bea172ef8d9990b5 P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:21.469256 880 tablet_service.cc:1514] Processing DeleteTablet for tablet fb183b4588e84212a3b4b096f430d47d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:20 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:21.469964 1196 tablet_replica.cc:331] T 53cc528d282c47788d5d558569f074c1 P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:21.470173 1196 raft_consensus.cc:2238] T 53cc528d282c47788d5d558569f074c1 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:21.470357 1196 raft_consensus.cc:2267] T 53cc528d282c47788d5d558569f074c1 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
W20250113 02:51:21.470542 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet c71511a100f14322b5091a9804720fa0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c71511a100f14322b5091a9804720fa0 already in progress: creating tablet
W20250113 02:51:21.470815 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet de4a58bf63054890808e2e2fe46e43d2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet de4a58bf63054890808e2e2fe46e43d2 already in progress: creating tablet
I20250113 02:51:21.470887 1196 ts_tablet_manager.cc:1905] T 53cc528d282c47788d5d558569f074c1 P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:21.475185 447 catalog_manager.cc:5526] T ad9b95c5450a4c9c89a0930862348d49 P 1186fa869eb14812832fa6562531f196 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1186fa869eb14812832fa6562531f196 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "1186fa869eb14812832fa6562531f196" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } health_report { overall_health: UNKNOWN } } }
I20250113 02:51:21.475649 447 catalog_manager.cc:5618] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095) does not have the latest schema for tablet ad9b95c5450a4c9c89a0930862348d49 (table default.Table_936e73f0b76046d285567ac0778e36a9 [id=219c739889d04a97aafa4fe1b55b0a24]). Expected version 2 got 0
I20250113 02:51:21.479856 453 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_d05b7517b14c4168bea99385f1a7d8b6" } modify_external_catalogs: true
I20250113 02:51:21.480084 453 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_d05b7517b14c4168bea99385f1a7d8b6" } modify_external_catalogs: true
I20250113 02:51:21.480965 1033 tablet_service.cc:1467] Processing CreateTablet for tablet 7213e3284b4d4cb581d9fb946a9f87e9 (DEFAULT_TABLE table=default.Table_ff6df4afcf4d4f9a964d079ee24f02b5 [id=8a5901fc685345579a254a0f3af32051]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:21.481289 881 tablet_service.cc:1514] Processing DeleteTablet for tablet 076ed7744da54ddebd8fc99686beb6ae with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:17 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:21.481516 880 tablet_service.cc:1514] Processing DeleteTablet for tablet 18e2d79d731142f89b9089869bc736e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:18 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:21.481998 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7213e3284b4d4cb581d9fb946a9f87e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:21.485769 1196 ts_tablet_manager.cc:1918] T 53cc528d282c47788d5d558569f074c1 P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:21.486425 1196 log.cc:1198] T 53cc528d282c47788d5d558569f074c1 P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/53cc528d282c47788d5d558569f074c1
I20250113 02:51:21.486809 1196 ts_tablet_manager.cc:1939] T 53cc528d282c47788d5d558569f074c1 P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
W20250113 02:51:21.487715 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f68f9f4076d9475d938e795e72bdd468 consensus state has no leader
I20250113 02:51:21.488173 1196 tablet_replica.cc:331] T 107e6e9e2d84496f8fce92b9ba377189 P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
W20250113 02:51:21.488243 451 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4b07a985897641c3ac48021c2a4daedb consensus state has no leader
I20250113 02:51:21.488384 1196 raft_consensus.cc:2238] T 107e6e9e2d84496f8fce92b9ba377189 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:21.488446 451 master_service.cc:946] ReplaceTablet: received request to replace tablet 157905316fc545b6a7864ae1c4a2a584 from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:21.488564 1196 raft_consensus.cc:2267] T 107e6e9e2d84496f8fce92b9ba377189 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:21.488984 452 master_service.cc:946] ReplaceTablet: received request to replace tablet 4efa63b8be794e1db4b340f4c0c4cd5a from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:21.489087 1196 ts_tablet_manager.cc:1905] T 107e6e9e2d84496f8fce92b9ba377189 P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:21.489778 455 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_37b69086d7374e7ea7b83b40e8f202db" } modify_external_catalogs: true
I20250113 02:51:21.489946 455 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_37b69086d7374e7ea7b83b40e8f202db" } modify_external_catalogs: true
I20250113 02:51:21.490330 1198 ts_tablet_manager.cc:1918] T f0e7cc161f354f86b8c2fef303e87c5b P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:21.490511 1198 log.cc:1198] T f0e7cc161f354f86b8c2fef303e87c5b P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/f0e7cc161f354f86b8c2fef303e87c5b
I20250113 02:51:21.491787 451 master_service.cc:946] ReplaceTablet: received request to replace tablet 1d81eec487dd4016b3b804451da31c49 from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:21.494618 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet 53cc528d282c47788d5d558569f074c1 (table default.Table_06f81a97cb014a3bad594e88c8cf4733 [id=7c2bc6d4232f4585b84d71e3e1f85794]) successfully deleted
W20250113 02:51:21.494908 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet bcb59573e82743858bd39bde64627d6d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bcb59573e82743858bd39bde64627d6d already in progress: creating tablet
I20250113 02:51:21.495954 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_b581b1a063a54551a8535adf01983fb8" } modify_external_catalogs: true
I20250113 02:51:21.496183 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_b581b1a063a54551a8535adf01983fb8" } modify_external_catalogs: true
I20250113 02:51:21.499424 1192 tablet_bootstrap.cc:654] T d101e98b72544c30a3a4bc60d1576aa3 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:21.501839 755 tablet_service.cc:1467] Processing CreateTablet for tablet 7213e3284b4d4cb581d9fb946a9f87e9 (DEFAULT_TABLE table=default.Table_ff6df4afcf4d4f9a964d079ee24f02b5 [id=8a5901fc685345579a254a0f3af32051]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:21.502272 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7213e3284b4d4cb581d9fb946a9f87e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:21.503721 1191 tablet_bootstrap.cc:492] T fb183b4588e84212a3b4b096f430d47d P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:21.503880 1191 ts_tablet_manager.cc:1397] T fb183b4588e84212a3b4b096f430d47d P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.112s user 0.003s sys 0.000s
I20250113 02:51:21.505125 1191 raft_consensus.cc:357] T fb183b4588e84212a3b4b096f430d47d P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:21.505340 1191 raft_consensus.cc:383] T fb183b4588e84212a3b4b096f430d47d P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:21.505417 1191 raft_consensus.cc:738] T fb183b4588e84212a3b4b096f430d47d P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:21.509793 880 tablet_service.cc:1467] Processing CreateTablet for tablet 7213e3284b4d4cb581d9fb946a9f87e9 (DEFAULT_TABLE table=default.Table_ff6df4afcf4d4f9a964d079ee24f02b5 [id=8a5901fc685345579a254a0f3af32051]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:21.510221 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7213e3284b4d4cb581d9fb946a9f87e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:21.511518 1196 ts_tablet_manager.cc:1918] T 107e6e9e2d84496f8fce92b9ba377189 P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:21.511655 1196 log.cc:1198] T 107e6e9e2d84496f8fce92b9ba377189 P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/107e6e9e2d84496f8fce92b9ba377189
I20250113 02:51:21.512080 1196 ts_tablet_manager.cc:1939] T 107e6e9e2d84496f8fce92b9ba377189 P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:21.519919 1023 tablet_service.cc:1467] Processing CreateTablet for tablet df87e25053a74990959f85288a85217b (DEFAULT_TABLE table=default.Table_5cb0c9e84e72464c895d03166a1d18d7 [id=b2c8431e06ac46be9840edf78b6c19a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:21.520007 1191 consensus_queue.cc:260] T fb183b4588e84212a3b4b096f430d47d P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:21.520409 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df87e25053a74990959f85288a85217b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:21.520607 1191 ts_tablet_manager.cc:1428] T fb183b4588e84212a3b4b096f430d47d P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.017s user 0.003s sys 0.000s
I20250113 02:51:21.521010 1191 tablet_bootstrap.cc:492] T 9041b28402664a08b8761fa155c9c229 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:21.521908 822 heartbeater.cc:346] Connected to a master server at 127.0.104.61:35271
I20250113 02:51:21.526207 1109 heartbeater.cc:502] Master 127.0.104.61:35271 was elected leader, sending a full tablet report...
I20250113 02:51:21.528038 1191 tablet_bootstrap.cc:654] T 9041b28402664a08b8761fa155c9c229 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:21.529094 1198 ts_tablet_manager.cc:1939] T f0e7cc161f354f86b8c2fef303e87c5b P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:21.529698 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 261) took 1018 ms. Trace:
I20250113 02:51:21.529896 1198 rpcz_store.cc:276] 0113 02:51:20.510992 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:20.511040 (+ 48us) service_pool.cc:225] Handling call
0113 02:51:21.337903 (+826863us) ts_tablet_manager.cc:1145] Deleting tablet f0e7cc161f354f86b8c2fef303e87c5b
0113 02:51:21.337918 (+ 15us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:21.458432 (+120514us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:21.490208 (+ 31776us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:21.529625 (+ 39417us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":826679}
I20250113 02:51:21.531862 1198 tablet_replica.cc:331] T 15a05a32febd443dbb519c186048428a P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:21.531936 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet f0e7cc161f354f86b8c2fef303e87c5b (table default.Table_20ca05c407b14ba0a92f76f4596ae2ef [id=75cb0745bbb84e1c883cb13b546ecb11]) successfully deleted
I20250113 02:51:21.532191 1198 raft_consensus.cc:2238] T 15a05a32febd443dbb519c186048428a P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:21.532433 1198 raft_consensus.cc:2267] T 15a05a32febd443dbb519c186048428a P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:21.533102 1198 ts_tablet_manager.cc:1905] T 15a05a32febd443dbb519c186048428a P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:21.538370 447 master_service.cc:946] ReplaceTablet: received request to replace tablet a92f9b1c9c1a4cbdbf0cc7b2a3f2275b from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:21.539702 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 1500e4f9c0664a879503afd3edf14c66
I20250113 02:51:21.540499 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_157d1350cce74502a990462ac9bd0e5f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:21.541067 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_157d1350cce74502a990462ac9bd0e5f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:21.542927 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_029fb0ede6224bf4b0cfa388bee300a7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:21.543292 750 tablet_service.cc:1514] Processing DeleteTablet for tablet 1500e4f9c0664a879503afd3edf14c66 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:21 UTC) from {username='slave'} at 127.0.0.1:55398
W20250113 02:51:21.543427 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_029fb0ede6224bf4b0cfa388bee300a7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250113 02:51:21.543916 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6666922d677a4c678f80d7c6c47fd2a5 consensus state has no leader
I20250113 02:51:21.545097 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet efccefd6787943469131492741ee2bd5
I20250113 02:51:21.546018 1196 tablet_replica.cc:331] T 9041b28402664a08b8761fa155c9c229 P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:21.546301 750 tablet_service.cc:1514] Processing DeleteTablet for tablet efccefd6787943469131492741ee2bd5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:21 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:21.546347 1196 raft_consensus.cc:2238] T 9041b28402664a08b8761fa155c9c229 P df69f48656b44bb890f76587ee69cc0f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:21.546669 1196 raft_consensus.cc:2267] T 9041b28402664a08b8761fa155c9c229 P df69f48656b44bb890f76587ee69cc0f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:21.546927 1031 tablet_service.cc:1514] Processing DeleteTablet for tablet efccefd6787943469131492741ee2bd5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:21 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:21.547276 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet bffec36d567544c5a1c87e64a7e9730b
I20250113 02:51:21.547430 1196 ts_tablet_manager.cc:1905] T 9041b28402664a08b8761fa155c9c229 P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:21.548596 1031 tablet_service.cc:1514] Processing DeleteTablet for tablet bffec36d567544c5a1c87e64a7e9730b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:21 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:21.549170 750 tablet_service.cc:1514] Processing DeleteTablet for tablet bffec36d567544c5a1c87e64a7e9730b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:21 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:21.552531 750 tablet_service.cc:1514] Processing DeleteTablet for tablet c62a5e9e43d14b5c94eb87260528469d with delete_type TABLET_DATA_DELETED (replaced by tablet 6c8ca30720664f3a819c4bff2493934d) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:21.553794 881 tablet_service.cc:1514] Processing DeleteTablet for tablet efccefd6787943469131492741ee2bd5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:21 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:21.554459 881 tablet_service.cc:1514] Processing DeleteTablet for tablet bffec36d567544c5a1c87e64a7e9730b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:21 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:21.555349 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet 107e6e9e2d84496f8fce92b9ba377189 (table default.Table_74627d276edf4d75a2fcaa627a7f921b [id=de6b4835f6c04c20b6a5ef8404dc78f7]) successfully deleted
I20250113 02:51:21.556409 448 master_service.cc:946] ReplaceTablet: received request to replace tablet 18e2d79d731142f89b9089869bc736e5 from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:21.557191 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_12a092a4543e4b04a466945f4b694c53"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:21.557588 1192 tablet_bootstrap.cc:492] T d101e98b72544c30a3a4bc60d1576aa3 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:21.557832 1192 ts_tablet_manager.cc:1397] T d101e98b72544c30a3a4bc60d1576aa3 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.093s user 0.004s sys 0.001s
W20250113 02:51:21.557874 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_12a092a4543e4b04a466945f4b694c53 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:21.558861 1192 raft_consensus.cc:357] T d101e98b72544c30a3a4bc60d1576aa3 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:21.559159 1192 raft_consensus.cc:383] T d101e98b72544c30a3a4bc60d1576aa3 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:21.559250 1192 raft_consensus.cc:738] T d101e98b72544c30a3a4bc60d1576aa3 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:21.559808 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_3c637eb181574498b44ecb4fc41f0117"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:21.560449 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3c637eb181574498b44ecb4fc41f0117 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:21.559546 1192 consensus_queue.cc:260] T d101e98b72544c30a3a4bc60d1576aa3 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:21.562973 755 tablet_service.cc:1467] Processing CreateTablet for tablet df87e25053a74990959f85288a85217b (DEFAULT_TABLE table=default.Table_5cb0c9e84e72464c895d03166a1d18d7 [id=b2c8431e06ac46be9840edf78b6c19a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:21.566537 1196 ts_tablet_manager.cc:1918] T 9041b28402664a08b8761fa155c9c229 P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:21.566556 1031 tablet_service.cc:1514] Processing DeleteTablet for tablet d101e98b72544c30a3a4bc60d1576aa3 with delete_type TABLET_DATA_DELETED (replaced by tablet 5b3f012b88c04cb68ed7a1848147b70b) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:21.566676 1196 log.cc:1198] T 9041b28402664a08b8761fa155c9c229 P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/9041b28402664a08b8761fa155c9c229
I20250113 02:51:21.566920 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet 1500e4f9c0664a879503afd3edf14c66 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:21 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:21.567080 1196 ts_tablet_manager.cc:1939] T 9041b28402664a08b8761fa155c9c229 P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:21.567278 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df87e25053a74990959f85288a85217b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:21.567723 1192 ts_tablet_manager.cc:1428] T d101e98b72544c30a3a4bc60d1576aa3 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.010s user 0.002s sys 0.001s
I20250113 02:51:21.568114 1109 heartbeater.cc:502] Master 127.0.104.61:35271 was elected leader, sending a full tablet report...
I20250113 02:51:21.568562 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet 9041b28402664a08b8761fa155c9c229 (table default.Table_1b75b69f143d4fb2a97f31fd75657b96 [id=7e2f491b17c94a869837d457888bf71b]) successfully deleted
I20250113 02:51:21.568945 880 tablet_service.cc:1514] Processing DeleteTablet for tablet 1500e4f9c0664a879503afd3edf14c66 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:21 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:21.574818 1195 ts_tablet_manager.cc:1918] T 15a05a32febd443dbb519c186048428a P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:21.575030 1195 log.cc:1198] T 15a05a32febd443dbb519c186048428a P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/15a05a32febd443dbb519c186048428a
I20250113 02:51:21.575481 1195 ts_tablet_manager.cc:1939] T 15a05a32febd443dbb519c186048428a P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:21.577706 1196 tablet_replica.cc:331] T 2fb62cc83f9c4dcfa9aa01ae8b46c025 P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
W20250113 02:51:21.578001 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet be80b631ccd747f1b9355a00787a7b55 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet be80b631ccd747f1b9355a00787a7b55 already in progress: creating tablet
W20250113 02:51:21.578414 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 33c665e149dd48e8868acd935de581bf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 33c665e149dd48e8868acd935de581bf already in progress: creating tablet
W20250113 02:51:21.578727 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 64cda8796dff46ce95c37439395c77ec because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 64cda8796dff46ce95c37439395c77ec already in progress: creating tablet
W20250113 02:51:21.579123 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet b2a35fd891c645d68d79c61a2c832ba7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b2a35fd891c645d68d79c61a2c832ba7 already in progress: creating tablet
W20250113 02:51:21.579407 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet c94eedb5dce44e6d896192c93127903a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c94eedb5dce44e6d896192c93127903a already in progress: creating tablet
W20250113 02:51:21.579675 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet ff8ef6290698477391721fd65b9eeb28 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ff8ef6290698477391721fd65b9eeb28 already in progress: creating tablet
I20250113 02:51:21.580964 1196 raft_consensus.cc:2238] T 2fb62cc83f9c4dcfa9aa01ae8b46c025 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:21.581539 1196 raft_consensus.cc:2267] T 2fb62cc83f9c4dcfa9aa01ae8b46c025 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:21.581909 750 tablet_service.cc:1467] Processing CreateTablet for tablet 0b119704bc774ca9b4155c9619d4765b (DEFAULT_TABLE table=default.Table_707144e534a1463986168c22b636e208 [id=843aabcff7074acd98e5801b21282faa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:21.582336 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b119704bc774ca9b4155c9619d4765b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:21.582448 1196 ts_tablet_manager.cc:1905] T 2fb62cc83f9c4dcfa9aa01ae8b46c025 P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:21.587306 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 4efa63b8be794e1db4b340f4c0c4cd5a
I20250113 02:51:21.587915 452 catalog_manager.cc:6596] ReplaceTablet: tablet 4efa63b8be794e1db4b340f4c0c4cd5a deleted and replaced by tablet eceed0f52c9f46bbaa78ca773a7afde5
I20250113 02:51:21.588559 455 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet 234dab42f15f4043a7c74af4f267546b
I20250113 02:51:21.592711 741 tablet_service.cc:1514] Processing DeleteTablet for tablet 4efa63b8be794e1db4b340f4c0c4cd5a with delete_type TABLET_DATA_DELETED (replaced by tablet eceed0f52c9f46bbaa78ca773a7afde5) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:21.593441 1196 ts_tablet_manager.cc:1918] T 2fb62cc83f9c4dcfa9aa01ae8b46c025 P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:21.594419 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 7f3cbef7d91d4576822d913071a88248
I20250113 02:51:21.595713 741 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f3cbef7d91d4576822d913071a88248 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:21 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:21.596192 1196 log.cc:1198] T 2fb62cc83f9c4dcfa9aa01ae8b46c025 P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/2fb62cc83f9c4dcfa9aa01ae8b46c025
I20250113 02:51:21.596850 1196 ts_tablet_manager.cc:1939] T 2fb62cc83f9c4dcfa9aa01ae8b46c025 P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:21.598554 1196 tablet_replica.cc:331] T 076ed7744da54ddebd8fc99686beb6ae P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:21.600179 1196 raft_consensus.cc:2238] T 076ed7744da54ddebd8fc99686beb6ae P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:21.600939 1196 raft_consensus.cc:2267] T 076ed7744da54ddebd8fc99686beb6ae P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:21.600906 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 275) took 1057 ms. Trace:
I20250113 02:51:21.600457 881 tablet_service.cc:1514] Processing DeleteTablet for tablet 4efa63b8be794e1db4b340f4c0c4cd5a with delete_type TABLET_DATA_DELETED (replaced by tablet eceed0f52c9f46bbaa78ca773a7afde5) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:21.602720 1196 ts_tablet_manager.cc:1905] T 076ed7744da54ddebd8fc99686beb6ae P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:21.602671 1442 raft_consensus.cc:491] T 9b5dc06ba3ea4056822ca3e4b1249587 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:21.602932 1192 tablet_bootstrap.cc:492] T 3d3a6d9f687e45218d5875a7c5597648 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:21.607044 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f3cbef7d91d4576822d913071a88248 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:21 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:21.608071 1442 raft_consensus.cc:513] T 9b5dc06ba3ea4056822ca3e4b1249587 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:21.608798 1442 leader_election.cc:290] T 9b5dc06ba3ea4056822ca3e4b1249587 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:21.609477 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b5dc06ba3ea4056822ca3e4b1249587" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
I20250113 02:51:21.601042 1195 rpcz_store.cc:276] 0113 02:51:20.543579 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:20.543774 (+ 195us) service_pool.cc:225] Handling call
0113 02:51:21.422914 (+879140us) ts_tablet_manager.cc:1145] Deleting tablet 15a05a32febd443dbb519c186048428a
0113 02:51:21.422922 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:21.520302 (+ 97380us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:21.574636 (+ 54334us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:21.600834 (+ 26198us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":36891,"tablet-delete.queue_time_us":805269}
I20250113 02:51:21.612478 1031 tablet_service.cc:1514] Processing DeleteTablet for tablet 4efa63b8be794e1db4b340f4c0c4cd5a with delete_type TABLET_DATA_DELETED (replaced by tablet eceed0f52c9f46bbaa78ca773a7afde5) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:21.612545 1109 heartbeater.cc:502] Master 127.0.104.61:35271 was elected leader, sending a full tablet report...
I20250113 02:51:21.612936 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet a92f9b1c9c1a4cbdbf0cc7b2a3f2275b
I20250113 02:51:21.613399 447 catalog_manager.cc:6596] ReplaceTablet: tablet a92f9b1c9c1a4cbdbf0cc7b2a3f2275b deleted and replaced by tablet bc6a7322c0c34adb85457750526d8955
I20250113 02:51:21.613903 1192 tablet_bootstrap.cc:654] T 3d3a6d9f687e45218d5875a7c5597648 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
W20250113 02:51:21.614343 995 leader_election.cc:343] T 9b5dc06ba3ea4056822ca3e4b1249587 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:21.614167 1195 tablet_replica.cc:331] T fb183b4588e84212a3b4b096f430d47d P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:21.616251 1195 raft_consensus.cc:2238] T fb183b4588e84212a3b4b096f430d47d P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:21.616436 1195 raft_consensus.cc:2267] T fb183b4588e84212a3b4b096f430d47d P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:21.617405 1031 tablet_service.cc:1514] Processing DeleteTablet for tablet a92f9b1c9c1a4cbdbf0cc7b2a3f2275b with delete_type TABLET_DATA_DELETED (replaced by tablet bc6a7322c0c34adb85457750526d8955) from {username='slave'} at 127.0.0.1:60230
W20250113 02:51:21.617780 998 leader_election.cc:343] T 9b5dc06ba3ea4056822ca3e4b1249587 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Not found: Tablet not found: 9b5dc06ba3ea4056822ca3e4b1249587
I20250113 02:51:21.618438 741 tablet_service.cc:1514] Processing DeleteTablet for tablet a92f9b1c9c1a4cbdbf0cc7b2a3f2275b with delete_type TABLET_DATA_DELETED (replaced by tablet bc6a7322c0c34adb85457750526d8955) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:21.618830 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet 15a05a32febd443dbb519c186048428a (table default.Table_f9aa5df6ba3d43999ebd6570df3108d1 [id=fa5a1cda58f54103ac637c12d99e4e38]) successfully deleted
W20250113 02:51:21.619127 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet a97efc70946c4af487e33a85abf6b90c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a97efc70946c4af487e33a85abf6b90c already in progress: creating tablet
W20250113 02:51:21.619405 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet bcb59573e82743858bd39bde64627d6d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bcb59573e82743858bd39bde64627d6d already in progress: creating tablet
I20250113 02:51:21.619650 1196 ts_tablet_manager.cc:1918] T 076ed7744da54ddebd8fc99686beb6ae P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250113 02:51:21.619722 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet be80b631ccd747f1b9355a00787a7b55 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet be80b631ccd747f1b9355a00787a7b55 already in progress: creating tablet
I20250113 02:51:21.621927 1196 log.cc:1198] T 076ed7744da54ddebd8fc99686beb6ae P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/076ed7744da54ddebd8fc99686beb6ae
I20250113 02:51:21.622339 998 leader_election.cc:304] T 9b5dc06ba3ea4056822ca3e4b1249587 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4216a7f853d642b7bc3e5cff458f3aca; no voters: 1186fa869eb14812832fa6562531f196, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:21.622923 1196 ts_tablet_manager.cc:1939] T 076ed7744da54ddebd8fc99686beb6ae P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:21.623318 1442 raft_consensus.cc:2743] T 9b5dc06ba3ea4056822ca3e4b1249587 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:21.624147 881 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f3cbef7d91d4576822d913071a88248 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:21 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:21.624374 1195 ts_tablet_manager.cc:1905] T fb183b4588e84212a3b4b096f430d47d P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:21.625429 1196 tablet_replica.cc:331] T 18e2d79d731142f89b9089869bc736e5 P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
W20250113 02:51:21.625447 442 catalog_manager.cc:4839] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet fb183b4588e84212a3b4b096f430d47d because the tablet was not found. No further retry: Not found: Tablet not found: fb183b4588e84212a3b4b096f430d47d
I20250113 02:51:21.625684 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet 076ed7744da54ddebd8fc99686beb6ae (table default.Table_8b98cd7fbceb4d3f8b138e4da9bb0e05 [id=4100a37b30624d0bab2ec027b783be00]) successfully deleted
I20250113 02:51:21.626418 881 tablet_service.cc:1514] Processing DeleteTablet for tablet a92f9b1c9c1a4cbdbf0cc7b2a3f2275b with delete_type TABLET_DATA_DELETED (replaced by tablet bc6a7322c0c34adb85457750526d8955) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:21.626783 1196 raft_consensus.cc:2238] T 18e2d79d731142f89b9089869bc736e5 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:21.577394 880 tablet_service.cc:1467] Processing CreateTablet for tablet df87e25053a74990959f85288a85217b (DEFAULT_TABLE table=default.Table_5cb0c9e84e72464c895d03166a1d18d7 [id=b2c8431e06ac46be9840edf78b6c19a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:21.627107 1196 raft_consensus.cc:2267] T 18e2d79d731142f89b9089869bc736e5 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:21.627543 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df87e25053a74990959f85288a85217b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:21.628418 1196 ts_tablet_manager.cc:1905] T 18e2d79d731142f89b9089869bc736e5 P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:21.630366 1194 tablet_bootstrap.cc:654] T d35ec7e4a5084a74bea172ef8d9990b5 P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:21.632018 1023 tablet_service.cc:1467] Processing CreateTablet for tablet 0b119704bc774ca9b4155c9619d4765b (DEFAULT_TABLE table=default.Table_707144e534a1463986168c22b636e208 [id=843aabcff7074acd98e5801b21282faa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:21.632546 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b119704bc774ca9b4155c9619d4765b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:21.635345 1031 tablet_service.cc:1467] Processing CreateTablet for tablet f21eb3fa0f414816a29e3d4560cdc0fe (DEFAULT_TABLE table=default.Table_6a3d6ef326e74c4ab0a5ed1e1bec2ecc [id=02e7cb8431884f67989a39c2ffab28ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:21.635810 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f21eb3fa0f414816a29e3d4560cdc0fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:21.644584 881 tablet_service.cc:1467] Processing CreateTablet for tablet 0b119704bc774ca9b4155c9619d4765b (DEFAULT_TABLE table=default.Table_707144e534a1463986168c22b636e208 [id=843aabcff7074acd98e5801b21282faa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:21.645066 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b119704bc774ca9b4155c9619d4765b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:21.645550 750 tablet_service.cc:1467] Processing CreateTablet for tablet f21eb3fa0f414816a29e3d4560cdc0fe (DEFAULT_TABLE table=default.Table_6a3d6ef326e74c4ab0a5ed1e1bec2ecc [id=02e7cb8431884f67989a39c2ffab28ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:21.648018 1198 ts_tablet_manager.cc:1918] T 15a05a32febd443dbb519c186048428a P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:21.648192 1198 log.cc:1198] T 15a05a32febd443dbb519c186048428a P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/15a05a32febd443dbb519c186048428a
I20250113 02:51:21.648406 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f21eb3fa0f414816a29e3d4560cdc0fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:21.648664 1198 ts_tablet_manager.cc:1939] T 15a05a32febd443dbb519c186048428a P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:21.649467 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 262) took 1137 ms. Trace:
I20250113 02:51:21.650840 1031 tablet_service.cc:1467] Processing CreateTablet for tablet 4eb5f1944cb44fd5a40ca92a6a3d7214 (DEFAULT_TABLE table=default.Table_1065dceda1dd4999a868ec77b48f6ab3 [id=4f761dbea01a42e4aecf9f0367b8e743]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:21.651285 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4eb5f1944cb44fd5a40ca92a6a3d7214. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:21.651790 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet 98f038ad31ff4e05bb88ac68ae341cc2 with delete_type TABLET_DATA_DELETED (replaced by tablet f129bb3fbbfd492d98822be0a6626e44) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:21.656555 881 tablet_service.cc:1467] Processing CreateTablet for tablet f21eb3fa0f414816a29e3d4560cdc0fe (DEFAULT_TABLE table=default.Table_6a3d6ef326e74c4ab0a5ed1e1bec2ecc [id=02e7cb8431884f67989a39c2ffab28ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:21.657043 1191 tablet_bootstrap.cc:492] T 9041b28402664a08b8761fa155c9c229 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:21.657083 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f21eb3fa0f414816a29e3d4560cdc0fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:21.657740 1194 tablet_bootstrap.cc:492] T d35ec7e4a5084a74bea172ef8d9990b5 P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:21.657959 1194 ts_tablet_manager.cc:1397] T d35ec7e4a5084a74bea172ef8d9990b5 P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.189s user 0.004s sys 0.009s
I20250113 02:51:21.658847 1194 raft_consensus.cc:357] T d35ec7e4a5084a74bea172ef8d9990b5 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:21.659152 1194 raft_consensus.cc:383] T d35ec7e4a5084a74bea172ef8d9990b5 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:21.659786 1194 raft_consensus.cc:738] T d35ec7e4a5084a74bea172ef8d9990b5 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:21.660156 1194 consensus_queue.cc:260] T d35ec7e4a5084a74bea172ef8d9990b5 P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:21.649590 1198 rpcz_store.cc:276] 0113 02:51:20.511534 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:20.511578 (+ 44us) service_pool.cc:225] Handling call
0113 02:51:21.531760 (+1020182us) ts_tablet_manager.cc:1145] Deleting tablet 15a05a32febd443dbb519c186048428a
0113 02:51:21.531773 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:21.611182 (+ 79409us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:21.647836 (+ 36654us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:21.649435 (+ 1599us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1020006}
I20250113 02:51:21.663671 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet 15a05a32febd443dbb519c186048428a (table default.Table_f9aa5df6ba3d43999ebd6570df3108d1 [id=fa5a1cda58f54103ac637c12d99e4e38]) successfully deleted
I20250113 02:51:21.663516 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 258) took 1154 ms. Trace:
I20250113 02:51:21.663945 1198 rpcz_store.cc:276] 0113 02:51:20.509560 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:20.514329 (+ 4769us) service_pool.cc:225] Handling call
0113 02:51:21.663351 (+1149022us) ts_tablet_manager.cc:1145] Deleting tablet 23e04961f072440fa3e3f9485e40f79c
0113 02:51:21.663363 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:21.663485 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1148827}
I20250113 02:51:21.664139 1194 ts_tablet_manager.cc:1428] T d35ec7e4a5084a74bea172ef8d9990b5 P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250113 02:51:21.666535 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 265) took 1127 ms. Trace:
I20250113 02:51:21.667124 1198 rpcz_store.cc:276] 0113 02:51:20.538758 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:20.538817 (+ 59us) service_pool.cc:225] Handling call
0113 02:51:21.666356 (+1127539us) ts_tablet_manager.cc:1145] Deleting tablet 0810b9671f824d0a84741bc1904df247
0113 02:51:21.666370 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:21.666497 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1127302}
W20250113 02:51:21.667675 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 23e04961f072440fa3e3f9485e40f79c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 23e04961f072440fa3e3f9485e40f79c already in progress: creating tablet
I20250113 02:51:21.668089 1194 tablet_bootstrap.cc:492] T c71511a100f14322b5091a9804720fa0 P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
W20250113 02:51:21.668052 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 0810b9671f824d0a84741bc1904df247 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0810b9671f824d0a84741bc1904df247 already in progress: creating tablet
I20250113 02:51:21.668074 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 276) took 1101 ms. Trace:
I20250113 02:51:21.668208 1198 rpcz_store.cc:276] 0113 02:51:20.566702 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:20.572067 (+ 5365us) service_pool.cc:225] Handling call
0113 02:51:21.667873 (+1095806us) ts_tablet_manager.cc:1145] Deleting tablet c215ffe61b9b426286cc5e808463078d
0113 02:51:21.667881 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:21.668036 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1095606}
I20250113 02:51:21.670024 892 tablet_service.cc:1514] Processing DeleteTablet for tablet d35ec7e4a5084a74bea172ef8d9990b5 with delete_type TABLET_DATA_DELETED (replaced by tablet b4806b18a4194f9c95350d89ba62e13a) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:21.670176 1191 ts_tablet_manager.cc:1397] T 9041b28402664a08b8761fa155c9c229 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.149s user 0.019s sys 0.018s
I20250113 02:51:21.670977 1191 raft_consensus.cc:357] T 9041b28402664a08b8761fa155c9c229 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:21.671177 1191 raft_consensus.cc:383] T 9041b28402664a08b8761fa155c9c229 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:21.671272 1191 raft_consensus.cc:738] T 9041b28402664a08b8761fa155c9c229 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:21.672137 454 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_5cb0c9e84e72464c895d03166a1d18d7" } new_table_name: "default.Table_9613de5a21a8447eb2d5ab0076b77d5c" modify_external_catalogs: true
I20250113 02:51:21.673709 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_2d201bcde7f04c76b66dd5c6a09ddbff"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:21.676177 1191 consensus_queue.cc:260] T 9041b28402664a08b8761fa155c9c229 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
W20250113 02:51:21.677707 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet c215ffe61b9b426286cc5e808463078d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c215ffe61b9b426286cc5e808463078d already in progress: creating tablet
I20250113 02:51:21.677765 1194 tablet_bootstrap.cc:654] T c71511a100f14322b5091a9804720fa0 P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:21.679466 1191 ts_tablet_manager.cc:1428] T 9041b28402664a08b8761fa155c9c229 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20250113 02:51:21.679772 1191 tablet_bootstrap.cc:492] T 2fb62cc83f9c4dcfa9aa01ae8b46c025 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:21.681694 1191 tablet_bootstrap.cc:654] T 2fb62cc83f9c4dcfa9aa01ae8b46c025 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:21.683409 1198 tablet_replica.cc:331] T 9041b28402664a08b8761fa155c9c229 P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:21.683725 1198 raft_consensus.cc:2238] T 9041b28402664a08b8761fa155c9c229 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: Raft consensus shutting down.
I20250113 02:51:21.684092 1198 pending_rounds.cc:62] T 9041b28402664a08b8761fa155c9c229 P 4216a7f853d642b7bc3e5cff458f3aca: Trying to abort 1 pending ops.
I20250113 02:51:21.682775 755 tablet_service.cc:1514] Processing DeleteTablet for tablet 9041b28402664a08b8761fa155c9c229 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:20 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:21.685285 1196 ts_tablet_manager.cc:1918] T 18e2d79d731142f89b9089869bc736e5 P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:21.687434 1196 log.cc:1198] T 18e2d79d731142f89b9089869bc736e5 P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/18e2d79d731142f89b9089869bc736e5
W20250113 02:51:21.687832 454 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet df87e25053a74990959f85288a85217b consensus state has no leader
I20250113 02:51:21.688088 1195 ts_tablet_manager.cc:1918] T fb183b4588e84212a3b4b096f430d47d P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:21.688220 1195 log.cc:1198] T fb183b4588e84212a3b4b096f430d47d P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/fb183b4588e84212a3b4b096f430d47d
W20250113 02:51:21.676304 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2d201bcde7f04c76b66dd5c6a09ddbff in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:21.688231 1196 ts_tablet_manager.cc:1939] T 18e2d79d731142f89b9089869bc736e5 P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:21.688606 1195 ts_tablet_manager.cc:1939] T fb183b4588e84212a3b4b096f430d47d P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:21.689589 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet fb183b4588e84212a3b4b096f430d47d (table default.Table_53b46860bb68469fa08f3fd02b2408e5 [id=33bf7fd42ab5458db0f14347353c0db4]) successfully deleted
I20250113 02:51:21.684183 1198 pending_rounds.cc:69] T 9041b28402664a08b8761fa155c9c229 P 4216a7f853d642b7bc3e5cff458f3aca: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113673446682157056 op_type: NO_OP noop_request { }
I20250113 02:51:21.690646 1198 raft_consensus.cc:2883] T 9041b28402664a08b8761fa155c9c229 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250113 02:51:21.690001 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_367f2846b81e40eb82c5dd077f36017c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:21.690775 1438 raft_consensus.cc:491] T d35ec7e4a5084a74bea172ef8d9990b5 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:21.690953 1438 raft_consensus.cc:513] T d35ec7e4a5084a74bea172ef8d9990b5 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
W20250113 02:51:21.691378 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_367f2846b81e40eb82c5dd077f36017c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:21.691651 1438 leader_election.cc:290] T d35ec7e4a5084a74bea172ef8d9990b5 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587), 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
I20250113 02:51:21.691704 1198 raft_consensus.cc:2267] T 9041b28402664a08b8761fa155c9c229 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:21.692360 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d35ec7e4a5084a74bea172ef8d9990b5" candidate_uuid: "df69f48656b44bb890f76587ee69cc0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
W20250113 02:51:21.692967 852 leader_election.cc:343] T d35ec7e4a5084a74bea172ef8d9990b5 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:21.693593 1198 ts_tablet_manager.cc:1905] T 9041b28402664a08b8761fa155c9c229 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:21.693622 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d35ec7e4a5084a74bea172ef8d9990b5" candidate_uuid: "df69f48656b44bb890f76587ee69cc0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
W20250113 02:51:21.694298 851 leader_election.cc:343] T d35ec7e4a5084a74bea172ef8d9990b5 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:21.694458 851 leader_election.cc:304] T d35ec7e4a5084a74bea172ef8d9990b5 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df69f48656b44bb890f76587ee69cc0f; no voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca
I20250113 02:51:21.694940 1438 raft_consensus.cc:2743] T d35ec7e4a5084a74bea172ef8d9990b5 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:21.698776 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_65c5f2ca0aea479b9edd500ceb8c01bc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:21.699534 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_65c5f2ca0aea479b9edd500ceb8c01bc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250113 02:51:21.699813 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet c71511a100f14322b5091a9804720fa0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c71511a100f14322b5091a9804720fa0 already in progress: creating tablet
W20250113 02:51:21.700826 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet de4a58bf63054890808e2e2fe46e43d2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet de4a58bf63054890808e2e2fe46e43d2 already in progress: creating tablet
I20250113 02:51:21.704988 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_8b93113e8f8f489eb5030ebde0889e72" } new_table_name: "default.Table_32e88abaae96492fbcce39fb2cd7c221" modify_external_catalogs: true
W20250113 02:51:21.705657 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 33c665e149dd48e8868acd935de581bf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 33c665e149dd48e8868acd935de581bf already in progress: creating tablet
W20250113 02:51:21.707088 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet c94eedb5dce44e6d896192c93127903a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c94eedb5dce44e6d896192c93127903a already in progress: creating tablet
I20250113 02:51:21.709231 454 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_6167b33d8e4246cab5fbe98cb57e9a3c" } new_table_name: "default.Table_9a9b1ced3bef4e648088e0781851128e" modify_external_catalogs: true
W20250113 02:51:21.710141 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 64cda8796dff46ce95c37439395c77ec because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 64cda8796dff46ce95c37439395c77ec already in progress: creating tablet
W20250113 02:51:21.711360 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet b2a35fd891c645d68d79c61a2c832ba7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b2a35fd891c645d68d79c61a2c832ba7 already in progress: creating tablet
I20250113 02:51:21.711417 1196 tablet_replica.cc:331] T 7f3cbef7d91d4576822d913071a88248 P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:21.711609 1196 raft_consensus.cc:2238] T 7f3cbef7d91d4576822d913071a88248 P df69f48656b44bb890f76587ee69cc0f [term 1 FOLLOWER]: Raft consensus shutting down.
W20250113 02:51:21.712038 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet ff8ef6290698477391721fd65b9eeb28 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ff8ef6290698477391721fd65b9eeb28 already in progress: creating tablet
W20250113 02:51:21.712626 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1cfbcd682b944c89847b40fe69f0a75e consensus state has no leader
I20250113 02:51:21.712985 1191 tablet_bootstrap.cc:492] T 2fb62cc83f9c4dcfa9aa01ae8b46c025 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:21.713161 1191 ts_tablet_manager.cc:1397] T 2fb62cc83f9c4dcfa9aa01ae8b46c025 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.033s user 0.005s sys 0.001s
I20250113 02:51:21.713994 1191 raft_consensus.cc:357] T 2fb62cc83f9c4dcfa9aa01ae8b46c025 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:21.714460 1196 raft_consensus.cc:2267] T 7f3cbef7d91d4576822d913071a88248 P df69f48656b44bb890f76587ee69cc0f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:21.715586 1196 ts_tablet_manager.cc:1905] T 7f3cbef7d91d4576822d913071a88248 P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:21.716128 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_20ca05c407b14ba0a92f76f4596ae2ef" } new_table_name: "default.Table_addfcf0ba42e4d218b80a9748c704587" modify_external_catalogs: true
I20250113 02:51:21.715276 755 tablet_service.cc:1467] Processing CreateTablet for tablet 4eb5f1944cb44fd5a40ca92a6a3d7214 (DEFAULT_TABLE table=default.Table_1065dceda1dd4999a868ec77b48f6ab3 [id=4f761dbea01a42e4aecf9f0367b8e743]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:21.718631 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4eb5f1944cb44fd5a40ca92a6a3d7214. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:21.720379 454 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8adfa738c17a4169a5d99f6ceca75100 consensus state has no leader
I20250113 02:51:21.720834 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_059d10dd368b428db10509d40437287b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:21.722747 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 572e4eed1e924381bada27f8389821a7 consensus state has no leader
I20250113 02:51:21.724475 1023 tablet_service.cc:1467] Processing CreateTablet for tablet 9438472464304da9a0c7658d4f0736b6 (DEFAULT_TABLE table=default.Table_ac80d162769b4fc183613eb52f1bc4b7 [id=aaf6558a15124f2b8eb9f01d692b845d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:21.724910 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9438472464304da9a0c7658d4f0736b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:21.716185 1195 tablet_replica.cc:331] T 3455f010e51b4db1a9804a57fd1a4fb8 P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:21.725378 1195 raft_consensus.cc:2238] T 3455f010e51b4db1a9804a57fd1a4fb8 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:21.725546 1195 raft_consensus.cc:2267] T 3455f010e51b4db1a9804a57fd1a4fb8 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:21.726102 1195 ts_tablet_manager.cc:1905] T 3455f010e51b4db1a9804a57fd1a4fb8 P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:21.727643 447 master_service.cc:946] ReplaceTablet: received request to replace tablet c826a0b8327f44f9a00e5afe3a36cd8a from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:21.728096 1191 raft_consensus.cc:383] T 2fb62cc83f9c4dcfa9aa01ae8b46c025 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:21.728355 1191 raft_consensus.cc:738] T 2fb62cc83f9c4dcfa9aa01ae8b46c025 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:21.728761 1191 consensus_queue.cc:260] T 2fb62cc83f9c4dcfa9aa01ae8b46c025 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:21.729970 454 master_service.cc:946] ReplaceTablet: received request to replace tablet 2c399784af46426b8327d175c7a6cce8 from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:21.733603 1191 ts_tablet_manager.cc:1428] T 2fb62cc83f9c4dcfa9aa01ae8b46c025 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.020s user 0.001s sys 0.001s
I20250113 02:51:21.734416 1191 tablet_bootstrap.cc:492] T 98f038ad31ff4e05bb88ac68ae341cc2 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:21.740839 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet c826a0b8327f44f9a00e5afe3a36cd8a
I20250113 02:51:21.741436 447 catalog_manager.cc:6596] ReplaceTablet: tablet c826a0b8327f44f9a00e5afe3a36cd8a deleted and replaced by tablet bc533518faa24dffa3306ff092e8d507
I20250113 02:51:21.741580 1033 tablet_service.cc:1514] Processing DeleteTablet for tablet c826a0b8327f44f9a00e5afe3a36cd8a with delete_type TABLET_DATA_DELETED (replaced by tablet bc533518faa24dffa3306ff092e8d507) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:21.742594 1191 tablet_bootstrap.cc:654] T 98f038ad31ff4e05bb88ac68ae341cc2 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:21.742563 741 tablet_service.cc:1514] Processing DeleteTablet for tablet c826a0b8327f44f9a00e5afe3a36cd8a with delete_type TABLET_DATA_DELETED (replaced by tablet bc533518faa24dffa3306ff092e8d507) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:21.743497 755 tablet_service.cc:1514] Processing DeleteTablet for tablet 2fb62cc83f9c4dcfa9aa01ae8b46c025 with delete_type TABLET_DATA_DELETED (replaced by tablet 234dab42f15f4043a7c74af4f267546b) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:21.748258 1196 ts_tablet_manager.cc:1918] T 7f3cbef7d91d4576822d913071a88248 P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250113 02:51:21.751185 755 tablet_service.cc:1467] Processing CreateTablet for tablet 9438472464304da9a0c7658d4f0736b6 (DEFAULT_TABLE table=default.Table_ac80d162769b4fc183613eb52f1bc4b7 [id=aaf6558a15124f2b8eb9f01d692b845d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:21.751686 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9438472464304da9a0c7658d4f0736b6. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:21.732347 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_059d10dd368b428db10509d40437287b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:21.754042 1198 ts_tablet_manager.cc:1918] T 9041b28402664a08b8761fa155c9c229 P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250113 02:51:21.755934 741 tablet_service.cc:1467] Processing CreateTablet for tablet d8bb60fdd06946739b75b255303d9d24 (DEFAULT_TABLE table=default.Table_ba72b5bfcbe6447196e4d8640ab0ff65 [id=7c84304f6d1e4b4ea99f0bc38215d74c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:21.756387 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8bb60fdd06946739b75b255303d9d24. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:21.755932 1023 tablet_service.cc:1467] Processing CreateTablet for tablet d8bb60fdd06946739b75b255303d9d24 (DEFAULT_TABLE table=default.Table_ba72b5bfcbe6447196e4d8640ab0ff65 [id=7c84304f6d1e4b4ea99f0bc38215d74c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:21.757089 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8bb60fdd06946739b75b255303d9d24. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:21.758704 1191 tablet_bootstrap.cc:492] T 98f038ad31ff4e05bb88ac68ae341cc2 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:21.758859 1191 ts_tablet_manager.cc:1397] T 98f038ad31ff4e05bb88ac68ae341cc2 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.025s user 0.000s sys 0.004s
I20250113 02:51:21.759620 1191 raft_consensus.cc:357] T 98f038ad31ff4e05bb88ac68ae341cc2 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:21.759825 1191 raft_consensus.cc:383] T 98f038ad31ff4e05bb88ac68ae341cc2 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:21.759925 1191 raft_consensus.cc:738] T 98f038ad31ff4e05bb88ac68ae341cc2 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:21.760071 1198 log.cc:1198] T 9041b28402664a08b8761fa155c9c229 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/9041b28402664a08b8761fa155c9c229
I20250113 02:51:21.760160 1191 consensus_queue.cc:260] T 98f038ad31ff4e05bb88ac68ae341cc2 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:21.760535 1198 ts_tablet_manager.cc:1939] T 9041b28402664a08b8761fa155c9c229 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:21.761323 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 277) took 1194 ms. Trace:
I20250113 02:51:21.762126 1196 log.cc:1198] T 7f3cbef7d91d4576822d913071a88248 P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/7f3cbef7d91d4576822d913071a88248
I20250113 02:51:21.762820 1196 ts_tablet_manager.cc:1939] T 7f3cbef7d91d4576822d913071a88248 P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:21.763408 1195 ts_tablet_manager.cc:1918] T 3455f010e51b4db1a9804a57fd1a4fb8 P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:21.763842 1198 rpcz_store.cc:276] 0113 02:51:20.566971 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:20.572311 (+ 5340us) service_pool.cc:225] Handling call
0113 02:51:21.676547 (+1104236us) ts_tablet_manager.cc:1145] Deleting tablet 9041b28402664a08b8761fa155c9c229
0113 02:51:21.676559 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:21.719675 (+ 43116us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:21.749623 (+ 29948us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:21.761288 (+ 11665us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1104062}
I20250113 02:51:21.765995 1195 log.cc:1198] T 3455f010e51b4db1a9804a57fd1a4fb8 P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/3455f010e51b4db1a9804a57fd1a4fb8
I20250113 02:51:21.766358 1195 ts_tablet_manager.cc:1939] T 3455f010e51b4db1a9804a57fd1a4fb8 P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:21.767617 1191 ts_tablet_manager.cc:1428] T 98f038ad31ff4e05bb88ac68ae341cc2 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20250113 02:51:21.768500 1191 tablet_bootstrap.cc:492] T 9b5dc06ba3ea4056822ca3e4b1249587 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:21.769690 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet 3455f010e51b4db1a9804a57fd1a4fb8 (table default.Table_44f1c4e8cc2b4e83adce8d411ae53448 [id=a9e0aae9f5af42b796d29a75140f9c29]) successfully deleted
I20250113 02:51:21.772256 1196 tablet_replica.cc:331] T d35ec7e4a5084a74bea172ef8d9990b5 P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:21.772716 1195 tablet_replica.cc:331] T 8193357427f4491c8c6c1396ceeef2dc P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:21.773022 1196 raft_consensus.cc:2238] T d35ec7e4a5084a74bea172ef8d9990b5 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:21.773509 1195 raft_consensus.cc:2238] T 8193357427f4491c8c6c1396ceeef2dc P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:21.773835 1196 raft_consensus.cc:2267] T d35ec7e4a5084a74bea172ef8d9990b5 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:21.774737 1195 raft_consensus.cc:2267] T 8193357427f4491c8c6c1396ceeef2dc P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:21.774966 892 tablet_service.cc:1467] Processing CreateTablet for tablet 4eb5f1944cb44fd5a40ca92a6a3d7214 (DEFAULT_TABLE table=default.Table_1065dceda1dd4999a868ec77b48f6ab3 [id=4f761dbea01a42e4aecf9f0367b8e743]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:21.775401 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4eb5f1944cb44fd5a40ca92a6a3d7214. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:21.775386 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet 9041b28402664a08b8761fa155c9c229 (table default.Table_1b75b69f143d4fb2a97f31fd75657b96 [id=7e2f491b17c94a869837d457888bf71b]) successfully deleted
I20250113 02:51:21.775785 1195 ts_tablet_manager.cc:1905] T 8193357427f4491c8c6c1396ceeef2dc P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:21.775899 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 281) took 1066 ms. Trace:
I20250113 02:51:21.776052 1198 rpcz_store.cc:276] 0113 02:51:20.727544 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:20.727615 (+ 71us) service_pool.cc:225] Handling call
0113 02:51:21.775729 (+1048114us) ts_tablet_manager.cc:1145] Deleting tablet a97efc70946c4af487e33a85abf6b90c
0113 02:51:21.775741 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:21.775849 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1041507}
W20250113 02:51:21.776515 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet a97efc70946c4af487e33a85abf6b90c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a97efc70946c4af487e33a85abf6b90c already in progress: creating tablet
I20250113 02:51:21.777137 1196 ts_tablet_manager.cc:1905] T d35ec7e4a5084a74bea172ef8d9990b5 P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:21.779572 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 283) took 1045 ms. Trace:
I20250113 02:51:21.779995 1198 rpcz_store.cc:276] 0113 02:51:20.733802 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:20.734012 (+ 210us) service_pool.cc:225] Handling call
0113 02:51:21.779443 (+1045431us) ts_tablet_manager.cc:1145] Deleting tablet bcb59573e82743858bd39bde64627d6d
0113 02:51:21.779453 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:21.779541 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1041680}
I20250113 02:51:21.780447 1191 tablet_bootstrap.cc:654] T 9b5dc06ba3ea4056822ca3e4b1249587 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
W20250113 02:51:21.780900 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet bcb59573e82743858bd39bde64627d6d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bcb59573e82743858bd39bde64627d6d already in progress: creating tablet
I20250113 02:51:21.781303 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 287) took 1040 ms. Trace:
I20250113 02:51:21.782040 1198 rpcz_store.cc:276] 0113 02:51:20.740926 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:20.740983 (+ 57us) service_pool.cc:225] Handling call
0113 02:51:21.781176 (+1040193us) ts_tablet_manager.cc:1145] Deleting tablet c71511a100f14322b5091a9804720fa0
0113 02:51:21.781186 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:21.781273 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1028592}
I20250113 02:51:21.784365 741 tablet_service.cc:1514] Processing DeleteTablet for tablet 98f038ad31ff4e05bb88ac68ae341cc2 with delete_type TABLET_DATA_DELETED (replaced by tablet f129bb3fbbfd492d98822be0a6626e44) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:21.784796 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 2c399784af46426b8327d175c7a6cce8
W20250113 02:51:21.784967 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet c71511a100f14322b5091a9804720fa0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c71511a100f14322b5091a9804720fa0 already in progress: creating tablet
I20250113 02:51:21.785413 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 282) took 1054 ms. Trace:
I20250113 02:51:21.785677 741 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c399784af46426b8327d175c7a6cce8 with delete_type TABLET_DATA_DELETED (replaced by tablet 9e736211eb3c4536b9fd1d90d2d83838) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:21.785852 1198 rpcz_store.cc:276] 0113 02:51:20.731285 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:20.731371 (+ 86us) service_pool.cc:225] Handling call
0113 02:51:21.785288 (+1053917us) ts_tablet_manager.cc:1145] Deleting tablet be80b631ccd747f1b9355a00787a7b55
0113 02:51:21.785297 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:21.785385 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1032551}
I20250113 02:51:21.785948 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet 18e2d79d731142f89b9089869bc736e5 (table default.Table_f9aa5df6ba3d43999ebd6570df3108d1 [id=fa5a1cda58f54103ac637c12d99e4e38]) successfully deleted
W20250113 02:51:21.786334 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet efccefd6787943469131492741ee2bd5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet efccefd6787943469131492741ee2bd5 already in progress: creating tablet
W20250113 02:51:21.786528 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet bffec36d567544c5a1c87e64a7e9730b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bffec36d567544c5a1c87e64a7e9730b already in progress: creating tablet
W20250113 02:51:21.786752 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 1500e4f9c0664a879503afd3edf14c66 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1500e4f9c0664a879503afd3edf14c66 already in progress: creating tablet
W20250113 02:51:21.786854 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet be80b631ccd747f1b9355a00787a7b55 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet be80b631ccd747f1b9355a00787a7b55 already in progress: creating tablet
W20250113 02:51:21.786974 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 4efa63b8be794e1db4b340f4c0c4cd5a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4efa63b8be794e1db4b340f4c0c4cd5a already in progress: creating tablet
I20250113 02:51:21.787149 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet 7f3cbef7d91d4576822d913071a88248 (table default.Table_b581b1a063a54551a8535adf01983fb8 [id=54e4e3810b3b412bbec3b5bf5b28c1a2]) successfully deleted
W20250113 02:51:21.787401 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet a92f9b1c9c1a4cbdbf0cc7b2a3f2275b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a92f9b1c9c1a4cbdbf0cc7b2a3f2275b already in progress: creating tablet
I20250113 02:51:21.788080 892 tablet_service.cc:1514] Processing DeleteTablet for tablet c826a0b8327f44f9a00e5afe3a36cd8a with delete_type TABLET_DATA_DELETED (replaced by tablet bc533518faa24dffa3306ff092e8d507) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:21.788200 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 288) took 1032 ms. Trace:
I20250113 02:51:21.788544 1198 rpcz_store.cc:276] 0113 02:51:20.755288 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:20.755341 (+ 53us) service_pool.cc:225] Handling call
0113 02:51:21.787368 (+1032027us) ts_tablet_manager.cc:1145] Deleting tablet de4a58bf63054890808e2e2fe46e43d2
0113 02:51:21.787380 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:21.788156 (+ 776us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1031829}
W20250113 02:51:21.789610 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet de4a58bf63054890808e2e2fe46e43d2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet de4a58bf63054890808e2e2fe46e43d2 already in progress: creating tablet
I20250113 02:51:21.790163 1198 tablet_replica.cc:331] T fb183b4588e84212a3b4b096f430d47d P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:21.791743 1198 raft_consensus.cc:2238] T fb183b4588e84212a3b4b096f430d47d P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:21.792237 1198 raft_consensus.cc:2267] T fb183b4588e84212a3b4b096f430d47d P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:21.793640 1195 ts_tablet_manager.cc:1918] T 8193357427f4491c8c6c1396ceeef2dc P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:21.793167 1192 tablet_bootstrap.cc:492] T 3d3a6d9f687e45218d5875a7c5597648 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:21.793758 1195 log.cc:1198] T 8193357427f4491c8c6c1396ceeef2dc P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/8193357427f4491c8c6c1396ceeef2dc
I20250113 02:51:21.794112 1195 ts_tablet_manager.cc:1939] T 8193357427f4491c8c6c1396ceeef2dc P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:21.794207 1192 ts_tablet_manager.cc:1397] T 3d3a6d9f687e45218d5875a7c5597648 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.212s user 0.012s sys 0.013s
I20250113 02:51:21.795148 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet 8193357427f4491c8c6c1396ceeef2dc (table default.Table_8b87c8469fbb465e80375aa0446de6c2 [id=f85f0c435ffa4fdca81d86de4aa4fd5f]) successfully deleted
I20250113 02:51:21.795238 1192 raft_consensus.cc:357] T 3d3a6d9f687e45218d5875a7c5597648 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:21.796363 1192 raft_consensus.cc:383] T 3d3a6d9f687e45218d5875a7c5597648 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:21.796509 1192 raft_consensus.cc:738] T 3d3a6d9f687e45218d5875a7c5597648 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:21.796741 454 catalog_manager.cc:6596] ReplaceTablet: tablet 2c399784af46426b8327d175c7a6cce8 deleted and replaced by tablet 9e736211eb3c4536b9fd1d90d2d83838
I20250113 02:51:21.796797 1192 consensus_queue.cc:260] T 3d3a6d9f687e45218d5875a7c5597648 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:21.799253 1109 heartbeater.cc:502] Master 127.0.104.61:35271 was elected leader, sending a full tablet report...
I20250113 02:51:21.799790 1196 ts_tablet_manager.cc:1918] T d35ec7e4a5084a74bea172ef8d9990b5 P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:21.799928 1196 log.cc:1198] T d35ec7e4a5084a74bea172ef8d9990b5 P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/d35ec7e4a5084a74bea172ef8d9990b5
I20250113 02:51:21.800835 1196 ts_tablet_manager.cc:1939] T d35ec7e4a5084a74bea172ef8d9990b5 P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:21.803181 1033 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d3a6d9f687e45218d5875a7c5597648 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:17 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:21.807610 1195 tablet_replica.cc:331] T 0c55662a71444a3683a825bcec6c5b68 P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:21.807822 1195 raft_consensus.cc:2238] T 0c55662a71444a3683a825bcec6c5b68 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:21.808025 1195 raft_consensus.cc:2267] T 0c55662a71444a3683a825bcec6c5b68 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:21.808588 1195 ts_tablet_manager.cc:1905] T 0c55662a71444a3683a825bcec6c5b68 P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:21.812456 1192 ts_tablet_manager.cc:1428] T 3d3a6d9f687e45218d5875a7c5597648 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.018s user 0.001s sys 0.001s
I20250113 02:51:21.815532 880 tablet_service.cc:1467] Processing CreateTablet for tablet 9438472464304da9a0c7658d4f0736b6 (DEFAULT_TABLE table=default.Table_ac80d162769b4fc183613eb52f1bc4b7 [id=aaf6558a15124f2b8eb9f01d692b845d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:21.815793 447 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_5175e117e3f34778a5ef456bed6b5e28" } modify_external_catalogs: true
I20250113 02:51:21.816047 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9438472464304da9a0c7658d4f0736b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:21.816085 447 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_5175e117e3f34778a5ef456bed6b5e28" } modify_external_catalogs: true
I20250113 02:51:21.816812 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_b89481a4a7be4e49b13f3260d8a31d5d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:21.817736 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b89481a4a7be4e49b13f3260d8a31d5d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:21.821957 446 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_abd20980e42242719c1529668d68128b" } modify_external_catalogs: true
I20250113 02:51:21.822182 1194 tablet_bootstrap.cc:492] T c71511a100f14322b5091a9804720fa0 P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:21.822135 446 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_abd20980e42242719c1529668d68128b" } modify_external_catalogs: true
I20250113 02:51:21.822347 1194 ts_tablet_manager.cc:1397] T c71511a100f14322b5091a9804720fa0 P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.154s user 0.019s sys 0.007s
I20250113 02:51:21.823109 1194 raft_consensus.cc:357] T c71511a100f14322b5091a9804720fa0 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:21.823331 1194 raft_consensus.cc:383] T c71511a100f14322b5091a9804720fa0 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:21.823410 1194 raft_consensus.cc:738] T c71511a100f14322b5091a9804720fa0 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:21.823655 1194 consensus_queue.cc:260] T c71511a100f14322b5091a9804720fa0 P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:21.818884 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c399784af46426b8327d175c7a6cce8 with delete_type TABLET_DATA_DELETED (replaced by tablet 9e736211eb3c4536b9fd1d90d2d83838) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:21.830278 455 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_06f81a97cb014a3bad594e88c8cf4733" } modify_external_catalogs: true
I20250113 02:51:21.830507 455 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_06f81a97cb014a3bad594e88c8cf4733" } modify_external_catalogs: true
I20250113 02:51:21.830732 1194 ts_tablet_manager.cc:1428] T c71511a100f14322b5091a9804720fa0 P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20250113 02:51:21.833496 451 master_service.cc:946] ReplaceTablet: received request to replace tablet 2c399784af46426b8327d175c7a6cce8 from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:21.833979 1194 tablet_bootstrap.cc:492] T 1d81eec487dd4016b3b804451da31c49 P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:21.833992 1192 tablet_bootstrap.cc:492] T d35ec7e4a5084a74bea172ef8d9990b5 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:21.834457 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_93a7b6abed0446a4baf21e5e7c8f0c1b" } modify_external_catalogs: true
I20250113 02:51:21.834632 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_93a7b6abed0446a4baf21e5e7c8f0c1b" } modify_external_catalogs: true
I20250113 02:51:21.834462 1198 ts_tablet_manager.cc:1905] T fb183b4588e84212a3b4b096f430d47d P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:21.835328 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_b5da750f5930453fa3951ffba96c34d4" } modify_external_catalogs: true
I20250113 02:51:21.835480 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_b5da750f5930453fa3951ffba96c34d4" } modify_external_catalogs: true
I20250113 02:51:21.836141 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_a7c3697e25a94815ac03a3cb1a57bc0c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:21.836699 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a7c3697e25a94815ac03a3cb1a57bc0c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:21.838035 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_437ebdf134da491e947f52a1bcc89f01"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:21.838491 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_437ebdf134da491e947f52a1bcc89f01 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:21.839677 1194 tablet_bootstrap.cc:654] T 1d81eec487dd4016b3b804451da31c49 P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:21.841166 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet eebad045e7bb43fa9ca9c9af8b3b7ea5
I20250113 02:51:21.842469 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet eebad045e7bb43fa9ca9c9af8b3b7ea5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:21 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:21.844609 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 7d3562a48c6e4badbbcf37640e91b5ba
I20250113 02:51:21.845970 741 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d3562a48c6e4badbbcf37640e91b5ba with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:21 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:21.848105 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d3562a48c6e4badbbcf37640e91b5ba with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:21 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:21.845980 1195 ts_tablet_manager.cc:1918] T 0c55662a71444a3683a825bcec6c5b68 P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:21.849135 1195 log.cc:1198] T 0c55662a71444a3683a825bcec6c5b68 P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/0c55662a71444a3683a825bcec6c5b68
I20250113 02:51:21.849529 1195 ts_tablet_manager.cc:1939] T 0c55662a71444a3683a825bcec6c5b68 P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:21.850487 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet 0c55662a71444a3683a825bcec6c5b68 (table default.Table_3865c1517889457a87be9ea12b3a156f [id=79b4c84e05ce4571b42e2b4e40fae978]) successfully deleted
I20250113 02:51:21.851483 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_0a251c1b92404b18a7dbce48963cdede" } new_table_name: "default.Table_4484fe40b4454c5ea4943a4d38f49c5e" modify_external_catalogs: true
I20250113 02:51:21.852327 1109 heartbeater.cc:502] Master 127.0.104.61:35271 was elected leader, sending a full tablet report...
I20250113 02:51:21.853262 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_3a73b4672b154b99af1a24e962007ea0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:21.854188 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 1500e4f9c0664a879503afd3edf14c66 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1500e4f9c0664a879503afd3edf14c66 already in progress: creating tablet
I20250113 02:51:21.852486 1192 tablet_bootstrap.cc:654] T d35ec7e4a5084a74bea172ef8d9990b5 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
W20250113 02:51:21.855583 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet efccefd6787943469131492741ee2bd5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet efccefd6787943469131492741ee2bd5 already in progress: creating tablet
I20250113 02:51:21.855836 1195 tablet_replica.cc:331] T c62a5e9e43d14b5c94eb87260528469d P 1186fa869eb14812832fa6562531f196: stopping tablet replica
W20250113 02:51:21.855849 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet bffec36d567544c5a1c87e64a7e9730b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bffec36d567544c5a1c87e64a7e9730b already in progress: creating tablet
I20250113 02:51:21.856062 1195 raft_consensus.cc:2238] T c62a5e9e43d14b5c94eb87260528469d P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:21.856251 1195 raft_consensus.cc:2267] T c62a5e9e43d14b5c94eb87260528469d P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250113 02:51:21.856549 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3a73b4672b154b99af1a24e962007ea0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:21.856792 1195 ts_tablet_manager.cc:1905] T c62a5e9e43d14b5c94eb87260528469d P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:21.860433 880 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c399784af46426b8327d175c7a6cce8 with delete_type TABLET_DATA_DELETED (replaced by tablet 9e736211eb3c4536b9fd1d90d2d83838) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:21.862502 892 tablet_service.cc:1514] Processing DeleteTablet for tablet eebad045e7bb43fa9ca9c9af8b3b7ea5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:21 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:21.863395 892 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d3562a48c6e4badbbcf37640e91b5ba with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:21 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:21.866487 750 tablet_service.cc:1514] Processing DeleteTablet for tablet eebad045e7bb43fa9ca9c9af8b3b7ea5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:21 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:21.859176 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_25a2c280183748df92b064c40756f5ac"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:21.868111 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_25a2c280183748df92b064c40756f5ac in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:21.868417 881 tablet_service.cc:1467] Processing CreateTablet for tablet d8bb60fdd06946739b75b255303d9d24 (DEFAULT_TABLE table=default.Table_ba72b5bfcbe6447196e4d8640ab0ff65 [id=7c84304f6d1e4b4ea99f0bc38215d74c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:21.868857 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8bb60fdd06946739b75b255303d9d24. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:21.859567 448 master_service.cc:946] ReplaceTablet: received request to replace tablet 21c4b419bf2148e0b140301fa80622bb from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:21.871397 880 tablet_service.cc:1514] Processing DeleteTablet for tablet c71511a100f14322b5091a9804720fa0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:20 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:21.871740 1196 tablet_replica.cc:331] T c71511a100f14322b5091a9804720fa0 P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:21.871956 1196 raft_consensus.cc:2238] T c71511a100f14322b5091a9804720fa0 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:21.872196 1196 raft_consensus.cc:2267] T c71511a100f14322b5091a9804720fa0 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
W20250113 02:51:21.872309 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f8a6eebb6b6d4b63903e68750e181707 has no consensus state
I20250113 02:51:21.867070 1191 tablet_bootstrap.cc:492] T 9b5dc06ba3ea4056822ca3e4b1249587 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:21.872639 1191 ts_tablet_manager.cc:1397] T 9b5dc06ba3ea4056822ca3e4b1249587 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.104s user 0.006s sys 0.013s
I20250113 02:51:21.873070 1196 ts_tablet_manager.cc:1905] T c71511a100f14322b5091a9804720fa0 P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:21.873553 1191 raft_consensus.cc:357] T 9b5dc06ba3ea4056822ca3e4b1249587 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:21.873793 1191 raft_consensus.cc:383] T 9b5dc06ba3ea4056822ca3e4b1249587 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:21.873867 1191 raft_consensus.cc:738] T 9b5dc06ba3ea4056822ca3e4b1249587 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:21.874387 1191 consensus_queue.cc:260] T 9b5dc06ba3ea4056822ca3e4b1249587 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:21.875219 1191 ts_tablet_manager.cc:1428] T 9b5dc06ba3ea4056822ca3e4b1249587 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250113 02:51:21.869976 1195 ts_tablet_manager.cc:1918] T c62a5e9e43d14b5c94eb87260528469d P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:21.875684 1191 tablet_bootstrap.cc:492] T 3d3a6d9f687e45218d5875a7c5597648 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:21.877359 1192 tablet_bootstrap.cc:492] T d35ec7e4a5084a74bea172ef8d9990b5 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:21.877522 1192 ts_tablet_manager.cc:1397] T d35ec7e4a5084a74bea172ef8d9990b5 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.044s user 0.002s sys 0.001s
I20250113 02:51:21.879174 1195 log.cc:1198] T c62a5e9e43d14b5c94eb87260528469d P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/c62a5e9e43d14b5c94eb87260528469d
I20250113 02:51:21.879247 1192 raft_consensus.cc:357] T d35ec7e4a5084a74bea172ef8d9990b5 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:21.879487 1192 raft_consensus.cc:383] T d35ec7e4a5084a74bea172ef8d9990b5 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:21.879575 1192 raft_consensus.cc:738] T d35ec7e4a5084a74bea172ef8d9990b5 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:21.879843 1192 consensus_queue.cc:260] T d35ec7e4a5084a74bea172ef8d9990b5 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:21.880162 1191 tablet_bootstrap.cc:654] T 3d3a6d9f687e45218d5875a7c5597648 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:21.882961 1195 ts_tablet_manager.cc:1939] T c62a5e9e43d14b5c94eb87260528469d P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:21.885149 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet c62a5e9e43d14b5c94eb87260528469d (table default.Table_8aa63512c7bb4bdb86f37c9ee24df209 [id=aadd55280b654ceb96c290786a0bfce0]) successfully deleted
W20250113 02:51:21.885579 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 4efa63b8be794e1db4b340f4c0c4cd5a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4efa63b8be794e1db4b340f4c0c4cd5a already in progress: creating tablet
I20250113 02:51:21.885932 1195 tablet_replica.cc:331] T 7f3cbef7d91d4576822d913071a88248 P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:21.886164 1195 raft_consensus.cc:2238] T 7f3cbef7d91d4576822d913071a88248 P 1186fa869eb14812832fa6562531f196 [term 1 LEADER]: Raft consensus shutting down.
I20250113 02:51:21.886487 1195 raft_consensus.cc:2267] T 7f3cbef7d91d4576822d913071a88248 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:21.887123 1195 ts_tablet_manager.cc:1905] T 7f3cbef7d91d4576822d913071a88248 P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:21.887539 1192 ts_tablet_manager.cc:1428] T d35ec7e4a5084a74bea172ef8d9990b5 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.010s user 0.003s sys 0.000s
I20250113 02:51:21.887662 1198 ts_tablet_manager.cc:1918] T fb183b4588e84212a3b4b096f430d47d P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:21.887945 1198 log.cc:1198] T fb183b4588e84212a3b4b096f430d47d P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/fb183b4588e84212a3b4b096f430d47d
I20250113 02:51:21.888407 1198 ts_tablet_manager.cc:1939] T fb183b4588e84212a3b4b096f430d47d P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:21.865012 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet d35ec7e4a5084a74bea172ef8d9990b5 (table default.Table_4369a143c72643b084df58d25314c6dd [id=36ff239c08a54be99ca307354810fff3]) successfully deleted
W20250113 02:51:21.896314 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet c826a0b8327f44f9a00e5afe3a36cd8a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c826a0b8327f44f9a00e5afe3a36cd8a already in progress: creating tablet
I20250113 02:51:21.897132 750 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b5dc06ba3ea4056822ca3e4b1249587 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:18 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:21.888818 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet d35ec7e4a5084a74bea172ef8d9990b5 with delete_type TABLET_DATA_DELETED (replaced by tablet b4806b18a4194f9c95350d89ba62e13a) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:21.892458 1192 tablet_bootstrap.cc:492] T d5bd09a1ede644c59e400e49c8125793 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
W20250113 02:51:21.897974 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 2c399784af46426b8327d175c7a6cce8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2c399784af46426b8327d175c7a6cce8 already in progress: creating tablet
W20250113 02:51:21.898281 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet eebad045e7bb43fa9ca9c9af8b3b7ea5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet eebad045e7bb43fa9ca9c9af8b3b7ea5 already in progress: creating tablet
W20250113 02:51:21.898533 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 7d3562a48c6e4badbbcf37640e91b5ba because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7d3562a48c6e4badbbcf37640e91b5ba already in progress: creating tablet
I20250113 02:51:21.865495 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_6716ec8ca07b476fbaa9480c4d821568"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:21.899688 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6716ec8ca07b476fbaa9480c4d821568 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:21.900287 1192 tablet_bootstrap.cc:654] T d5bd09a1ede644c59e400e49c8125793 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:21.900580 1195 ts_tablet_manager.cc:1918] T 7f3cbef7d91d4576822d913071a88248 P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250113 02:51:21.900767 1195 log.cc:1198] T 7f3cbef7d91d4576822d913071a88248 P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/7f3cbef7d91d4576822d913071a88248
I20250113 02:51:21.901264 1195 ts_tablet_manager.cc:1939] T 7f3cbef7d91d4576822d913071a88248 P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:21.902330 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet 7f3cbef7d91d4576822d913071a88248 (table default.Table_b581b1a063a54551a8535adf01983fb8 [id=54e4e3810b3b412bbec3b5bf5b28c1a2]) successfully deleted
W20250113 02:51:21.905956 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet a92f9b1c9c1a4cbdbf0cc7b2a3f2275b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a92f9b1c9c1a4cbdbf0cc7b2a3f2275b already in progress: creating tablet
I20250113 02:51:21.906349 1196 ts_tablet_manager.cc:1918] T c71511a100f14322b5091a9804720fa0 P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:21.906558 1196 log.cc:1198] T c71511a100f14322b5091a9804720fa0 P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/c71511a100f14322b5091a9804720fa0
I20250113 02:51:21.906584 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 286) took 1164 ms. Trace:
I20250113 02:51:21.906701 1195 tablet_replica.cc:331] T 9041b28402664a08b8761fa155c9c229 P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:21.906706 1198 rpcz_store.cc:276] 0113 02:51:20.739805 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:20.739869 (+ 64us) service_pool.cc:225] Handling call
0113 02:51:21.790090 (+1050221us) ts_tablet_manager.cc:1145] Deleting tablet fb183b4588e84212a3b4b096f430d47d
0113 02:51:21.790102 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:21.847121 (+ 57019us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:21.886075 (+ 38954us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:21.904439 (+ 18364us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1019784}
I20250113 02:51:21.906939 1195 raft_consensus.cc:2238] T 9041b28402664a08b8761fa155c9c229 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:21.907172 1195 raft_consensus.cc:2267] T 9041b28402664a08b8761fa155c9c229 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:21.907147 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet fb183b4588e84212a3b4b096f430d47d (table default.Table_53b46860bb68469fa08f3fd02b2408e5 [id=33bf7fd42ab5458db0f14347353c0db4]) successfully deleted
I20250113 02:51:21.907209 1198 tablet_replica.cc:331] T c62a5e9e43d14b5c94eb87260528469d P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:21.907377 1198 raft_consensus.cc:2238] T c62a5e9e43d14b5c94eb87260528469d P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:21.907836 1196 ts_tablet_manager.cc:1939] T c71511a100f14322b5091a9804720fa0 P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:21.908795 1195 ts_tablet_manager.cc:1905] T 9041b28402664a08b8761fa155c9c229 P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:21.913913 1345 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39310 (request call id 0) took 1268 ms. Trace:
I20250113 02:51:21.914315 1198 raft_consensus.cc:2267] T c62a5e9e43d14b5c94eb87260528469d P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:21.914942 1198 ts_tablet_manager.cc:1905] T c62a5e9e43d14b5c94eb87260528469d P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:21.909320 1109 heartbeater.cc:502] Master 127.0.104.61:35271 was elected leader, sending a full tablet report...
I20250113 02:51:21.924196 1195 ts_tablet_manager.cc:1918] T 9041b28402664a08b8761fa155c9c229 P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:21.914254 1446 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bbc3541604a455584bb32dfc2cc4efd [sys.catalog]: SysCatalogTable state changed. Reason: New leader e13f5bcdcee34225a46c6d2651e5ede9. Latest consensus state: current_term: 4 leader_uuid: "e13f5bcdcee34225a46c6d2651e5ede9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e13f5bcdcee34225a46c6d2651e5ede9" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44217 } } peers { permanent_uuid: "6bbc3541604a455584bb32dfc2cc4efd" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 35271 } } peers { permanent_uuid: "45e9d43ecac0401fb2c18fe5eaf089b5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42067 } } }
I20250113 02:51:21.924369 1195 log.cc:1198] T 9041b28402664a08b8761fa155c9c229 P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/9041b28402664a08b8761fa155c9c229
I20250113 02:51:21.924453 1446 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bbc3541604a455584bb32dfc2cc4efd [sys.catalog]: This master's current role is: FOLLOWER
I20250113 02:51:21.915424 1480 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6bbc3541604a455584bb32dfc2cc4efd [term 4 FOLLOWER]: Starting pre-election (detected failure of leader e13f5bcdcee34225a46c6d2651e5ede9)
I20250113 02:51:21.924763 1480 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6bbc3541604a455584bb32dfc2cc4efd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e13f5bcdcee34225a46c6d2651e5ede9" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44217 } } peers { permanent_uuid: "6bbc3541604a455584bb32dfc2cc4efd" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 35271 } } peers { permanent_uuid: "45e9d43ecac0401fb2c18fe5eaf089b5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42067 } }
I20250113 02:51:21.926770 1191 tablet_bootstrap.cc:492] T 3d3a6d9f687e45218d5875a7c5597648 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:21.926730 454 master_service.cc:946] ReplaceTablet: received request to replace tablet 21c4b419bf2148e0b140301fa80622bb from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:21.926928 1191 ts_tablet_manager.cc:1397] T 3d3a6d9f687e45218d5875a7c5597648 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.051s user 0.004s sys 0.017s
I20250113 02:51:21.927515 1480 leader_election.cc:290] T 00000000000000000000000000000000 P 6bbc3541604a455584bb32dfc2cc4efd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers e13f5bcdcee34225a46c6d2651e5ede9 (127.0.104.62:44217), 45e9d43ecac0401fb2c18fe5eaf089b5 (127.0.104.60:42067)
I20250113 02:51:21.927829 1191 raft_consensus.cc:357] T 3d3a6d9f687e45218d5875a7c5597648 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:21.928128 1191 raft_consensus.cc:383] T 3d3a6d9f687e45218d5875a7c5597648 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:21.928227 1191 raft_consensus.cc:738] T 3d3a6d9f687e45218d5875a7c5597648 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:21.928516 1191 consensus_queue.cc:260] T 3d3a6d9f687e45218d5875a7c5597648 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:21.929234 1462 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bbc3541604a455584bb32dfc2cc4efd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "e13f5bcdcee34225a46c6d2651e5ede9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e13f5bcdcee34225a46c6d2651e5ede9" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44217 } } peers { permanent_uuid: "6bbc3541604a455584bb32dfc2cc4efd" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 35271 } } peers { permanent_uuid: "45e9d43ecac0401fb2c18fe5eaf089b5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42067 } } }
I20250113 02:51:21.929519 1462 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bbc3541604a455584bb32dfc2cc4efd [sys.catalog]: This master's current role is: FOLLOWER
I20250113 02:51:21.933563 1191 ts_tablet_manager.cc:1428] T 3d3a6d9f687e45218d5875a7c5597648 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.007s user 0.000s sys 0.003s
I20250113 02:51:21.934074 1191 tablet_bootstrap.cc:492] T 68023c30a7f644eaa5b4ddd60d6c5ef5 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:21.934557 1194 tablet_bootstrap.cc:492] T 1d81eec487dd4016b3b804451da31c49 P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:21.934784 1194 ts_tablet_manager.cc:1397] T 1d81eec487dd4016b3b804451da31c49 P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.101s user 0.007s sys 0.002s
I20250113 02:51:21.934789 1195 ts_tablet_manager.cc:1939] T 9041b28402664a08b8761fa155c9c229 P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:21.935599 1194 raft_consensus.cc:357] T 1d81eec487dd4016b3b804451da31c49 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:21.935846 1194 raft_consensus.cc:383] T 1d81eec487dd4016b3b804451da31c49 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:21.935963 1194 raft_consensus.cc:738] T 1d81eec487dd4016b3b804451da31c49 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:21.936266 1194 consensus_queue.cc:260] T 1d81eec487dd4016b3b804451da31c49 P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:21.936900 1194 ts_tablet_manager.cc:1428] T 1d81eec487dd4016b3b804451da31c49 P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250113 02:51:21.938144 1194 tablet_bootstrap.cc:492] T 3d3a6d9f687e45218d5875a7c5597648 P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:21.940497 1194 tablet_bootstrap.cc:654] T 3d3a6d9f687e45218d5875a7c5597648 P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:21.940649 1192 tablet_bootstrap.cc:492] T d5bd09a1ede644c59e400e49c8125793 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:21.940815 1192 ts_tablet_manager.cc:1397] T d5bd09a1ede644c59e400e49c8125793 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.048s user 0.003s sys 0.000s
I20250113 02:51:21.941617 1192 raft_consensus.cc:357] T d5bd09a1ede644c59e400e49c8125793 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:21.941836 1192 raft_consensus.cc:383] T d5bd09a1ede644c59e400e49c8125793 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:21.941974 1192 raft_consensus.cc:738] T d5bd09a1ede644c59e400e49c8125793 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:21.942658 822 heartbeater.cc:346] Connected to a master server at 127.0.104.61:35271
I20250113 02:51:21.945566 1329 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" instance_seqno: 1736736674635228) as {username='slave'} at 127.0.104.2:59285; Asking this server to re-register.
I20250113 02:51:21.945653 1195 tablet_replica.cc:331] T 2fb62cc83f9c4dcfa9aa01ae8b46c025 P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:21.947183 966 heartbeater.cc:463] Registering TS with master...
I20250113 02:51:21.947180 1329 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "1186fa869eb14812832fa6562531f196" instance_seqno: 1736736674118640) as {username='slave'} at 127.0.104.1:58961; Asking this server to re-register.
I20250113 02:51:21.948293 822 heartbeater.cc:463] Registering TS with master...
I20250113 02:51:21.948843 1191 tablet_bootstrap.cc:654] T 68023c30a7f644eaa5b4ddd60d6c5ef5 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:21.949769 1195 raft_consensus.cc:2238] T 2fb62cc83f9c4dcfa9aa01ae8b46c025 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:21.950795 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet c71511a100f14322b5091a9804720fa0 (table default.Table_8606b4caf08b43b18b25ee8849eebd07 [id=85bca730e1274dbebb2f7c2e6eaea7c7]) successfully deleted
I20250113 02:51:21.950883 1329 ts_manager.cc:194] Registered new tserver with Master: 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
I20250113 02:51:21.951272 1195 raft_consensus.cc:2267] T 2fb62cc83f9c4dcfa9aa01ae8b46c025 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:21.953320 1323 ts_manager.cc:194] Registered new tserver with Master: df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:21.953478 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_37b52d3cfe0f4b0fb1ef1468117512dd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:21.955042 454 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_9613de5a21a8447eb2d5ab0076b77d5c" } new_table_name: "default.Table_0bf61ef424c949cfaff504365b82c4cf" modify_external_catalogs: true
I20250113 02:51:21.955283 1192 consensus_queue.cc:260] T d5bd09a1ede644c59e400e49c8125793 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:21.960742 1192 ts_tablet_manager.cc:1428] T d5bd09a1ede644c59e400e49c8125793 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.020s user 0.003s sys 0.000s
I20250113 02:51:21.960443 1195 ts_tablet_manager.cc:1905] T 2fb62cc83f9c4dcfa9aa01ae8b46c025 P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:21.961208 1192 tablet_bootstrap.cc:492] T 076ed7744da54ddebd8fc99686beb6ae P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:21.961460 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_719ed613a9354add97b036faad401762"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:21.962641 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_384366cb310c4e248cdc81ecede00d18" } new_table_name: "default.Table_5ddfdff2cc8c4d12a1c4552e9a302027" modify_external_catalogs: true
I20250113 02:51:21.964352 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_0ea497aeeb3742fd946679cb72cdf291" } new_table_name: "default.Table_67ba9f96c20f458ebe9d6529f3d97852" modify_external_catalogs: true
I20250113 02:51:21.971825 1192 tablet_bootstrap.cc:654] T 076ed7744da54ddebd8fc99686beb6ae P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:21.974016 1194 tablet_bootstrap.cc:492] T 3d3a6d9f687e45218d5875a7c5597648 P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:21.974289 1194 ts_tablet_manager.cc:1397] T 3d3a6d9f687e45218d5875a7c5597648 P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.036s user 0.002s sys 0.007s
I20250113 02:51:21.975123 1194 raft_consensus.cc:357] T 3d3a6d9f687e45218d5875a7c5597648 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:21.975366 1194 raft_consensus.cc:383] T 3d3a6d9f687e45218d5875a7c5597648 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:21.975445 1194 raft_consensus.cc:738] T 3d3a6d9f687e45218d5875a7c5597648 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:21.975878 1194 consensus_queue.cc:260] T 3d3a6d9f687e45218d5875a7c5597648 P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:21.976272 1191 tablet_bootstrap.cc:492] T 68023c30a7f644eaa5b4ddd60d6c5ef5 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:21.976652 1191 ts_tablet_manager.cc:1397] T 68023c30a7f644eaa5b4ddd60d6c5ef5 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.043s user 0.005s sys 0.001s
I20250113 02:51:21.977653 1194 ts_tablet_manager.cc:1428] T 3d3a6d9f687e45218d5875a7c5597648 P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250113 02:51:21.977978 1194 tablet_bootstrap.cc:492] T d101e98b72544c30a3a4bc60d1576aa3 P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:21.978094 1191 raft_consensus.cc:357] T 68023c30a7f644eaa5b4ddd60d6c5ef5 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:21.978441 1191 raft_consensus.cc:383] T 68023c30a7f644eaa5b4ddd60d6c5ef5 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:21.979158 1191 raft_consensus.cc:738] T 68023c30a7f644eaa5b4ddd60d6c5ef5 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:21.980052 1323 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" instance_seqno: 1736736675135870) as {username='slave'} at 127.0.104.3:34535; Asking this server to re-register.
I20250113 02:51:21.980826 1191 consensus_queue.cc:260] T 68023c30a7f644eaa5b4ddd60d6c5ef5 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:21.985014 1191 ts_tablet_manager.cc:1428] T 68023c30a7f644eaa5b4ddd60d6c5ef5 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.008s user 0.003s sys 0.000s
I20250113 02:51:21.985321 1494 catalog_manager.cc:1260] Loaded cluster ID: 00860a399ece46318a55d1901f0159c4
I20250113 02:51:21.988873 1494 catalog_manager.cc:1553] T 00000000000000000000000000000000 P 6bbc3541604a455584bb32dfc2cc4efd: loading cluster ID for follower catalog manager: success
I20250113 02:51:21.990576 1191 tablet_bootstrap.cc:492] T c71511a100f14322b5091a9804720fa0 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:21.990852 1198 ts_tablet_manager.cc:1918] T c62a5e9e43d14b5c94eb87260528469d P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:21.991075 1198 log.cc:1198] T c62a5e9e43d14b5c94eb87260528469d P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/c62a5e9e43d14b5c94eb87260528469d
I20250113 02:51:21.993590 1198 ts_tablet_manager.cc:1939] T c62a5e9e43d14b5c94eb87260528469d P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:21.995106 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 294) took 1090 ms. Trace:
I20250113 02:51:21.999017 1198 rpcz_store.cc:276] 0113 02:51:20.904916 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:20.905097 (+ 181us) service_pool.cc:225] Handling call
0113 02:51:21.907132 (+1002035us) ts_tablet_manager.cc:1145] Deleting tablet c62a5e9e43d14b5c94eb87260528469d
0113 02:51:21.907140 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:21.918067 (+ 10927us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:21.990698 (+ 72631us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:21.995059 (+ 4361us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":959698}
I20250113 02:51:21.999677 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet c62a5e9e43d14b5c94eb87260528469d (table default.Table_8aa63512c7bb4bdb86f37c9ee24df209 [id=aadd55280b654ceb96c290786a0bfce0]) successfully deleted
I20250113 02:51:21.999912 1191 tablet_bootstrap.cc:654] T c71511a100f14322b5091a9804720fa0 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
W20250113 02:51:22.000578 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 64cda8796dff46ce95c37439395c77ec because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 64cda8796dff46ce95c37439395c77ec already in progress: creating tablet
W20250113 02:51:22.001274 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet b2a35fd891c645d68d79c61a2c832ba7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b2a35fd891c645d68d79c61a2c832ba7 already in progress: creating tablet
W20250113 02:51:22.001911 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet c94eedb5dce44e6d896192c93127903a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c94eedb5dce44e6d896192c93127903a already in progress: creating tablet
W20250113 02:51:22.002583 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 33c665e149dd48e8868acd935de581bf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 33c665e149dd48e8868acd935de581bf already in progress: creating tablet
W20250113 02:51:22.003221 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet ff8ef6290698477391721fd65b9eeb28 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ff8ef6290698477391721fd65b9eeb28 already in progress: creating tablet
I20250113 02:51:21.997170 1109 heartbeater.cc:463] Registering TS with master...
I20250113 02:51:22.007947 1198 tablet_replica.cc:331] T 2fb62cc83f9c4dcfa9aa01ae8b46c025 P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:22.008262 1198 raft_consensus.cc:2238] T 2fb62cc83f9c4dcfa9aa01ae8b46c025 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:22.010398 1329 ts_manager.cc:194] Registered new tserver with Master: 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:21.942418 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet 9041b28402664a08b8761fa155c9c229 (table default.Table_1b75b69f143d4fb2a97f31fd75657b96 [id=7e2f491b17c94a869837d457888bf71b]) successfully deleted
W20250113 02:51:22.011845 439 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.104.1:40095, user_credentials={real_user=slave}} blocked reactor thread for 69422.2us
W20250113 02:51:22.012094 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_719ed613a9354add97b036faad401762 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:22.012245 1198 raft_consensus.cc:2267] T 2fb62cc83f9c4dcfa9aa01ae8b46c025 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus is shut down!
W20250113 02:51:22.012346 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet c826a0b8327f44f9a00e5afe3a36cd8a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c826a0b8327f44f9a00e5afe3a36cd8a already in progress: creating tablet
I20250113 02:51:22.012805 1198 ts_tablet_manager.cc:1905] T 2fb62cc83f9c4dcfa9aa01ae8b46c025 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250113 02:51:22.013443 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_37b52d3cfe0f4b0fb1ef1468117512dd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:22.012071 1192 tablet_bootstrap.cc:492] T 076ed7744da54ddebd8fc99686beb6ae P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:22.014302 1192 ts_tablet_manager.cc:1397] T 076ed7744da54ddebd8fc99686beb6ae P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.053s user 0.002s sys 0.002s
I20250113 02:51:22.015736 1192 raft_consensus.cc:357] T 076ed7744da54ddebd8fc99686beb6ae P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:22.016001 1192 raft_consensus.cc:383] T 076ed7744da54ddebd8fc99686beb6ae P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:22.016074 1192 raft_consensus.cc:738] T 076ed7744da54ddebd8fc99686beb6ae P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:22.016315 1192 consensus_queue.cc:260] T 076ed7744da54ddebd8fc99686beb6ae P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
W20250113 02:51:22.018908 454 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet df87e25053a74990959f85288a85217b consensus state has no leader
I20250113 02:51:22.019243 1192 ts_tablet_manager.cc:1428] T 076ed7744da54ddebd8fc99686beb6ae P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250113 02:51:22.021896 1192 tablet_bootstrap.cc:492] T 18e2d79d731142f89b9089869bc736e5 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
W20250113 02:51:22.023459 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 377ebf66bbef40988043f6200ae0893c consensus state has no leader
I20250113 02:51:22.023897 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_694391751b064302a0b020dbea9d842d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:22.024222 1192 tablet_bootstrap.cc:654] T 18e2d79d731142f89b9089869bc736e5 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
W20250113 02:51:22.024439 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_694391751b064302a0b020dbea9d842d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:22.025381 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_4757a3729a5c41d2a28c09572ef23181" } new_table_name: "default.Table_2155bf71d1a04a93a28b13c97f636fab" modify_external_catalogs: true
W20250113 02:51:22.025779 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7f08b72301dc431396cf8cc7790d56d4 consensus state has no leader
I20250113 02:51:22.026378 1194 tablet_bootstrap.cc:654] T d101e98b72544c30a3a4bc60d1576aa3 P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:22.029783 750 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d3a6d9f687e45218d5875a7c5597648 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:17 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:22.034042 1442 raft_consensus.cc:491] T d5bd09a1ede644c59e400e49c8125793 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:22.034220 1442 raft_consensus.cc:513] T d5bd09a1ede644c59e400e49c8125793 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:22.034631 464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6bbc3541604a455584bb32dfc2cc4efd" candidate_term: 5 candidate_status { last_received { term: 4 index: 318 } } ignore_live_leader: false dest_uuid: "e13f5bcdcee34225a46c6d2651e5ede9" is_pre_election: true
I20250113 02:51:22.034927 1442 leader_election.cc:290] T d5bd09a1ede644c59e400e49c8125793 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
I20250113 02:51:22.035292 605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6bbc3541604a455584bb32dfc2cc4efd" candidate_term: 5 candidate_status { last_received { term: 4 index: 318 } } ignore_live_leader: false dest_uuid: "45e9d43ecac0401fb2c18fe5eaf089b5" is_pre_election: true
I20250113 02:51:22.035681 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bd09a1ede644c59e400e49c8125793" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:22.036592 1290 leader_election.cc:304] T 00000000000000000000000000000000 P 6bbc3541604a455584bb32dfc2cc4efd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6bbc3541604a455584bb32dfc2cc4efd; no voters: 45e9d43ecac0401fb2c18fe5eaf089b5, e13f5bcdcee34225a46c6d2651e5ede9
I20250113 02:51:22.036893 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bd09a1ede644c59e400e49c8125793" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
I20250113 02:51:22.037241 1462 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 6bbc3541604a455584bb32dfc2cc4efd [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250113 02:51:22.037350 750 tablet_service.cc:1514] Processing DeleteTablet for tablet 68023c30a7f644eaa5b4ddd60d6c5ef5 with delete_type TABLET_DATA_DELETED (replaced by tablet 32351d6de1574353a6e8073d333da477) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:22.035924 919 raft_consensus.cc:2463] T d5bd09a1ede644c59e400e49c8125793 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 0.
I20250113 02:51:22.037909 998 leader_election.cc:304] T d5bd09a1ede644c59e400e49c8125793 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f; no voters:
W20250113 02:51:22.038388 995 leader_election.cc:343] T d5bd09a1ede644c59e400e49c8125793 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:22.038910 1442 raft_consensus.cc:2798] T d5bd09a1ede644c59e400e49c8125793 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 02:51:22.039000 1442 raft_consensus.cc:491] T d5bd09a1ede644c59e400e49c8125793 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 02:51:22.039086 1442 raft_consensus.cc:3054] T d5bd09a1ede644c59e400e49c8125793 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:22.042332 1442 raft_consensus.cc:513] T d5bd09a1ede644c59e400e49c8125793 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:22.043466 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bd09a1ede644c59e400e49c8125793" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f"
I20250113 02:51:22.043656 919 raft_consensus.cc:3054] T d5bd09a1ede644c59e400e49c8125793 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:22.046931 919 raft_consensus.cc:2463] T d5bd09a1ede644c59e400e49c8125793 P df69f48656b44bb890f76587ee69cc0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 1.
I20250113 02:51:22.046964 448 master_service.cc:946] ReplaceTablet: received request to replace tablet f129bb3fbbfd492d98822be0a6626e44 from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:22.047360 998 leader_election.cc:304] T d5bd09a1ede644c59e400e49c8125793 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f; no voters:
I20250113 02:51:22.049615 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bd09a1ede644c59e400e49c8125793" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196"
W20250113 02:51:22.050237 995 leader_election.cc:343] T d5bd09a1ede644c59e400e49c8125793 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:22.050845 1442 leader_election.cc:290] T d5bd09a1ede644c59e400e49c8125793 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Requested vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
I20250113 02:51:22.051199 1442 raft_consensus.cc:2798] T d5bd09a1ede644c59e400e49c8125793 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Leader election won for term 1
I20250113 02:51:22.051945 881 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d81eec487dd4016b3b804451da31c49 with delete_type TABLET_DATA_DELETED (replaced by tablet 438d1d2300e044f39a15586f335935ba) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:22.052287 1196 tablet_replica.cc:331] T 1d81eec487dd4016b3b804451da31c49 P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:22.052541 1196 raft_consensus.cc:2238] T 1d81eec487dd4016b3b804451da31c49 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:22.052719 1196 raft_consensus.cc:2267] T 1d81eec487dd4016b3b804451da31c49 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:22.053563 1196 ts_tablet_manager.cc:1905] T 1d81eec487dd4016b3b804451da31c49 P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:22.056182 1195 ts_tablet_manager.cc:1918] T 2fb62cc83f9c4dcfa9aa01ae8b46c025 P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:22.056334 1195 log.cc:1198] T 2fb62cc83f9c4dcfa9aa01ae8b46c025 P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/2fb62cc83f9c4dcfa9aa01ae8b46c025
I20250113 02:51:22.056711 1195 ts_tablet_manager.cc:1939] T 2fb62cc83f9c4dcfa9aa01ae8b46c025 P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:22.057703 1195 tablet_replica.cc:331] T 98f038ad31ff4e05bb88ac68ae341cc2 P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:22.057883 1195 raft_consensus.cc:2238] T 98f038ad31ff4e05bb88ac68ae341cc2 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:22.057904 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet 2fb62cc83f9c4dcfa9aa01ae8b46c025 (table default.Table_37b69086d7374e7ea7b83b40e8f202db [id=dd19d0450abb4862a408f2f5f5922a4f]) successfully deleted
I20250113 02:51:22.058043 1195 raft_consensus.cc:2267] T 98f038ad31ff4e05bb88ac68ae341cc2 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:22.051923 1442 raft_consensus.cc:695] T d5bd09a1ede644c59e400e49c8125793 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: Becoming Leader. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Running, Role: LEADER
I20250113 02:51:22.059918 1442 consensus_queue.cc:237] T d5bd09a1ede644c59e400e49c8125793 P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:22.062703 1196 ts_tablet_manager.cc:1918] T 1d81eec487dd4016b3b804451da31c49 P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:22.062893 1196 log.cc:1198] T 1d81eec487dd4016b3b804451da31c49 P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/1d81eec487dd4016b3b804451da31c49
I20250113 02:51:22.063961 1196 ts_tablet_manager.cc:1939] T 1d81eec487dd4016b3b804451da31c49 P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:22.065857 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_08def01561d44ec5a88f5b5a82d47e3b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:22.066463 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_08def01561d44ec5a88f5b5a82d47e3b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:22.068853 1195 ts_tablet_manager.cc:1905] T 98f038ad31ff4e05bb88ac68ae341cc2 P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:22.069036 446 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_fb08fe0ed5cd4aaebce3169c264441cb" } modify_external_catalogs: true
I20250113 02:51:22.069214 446 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_fb08fe0ed5cd4aaebce3169c264441cb" } modify_external_catalogs: true
I20250113 02:51:22.069442 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet 1d81eec487dd4016b3b804451da31c49 (table default.Table_d7d1f5a34e5f4c05b474b0b61f912cd9 [id=2c7925ec0d8d4cd7b8761268d3962f15]) successfully deleted
I20250113 02:51:22.073923 455 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_3c637eb181574498b44ecb4fc41f0117" } modify_external_catalogs: true
I20250113 02:51:22.074118 455 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_3c637eb181574498b44ecb4fc41f0117" } modify_external_catalogs: true
W20250113 02:51:22.076190 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d9df314ffaf042ec80180868d7e7bdf7 has no consensus state
I20250113 02:51:22.076689 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_b21bac46382e43c4bb99fc94778daeeb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:22.076946 1194 tablet_bootstrap.cc:492] T d101e98b72544c30a3a4bc60d1576aa3 P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:22.076956 1191 tablet_bootstrap.cc:492] T c71511a100f14322b5091a9804720fa0 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:22.077090 1194 ts_tablet_manager.cc:1397] T d101e98b72544c30a3a4bc60d1576aa3 P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.099s user 0.002s sys 0.001s
I20250113 02:51:22.077090 1191 ts_tablet_manager.cc:1397] T c71511a100f14322b5091a9804720fa0 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.087s user 0.004s sys 0.000s
I20250113 02:51:22.077893 1191 raft_consensus.cc:357] T c71511a100f14322b5091a9804720fa0 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:22.077863 1194 raft_consensus.cc:357] T d101e98b72544c30a3a4bc60d1576aa3 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:22.078151 1194 raft_consensus.cc:383] T d101e98b72544c30a3a4bc60d1576aa3 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:22.078194 1191 raft_consensus.cc:383] T c71511a100f14322b5091a9804720fa0 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:22.078251 1194 raft_consensus.cc:738] T d101e98b72544c30a3a4bc60d1576aa3 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
W20250113 02:51:22.078289 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b21bac46382e43c4bb99fc94778daeeb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:22.078404 1191 raft_consensus.cc:738] T c71511a100f14322b5091a9804720fa0 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:22.078508 1194 consensus_queue.cc:260] T d101e98b72544c30a3a4bc60d1576aa3 P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:22.078647 1191 consensus_queue.cc:260] T c71511a100f14322b5091a9804720fa0 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:22.079329 1191 ts_tablet_manager.cc:1428] T c71511a100f14322b5091a9804720fa0 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250113 02:51:22.079643 1191 tablet_bootstrap.cc:492] T d101e98b72544c30a3a4bc60d1576aa3 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:22.080017 1194 ts_tablet_manager.cc:1428] T d101e98b72544c30a3a4bc60d1576aa3 P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250113 02:51:22.080271 1194 tablet_bootstrap.cc:492] T 34ab7ee573284550aaac9f814e8bd7ab P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:22.082033 1194 tablet_bootstrap.cc:654] T 34ab7ee573284550aaac9f814e8bd7ab P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:22.083922 1191 tablet_bootstrap.cc:654] T d101e98b72544c30a3a4bc60d1576aa3 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:22.090639 1508 raft_consensus.cc:491] T 98f038ad31ff4e05bb88ac68ae341cc2 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:22.090852 1508 raft_consensus.cc:513] T 98f038ad31ff4e05bb88ac68ae341cc2 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:22.092389 1508 leader_election.cc:290] T 98f038ad31ff4e05bb88ac68ae341cc2 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
W20250113 02:51:22.093132 998 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:38467, user_credentials={real_user=slave}} blocked reactor thread for 45814.9us
I20250113 02:51:22.098948 1194 tablet_bootstrap.cc:492] T 34ab7ee573284550aaac9f814e8bd7ab P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:22.099157 1194 ts_tablet_manager.cc:1397] T 34ab7ee573284550aaac9f814e8bd7ab P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.019s user 0.000s sys 0.003s
I20250113 02:51:22.100075 1192 tablet_bootstrap.cc:492] T 18e2d79d731142f89b9089869bc736e5 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:22.099952 1194 raft_consensus.cc:357] T 34ab7ee573284550aaac9f814e8bd7ab P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:22.100183 1194 raft_consensus.cc:383] T 34ab7ee573284550aaac9f814e8bd7ab P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:22.100265 1194 raft_consensus.cc:738] T 34ab7ee573284550aaac9f814e8bd7ab P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:22.100382 1192 ts_tablet_manager.cc:1397] T 18e2d79d731142f89b9089869bc736e5 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.079s user 0.002s sys 0.002s
I20250113 02:51:22.100500 1194 consensus_queue.cc:260] T 34ab7ee573284550aaac9f814e8bd7ab P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:22.101174 1192 raft_consensus.cc:357] T 18e2d79d731142f89b9089869bc736e5 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:22.101426 1192 raft_consensus.cc:383] T 18e2d79d731142f89b9089869bc736e5 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:22.101562 1192 raft_consensus.cc:738] T 18e2d79d731142f89b9089869bc736e5 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:22.101815 1192 consensus_queue.cc:260] T 18e2d79d731142f89b9089869bc736e5 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:22.102571 1192 ts_tablet_manager.cc:1428] T 18e2d79d731142f89b9089869bc736e5 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250113 02:51:22.102893 1192 tablet_bootstrap.cc:492] T 68023c30a7f644eaa5b4ddd60d6c5ef5 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:22.092072 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98f038ad31ff4e05bb88ac68ae341cc2" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
I20250113 02:51:22.103377 1194 ts_tablet_manager.cc:1428] T 34ab7ee573284550aaac9f814e8bd7ab P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
W20250113 02:51:22.103837 995 leader_election.cc:343] T 98f038ad31ff4e05bb88ac68ae341cc2 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Illegal state: Tablet not RUNNING: STOPPED
I20250113 02:51:22.105437 1192 tablet_bootstrap.cc:654] T 68023c30a7f644eaa5b4ddd60d6c5ef5 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:22.105726 1194 tablet_bootstrap.cc:492] T 4efa63b8be794e1db4b340f4c0c4cd5a P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:22.101585 1195 ts_tablet_manager.cc:1918] T 98f038ad31ff4e05bb88ac68ae341cc2 P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:22.106995 1195 log.cc:1198] T 98f038ad31ff4e05bb88ac68ae341cc2 P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/98f038ad31ff4e05bb88ac68ae341cc2
I20250113 02:51:22.109117 750 tablet_service.cc:1514] Processing DeleteTablet for tablet c71511a100f14322b5091a9804720fa0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:20 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:22.110142 1191 tablet_bootstrap.cc:492] T d101e98b72544c30a3a4bc60d1576aa3 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:22.110333 1191 ts_tablet_manager.cc:1397] T d101e98b72544c30a3a4bc60d1576aa3 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.031s user 0.004s sys 0.001s
I20250113 02:51:22.110394 1438 raft_consensus.cc:491] T 3d3a6d9f687e45218d5875a7c5597648 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:22.110569 1438 raft_consensus.cc:513] T 3d3a6d9f687e45218d5875a7c5597648 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:22.111259 1438 leader_election.cc:290] T 3d3a6d9f687e45218d5875a7c5597648 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587), 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
I20250113 02:51:22.111153 1191 raft_consensus.cc:357] T d101e98b72544c30a3a4bc60d1576aa3 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:22.111371 1191 raft_consensus.cc:383] T d101e98b72544c30a3a4bc60d1576aa3 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:22.111445 1191 raft_consensus.cc:738] T d101e98b72544c30a3a4bc60d1576aa3 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:22.111663 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d3a6d9f687e45218d5875a7c5597648" candidate_uuid: "df69f48656b44bb890f76587ee69cc0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
I20250113 02:51:22.111995 772 raft_consensus.cc:2463] T 3d3a6d9f687e45218d5875a7c5597648 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df69f48656b44bb890f76587ee69cc0f in term 0.
I20250113 02:51:22.111900 1191 consensus_queue.cc:260] T d101e98b72544c30a3a4bc60d1576aa3 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:22.112471 851 leader_election.cc:304] T 3d3a6d9f687e45218d5875a7c5597648 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1186fa869eb14812832fa6562531f196, df69f48656b44bb890f76587ee69cc0f; no voters:
I20250113 02:51:22.112869 1438 raft_consensus.cc:2798] T 3d3a6d9f687e45218d5875a7c5597648 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 02:51:22.112978 1438 raft_consensus.cc:491] T 3d3a6d9f687e45218d5875a7c5597648 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 02:51:22.113075 1438 raft_consensus.cc:3054] T 3d3a6d9f687e45218d5875a7c5597648 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:22.114391 1191 ts_tablet_manager.cc:1428] T d101e98b72544c30a3a4bc60d1576aa3 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250113 02:51:22.114715 1191 tablet_bootstrap.cc:492] T 8547b63c40ec4ccebd962b5a3b2c5e0f P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:22.115773 1438 raft_consensus.cc:513] T 3d3a6d9f687e45218d5875a7c5597648 P df69f48656b44bb890f76587ee69cc0f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:22.116498 1191 tablet_bootstrap.cc:654] T 8547b63c40ec4ccebd962b5a3b2c5e0f P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:22.116847 1194 tablet_bootstrap.cc:654] T 4efa63b8be794e1db4b340f4c0c4cd5a P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:22.116822 1195 ts_tablet_manager.cc:1939] T 98f038ad31ff4e05bb88ac68ae341cc2 P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:22.117729 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d3a6d9f687e45218d5875a7c5597648" candidate_uuid: "df69f48656b44bb890f76587ee69cc0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196"
I20250113 02:51:22.117933 772 raft_consensus.cc:3054] T 3d3a6d9f687e45218d5875a7c5597648 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:22.118005 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet 98f038ad31ff4e05bb88ac68ae341cc2 (table default.Table_5182a1c5c7a2462ba930940ae04b505f [id=6666a614063942c99a78b95d35add037]) successfully deleted
W20250113 02:51:22.118718 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 2c399784af46426b8327d175c7a6cce8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2c399784af46426b8327d175c7a6cce8 already in progress: creating tablet
W20250113 02:51:22.119709 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 7d3562a48c6e4badbbcf37640e91b5ba because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7d3562a48c6e4badbbcf37640e91b5ba already in progress: creating tablet
I20250113 02:51:22.120205 1438 leader_election.cc:290] T 3d3a6d9f687e45218d5875a7c5597648 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 election: Requested vote from peers 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587), 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
W20250113 02:51:22.120726 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet eebad045e7bb43fa9ca9c9af8b3b7ea5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet eebad045e7bb43fa9ca9c9af8b3b7ea5 already in progress: creating tablet
I20250113 02:51:22.121040 1195 tablet_replica.cc:331] T 9b5dc06ba3ea4056822ca3e4b1249587 P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:22.121353 772 raft_consensus.cc:2463] T 3d3a6d9f687e45218d5875a7c5597648 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df69f48656b44bb890f76587ee69cc0f in term 1.
I20250113 02:51:22.121807 1195 raft_consensus.cc:2238] T 9b5dc06ba3ea4056822ca3e4b1249587 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:22.121909 851 leader_election.cc:304] T 3d3a6d9f687e45218d5875a7c5597648 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1186fa869eb14812832fa6562531f196, df69f48656b44bb890f76587ee69cc0f; no voters:
I20250113 02:51:22.122206 1438 raft_consensus.cc:2798] T 3d3a6d9f687e45218d5875a7c5597648 P df69f48656b44bb890f76587ee69cc0f [term 1 FOLLOWER]: Leader election won for term 1
I20250113 02:51:22.122406 1195 raft_consensus.cc:2267] T 9b5dc06ba3ea4056822ca3e4b1249587 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:22.125950 1195 ts_tablet_manager.cc:1905] T 9b5dc06ba3ea4056822ca3e4b1249587 P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:22.128607 881 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d81eec487dd4016b3b804451da31c49 with delete_type TABLET_DATA_DELETED (replaced by tablet 438d1d2300e044f39a15586f335935ba) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:22.130718 881 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d3a6d9f687e45218d5875a7c5597648 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:17 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:22.131006 1196 tablet_replica.cc:331] T 3d3a6d9f687e45218d5875a7c5597648 P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:22.132489 881 tablet_service.cc:1514] Processing DeleteTablet for tablet d101e98b72544c30a3a4bc60d1576aa3 with delete_type TABLET_DATA_DELETED (replaced by tablet 5b3f012b88c04cb68ed7a1848147b70b) from {username='slave'} at 127.0.0.1:37366
W20250113 02:51:22.133095 442 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet df87e25053a74990959f85288a85217b (table default.Table_0bf61ef424c949cfaff504365b82c4cf [id=b2c8431e06ac46be9840edf78b6c19a3], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet df87e25053a74990959f85288a85217b consensus state has no leader
I20250113 02:51:22.137634 1191 tablet_bootstrap.cc:492] T 8547b63c40ec4ccebd962b5a3b2c5e0f P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:22.137863 1191 ts_tablet_manager.cc:1397] T 8547b63c40ec4ccebd962b5a3b2c5e0f P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.023s user 0.003s sys 0.000s
I20250113 02:51:22.138612 1494 catalog_manager.cc:1575] T 00000000000000000000000000000000 P 6bbc3541604a455584bb32dfc2cc4efd: acquiring CA information for follower catalog manager: success
I20250113 02:51:22.138939 1191 raft_consensus.cc:357] T 8547b63c40ec4ccebd962b5a3b2c5e0f P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:22.139228 1191 raft_consensus.cc:383] T 8547b63c40ec4ccebd962b5a3b2c5e0f P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:22.139341 1191 raft_consensus.cc:738] T 8547b63c40ec4ccebd962b5a3b2c5e0f P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:22.139642 1191 consensus_queue.cc:260] T 8547b63c40ec4ccebd962b5a3b2c5e0f P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:22.140377 1191 ts_tablet_manager.cc:1428] T 8547b63c40ec4ccebd962b5a3b2c5e0f P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250113 02:51:22.140744 1191 tablet_bootstrap.cc:492] T 1d81eec487dd4016b3b804451da31c49 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:22.141077 881 tablet_service.cc:1514] Processing DeleteTablet for tablet 34ab7ee573284550aaac9f814e8bd7ab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:17 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:22.141847 750 tablet_service.cc:1514] Processing DeleteTablet for tablet d101e98b72544c30a3a4bc60d1576aa3 with delete_type TABLET_DATA_DELETED (replaced by tablet 5b3f012b88c04cb68ed7a1848147b70b) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:22.141866 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_d84fa5147d8a430ca92f49e98b2f7839"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:22.142441 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d84fa5147d8a430ca92f49e98b2f7839 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:22.144254 1191 tablet_bootstrap.cc:654] T 1d81eec487dd4016b3b804451da31c49 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
W20250113 02:51:22.110452 998 leader_election.cc:343] T 98f038ad31ff4e05bb88ac68ae341cc2 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Not found: Tablet not found: 98f038ad31ff4e05bb88ac68ae341cc2
I20250113 02:51:22.111892 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d3a6d9f687e45218d5875a7c5597648" candidate_uuid: "df69f48656b44bb890f76587ee69cc0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:22.145323 1059 raft_consensus.cc:2463] T 3d3a6d9f687e45218d5875a7c5597648 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df69f48656b44bb890f76587ee69cc0f in term 0.
I20250113 02:51:22.117020 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d3a6d9f687e45218d5875a7c5597648" candidate_uuid: "df69f48656b44bb890f76587ee69cc0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca"
I20250113 02:51:22.145677 1061 raft_consensus.cc:3054] T 3d3a6d9f687e45218d5875a7c5597648 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:22.146708 1194 tablet_bootstrap.cc:492] T 4efa63b8be794e1db4b340f4c0c4cd5a P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:22.146845 1194 ts_tablet_manager.cc:1397] T 4efa63b8be794e1db4b340f4c0c4cd5a P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.041s user 0.004s sys 0.001s
I20250113 02:51:22.147588 1194 raft_consensus.cc:357] T 4efa63b8be794e1db4b340f4c0c4cd5a P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:22.147769 1194 raft_consensus.cc:383] T 4efa63b8be794e1db4b340f4c0c4cd5a P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:22.147847 1194 raft_consensus.cc:738] T 4efa63b8be794e1db4b340f4c0c4cd5a P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:22.148108 1194 consensus_queue.cc:260] T 4efa63b8be794e1db4b340f4c0c4cd5a P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:22.148577 1195 ts_tablet_manager.cc:1918] T 9b5dc06ba3ea4056822ca3e4b1249587 P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:22.148708 1195 log.cc:1198] T 9b5dc06ba3ea4056822ca3e4b1249587 P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/9b5dc06ba3ea4056822ca3e4b1249587
I20250113 02:51:22.149040 1195 ts_tablet_manager.cc:1939] T 9b5dc06ba3ea4056822ca3e4b1249587 P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:22.149271 1061 raft_consensus.cc:2463] T 3d3a6d9f687e45218d5875a7c5597648 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df69f48656b44bb890f76587ee69cc0f in term 1.
I20250113 02:51:22.149893 998 leader_election.cc:304] T 98f038ad31ff4e05bb88ac68ae341cc2 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4216a7f853d642b7bc3e5cff458f3aca; no voters: 1186fa869eb14812832fa6562531f196, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:22.150339 1195 tablet_replica.cc:331] T 3d3a6d9f687e45218d5875a7c5597648 P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:22.150555 1508 raft_consensus.cc:2743] T 98f038ad31ff4e05bb88ac68ae341cc2 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:22.150588 1195 raft_consensus.cc:2238] T 3d3a6d9f687e45218d5875a7c5597648 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:22.131774 1198 ts_tablet_manager.cc:1918] T 2fb62cc83f9c4dcfa9aa01ae8b46c025 P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:22.150803 1195 raft_consensus.cc:2267] T 3d3a6d9f687e45218d5875a7c5597648 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:22.150979 1198 log.cc:1198] T 2fb62cc83f9c4dcfa9aa01ae8b46c025 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/2fb62cc83f9c4dcfa9aa01ae8b46c025
I20250113 02:51:22.152531 1195 ts_tablet_manager.cc:1905] T 3d3a6d9f687e45218d5875a7c5597648 P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:22.154659 1198 ts_tablet_manager.cc:1939] T 2fb62cc83f9c4dcfa9aa01ae8b46c025 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:22.155889 1198 tablet_replica.cc:331] T 9b5dc06ba3ea4056822ca3e4b1249587 P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:22.158268 1194 ts_tablet_manager.cc:1428] T 4efa63b8be794e1db4b340f4c0c4cd5a P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.011s user 0.004s sys 0.000s
I20250113 02:51:22.158916 1194 tablet_bootstrap.cc:492] T 6c8ca30720664f3a819c4bff2493934d P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:22.160543 1191 tablet_bootstrap.cc:492] T 1d81eec487dd4016b3b804451da31c49 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:22.160770 1191 ts_tablet_manager.cc:1397] T 1d81eec487dd4016b3b804451da31c49 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.003s
I20250113 02:51:22.160933 1194 tablet_bootstrap.cc:654] T 6c8ca30720664f3a819c4bff2493934d P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:22.161680 1191 raft_consensus.cc:357] T 1d81eec487dd4016b3b804451da31c49 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:22.162014 1191 raft_consensus.cc:383] T 1d81eec487dd4016b3b804451da31c49 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:22.162132 1191 raft_consensus.cc:738] T 1d81eec487dd4016b3b804451da31c49 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:22.162384 1198 raft_consensus.cc:2238] T 9b5dc06ba3ea4056822ca3e4b1249587 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:22.162402 1191 consensus_queue.cc:260] T 1d81eec487dd4016b3b804451da31c49 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:22.162968 1191 ts_tablet_manager.cc:1428] T 1d81eec487dd4016b3b804451da31c49 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250113 02:51:22.163437 1198 raft_consensus.cc:2267] T 9b5dc06ba3ea4056822ca3e4b1249587 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:22.164542 1191 tablet_bootstrap.cc:492] T 076ed7744da54ddebd8fc99686beb6ae P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:22.159359 1192 tablet_bootstrap.cc:492] T 68023c30a7f644eaa5b4ddd60d6c5ef5 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:22.166261 1192 ts_tablet_manager.cc:1397] T 68023c30a7f644eaa5b4ddd60d6c5ef5 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.063s user 0.002s sys 0.001s
I20250113 02:51:22.167024 1192 raft_consensus.cc:357] T 68023c30a7f644eaa5b4ddd60d6c5ef5 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
W20250113 02:51:22.141968 442 catalog_manager.cc:4839] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 1d81eec487dd4016b3b804451da31c49 because the tablet was not found. No further retry: Not found: Tablet not found: 1d81eec487dd4016b3b804451da31c49
I20250113 02:51:22.167223 1192 raft_consensus.cc:383] T 68023c30a7f644eaa5b4ddd60d6c5ef5 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:22.167286 1192 raft_consensus.cc:738] T 68023c30a7f644eaa5b4ddd60d6c5ef5 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:22.149979 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet 9b5dc06ba3ea4056822ca3e4b1249587 (table default.Table_20efb43987cc47c7a77436e076cc70f6 [id=8d889764fb3046369d87cef8998a5a87]) successfully deleted
I20250113 02:51:22.167496 1192 consensus_queue.cc:260] T 68023c30a7f644eaa5b4ddd60d6c5ef5 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:22.167994 1508 raft_consensus.cc:491] T 076ed7744da54ddebd8fc99686beb6ae P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:22.168174 1508 raft_consensus.cc:513] T 076ed7744da54ddebd8fc99686beb6ae P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:22.168715 1191 tablet_bootstrap.cc:654] T 076ed7744da54ddebd8fc99686beb6ae P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:22.169379 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "076ed7744da54ddebd8fc99686beb6ae" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
W20250113 02:51:22.171021 995 leader_election.cc:343] T 076ed7744da54ddebd8fc99686beb6ae P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:22.172860 1508 leader_election.cc:290] T 076ed7744da54ddebd8fc99686beb6ae P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
W20250113 02:51:22.173789 998 leader_election.cc:343] T 076ed7744da54ddebd8fc99686beb6ae P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Not found: Tablet not found: 076ed7744da54ddebd8fc99686beb6ae
I20250113 02:51:22.173957 998 leader_election.cc:304] T 076ed7744da54ddebd8fc99686beb6ae P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4216a7f853d642b7bc3e5cff458f3aca; no voters: 1186fa869eb14812832fa6562531f196, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:22.174355 1508 raft_consensus.cc:2743] T 076ed7744da54ddebd8fc99686beb6ae P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:22.174666 1194 tablet_bootstrap.cc:492] T 6c8ca30720664f3a819c4bff2493934d P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:22.174827 1194 ts_tablet_manager.cc:1397] T 6c8ca30720664f3a819c4bff2493934d P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.000s
I20250113 02:51:22.174961 1438 raft_consensus.cc:695] T 3d3a6d9f687e45218d5875a7c5597648 P df69f48656b44bb890f76587ee69cc0f [term 1 LEADER]: Becoming Leader. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Running, Role: LEADER
I20250113 02:51:22.175292 1438 consensus_queue.cc:237] T 3d3a6d9f687e45218d5875a7c5597648 P df69f48656b44bb890f76587ee69cc0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:22.175606 1194 raft_consensus.cc:357] T 6c8ca30720664f3a819c4bff2493934d P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:22.175804 1194 raft_consensus.cc:383] T 6c8ca30720664f3a819c4bff2493934d P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:22.175881 1194 raft_consensus.cc:738] T 6c8ca30720664f3a819c4bff2493934d P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:22.176201 1192 ts_tablet_manager.cc:1428] T 68023c30a7f644eaa5b4ddd60d6c5ef5 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20250113 02:51:22.176148 1194 consensus_queue.cc:260] T 6c8ca30720664f3a819c4bff2493934d P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:22.176759 1194 ts_tablet_manager.cc:1428] T 6c8ca30720664f3a819c4bff2493934d P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250113 02:51:22.177048 1194 tablet_bootstrap.cc:492] T 4fe54131aafb40ee9d8bc5ea299af193 P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:22.177445 1198 ts_tablet_manager.cc:1905] T 9b5dc06ba3ea4056822ca3e4b1249587 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:22.179034 1194 tablet_bootstrap.cc:654] T 4fe54131aafb40ee9d8bc5ea299af193 P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:22.179221 1192 tablet_bootstrap.cc:492] T 8547b63c40ec4ccebd962b5a3b2c5e0f P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:22.181445 1192 tablet_bootstrap.cc:654] T 8547b63c40ec4ccebd962b5a3b2c5e0f P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:22.183362 1195 ts_tablet_manager.cc:1918] T 3d3a6d9f687e45218d5875a7c5597648 P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:22.183497 1195 log.cc:1198] T 3d3a6d9f687e45218d5875a7c5597648 P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/3d3a6d9f687e45218d5875a7c5597648
I20250113 02:51:22.183928 1195 ts_tablet_manager.cc:1939] T 3d3a6d9f687e45218d5875a7c5597648 P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:22.185127 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet 3d3a6d9f687e45218d5875a7c5597648 (table default.Table_21dd1d15c2bb4849af81bb4d8e1a5f04 [id=219903e6141d465bbf5396197210c420]) successfully deleted
I20250113 02:51:22.186811 1368 raft_consensus.cc:491] T d101e98b72544c30a3a4bc60d1576aa3 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:22.186955 1368 raft_consensus.cc:513] T d101e98b72544c30a3a4bc60d1576aa3 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:22.187626 1368 leader_election.cc:290] T d101e98b72544c30a3a4bc60d1576aa3 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:22.188052 1191 tablet_bootstrap.cc:492] T 076ed7744da54ddebd8fc99686beb6ae P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:22.188091 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d101e98b72544c30a3a4bc60d1576aa3" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:22.188189 1191 ts_tablet_manager.cc:1397] T 076ed7744da54ddebd8fc99686beb6ae P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.024s user 0.002s sys 0.001s
I20250113 02:51:22.188297 1061 raft_consensus.cc:2463] T d101e98b72544c30a3a4bc60d1576aa3 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1186fa869eb14812832fa6562531f196 in term 0.
I20250113 02:51:22.188623 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d101e98b72544c30a3a4bc60d1576aa3" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:22.188850 919 raft_consensus.cc:2463] T d101e98b72544c30a3a4bc60d1576aa3 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1186fa869eb14812832fa6562531f196 in term 0.
I20250113 02:51:22.188952 1191 raft_consensus.cc:357] T 076ed7744da54ddebd8fc99686beb6ae P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:22.189170 1191 raft_consensus.cc:383] T 076ed7744da54ddebd8fc99686beb6ae P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:22.189296 1191 raft_consensus.cc:738] T 076ed7744da54ddebd8fc99686beb6ae P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:22.189425 711 leader_election.cc:304] T d101e98b72544c30a3a4bc60d1576aa3 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1186fa869eb14812832fa6562531f196, df69f48656b44bb890f76587ee69cc0f; no voters:
I20250113 02:51:22.189688 1368 raft_consensus.cc:2798] T d101e98b72544c30a3a4bc60d1576aa3 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 02:51:22.189850 1368 raft_consensus.cc:491] T d101e98b72544c30a3a4bc60d1576aa3 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 02:51:22.189955 1368 raft_consensus.cc:3054] T d101e98b72544c30a3a4bc60d1576aa3 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:22.193262 1368 raft_consensus.cc:513] T d101e98b72544c30a3a4bc60d1576aa3 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:22.193966 1368 leader_election.cc:290] T d101e98b72544c30a3a4bc60d1576aa3 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 election: Requested vote from peers 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:22.194118 1191 consensus_queue.cc:260] T 076ed7744da54ddebd8fc99686beb6ae P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:22.194447 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d101e98b72544c30a3a4bc60d1576aa3" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f"
I20250113 02:51:22.194676 919 raft_consensus.cc:3054] T d101e98b72544c30a3a4bc60d1576aa3 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:22.194896 1191 ts_tablet_manager.cc:1428] T 076ed7744da54ddebd8fc99686beb6ae P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20250113 02:51:22.195225 1191 tablet_bootstrap.cc:492] T d35ec7e4a5084a74bea172ef8d9990b5 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:22.197280 919 raft_consensus.cc:2463] T d101e98b72544c30a3a4bc60d1576aa3 P df69f48656b44bb890f76587ee69cc0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1186fa869eb14812832fa6562531f196 in term 1.
I20250113 02:51:22.198244 1192 tablet_bootstrap.cc:492] T 8547b63c40ec4ccebd962b5a3b2c5e0f P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:22.198309 711 leader_election.cc:304] T d101e98b72544c30a3a4bc60d1576aa3 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1186fa869eb14812832fa6562531f196, df69f48656b44bb890f76587ee69cc0f; no voters:
I20250113 02:51:22.198408 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d101e98b72544c30a3a4bc60d1576aa3" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca"
I20250113 02:51:22.198545 1368 raft_consensus.cc:2798] T d101e98b72544c30a3a4bc60d1576aa3 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Leader election won for term 1
I20250113 02:51:22.198638 1061 raft_consensus.cc:3054] T d101e98b72544c30a3a4bc60d1576aa3 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:22.198410 1192 ts_tablet_manager.cc:1397] T 8547b63c40ec4ccebd962b5a3b2c5e0f P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.000s
I20250113 02:51:22.202237 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_0fcdb63a92324044972d1e69630f524d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:22.202883 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0fcdb63a92324044972d1e69630f524d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:22.204515 1192 raft_consensus.cc:357] T 8547b63c40ec4ccebd962b5a3b2c5e0f P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:22.204763 1192 raft_consensus.cc:383] T 8547b63c40ec4ccebd962b5a3b2c5e0f P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:22.204859 1192 raft_consensus.cc:738] T 8547b63c40ec4ccebd962b5a3b2c5e0f P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:22.205081 1192 consensus_queue.cc:260] T 8547b63c40ec4ccebd962b5a3b2c5e0f P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:22.205637 1023 tablet_service.cc:1467] Processing CreateTablet for tablet 80d31d5ead804ddaa032969e401a8f7e (DEFAULT_TABLE table=default.Table_2d201bcde7f04c76b66dd5c6a09ddbff [id=e0d56fb485af49c8bcf9152f95f89b2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:22.205880 1195 tablet_replica.cc:331] T 68023c30a7f644eaa5b4ddd60d6c5ef5 P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:22.206075 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 80d31d5ead804ddaa032969e401a8f7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:22.206100 1195 raft_consensus.cc:2238] T 68023c30a7f644eaa5b4ddd60d6c5ef5 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:22.206262 1195 raft_consensus.cc:2267] T 68023c30a7f644eaa5b4ddd60d6c5ef5 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:22.207000 1191 tablet_bootstrap.cc:654] T d35ec7e4a5084a74bea172ef8d9990b5 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:22.207931 750 tablet_service.cc:1467] Processing CreateTablet for tablet 80d31d5ead804ddaa032969e401a8f7e (DEFAULT_TABLE table=default.Table_2d201bcde7f04c76b66dd5c6a09ddbff [id=e0d56fb485af49c8bcf9152f95f89b2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:22.208333 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 80d31d5ead804ddaa032969e401a8f7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:22.208271 1031 tablet_service.cc:1467] Processing CreateTablet for tablet 30ca8a716bfc4ab789b216e17567785e (DEFAULT_TABLE table=default.Table_65c5f2ca0aea479b9edd500ceb8c01bc [id=9eaf97b701fd4e79847dce3a9d259beb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:22.208632 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30ca8a716bfc4ab789b216e17567785e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:22.209237 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_029fb0ede6224bf4b0cfa388bee300a7" } new_table_name: "default.Table_db2a6ff23fc94f67b8883df9728be422" modify_external_catalogs: true
I20250113 02:51:22.212280 741 tablet_service.cc:1467] Processing CreateTablet for tablet 30ca8a716bfc4ab789b216e17567785e (DEFAULT_TABLE table=default.Table_65c5f2ca0aea479b9edd500ceb8c01bc [id=9eaf97b701fd4e79847dce3a9d259beb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:22.212342 1033 tablet_service.cc:1467] Processing CreateTablet for tablet 44d2c7d5334c476fac60fdd57e2c02b6 (DEFAULT_TABLE table=default.Table_367f2846b81e40eb82c5dd077f36017c [id=2931f217041c46cc8ae52bd6ce8cfd17]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:22.212705 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30ca8a716bfc4ab789b216e17567785e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:22.212831 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44d2c7d5334c476fac60fdd57e2c02b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:22.216102 755 tablet_service.cc:1467] Processing CreateTablet for tablet 44d2c7d5334c476fac60fdd57e2c02b6 (DEFAULT_TABLE table=default.Table_367f2846b81e40eb82c5dd077f36017c [id=2931f217041c46cc8ae52bd6ce8cfd17]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:22.216454 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44d2c7d5334c476fac60fdd57e2c02b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:22.216995 1508 raft_consensus.cc:491] T d35ec7e4a5084a74bea172ef8d9990b5 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:22.217304 1508 raft_consensus.cc:513] T d35ec7e4a5084a74bea172ef8d9990b5 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:22.219077 1508 leader_election.cc:290] T d35ec7e4a5084a74bea172ef8d9990b5 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
I20250113 02:51:22.219283 1508 raft_consensus.cc:491] T 18e2d79d731142f89b9089869bc736e5 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:22.219436 1508 raft_consensus.cc:513] T 18e2d79d731142f89b9089869bc736e5 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:22.220196 746 tablet_service.cc:1467] Processing CreateTablet for tablet d9df314ffaf042ec80180868d7e7bdf7 (DEFAULT_TABLE table=default.Table_2155bf71d1a04a93a28b13c97f636fab [id=00107bce6ce24feba58176198a222970]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:22.221055 1508 leader_election.cc:290] T 18e2d79d731142f89b9089869bc736e5 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:22.221098 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d9df314ffaf042ec80180868d7e7bdf7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:22.224283 754 tablet_service.cc:1467] Processing CreateTablet for tablet 0644f5600c5b47e29f3fa73060ed96e3 (DEFAULT_TABLE table=default.Table_cd499f4a8487400690a9ab957c606b7f [id=be5f659ed070438bb6d0e61a3820eb58]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:22.224684 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0644f5600c5b47e29f3fa73060ed96e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:22.225536 748 tablet_service.cc:1467] Processing CreateTablet for tablet b0113b16110e4787ac1c4676b2c8bd1e (DEFAULT_TABLE table=default.Table_d5c2b805dfbf4227bb1ac190525b7dee [id=bf855203fc20403294cc733cdbc593e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:22.225934 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b0113b16110e4787ac1c4676b2c8bd1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:22.224295 742 tablet_service.cc:1467] Processing CreateTablet for tablet 655b63ff61ff4f4aa08e0449df39352a (DEFAULT_TABLE table=default.Table_c6c64517b49e4843ba63567a871d22bd [id=ec4bdc424404427589852603bb6d8112]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:22.228241 738 tablet_service.cc:1467] Processing CreateTablet for tablet 1014e51f1c184d729552a5b34292f232 (DEFAULT_TABLE table=default.Table_dd19b1ec1af841a7a475c3ace96712a6 [id=f7e15a7c1fe848d591ba3232281c45ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:22.228332 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 655b63ff61ff4f4aa08e0449df39352a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:22.228806 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d35ec7e4a5084a74bea172ef8d9990b5" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
I20250113 02:51:22.229161 747 tablet_service.cc:1467] Processing CreateTablet for tablet bbc7832c50114d6f890fec45a3d994e9 (DEFAULT_TABLE table=default.Table_3c637eb181574498b44ecb4fc41f0117 [id=10c0207c68d8463e91f09972031ab8d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:22.229615 751 tablet_service.cc:1467] Processing CreateTablet for tablet dd566a49d3ff4d3889b80d8d2ccbee0e (DEFAULT_TABLE table=default.Table_3bb52edb283b4667b984b8e0370430a5 [id=0e2055968e5a492492305d4513f0edba]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:22.230118 744 tablet_service.cc:1467] Processing CreateTablet for tablet f1e32e09a8244957a99bdfea5812ff31 (DEFAULT_TABLE table=default.Table_12a092a4543e4b04a466945f4b694c53 [id=c84b256e3b5744edbbf4de12eeaff0a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:22.230480 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18e2d79d731142f89b9089869bc736e5" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
I20250113 02:51:22.231011 1029 tablet_service.cc:1467] Processing CreateTablet for tablet d9df314ffaf042ec80180868d7e7bdf7 (DEFAULT_TABLE table=default.Table_2155bf71d1a04a93a28b13c97f636fab [id=00107bce6ce24feba58176198a222970]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:22.231061 1027 tablet_service.cc:1467] Processing CreateTablet for tablet 0644f5600c5b47e29f3fa73060ed96e3 (DEFAULT_TABLE table=default.Table_cd499f4a8487400690a9ab957c606b7f [id=be5f659ed070438bb6d0e61a3820eb58]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:22.231446 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d9df314ffaf042ec80180868d7e7bdf7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:22.231491 1035 tablet_service.cc:1467] Processing CreateTablet for tablet f8a6eebb6b6d4b63903e68750e181707 (DEFAULT_TABLE table=default.Table_4484fe40b4454c5ea4943a4d38f49c5e [id=a0e91da6e0ee449b92b7afbc6f68087d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:22.232054 1194 tablet_bootstrap.cc:492] T 4fe54131aafb40ee9d8bc5ea299af193 P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:22.232209 1194 ts_tablet_manager.cc:1397] T 4fe54131aafb40ee9d8bc5ea299af193 P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.055s user 0.004s sys 0.003s
I20250113 02:51:22.232225 752 tablet_service.cc:1467] Processing CreateTablet for tablet eceed0f52c9f46bbaa78ca773a7afde5 (DEFAULT_TABLE table=default.Table_6a0195b054a043269e133f6ed8dd5979 [id=4b2bd999368c4b628c70cb3ee8c2b1f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:22.232259 1025 tablet_service.cc:1467] Processing CreateTablet for tablet 655b63ff61ff4f4aa08e0449df39352a (DEFAULT_TABLE table=default.Table_c6c64517b49e4843ba63567a871d22bd [id=ec4bdc424404427589852603bb6d8112]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:22.232615 1032 tablet_service.cc:1467] Processing CreateTablet for tablet b0113b16110e4787ac1c4676b2c8bd1e (DEFAULT_TABLE table=default.Table_d5c2b805dfbf4227bb1ac190525b7dee [id=bf855203fc20403294cc733cdbc593e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:22.232654 745 tablet_service.cc:1467] Processing CreateTablet for tablet 9e736211eb3c4536b9fd1d90d2d83838 (DEFAULT_TABLE table=default.Table_f8af5d3e0dc64d3fae054f82a7b33ccf [id=4b739228564344b8b394a4718750dbb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:22.232949 1039 tablet_service.cc:1467] Processing CreateTablet for tablet 1014e51f1c184d729552a5b34292f232 (DEFAULT_TABLE table=default.Table_dd19b1ec1af841a7a475c3ace96712a6 [id=f7e15a7c1fe848d591ba3232281c45ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:22.228642 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1014e51f1c184d729552a5b34292f232. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:22.233263 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1e32e09a8244957a99bdfea5812ff31. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:22.233260 1036 tablet_service.cc:1467] Processing CreateTablet for tablet eceed0f52c9f46bbaa78ca773a7afde5 (DEFAULT_TABLE table=default.Table_6a0195b054a043269e133f6ed8dd5979 [id=4b2bd999368c4b628c70cb3ee8c2b1f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:22.233580 1026 tablet_service.cc:1467] Processing CreateTablet for tablet 9e736211eb3c4536b9fd1d90d2d83838 (DEFAULT_TABLE table=default.Table_f8af5d3e0dc64d3fae054f82a7b33ccf [id=4b739228564344b8b394a4718750dbb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:22.235409 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f8a6eebb6b6d4b63903e68750e181707. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:22.236177 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd566a49d3ff4d3889b80d8d2ccbee0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:22.236943 1494 catalog_manager.cc:1603] T 00000000000000000000000000000000 P 6bbc3541604a455584bb32dfc2cc4efd: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250113 02:51:22.237972 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bbc7832c50114d6f890fec45a3d994e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:22.240283 1036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eceed0f52c9f46bbaa78ca773a7afde5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:22.240626 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eceed0f52c9f46bbaa78ca773a7afde5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:22.241351 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0644f5600c5b47e29f3fa73060ed96e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:22.243016 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e736211eb3c4536b9fd1d90d2d83838. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:22.244313 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e736211eb3c4536b9fd1d90d2d83838. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:22.245512 995 leader_election.cc:343] T d35ec7e4a5084a74bea172ef8d9990b5 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Illegal state: must be running to vote when last-logged opid is not known
W20250113 02:51:22.245718 995 leader_election.cc:343] T 18e2d79d731142f89b9089869bc736e5 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:22.246629 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1014e51f1c184d729552a5b34292f232. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:22.249998 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 655b63ff61ff4f4aa08e0449df39352a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:22.252316 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b0113b16110e4787ac1c4676b2c8bd1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:22.254041 736 tablet_service.cc:1467] Processing CreateTablet for tablet bc6a7322c0c34adb85457750526d8955 (DEFAULT_TABLE table=default.Table_6185ed1a90a44576bc65ed60294c2537 [id=5634c185da26460f93cde9c65fbdb24a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:22.254442 749 tablet_service.cc:1467] Processing CreateTablet for tablet ee240d838d4a4bbba97f8ed61f69de78 (DEFAULT_TABLE table=default.Table_deb7c4bd8a954e5081cdbee12274dbcf [id=383148ed836144c1bc5403dae46aad50]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:22.254863 737 tablet_service.cc:1467] Processing CreateTablet for tablet b5c1882125b5491ea79a29a6dac6771e (DEFAULT_TABLE table=default.Table_029fb0ede6224bf4b0cfa388bee300a7 [id=3237c1be7e1c48918ff095162d191dc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:22.255036 740 tablet_service.cc:1467] Processing CreateTablet for tablet c518f791b4d74c54ab0c0364d29b4a64 (DEFAULT_TABLE table=default.Table_157d1350cce74502a990462ac9bd0e5f [id=f96a39bfcdf442f7965a7b16d9aa432f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:22.257045 736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc6a7322c0c34adb85457750526d8955. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:22.258376 1194 raft_consensus.cc:357] T 4fe54131aafb40ee9d8bc5ea299af193 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:22.258618 1194 raft_consensus.cc:383] T 4fe54131aafb40ee9d8bc5ea299af193 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:22.258777 1194 raft_consensus.cc:738] T 4fe54131aafb40ee9d8bc5ea299af193 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:22.259230 1194 consensus_queue.cc:260] T 4fe54131aafb40ee9d8bc5ea299af193 P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:22.259857 1194 ts_tablet_manager.cc:1428] T 4fe54131aafb40ee9d8bc5ea299af193 P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.027s user 0.002s sys 0.000s
I20250113 02:51:22.260329 1194 tablet_bootstrap.cc:492] T d7c4c7b0cfc44ac1952bc73cbe1bced8 P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:22.261009 1037 tablet_service.cc:1467] Processing CreateTablet for tablet dd566a49d3ff4d3889b80d8d2ccbee0e (DEFAULT_TABLE table=default.Table_3bb52edb283b4667b984b8e0370430a5 [id=0e2055968e5a492492305d4513f0edba]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:22.261372 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c518f791b4d74c54ab0c0364d29b4a64. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:22.261427 1037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd566a49d3ff4d3889b80d8d2ccbee0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:22.262039 737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5c1882125b5491ea79a29a6dac6771e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:22.262094 1030 tablet_service.cc:1467] Processing CreateTablet for tablet ee240d838d4a4bbba97f8ed61f69de78 (DEFAULT_TABLE table=default.Table_deb7c4bd8a954e5081cdbee12274dbcf [id=383148ed836144c1bc5403dae46aad50]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:22.262476 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee240d838d4a4bbba97f8ed61f69de78. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:22.262825 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee240d838d4a4bbba97f8ed61f69de78. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:22.263257 1024 tablet_service.cc:1467] Processing CreateTablet for tablet f1e32e09a8244957a99bdfea5812ff31 (DEFAULT_TABLE table=default.Table_12a092a4543e4b04a466945f4b694c53 [id=c84b256e3b5744edbbf4de12eeaff0a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:22.263604 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1e32e09a8244957a99bdfea5812ff31. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:22.264343 1042 tablet_service.cc:1467] Processing CreateTablet for tablet b5c1882125b5491ea79a29a6dac6771e (DEFAULT_TABLE table=default.Table_029fb0ede6224bf4b0cfa388bee300a7 [id=3237c1be7e1c48918ff095162d191dc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:22.264673 1042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5c1882125b5491ea79a29a6dac6771e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:22.265249 1040 tablet_service.cc:1467] Processing CreateTablet for tablet bc6a7322c0c34adb85457750526d8955 (DEFAULT_TABLE table=default.Table_6185ed1a90a44576bc65ed60294c2537 [id=5634c185da26460f93cde9c65fbdb24a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:22.265584 1040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc6a7322c0c34adb85457750526d8955. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:22.265743 743 tablet_service.cc:1467] Processing CreateTablet for tablet bc533518faa24dffa3306ff092e8d507 (DEFAULT_TABLE table=default.Table_03babf194cc448f296512a09b2e3fe3e [id=64b42698782f41a589577af20b33501e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:22.266600 1038 tablet_service.cc:1467] Processing CreateTablet for tablet c518f791b4d74c54ab0c0364d29b4a64 (DEFAULT_TABLE table=default.Table_157d1350cce74502a990462ac9bd0e5f [id=f96a39bfcdf442f7965a7b16d9aa432f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:22.266934 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c518f791b4d74c54ab0c0364d29b4a64. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:22.267194 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc533518faa24dffa3306ff092e8d507. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:22.267449 1041 tablet_service.cc:1467] Processing CreateTablet for tablet bc533518faa24dffa3306ff092e8d507 (DEFAULT_TABLE table=default.Table_03babf194cc448f296512a09b2e3fe3e [id=64b42698782f41a589577af20b33501e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:22.201197 1061 raft_consensus.cc:2463] T d101e98b72544c30a3a4bc60d1576aa3 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1186fa869eb14812832fa6562531f196 in term 1.
I20250113 02:51:22.267814 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc533518faa24dffa3306ff092e8d507. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:22.268210 1034 tablet_service.cc:1467] Processing CreateTablet for tablet 788ab82626424b6e833e31761a15b9b2 (DEFAULT_TABLE table=default.Table_2d289859f3a2422b9f1c5bd6b3083088 [id=0f07147c73f24decb09399a1dd188e1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:22.268543 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 788ab82626424b6e833e31761a15b9b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:22.269073 1028 tablet_service.cc:1467] Processing CreateTablet for tablet c7842d6f2c074e3b9d1a3378db9856be (DEFAULT_TABLE table=default.Table_59401ddb940e4e789a5bd54f96548ba5 [id=0ac370ed485e4d64a3e1aa1e8d67ea23]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:22.269409 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c7842d6f2c074e3b9d1a3378db9856be. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:22.270076 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet f56230a2c75d4763a54671c79358f4d9
I20250113 02:51:22.270233 1192 ts_tablet_manager.cc:1428] T 8547b63c40ec4ccebd962b5a3b2c5e0f P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.005s user 0.000s sys 0.000s
I20250113 02:51:22.270816 446 master_service.cc:946] ReplaceTablet: received request to replace tablet 8aac142230e244c0b59c9c2b7a7300ea from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:22.271173 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet f129bb3fbbfd492d98822be0a6626e44
I20250113 02:51:22.271638 448 catalog_manager.cc:6596] ReplaceTablet: tablet f129bb3fbbfd492d98822be0a6626e44 deleted and replaced by tablet 67baaab1e36e415381579d7385d4ab5a
I20250113 02:51:22.272306 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet bbc7832c50114d6f890fec45a3d994e9
I20250113 02:51:22.272543 1194 tablet_bootstrap.cc:654] T d7c4c7b0cfc44ac1952bc73cbe1bced8 P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:22.274191 753 tablet_service.cc:1467] Processing CreateTablet for tablet c7842d6f2c074e3b9d1a3378db9856be (DEFAULT_TABLE table=default.Table_59401ddb940e4e789a5bd54f96548ba5 [id=0ac370ed485e4d64a3e1aa1e8d67ea23]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:22.274551 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c7842d6f2c074e3b9d1a3378db9856be. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:22.277886 739 tablet_service.cc:1467] Processing CreateTablet for tablet 788ab82626424b6e833e31761a15b9b2 (DEFAULT_TABLE table=default.Table_2d289859f3a2422b9f1c5bd6b3083088 [id=0f07147c73f24decb09399a1dd188e1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:22.278607 741 tablet_service.cc:1467] Processing CreateTablet for tablet f8a6eebb6b6d4b63903e68750e181707 (DEFAULT_TABLE table=default.Table_4484fe40b4454c5ea4943a4d38f49c5e [id=a0e91da6e0ee449b92b7afbc6f68087d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:22.279484 1516 raft_consensus.cc:491] T 34ab7ee573284550aaac9f814e8bd7ab P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:22.279641 1516 raft_consensus.cc:513] T 34ab7ee573284550aaac9f814e8bd7ab P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:22.280804 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34ab7ee573284550aaac9f814e8bd7ab" candidate_uuid: "df69f48656b44bb890f76587ee69cc0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:22.281157 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet f56230a2c75d4763a54671c79358f4d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:22 UTC) from {username='slave'} at 127.0.0.1:60230
W20250113 02:51:22.281291 852 leader_election.cc:343] T 34ab7ee573284550aaac9f814e8bd7ab P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:22.281361 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet f129bb3fbbfd492d98822be0a6626e44 with delete_type TABLET_DATA_DELETED (replaced by tablet 67baaab1e36e415381579d7385d4ab5a) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:22.281520 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet bbc7832c50114d6f890fec45a3d994e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:22 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:22.281698 1516 leader_election.cc:290] T 34ab7ee573284550aaac9f814e8bd7ab P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587), 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
I20250113 02:51:22.282541 1192 tablet_bootstrap.cc:492] T 1d81eec487dd4016b3b804451da31c49 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:22.282693 1195 ts_tablet_manager.cc:1905] T 68023c30a7f644eaa5b4ddd60d6c5ef5 P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:22.283833 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 0a7e7713bc8648378b668d8b769ac927
I20250113 02:51:22.284041 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 1a26e2b5e1054aa6afe7f44bf8d43108
I20250113 02:51:22.284607 452 master_service.cc:946] ReplaceTablet: received request to replace tablet 33c665e149dd48e8868acd935de581bf from {username='slave'} at 127.0.0.1:49258
W20250113 02:51:22.285157 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b5c1882125b5491ea79a29a6dac6771e consensus state has no leader
I20250113 02:51:22.285179 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 05bf5ab0df874dd5b6559f6d82069184
I20250113 02:51:22.285650 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_7fc5d3f1e17c47c9938bc53627275ae0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:22.286901 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_1505767419694e5ba79a90c46780bbb7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:22.287492 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1505767419694e5ba79a90c46780bbb7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250113 02:51:22.288947 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7fc5d3f1e17c47c9938bc53627275ae0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:22.288564 453 catalog_manager.cc:5526] T d5bd09a1ede644c59e400e49c8125793 P 4216a7f853d642b7bc3e5cff458f3aca reported cstate change: term changed from 0 to 1, leader changed from <none> to 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3). New cstate: current_term: 1 leader_uuid: "4216a7f853d642b7bc3e5cff458f3aca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } health_report { overall_health: UNKNOWN } } }
I20250113 02:51:22.290467 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_5a11b1fe188b4b4ba215476c50f5f974"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:22.290902 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5a11b1fe188b4b4ba215476c50f5f974 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:22.297896 1032 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a7e7713bc8648378b668d8b769ac927 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:21 UTC) from {username='slave'} at 127.0.0.1:60230
W20250113 02:51:22.298394 998 leader_election.cc:343] T 18e2d79d731142f89b9089869bc736e5 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Not found: Tablet not found: 18e2d79d731142f89b9089869bc736e5
I20250113 02:51:22.298586 998 leader_election.cc:304] T 18e2d79d731142f89b9089869bc736e5 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4216a7f853d642b7bc3e5cff458f3aca; no voters: 1186fa869eb14812832fa6562531f196, df69f48656b44bb890f76587ee69cc0f
W20250113 02:51:22.299067 998 leader_election.cc:343] T d35ec7e4a5084a74bea172ef8d9990b5 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Not found: Tablet not found: d35ec7e4a5084a74bea172ef8d9990b5
I20250113 02:51:22.299064 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34ab7ee573284550aaac9f814e8bd7ab" candidate_uuid: "df69f48656b44bb890f76587ee69cc0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
I20250113 02:51:22.299296 998 leader_election.cc:304] T d35ec7e4a5084a74bea172ef8d9990b5 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4216a7f853d642b7bc3e5cff458f3aca; no voters: 1186fa869eb14812832fa6562531f196, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:22.299530 1508 raft_consensus.cc:2743] T 18e2d79d731142f89b9089869bc736e5 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:22.299716 1508 raft_consensus.cc:2743] T d35ec7e4a5084a74bea172ef8d9990b5 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250113 02:51:22.299685 851 leader_election.cc:343] T 34ab7ee573284550aaac9f814e8bd7ab P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:22.299839 851 leader_election.cc:304] T 34ab7ee573284550aaac9f814e8bd7ab P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df69f48656b44bb890f76587ee69cc0f; no voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca
I20250113 02:51:22.301851 1516 raft_consensus.cc:2743] T 34ab7ee573284550aaac9f814e8bd7ab P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:22.303481 881 tablet_service.cc:1467] Processing CreateTablet for tablet 80d31d5ead804ddaa032969e401a8f7e (DEFAULT_TABLE table=default.Table_2d201bcde7f04c76b66dd5c6a09ddbff [id=e0d56fb485af49c8bcf9152f95f89b2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:22.303692 880 tablet_service.cc:1467] Processing CreateTablet for tablet 30ca8a716bfc4ab789b216e17567785e (DEFAULT_TABLE table=default.Table_65c5f2ca0aea479b9edd500ceb8c01bc [id=9eaf97b701fd4e79847dce3a9d259beb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:22.303923 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 80d31d5ead804ddaa032969e401a8f7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:22.304406 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30ca8a716bfc4ab789b216e17567785e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:22.304601 892 tablet_service.cc:1467] Processing CreateTablet for tablet 44d2c7d5334c476fac60fdd57e2c02b6 (DEFAULT_TABLE table=default.Table_367f2846b81e40eb82c5dd077f36017c [id=2931f217041c46cc8ae52bd6ce8cfd17]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:22.305035 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44d2c7d5334c476fac60fdd57e2c02b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:22.307482 896 tablet_service.cc:1467] Processing CreateTablet for tablet b0113b16110e4787ac1c4676b2c8bd1e (DEFAULT_TABLE table=default.Table_d5c2b805dfbf4227bb1ac190525b7dee [id=bf855203fc20403294cc733cdbc593e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:22.307794 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b0113b16110e4787ac1c4676b2c8bd1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:22.308225 884 tablet_service.cc:1467] Processing CreateTablet for tablet d9df314ffaf042ec80180868d7e7bdf7 (DEFAULT_TABLE table=default.Table_2155bf71d1a04a93a28b13c97f636fab [id=00107bce6ce24feba58176198a222970]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:22.308606 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d9df314ffaf042ec80180868d7e7bdf7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:22.310153 889 tablet_service.cc:1467] Processing CreateTablet for tablet 9e736211eb3c4536b9fd1d90d2d83838 (DEFAULT_TABLE table=default.Table_f8af5d3e0dc64d3fae054f82a7b33ccf [id=4b739228564344b8b394a4718750dbb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:22.310523 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e736211eb3c4536b9fd1d90d2d83838. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:22.312541 898 tablet_service.cc:1467] Processing CreateTablet for tablet 0644f5600c5b47e29f3fa73060ed96e3 (DEFAULT_TABLE table=default.Table_cd499f4a8487400690a9ab957c606b7f [id=be5f659ed070438bb6d0e61a3820eb58]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:22.312883 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0644f5600c5b47e29f3fa73060ed96e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:22.316360 882 tablet_service.cc:1467] Processing CreateTablet for tablet f8a6eebb6b6d4b63903e68750e181707 (DEFAULT_TABLE table=default.Table_4484fe40b4454c5ea4943a4d38f49c5e [id=a0e91da6e0ee449b92b7afbc6f68087d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:22.316756 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f8a6eebb6b6d4b63903e68750e181707. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:22.317806 888 tablet_service.cc:1467] Processing CreateTablet for tablet dd566a49d3ff4d3889b80d8d2ccbee0e (DEFAULT_TABLE table=default.Table_3bb52edb283b4667b984b8e0370430a5 [id=0e2055968e5a492492305d4513f0edba]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:22.318220 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd566a49d3ff4d3889b80d8d2ccbee0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:22.319715 886 tablet_service.cc:1467] Processing CreateTablet for tablet 1014e51f1c184d729552a5b34292f232 (DEFAULT_TABLE table=default.Table_dd19b1ec1af841a7a475c3ace96712a6 [id=f7e15a7c1fe848d591ba3232281c45ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:22.320106 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1014e51f1c184d729552a5b34292f232. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:22.320559 887 tablet_service.cc:1467] Processing CreateTablet for tablet f1e32e09a8244957a99bdfea5812ff31 (DEFAULT_TABLE table=default.Table_12a092a4543e4b04a466945f4b694c53 [id=c84b256e3b5744edbbf4de12eeaff0a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:22.321421 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1e32e09a8244957a99bdfea5812ff31. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:22.322710 895 tablet_service.cc:1467] Processing CreateTablet for tablet eceed0f52c9f46bbaa78ca773a7afde5 (DEFAULT_TABLE table=default.Table_6a0195b054a043269e133f6ed8dd5979 [id=4b2bd999368c4b628c70cb3ee8c2b1f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:22.323151 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eceed0f52c9f46bbaa78ca773a7afde5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:22.324298 889 tablet_service.cc:1467] Processing CreateTablet for tablet bbc7832c50114d6f890fec45a3d994e9 (DEFAULT_TABLE table=default.Table_3c637eb181574498b44ecb4fc41f0117 [id=10c0207c68d8463e91f09972031ab8d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:22.324618 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bbc7832c50114d6f890fec45a3d994e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:22.330536 894 tablet_service.cc:1467] Processing CreateTablet for tablet 655b63ff61ff4f4aa08e0449df39352a (DEFAULT_TABLE table=default.Table_c6c64517b49e4843ba63567a871d22bd [id=ec4bdc424404427589852603bb6d8112]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:22.330928 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 655b63ff61ff4f4aa08e0449df39352a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:22.332460 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet 05bf5ab0df874dd5b6559f6d82069184 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:21 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:22.339790 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 788ab82626424b6e833e31761a15b9b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:21.935585 1345 rpcz_store.cc:276] 0113 02:51:20.645649 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:20.646358 (+ 709us) service_pool.cc:225] Handling call
0113 02:51:20.963180 (+316822us) raft_consensus.cc:1466] Updating replica for 300 ops
0113 02:51:21.016346 (+ 53166us) raft_consensus.cc:1529] Early marking committed up to index 18
0113 02:51:21.029650 (+ 13304us) raft_consensus.cc:1534] Triggering prepare for 300 ops
0113 02:51:21.030114 (+ 464us) write_op.cc:270] Start()
0113 02:51:21.030167 (+ 53us) write_op.cc:276] Timestamp: P: 1736736675981440 usec, L: 0
0113 02:51:21.030461 (+ 294us) write_op.cc:270] Start()
0113 02:51:21.030500 (+ 39us) write_op.cc:276] Timestamp: P: 1736736675983069 usec, L: 0
0113 02:51:21.030637 (+ 137us) write_op.cc:270] Start()
0113 02:51:21.030658 (+ 21us) write_op.cc:276] Timestamp: P: 1736736675983667 usec, L: 0
0113 02:51:21.030837 (+ 179us) write_op.cc:270] Start()
0113 02:51:21.030859 (+ 22us) write_op.cc:276] Timestamp: P: 1736736675984192 usec, L: 0
0113 02:51:21.031164 (+ 305us) write_op.cc:270] Start()
0113 02:51:21.031189 (+ 25us) write_op.cc:276] Timestamp: P: 1736736675984705 usec, L: 0
0113 02:51:21.031311 (+ 122us) write_op.cc:270] Start()
0113 02:51:21.031334 (+ 23us) write_op.cc:276] Timestamp: P: 1736736675985167 usec, L: 0
0113 02:51:21.031446 (+ 112us) write_op.cc:270] Start()
0113 02:51:21.031484 (+ 38us) write_op.cc:276] Timestamp: P: 1736736675985645 usec, L: 0
0113 02:51:21.031597 (+ 113us) write_op.cc:270] Start()
0113 02:51:21.031613 (+ 16us) write_op.cc:276] Timestamp: P: 1736736675986110 usec, L: 0
0113 02:51:21.031732 (+ 119us) write_op.cc:270] Start()
0113 02:51:21.031751 (+ 19us) write_op.cc:276] Timestamp: P: 1736736675986571 usec, L: 0
0113 02:51:21.031941 (+ 190us) write_op.cc:270] Start()
0113 02:51:21.031962 (+ 21us) write_op.cc:276] Timestamp: P: 1736736675987144 usec, L: 0
0113 02:51:21.032141 (+ 179us) write_op.cc:270] Start()
0113 02:51:21.032160 (+ 19us) write_op.cc:276] Timestamp: P: 1736736676002023 usec, L: 0
0113 02:51:21.032314 (+ 154us) write_op.cc:270] Start()
0113 02:51:21.032333 (+ 19us) write_op.cc:276] Timestamp: P: 1736736676003429 usec, L: 0
0113 02:51:21.032472 (+ 139us) write_op.cc:270] Start()
0113 02:51:21.032497 (+ 25us) write_op.cc:276] Timestamp: P: 1736736676018680 usec, L: 0
0113 02:51:21.032615 (+ 118us) write_op.cc:270] Start()
0113 02:51:21.032633 (+ 18us) write_op.cc:276] Timestamp: P: 1736736676019312 usec, L: 0
0113 02:51:21.032778 (+ 145us) write_op.cc:270] Start()
0113 02:51:21.032797 (+ 19us) write_op.cc:276] Timestamp: P: 1736736676025722 usec, L: 0
0113 02:51:21.032937 (+ 140us) write_op.cc:270] Start()
0113 02:51:21.032962 (+ 25us) write_op.cc:276] Timestamp: P: 1736736676026344 usec, L: 0
0113 02:51:21.033104 (+ 142us) write_op.cc:270] Start()
0113 02:51:21.033123 (+ 19us) write_op.cc:276] Timestamp: P: 1736736676026826 usec, L: 0
0113 02:51:21.033249 (+ 126us) write_op.cc:270] Start()
0113 02:51:21.033267 (+ 18us) write_op.cc:276] Timestamp: P: 1736736676028577 usec, L: 0
0113 02:51:21.051700 (+ 18433us) write_op.cc:270] Start()
0113 02:51:21.051744 (+ 44us) write_op.cc:276] Timestamp: P: 1736736676030095 usec, L: 0
0113 02:51:21.051965 (+ 221us) write_op.cc:270] Start()
0113 02:51:21.052012 (+ 47us) write_op.cc:276] Timestamp: P: 1736736676030929 usec, L: 0
0113 02:51:21.052195 (+ 183us) write_op.cc:270] Start()
0113 02:51:21.052217 (+ 22us) write_op.cc:276] Timestamp: P: 1736736676031980 usec, L: 0
0113 02:51:21.052366 (+ 149us) write_op.cc:270] Start()
0113 02:51:21.052387 (+ 21us) write_op.cc:276] Timestamp: P: 1736736676050565 usec, L: 0
0113 02:51:21.052536 (+ 149us) write_op.cc:270] Start()
0113 02:51:21.052555 (+ 19us) write_op.cc:276] Timestamp: P: 1736736676143567 usec, L: 0
0113 02:51:21.052717 (+ 162us) write_op.cc:270] Start()
0113 02:51:21.052737 (+ 20us) write_op.cc:276] Timestamp: P: 1736736676146308 usec, L: 0
0113 02:51:21.052942 (+ 205us) write_op.cc:270] Start()
0113 02:51:21.052988 (+ 46us) write_op.cc:276] Timestamp: P: 1736736676149912 usec, L: 0
0113 02:51:21.053190 (+ 202us) write_op.cc:270] Start()
0113 02:51:21.053219 (+ 29us) write_op.cc:276] Timestamp: P: 1736736676175493 usec, L: 0
0113 02:51:21.053415 (+ 196us) write_op.cc:270] Start()
0113 02:51:21.053435 (+ 20us) write_op.cc:276] Timestamp: P: 1736736676203415 usec, L: 0
0113 02:51:21.053570 (+ 135us) write_op.cc:270] Start()
0113 02:51:21.053596 (+ 26us) write_op.cc:276] Timestamp: P: 1736736676215145 usec, L: 0
0113 02:51:21.053699 (+ 103us) write_op.cc:270] Start()
0113 02:51:21.053715 (+ 16us) write_op.cc:276] Timestamp: P: 1736736676216039 usec, L: 0
0113 02:51:21.053839 (+ 124us) write_op.cc:270] Start()
0113 02:51:21.075523 (+ 21684us) write_op.cc:276] Timestamp: P: 1736736676217095 usec, L: 0
0113 02:51:21.091437 (+ 15914us) write_op.cc:270] Start()
0113 02:51:21.091473 (+ 36us) write_op.cc:276] Timestamp: P: 1736736676225564 usec, L: 0
0113 02:51:21.091661 (+ 188us) write_op.cc:270] Start()
0113 02:51:21.091684 (+ 23us) write_op.cc:276] Timestamp: P: 1736736676226111 usec, L: 0
0113 02:51:21.091850 (+ 166us) write_op.cc:270] Start()
0113 02:51:21.091869 (+ 19us) write_op.cc:276] Timestamp: P: 1736736676228270 usec, L: 0
0113 02:51:21.091992 (+ 123us) write_op.cc:270] Start()
0113 02:51:21.092013 (+ 21us) write_op.cc:276] Timestamp: P: 1736736676229142 usec, L: 0
0113 02:51:21.092153 (+ 140us) write_op.cc:270] Start()
0113 02:51:21.092171 (+ 18us) write_op.cc:276] Timestamp: P: 1736736676237116 usec, L: 0
0113 02:51:21.092309 (+ 138us) write_op.cc:270] Start()
0113 02:51:21.092330 (+ 21us) write_op.cc:276] Timestamp: P: 1736736676238850 usec, L: 0
0113 02:51:21.092461 (+ 131us) write_op.cc:270] Start()
0113 02:51:21.092489 (+ 28us) write_op.cc:276] Timestamp: P: 1736736676239902 usec, L: 0
0113 02:51:21.092634 (+ 145us) write_op.cc:270] Start()
0113 02:51:21.092652 (+ 18us) write_op.cc:276] Timestamp: P: 1736736676248389 usec, L: 0
0113 02:51:21.092792 (+ 140us) write_op.cc:270] Start()
0113 02:51:21.092811 (+ 19us) write_op.cc:276] Timestamp: P: 1736736676249334 usec, L: 0
0113 02:51:21.092952 (+ 141us) write_op.cc:270] Start()
0113 02:51:21.092970 (+ 18us) write_op.cc:276] Timestamp: P: 1736736676250503 usec, L: 0
0113 02:51:21.093100 (+ 130us) write_op.cc:270] Start()
0113 02:51:21.093120 (+ 20us) write_op.cc:276] Timestamp: P: 1736736676268761 usec, L: 0
0113 02:51:21.093248 (+ 128us) write_op.cc:270] Start()
0113 02:51:21.093265 (+ 17us) write_op.cc:276] Timestamp: P: 1736736676272374 usec, L: 0
0113 02:51:21.093391 (+ 126us) write_op.cc:270] Start()
0113 02:51:21.093413 (+ 22us) write_op.cc:276] Timestamp: P: 1736736676278702 usec, L: 0
0113 02:51:21.093564 (+ 151us) write_op.cc:270] Start()
0113 02:51:21.093604 (+ 40us) write_op.cc:276] Timestamp: P: 1736736676285155 usec, L: 0
0113 02:51:21.093742 (+ 138us) write_op.cc:270] Start()
0113 02:51:21.093762 (+ 20us) write_op.cc:276] Timestamp: P: 1736736676286649 usec, L: 0
0113 02:51:21.093893 (+ 131us) write_op.cc:270] Start()
0113 02:51:21.093914 (+ 21us) write_op.cc:276] Timestamp: P: 1736736676288663 usec, L: 0
0113 02:51:21.094054 (+ 140us) write_op.cc:270] Start()
0113 02:51:21.094077 (+ 23us) write_op.cc:276] Timestamp: P: 1736736676289794 usec, L: 0
0113 02:51:21.094258 (+ 181us) write_op.cc:270] Start()
0113 02:51:21.094279 (+ 21us) write_op.cc:276] Timestamp: P: 1736736676290710 usec, L: 0
0113 02:51:21.094417 (+ 138us) write_op.cc:270] Start()
0113 02:51:21.094436 (+ 19us) write_op.cc:276] Timestamp: P: 1736736676338950 usec, L: 0
0113 02:51:21.094702 (+ 266us) write_op.cc:270] Start()
0113 02:51:21.094729 (+ 27us) write_op.cc:276] Timestamp: P: 1736736676385557 usec, L: 0
0113 02:51:21.094957 (+ 228us) write_op.cc:270] Start()
0113 02:51:21.094992 (+ 35us) write_op.cc:276] Timestamp: P: 1736736676401204 usec, L: 0
0113 02:51:21.095178 (+ 186us) write_op.cc:270] Start()
0113 02:51:21.095203 (+ 25us) write_op.cc:276] Timestamp: P: 1736736676430223 usec, L: 0
0113 02:51:21.095379 (+ 176us) write_op.cc:270] Start()
0113 02:51:21.095402 (+ 23us) write_op.cc:276] Timestamp: P: 1736736676441416 usec, L: 0
0113 02:51:21.095532 (+ 130us) write_op.cc:270] Start()
0113 02:51:21.095552 (+ 20us) write_op.cc:276] Timestamp: P: 1736736676442030 usec, L: 0
0113 02:51:21.095758 (+ 206us) write_op.cc:270] Start()
0113 02:51:21.095780 (+ 22us) write_op.cc:276] Timestamp: P: 1736736676456827 usec, L: 0
0113 02:51:21.095936 (+ 156us) write_op.cc:270] Start()
0113 02:51:21.095958 (+ 22us) write_op.cc:276] Timestamp: P: 1736736676459629 usec, L: 0
0113 02:51:21.100563 (+ 4605us) write_op.cc:270] Start()
0113 02:51:21.100602 (+ 39us) write_op.cc:276] Timestamp: P: 1736736676477466 usec, L: 0
0113 02:51:21.100751 (+ 149us) write_op.cc:270] Start()
0113 02:51:21.100771 (+ 20us) write_op.cc:276] Timestamp: P: 1736736676478198 usec, L: 0
0113 02:51:21.100891 (+ 120us) write_op.cc:270] Start()
0113 02:51:21.100908 (+ 17us) write_op.cc:276] Timestamp: P: 1736736676478824 usec, L: 0
0113 02:51:21.101013 (+ 105us) write_op.cc:270] Start()
0113 02:51:21.101048 (+ 35us) write_op.cc:276] Timestamp: P: 1736736676479364 usec, L: 0
0113 02:51:21.101152 (+ 104us) write_op.cc:270] Start()
0113 02:51:21.101167 (+ 15us) write_op.cc:276] Timestamp: P: 1736736676480464 usec, L: 0
0113 02:51:21.101499 (+ 332us) write_op.cc:270] Start()
0113 02:51:21.101521 (+ 22us) write_op.cc:276] Timestamp: P: 1736736676481274 usec, L: 0
0113 02:51:21.101662 (+ 141us) write_op.cc:270] Start()
0113 02:51:21.101679 (+ 17us) write_op.cc:276] Timestamp: P: 1736736676483069 usec, L: 0
0113 02:51:21.101789 (+ 110us) write_op.cc:270] Start()
0113 02:51:21.101805 (+ 16us) write_op.cc:276] Timestamp: P: 1736736676488699 usec, L: 0
0113 02:51:21.101934 (+ 129us) write_op.cc:270] Start()
0113 02:51:21.101950 (+ 16us) write_op.cc:276] Timestamp: P: 1736736676492131 usec, L: 0
0113 02:51:21.102050 (+ 100us) write_op.cc:270] Start()
0113 02:51:21.102066 (+ 16us) write_op.cc:276] Timestamp: P: 1736736676493036 usec, L: 0
0113 02:51:21.102173 (+ 107us) write_op.cc:270] Start()
0113 02:51:21.102197 (+ 24us) write_op.cc:276] Timestamp: P: 1736736676500979 usec, L: 0
0113 02:51:21.102319 (+ 122us) write_op.cc:270] Start()
0113 02:51:21.102334 (+ 15us) write_op.cc:276] Timestamp: P: 1736736676501880 usec, L: 0
0113 02:51:21.102444 (+ 110us) write_op.cc:270] Start()
0113 02:51:21.102468 (+ 24us) write_op.cc:276] Timestamp: P: 1736736676504920 usec, L: 0
0113 02:51:21.103001 (+ 533us) write_op.cc:270] Start()
0113 02:51:21.103028 (+ 27us) write_op.cc:276] Timestamp: P: 1736736676505916 usec, L: 0
0113 02:51:21.103163 (+ 135us) write_op.cc:270] Start()
0113 02:51:21.103180 (+ 17us) write_op.cc:276] Timestamp: P: 1736736676521313 usec, L: 0
0113 02:51:21.103443 (+ 263us) write_op.cc:270] Start()
0113 02:51:21.103462 (+ 19us) write_op.cc:276] Timestamp: P: 1736736676535105 usec, L: 0
0113 02:51:21.103597 (+ 135us) write_op.cc:270] Start()
0113 02:51:21.103618 (+ 21us) write_op.cc:276] Timestamp: P: 1736736676535709 usec, L: 0
0113 02:51:21.103756 (+ 138us) write_op.cc:270] Start()
0113 02:51:21.103775 (+ 19us) write_op.cc:276] Timestamp: P: 1736736676536319 usec, L: 0
0113 02:51:21.103906 (+ 131us) write_op.cc:270] Start()
0113 02:51:21.103926 (+ 20us) write_op.cc:276] Timestamp: P: 1736736676559756 usec, L: 0
0113 02:51:21.104075 (+ 149us) write_op.cc:270] Start()
0113 02:51:21.104094 (+ 19us) write_op.cc:276] Timestamp: P: 1736736676602660 usec, L: 0
0113 02:51:21.104223 (+ 129us) write_op.cc:270] Start()
0113 02:51:21.104243 (+ 20us) write_op.cc:276] Timestamp: P: 1736736676652285 usec, L: 0
0113 02:51:21.104355 (+ 112us) write_op.cc:270] Start()
0113 02:51:21.104372 (+ 17us) write_op.cc:276] Timestamp: P: 1736736676658976 usec, L: 0
0113 02:51:21.104491 (+ 119us) write_op.cc:270] Start()
0113 02:51:21.104536 (+ 45us) write_op.cc:276] Timestamp: P: 1736736676669096 usec, L: 0
0113 02:51:21.104712 (+ 176us) write_op.cc:270] Start()
0113 02:51:21.104730 (+ 18us) write_op.cc:276] Timestamp: P: 1736736676669832 usec, L: 0
0113 02:51:21.104833 (+ 103us) write_op.cc:270] Start()
0113 02:51:21.104848 (+ 15us) write_op.cc:276] Timestamp: P: 1736736676694139 usec, L: 0
0113 02:51:21.104957 (+ 109us) write_op.cc:270] Start()
0113 02:51:21.104973 (+ 16us) write_op.cc:276] Timestamp: P: 1736736676709596 usec, L: 0
0113 02:51:21.105117 (+ 144us) write_op.cc:270] Start()
0113 02:51:21.105134 (+ 17us) write_op.cc:276] Timestamp: P: 1736736676714572 usec, L: 0
0113 02:51:21.105250 (+ 116us) write_op.cc:270] Start()
0113 02:51:21.105272 (+ 22us) write_op.cc:276] Timestamp: P: 1736736676719488 usec, L: 0
0113 02:51:21.105428 (+ 156us) write_op.cc:270] Start()
0113 02:51:21.105450 (+ 22us) write_op.cc:276] Timestamp: P: 1736736676736166 usec, L: 0
0113 02:51:21.105579 (+ 129us) write_op.cc:270] Start()
0113 02:51:21.105613 (+ 34us) write_op.cc:276] Timestamp: P: 1736736676737562 usec, L: 0
0113 02:51:21.105755 (+ 142us) write_op.cc:270] Start()
0113 02:51:21.105775 (+ 20us) write_op.cc:276] Timestamp: P: 1736736676744010 usec, L: 0
0113 02:51:21.105905 (+ 130us) write_op.cc:270] Start()
0113 02:51:21.105924 (+ 19us) write_op.cc:276] Timestamp: P: 1736736676769241 usec, L: 0
0113 02:51:21.106045 (+ 121us) write_op.cc:270] Start()
0113 02:51:21.106070 (+ 25us) write_op.cc:276] Timestamp: P: 1736736676776360 usec, L: 0
0113 02:51:21.106196 (+ 126us) write_op.cc:270] Start()
0113 02:51:21.106212 (+ 16us) write_op.cc:276] Timestamp: P: 1736736676790641 usec, L: 0
0113 02:51:21.106331 (+ 119us) write_op.cc:270] Start()
0113 02:51:21.106353 (+ 22us) write_op.cc:276] Timestamp: P: 1736736676795523 usec, L: 0
0113 02:51:21.106451 (+ 98us) write_op.cc:270] Start()
0113 02:51:21.106466 (+ 15us) write_op.cc:276] Timestamp: P: 1736736676812476 usec, L: 0
0113 02:51:21.106577 (+ 111us) write_op.cc:270] Start()
0113 02:51:21.106605 (+ 28us) write_op.cc:276] Timestamp: P: 1736736676816477 usec, L: 0
0113 02:51:21.106706 (+ 101us) write_op.cc:270] Start()
0113 02:51:21.106721 (+ 15us) write_op.cc:276] Timestamp: P: 1736736676817042 usec, L: 0
0113 02:51:21.106825 (+ 104us) write_op.cc:270] Start()
0113 02:51:21.106846 (+ 21us) write_op.cc:276] Timestamp: P: 1736736676817555 usec, L: 0
0113 02:51:21.107008 (+ 162us) write_op.cc:270] Start()
0113 02:51:21.107039 (+ 31us) write_op.cc:276] Timestamp: P: 1736736676832819 usec, L: 0
0113 02:51:21.107181 (+ 142us) write_op.cc:270] Start()
0113 02:51:21.107201 (+ 20us) write_op.cc:276] Timestamp: P: 1736736676885125 usec, L: 0
0113 02:51:21.107338 (+ 137us) write_op.cc:270] Start()
0113 02:51:21.107357 (+ 19us) write_op.cc:276] Timestamp: P: 1736736676896798 usec, L: 0
0113 02:51:21.107514 (+ 157us) write_op.cc:270] Start()
0113 02:51:21.107532 (+ 18us) write_op.cc:276] Timestamp: P: 1736736676903316 usec, L: 0
0113 02:51:21.107665 (+ 133us) write_op.cc:270] Start()
0113 02:51:21.107682 (+ 17us) write_op.cc:276] Timestamp: P: 1736736676918655 usec, L: 0
0113 02:51:21.107808 (+ 126us) write_op.cc:270] Start()
0113 02:51:21.107827 (+ 19us) write_op.cc:276] Timestamp: P: 1736736676920291 usec, L: 0
0113 02:51:21.108012 (+ 185us) write_op.cc:270] Start()
0113 02:51:21.108033 (+ 21us) write_op.cc:276] Timestamp: P: 1736736676946091 usec, L: 0
0113 02:51:21.108201 (+ 168us) write_op.cc:270] Start()
0113 02:51:21.108222 (+ 21us) write_op.cc:276] Timestamp: P: 1736736676957362 usec, L: 0
0113 02:51:21.108376 (+ 154us) write_op.cc:270] Start()
0113 02:51:21.108398 (+ 22us) write_op.cc:276] Timestamp: P: 1736736676969092 usec, L: 0
0113 02:51:21.108578 (+ 180us) write_op.cc:270] Start()
0113 02:51:21.108608 (+ 30us) write_op.cc:276] Timestamp: P: 1736736676973469 usec, L: 0
0113 02:51:21.108747 (+ 139us) write_op.cc:270] Start()
0113 02:51:21.108763 (+ 16us) write_op.cc:276] Timestamp: P: 1736736676976464 usec, L: 0
0113 02:51:21.108886 (+ 123us) write_op.cc:270] Start()
0113 02:51:21.108905 (+ 19us) write_op.cc:276] Timestamp: P: 1736736676978121 usec, L: 0
0113 02:51:21.109025 (+ 120us) write_op.cc:270] Start()
0113 02:51:21.109042 (+ 17us) write_op.cc:276] Timestamp: P: 1736736677007170 usec, L: 0
0113 02:51:21.109176 (+ 134us) write_op.cc:270] Start()
0113 02:51:21.109195 (+ 19us) write_op.cc:276] Timestamp: P: 1736736677021201 usec, L: 0
0113 02:51:21.109310 (+ 115us) write_op.cc:270] Start()
0113 02:51:21.109327 (+ 17us) write_op.cc:276] Timestamp: P: 1736736677034954 usec, L: 0
0113 02:51:21.109459 (+ 132us) write_op.cc:270] Start()
0113 02:51:21.109479 (+ 20us) write_op.cc:276] Timestamp: P: 1736736677056690 usec, L: 0
0113 02:51:21.109636 (+ 157us) write_op.cc:270] Start()
0113 02:51:21.109654 (+ 18us) write_op.cc:276] Timestamp: P: 1736736677086567 usec, L: 0
0113 02:51:21.109765 (+ 111us) write_op.cc:270] Start()
0113 02:51:21.109781 (+ 16us) write_op.cc:276] Timestamp: P: 1736736677087225 usec, L: 0
0113 02:51:21.109913 (+ 132us) write_op.cc:270] Start()
0113 02:51:21.109929 (+ 16us) write_op.cc:276] Timestamp: P: 1736736677109562 usec, L: 0
0113 02:51:21.110044 (+ 115us) write_op.cc:270] Start()
0113 02:51:21.110060 (+ 16us) write_op.cc:276] Timestamp: P: 1736736677141844 usec, L: 0
0113 02:51:21.110177 (+ 117us) write_op.cc:270] Start()
0113 02:51:21.110197 (+ 20us) write_op.cc:276] Timestamp: P: 1736736677146803 usec, L: 0
0113 02:51:21.110360 (+ 163us) write_op.cc:270] Start()
0113 02:51:21.110382 (+ 22us) write_op.cc:276] Timestamp: P: 1736736677160848 usec, L: 0
0113 02:51:21.110505 (+ 123us) write_op.cc:270] Start()
0113 02:51:21.110522 (+ 17us) write_op.cc:276] Timestamp: P: 1736736677162513 usec, L: 0
0113 02:51:21.111276 (+ 754us) write_op.cc:270] Start()
0113 02:51:21.111300 (+ 24us) write_op.cc:276] Timestamp: P: 1736736677170640 usec, L: 0
0113 02:51:21.111472 (+ 172us) write_op.cc:270] Start()
0113 02:51:21.111491 (+ 19us) write_op.cc:276] Timestamp: P: 1736736677195422 usec, L: 0
0113 02:51:21.111629 (+ 138us) write_op.cc:270] Start()
0113 02:51:21.111648 (+ 19us) write_op.cc:276] Timestamp: P: 1736736677198589 usec, L: 0
0113 02:51:21.111773 (+ 125us) write_op.cc:270] Start()
0113 02:51:21.111790 (+ 17us) write_op.cc:276] Timestamp: P: 1736736677201748 usec, L: 0
0113 02:51:21.111925 (+ 135us) write_op.cc:270] Start()
0113 02:51:21.111948 (+ 23us) write_op.cc:276] Timestamp: P: 1736736677206020 usec, L: 0
0113 02:51:21.113355 (+ 1407us) write_op.cc:270] Start()
0113 02:51:21.113381 (+ 26us) write_op.cc:276] Timestamp: P: 1736736677267133 usec, L: 0
0113 02:51:21.113559 (+ 178us) write_op.cc:270] Start()
0113 02:51:21.113579 (+ 20us) write_op.cc:276] Timestamp: P: 1736736677267864 usec, L: 0
0113 02:51:21.113749 (+ 170us) write_op.cc:270] Start()
0113 02:51:21.113773 (+ 24us) write_op.cc:276] Timestamp: P: 1736736677297127 usec, L: 0
0113 02:51:21.113889 (+ 116us) write_op.cc:270] Start()
0113 02:51:21.113907 (+ 18us) write_op.cc:276] Timestamp: P: 1736736677344507 usec, L: 0
0113 02:51:21.114034 (+ 127us) write_op.cc:270] Start()
0113 02:51:21.114057 (+ 23us) write_op.cc:276] Timestamp: P: 1736736677346836 usec, L: 0
0113 02:51:21.114451 (+ 394us) write_op.cc:270] Start()
0113 02:51:21.114475 (+ 24us) write_op.cc:276] Timestamp: P: 1736736677352247 usec, L: 0
0113 02:51:21.114709 (+ 234us) write_op.cc:270] Start()
0113 02:51:21.114731 (+ 22us) write_op.cc:276] Timestamp: P: 1736736677353221 usec, L: 0
0113 02:51:21.114902 (+ 171us) write_op.cc:270] Start()
0113 02:51:21.114951 (+ 49us) write_op.cc:276] Timestamp: P: 1736736677384944 usec, L: 0
0113 02:51:21.115091 (+ 140us) write_op.cc:270] Start()
0113 02:51:21.115111 (+ 20us) write_op.cc:276] Timestamp: P: 1736736677389148 usec, L: 0
0113 02:51:21.115245 (+ 134us) write_op.cc:270] Start()
0113 02:51:21.115263 (+ 18us) write_op.cc:276] Timestamp: P: 1736736677408145 usec, L: 0
0113 02:51:21.115398 (+ 135us) write_op.cc:270] Start()
0113 02:51:21.115417 (+ 19us) write_op.cc:276] Timestamp: P: 1736736677409926 usec, L: 0
0113 02:51:21.115622 (+ 205us) write_op.cc:270] Start()
0113 02:51:21.115642 (+ 20us) write_op.cc:276] Timestamp: P: 1736736677412370 usec, L: 0
0113 02:51:21.115785 (+ 143us) write_op.cc:270] Start()
0113 02:51:21.115805 (+ 20us) write_op.cc:276] Timestamp: P: 1736736677415802 usec, L: 0
0113 02:51:21.115939 (+ 134us) write_op.cc:270] Start()
0113 02:51:21.115958 (+ 19us) write_op.cc:276] Timestamp: P: 1736736677416366 usec, L: 0
0113 02:51:21.116083 (+ 125us) write_op.cc:270] Start()
0113 02:51:21.116100 (+ 17us) write_op.cc:276] Timestamp: P: 1736736677416878 usec, L: 0
0113 02:51:21.116221 (+ 121us) write_op.cc:270] Start()
0113 02:51:21.116241 (+ 20us) write_op.cc:276] Timestamp: P: 1736736677417405 usec, L: 0
0113 02:51:21.116342 (+ 101us) write_op.cc:270] Start()
0113 02:51:21.116359 (+ 17us) write_op.cc:276] Timestamp: P: 1736736677494922 usec, L: 0
0113 02:51:21.116532 (+ 173us) write_op.cc:270] Start()
0113 02:51:21.116556 (+ 24us) write_op.cc:276] Timestamp: P: 1736736677495585 usec, L: 0
0113 02:51:21.116735 (+ 179us) write_op.cc:270] Start()
0113 02:51:21.116755 (+ 20us) write_op.cc:276] Timestamp: P: 1736736677535124 usec, L: 0
0113 02:51:21.116882 (+ 127us) write_op.cc:270] Start()
0113 02:51:21.116901 (+ 19us) write_op.cc:276] Timestamp: P: 1736736677537366 usec, L: 0
0113 02:51:21.117040 (+ 139us) write_op.cc:270] Start()
0113 02:51:21.117060 (+ 20us) write_op.cc:276] Timestamp: P: 1736736677548026 usec, L: 0
0113 02:51:21.117291 (+ 231us) write_op.cc:270] Start()
0113 02:51:21.117315 (+ 24us) write_op.cc:276] Timestamp: P: 1736736677552129 usec, L: 0
0113 02:51:21.117545 (+ 230us) write_op.cc:270] Start()
0113 02:51:21.117571 (+ 26us) write_op.cc:276] Timestamp: P: 1736736677552699 usec, L: 0
0113 02:51:21.117740 (+ 169us) write_op.cc:270] Start()
0113 02:51:21.117761 (+ 21us) write_op.cc:276] Timestamp: P: 1736736677553209 usec, L: 0
0113 02:51:21.117903 (+ 142us) write_op.cc:270] Start()
0113 02:51:21.117923 (+ 20us) write_op.cc:276] Timestamp: P: 1736736677553729 usec, L: 0
0113 02:51:21.118056 (+ 133us) write_op.cc:270] Start()
0113 02:51:21.118074 (+ 18us) write_op.cc:276] Timestamp: P: 1736736677561005 usec, L: 0
0113 02:51:21.118254 (+ 180us) write_op.cc:270] Start()
0113 02:51:21.118275 (+ 21us) write_op.cc:276] Timestamp: P: 1736736677590837 usec, L: 0
0113 02:51:21.118404 (+ 129us) write_op.cc:270] Start()
0113 02:51:21.118423 (+ 19us) write_op.cc:276] Timestamp: P: 1736736677608930 usec, L: 0
0113 02:51:21.118547 (+ 124us) write_op.cc:270] Start()
0113 02:51:21.118569 (+ 22us) write_op.cc:276] Timestamp: P: 1736736677609588 usec, L: 0
0113 02:51:21.118714 (+ 145us) write_op.cc:270] Start()
0113 02:51:21.118734 (+ 20us) write_op.cc:276] Timestamp: P: 1736736677610318 usec, L: 0
0113 02:51:21.118877 (+ 143us) write_op.cc:270] Start()
0113 02:51:21.118900 (+ 23us) write_op.cc:276] Timestamp: P: 1736736677610960 usec, L: 0
0113 02:51:21.119060 (+ 160us) write_op.cc:270] Start()
0113 02:51:21.119100 (+ 40us) write_op.cc:276] Timestamp: P: 1736736677629297 usec, L: 0
0113 02:51:21.119217 (+ 117us) write_op.cc:270] Start()
0113 02:51:21.119234 (+ 17us) write_op.cc:276] Timestamp: P: 1736736677677005 usec, L: 0
0113 02:51:21.119374 (+ 140us) write_op.cc:270] Start()
0113 02:51:21.119395 (+ 21us) write_op.cc:276] Timestamp: P: 1736736677678552 usec, L: 0
0113 02:51:21.119540 (+ 145us) write_op.cc:270] Start()
0113 02:51:21.119576 (+ 36us) write_op.cc:276] Timestamp: P: 1736736677697550 usec, L: 0
0113 02:51:21.119719 (+ 143us) write_op.cc:270] Start()
0113 02:51:21.119743 (+ 24us) write_op.cc:276] Timestamp: P: 1736736677702436 usec, L: 0
0113 02:51:21.119890 (+ 147us) write_op.cc:270] Start()
0113 02:51:21.119908 (+ 18us) write_op.cc:276] Timestamp: P: 1736736677732518 usec, L: 0
0113 02:51:21.120160 (+ 252us) write_op.cc:270] Start()
0113 02:51:21.120184 (+ 24us) write_op.cc:276] Timestamp: P: 1736736677733187 usec, L: 0
0113 02:51:21.120377 (+ 193us) write_op.cc:270] Start()
0113 02:51:21.120396 (+ 19us) write_op.cc:276] Timestamp: P: 1736736677733695 usec, L: 0
0113 02:51:21.120541 (+ 145us) write_op.cc:270] Start()
0113 02:51:21.120565 (+ 24us) write_op.cc:276] Timestamp: P: 1736736677734203 usec, L: 0
0113 02:51:21.120719 (+ 154us) write_op.cc:270] Start()
0113 02:51:21.120743 (+ 24us) write_op.cc:276] Timestamp: P: 1736736677765427 usec, L: 0
0113 02:51:21.120885 (+ 142us) write_op.cc:270] Start()
0113 02:51:21.120902 (+ 17us) write_op.cc:276] Timestamp: P: 1736736677792004 usec, L: 0
0113 02:51:21.124115 (+ 3213us) write_op.cc:270] Start()
0113 02:51:21.124146 (+ 31us) write_op.cc:276] Timestamp: P: 1736736677827297 usec, L: 0
0113 02:51:21.124329 (+ 183us) write_op.cc:270] Start()
0113 02:51:21.124385 (+ 56us) write_op.cc:276] Timestamp: P: 1736736677828393 usec, L: 0
0113 02:51:21.124559 (+ 174us) write_op.cc:270] Start()
0113 02:51:21.124576 (+ 17us) write_op.cc:276] Timestamp: P: 1736736677828954 usec, L: 0
0113 02:51:21.124726 (+ 150us) write_op.cc:270] Start()
0113 02:51:21.124746 (+ 20us) write_op.cc:276] Timestamp: P: 1736736677829484 usec, L: 0
0113 02:51:21.125113 (+ 367us) write_op.cc:270] Start()
0113 02:51:21.125146 (+ 33us) write_op.cc:276] Timestamp: P: 1736736677830025 usec, L: 0
0113 02:51:21.125778 (+ 632us) write_op.cc:270] Start()
0113 02:51:21.125802 (+ 24us) write_op.cc:276] Timestamp: P: 1736736677831839 usec, L: 0
0113 02:51:21.126083 (+ 281us) write_op.cc:270] Start()
0113 02:51:21.126105 (+ 22us) write_op.cc:276] Timestamp: P: 1736736677832367 usec, L: 0
0113 02:51:21.126253 (+ 148us) write_op.cc:270] Start()
0113 02:51:21.126271 (+ 18us) write_op.cc:276] Timestamp: P: 1736736677832973 usec, L: 0
0113 02:51:21.126480 (+ 209us) write_op.cc:270] Start()
0113 02:51:21.126505 (+ 25us) write_op.cc:276] Timestamp: P: 1736736677851370 usec, L: 0
0113 02:51:21.126656 (+ 151us) write_op.cc:270] Start()
0113 02:51:21.126680 (+ 24us) write_op.cc:276] Timestamp: P: 1736736677854468 usec, L: 0
0113 02:51:21.127058 (+ 378us) write_op.cc:270] Start()
0113 02:51:21.127084 (+ 26us) write_op.cc:276] Timestamp: P: 1736736677973934 usec, L: 0
0113 02:51:21.127234 (+ 150us) write_op.cc:270] Start()
0113 02:51:21.127258 (+ 24us) write_op.cc:276] Timestamp: P: 1736736677985979 usec, L: 0
0113 02:51:21.127409 (+ 151us) write_op.cc:270] Start()
0113 02:51:21.127429 (+ 20us) write_op.cc:276] Timestamp: P: 1736736677988424 usec, L: 0
0113 02:51:21.127661 (+ 232us) write_op.cc:270] Start()
0113 02:51:21.127684 (+ 23us) write_op.cc:276] Timestamp: P: 1736736677991056 usec, L: 0
0113 02:51:21.127834 (+ 150us) write_op.cc:270] Start()
0113 02:51:21.127854 (+ 20us) write_op.cc:276] Timestamp: P: 1736736678000111 usec, L: 0
0113 02:51:21.128038 (+ 184us) write_op.cc:270] Start()
0113 02:51:21.128058 (+ 20us) write_op.cc:276] Timestamp: P: 1736736678000616 usec, L: 0
0113 02:51:21.128191 (+ 133us) write_op.cc:270] Start()
0113 02:51:21.128217 (+ 26us) write_op.cc:276] Timestamp: P: 1736736678002146 usec, L: 0
0113 02:51:21.128337 (+ 120us) write_op.cc:270] Start()
0113 02:51:21.128354 (+ 17us) write_op.cc:276] Timestamp: P: 1736736678011627 usec, L: 0
0113 02:51:21.128519 (+ 165us) write_op.cc:270] Start()
0113 02:51:21.128539 (+ 20us) write_op.cc:276] Timestamp: P: 1736736678019458 usec, L: 0
0113 02:51:21.128698 (+ 159us) write_op.cc:270] Start()
0113 02:51:21.128724 (+ 26us) write_op.cc:276] Timestamp: P: 1736736678027169 usec, L: 0
0113 02:51:21.128855 (+ 131us) write_op.cc:270] Start()
0113 02:51:21.128877 (+ 22us) write_op.cc:276] Timestamp: P: 1736736678029742 usec, L: 0
0113 02:51:21.129037 (+ 160us) write_op.cc:270] Start()
0113 02:51:21.129054 (+ 17us) write_op.cc:276] Timestamp: P: 1736736678032643 usec, L: 0
0113 02:51:21.129187 (+ 133us) write_op.cc:270] Start()
0113 02:51:21.129203 (+ 16us) write_op.cc:276] Timestamp: P: 1736736678045978 usec, L: 0
0113 02:51:21.129349 (+ 146us) write_op.cc:270] Start()
0113 02:51:21.129368 (+ 19us) write_op.cc:276] Timestamp: P: 1736736678049414 usec, L: 0
0113 02:51:21.129539 (+ 171us) write_op.cc:270] Start()
0113 02:51:21.129562 (+ 23us) write_op.cc:276] Timestamp: P: 1736736678083998 usec, L: 0
0113 02:51:21.129702 (+ 140us) write_op.cc:270] Start()
0113 02:51:21.129724 (+ 22us) write_op.cc:276] Timestamp: P: 1736736678190182 usec, L: 0
0113 02:51:21.129855 (+ 131us) write_op.cc:270] Start()
0113 02:51:21.129878 (+ 23us) write_op.cc:276] Timestamp: P: 1736736678192641 usec, L: 0
0113 02:51:21.130002 (+ 124us) write_op.cc:270] Start()
0113 02:51:21.130022 (+ 20us) write_op.cc:276] Timestamp: P: 1736736678195534 usec, L: 0
0113 02:51:21.130148 (+ 126us) write_op.cc:270] Start()
0113 02:51:21.130168 (+ 20us) write_op.cc:276] Timestamp: P: 1736736678201954 usec, L: 0
0113 02:51:21.130314 (+ 146us) write_op.cc:270] Start()
0113 02:51:21.130331 (+ 17us) write_op.cc:276] Timestamp: P: 1736736678202580 usec, L: 0
0113 02:51:21.130459 (+ 128us) write_op.cc:270] Start()
0113 02:51:21.130480 (+ 21us) write_op.cc:276] Timestamp: P: 1736736678203150 usec, L: 0
0113 02:51:21.130632 (+ 152us) write_op.cc:270] Start()
0113 02:51:21.130656 (+ 24us) write_op.cc:276] Timestamp: P: 1736736
I20250113 02:51:22.353014 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f8a6eebb6b6d4b63903e68750e181707. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:22.356043 744 tablet_service.cc:1514] Processing DeleteTablet for tablet f56230a2c75d4763a54671c79358f4d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:22 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:22.356671 744 tablet_service.cc:1514] Processing DeleteTablet for tablet f129bb3fbbfd492d98822be0a6626e44 with delete_type TABLET_DATA_DELETED (replaced by tablet 67baaab1e36e415381579d7385d4ab5a) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:22.358158 744 tablet_service.cc:1514] Processing DeleteTablet for tablet bbc7832c50114d6f890fec45a3d994e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:22 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:22.358557 744 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a7e7713bc8648378b668d8b769ac927 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:21 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:22.365454 1326 sys_catalog.cc:455] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 4 leader_uuid: "e13f5bcdcee34225a46c6d2651e5ede9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e13f5bcdcee34225a46c6d2651e5ede9" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44217 } } peers { permanent_uuid: "6bbc3541604a455584bb32dfc2cc4efd" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 35271 } } peers { permanent_uuid: "45e9d43ecac0401fb2c18fe5eaf089b5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42067 } } }
I20250113 02:51:22.366300 1326 sys_catalog.cc:458] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9 [sys.catalog]: This master's current role is: LEADER
I20250113 02:51:22.369143 1368 raft_consensus.cc:695] T d101e98b72544c30a3a4bc60d1576aa3 P 1186fa869eb14812832fa6562531f196 [term 1 LEADER]: Becoming Leader. State: Replica: 1186fa869eb14812832fa6562531f196, State: Running, Role: LEADER
I20250113 02:51:22.376361 880 tablet_service.cc:1467] Processing CreateTablet for tablet b5c1882125b5491ea79a29a6dac6771e (DEFAULT_TABLE table=default.Table_029fb0ede6224bf4b0cfa388bee300a7 [id=3237c1be7e1c48918ff095162d191dc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:22.385078 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a26e2b5e1054aa6afe7f44bf8d43108 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:21 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:22.385270 1029 tablet_service.cc:1467] Processing CreateTablet for tablet bbc7832c50114d6f890fec45a3d994e9 (DEFAULT_TABLE table=default.Table_3c637eb181574498b44ecb4fc41f0117 [id=10c0207c68d8463e91f09972031ab8d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:22.385699 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bbc7832c50114d6f890fec45a3d994e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:22.397459 748 tablet_service.cc:1514] Processing DeleteTablet for tablet 05bf5ab0df874dd5b6559f6d82069184 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:21 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:22.397678 1192 tablet_bootstrap.cc:654] T 1d81eec487dd4016b3b804451da31c49 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:22.400862 1196 raft_consensus.cc:2238] T 3d3a6d9f687e45218d5875a7c5597648 P df69f48656b44bb890f76587ee69cc0f [term 1 LEADER]: Raft consensus shutting down.
I20250113 02:51:22.401180 454 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_dc98b9959d0c42b58d9f47e3c2c2b73d" } new_table_name: "default.Table_911794a03fd1463787a098ca7e61dbc8" modify_external_catalogs: true
I20250113 02:51:22.401157 881 tablet_service.cc:1467] Processing CreateTablet for tablet bc6a7322c0c34adb85457750526d8955 (DEFAULT_TABLE table=default.Table_6185ed1a90a44576bc65ed60294c2537 [id=5634c185da26460f93cde9c65fbdb24a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:22.402593 448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_28d44f0d1c6c4a2cbd27c902820195b9" } new_table_name: "default.Table_3d5fd97116b343b0a6c798e1f22327c2" modify_external_catalogs: true
I20250113 02:51:22.400019 1198 ts_tablet_manager.cc:1918] T 9b5dc06ba3ea4056822ca3e4b1249587 P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:22.404911 1198 log.cc:1198] T 9b5dc06ba3ea4056822ca3e4b1249587 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/9b5dc06ba3ea4056822ca3e4b1249587
I20250113 02:51:22.405519 1198 ts_tablet_manager.cc:1939] T 9b5dc06ba3ea4056822ca3e4b1249587 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:22.406456 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 308) took 1091 ms. Trace:
I20250113 02:51:22.408257 1198 rpcz_store.cc:276] 0113 02:51:21.316702 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:21.336891 (+ 20189us) service_pool.cc:225] Handling call
0113 02:51:22.155817 (+818926us) ts_tablet_manager.cc:1145] Deleting tablet 9b5dc06ba3ea4056822ca3e4b1249587
0113 02:51:22.155825 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:22.346878 (+191053us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:22.378829 (+ 31951us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:22.406422 (+ 27593us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":818639}
I20250113 02:51:22.410184 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_694391751b064302a0b020dbea9d842d" } new_table_name: "default.Table_b09c6cee53c64995a167959c15f5e288" modify_external_catalogs: true
I20250113 02:51:22.412348 1198 tablet_replica.cc:331] T d101e98b72544c30a3a4bc60d1576aa3 P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:22.412539 1198 raft_consensus.cc:2238] T d101e98b72544c30a3a4bc60d1576aa3 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:22.412694 1198 raft_consensus.cc:2267] T d101e98b72544c30a3a4bc60d1576aa3 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:22.413352 1198 ts_tablet_manager.cc:1905] T d101e98b72544c30a3a4bc60d1576aa3 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:22.415890 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet 9b5dc06ba3ea4056822ca3e4b1249587 (table default.Table_20efb43987cc47c7a77436e076cc70f6 [id=8d889764fb3046369d87cef8998a5a87]) successfully deleted
I20250113 02:51:22.416131 1192 tablet_bootstrap.cc:492] T 1d81eec487dd4016b3b804451da31c49 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
W20250113 02:51:22.416214 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet efccefd6787943469131492741ee2bd5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet efccefd6787943469131492741ee2bd5 already in progress: creating tablet
I20250113 02:51:22.416280 1192 ts_tablet_manager.cc:1397] T 1d81eec487dd4016b3b804451da31c49 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.134s user 0.001s sys 0.003s
I20250113 02:51:22.417142 1192 raft_consensus.cc:357] T 1d81eec487dd4016b3b804451da31c49 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:22.417444 1192 raft_consensus.cc:383] T 1d81eec487dd4016b3b804451da31c49 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:22.417400 888 tablet_service.cc:1467] Processing CreateTablet for tablet 788ab82626424b6e833e31761a15b9b2 (DEFAULT_TABLE table=default.Table_2d289859f3a2422b9f1c5bd6b3083088 [id=0f07147c73f24decb09399a1dd188e1f]), partition=RANGE (key) PARTITION UNBOUNDED
W20250113 02:51:22.417603 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet bffec36d567544c5a1c87e64a7e9730b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bffec36d567544c5a1c87e64a7e9730b already in progress: creating tablet
I20250113 02:51:22.417677 1192 raft_consensus.cc:738] T 1d81eec487dd4016b3b804451da31c49 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:22.417934 891 tablet_service.cc:1467] Processing CreateTablet for tablet c7842d6f2c074e3b9d1a3378db9856be (DEFAULT_TABLE table=default.Table_59401ddb940e4e789a5bd54f96548ba5 [id=0ac370ed485e4d64a3e1aa1e8d67ea23]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:22.417934 1192 consensus_queue.cc:260] T 1d81eec487dd4016b3b804451da31c49 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:22.417975 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_ccefabb7a6024c17897743e268c1ce49"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:22.418483 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ccefabb7a6024c17897743e268c1ce49 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:22.419531 897 tablet_service.cc:1467] Processing CreateTablet for tablet ee240d838d4a4bbba97f8ed61f69de78 (DEFAULT_TABLE table=default.Table_deb7c4bd8a954e5081cdbee12274dbcf [id=383148ed836144c1bc5403dae46aad50]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:22.420336 1192 ts_tablet_manager.cc:1428] T 1d81eec487dd4016b3b804451da31c49 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250113 02:51:22.421075 1192 tablet_bootstrap.cc:492] T 107e6e9e2d84496f8fce92b9ba377189 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:22.422322 447 master_service.cc:946] ReplaceTablet: received request to replace tablet c1fb2b36c0504cb4ba32671ae4c620d1 from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:22.422885 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_e1f19e9072094459a8aba69a14179bda"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:22.423426 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e1f19e9072094459a8aba69a14179bda in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:22.424726 1192 tablet_bootstrap.cc:654] T 107e6e9e2d84496f8fce92b9ba377189 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:22.424976 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_766a63106d8a4fd29bb338e76bd96089"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:22.425410 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_766a63106d8a4fd29bb338e76bd96089 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:22.426362 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_dd19b1ec1af841a7a475c3ace96712a6" } new_table_name: "default.Table_f621d2678f0e47c0997a05a2f1964785" modify_external_catalogs: true
I20250113 02:51:22.428758 893 tablet_service.cc:1514] Processing DeleteTablet for tablet bbc7832c50114d6f890fec45a3d994e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:22 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:22.429018 883 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a7e7713bc8648378b668d8b769ac927 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:21 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:22.429214 1198 ts_tablet_manager.cc:1918] T d101e98b72544c30a3a4bc60d1576aa3 P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:22.429419 1198 log.cc:1198] T d101e98b72544c30a3a4bc60d1576aa3 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/d101e98b72544c30a3a4bc60d1576aa3
I20250113 02:51:22.429824 1198 ts_tablet_manager.cc:1939] T d101e98b72544c30a3a4bc60d1576aa3 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:22.430912 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet d101e98b72544c30a3a4bc60d1576aa3 (table default.Table_ce0f77e091564c29bf54791dda458d78 [id=86a46eed92bc4bddad870e392232cb51]) successfully deleted
W20250113 02:51:22.432802 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 1500e4f9c0664a879503afd3edf14c66 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1500e4f9c0664a879503afd3edf14c66 already in progress: creating tablet
I20250113 02:51:22.433796 1198 tablet_replica.cc:331] T 7f3cbef7d91d4576822d913071a88248 P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:22.434181 1198 raft_consensus.cc:2238] T 7f3cbef7d91d4576822d913071a88248 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:22.434742 1198 raft_consensus.cc:2267] T 7f3cbef7d91d4576822d913071a88248 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:22.435631 1198 ts_tablet_manager.cc:1905] T 7f3cbef7d91d4576822d913071a88248 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:22.438314 1192 tablet_bootstrap.cc:492] T 107e6e9e2d84496f8fce92b9ba377189 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:22.438498 1192 ts_tablet_manager.cc:1397] T 107e6e9e2d84496f8fce92b9ba377189 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.002s
I20250113 02:51:22.439312 1192 raft_consensus.cc:357] T 107e6e9e2d84496f8fce92b9ba377189 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:22.439589 1192 raft_consensus.cc:383] T 107e6e9e2d84496f8fce92b9ba377189 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:22.439720 1192 raft_consensus.cc:738] T 107e6e9e2d84496f8fce92b9ba377189 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:22.440165 1192 consensus_queue.cc:260] T 107e6e9e2d84496f8fce92b9ba377189 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:22.441028 1192 ts_tablet_manager.cc:1428] T 107e6e9e2d84496f8fce92b9ba377189 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250113 02:51:22.441349 1192 tablet_bootstrap.cc:492] T 53cc528d282c47788d5d558569f074c1 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:22.442957 1192 tablet_bootstrap.cc:654] T 53cc528d282c47788d5d558569f074c1 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:22.449373 1198 ts_tablet_manager.cc:1918] T 7f3cbef7d91d4576822d913071a88248 P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250113 02:51:22.450407 1198 log.cc:1198] T 7f3cbef7d91d4576822d913071a88248 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/7f3cbef7d91d4576822d913071a88248
I20250113 02:51:22.451583 1198 ts_tablet_manager.cc:1939] T 7f3cbef7d91d4576822d913071a88248 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:22.453770 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet 7f3cbef7d91d4576822d913071a88248 (table default.Table_b581b1a063a54551a8535adf01983fb8 [id=54e4e3810b3b412bbec3b5bf5b28c1a2]) successfully deleted
W20250113 02:51:22.455410 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 4efa63b8be794e1db4b340f4c0c4cd5a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4efa63b8be794e1db4b340f4c0c4cd5a already in progress: creating tablet
W20250113 02:51:22.456383 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet a92f9b1c9c1a4cbdbf0cc7b2a3f2275b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a92f9b1c9c1a4cbdbf0cc7b2a3f2275b already in progress: creating tablet
I20250113 02:51:22.457412 1198 tablet_replica.cc:331] T 98f038ad31ff4e05bb88ac68ae341cc2 P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:22.457964 1198 raft_consensus.cc:2238] T 98f038ad31ff4e05bb88ac68ae341cc2 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:22.458449 1198 raft_consensus.cc:2267] T 98f038ad31ff4e05bb88ac68ae341cc2 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:22.459277 1198 ts_tablet_manager.cc:1905] T 98f038ad31ff4e05bb88ac68ae341cc2 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:22.470615 1198 ts_tablet_manager.cc:1918] T 98f038ad31ff4e05bb88ac68ae341cc2 P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:22.473749 1198 log.cc:1198] T 98f038ad31ff4e05bb88ac68ae341cc2 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/98f038ad31ff4e05bb88ac68ae341cc2
I20250113 02:51:22.474506 1198 ts_tablet_manager.cc:1939] T 98f038ad31ff4e05bb88ac68ae341cc2 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:22.476333 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet 98f038ad31ff4e05bb88ac68ae341cc2 (table default.Table_5182a1c5c7a2462ba930940ae04b505f [id=6666a614063942c99a78b95d35add037]) successfully deleted
I20250113 02:51:22.476372 882 tablet_service.cc:1467] Processing CreateTablet for tablet c518f791b4d74c54ab0c0364d29b4a64 (DEFAULT_TABLE table=default.Table_157d1350cce74502a990462ac9bd0e5f [id=f96a39bfcdf442f7965a7b16d9aa432f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:22.476882 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5c1882125b5491ea79a29a6dac6771e. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:22.478083 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet c826a0b8327f44f9a00e5afe3a36cd8a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c826a0b8327f44f9a00e5afe3a36cd8a already in progress: creating tablet
I20250113 02:51:22.478689 1198 tablet_replica.cc:331] T 3d3a6d9f687e45218d5875a7c5597648 P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:22.478878 1198 raft_consensus.cc:2238] T 3d3a6d9f687e45218d5875a7c5597648 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:22.479089 1198 raft_consensus.cc:2267] T 3d3a6d9f687e45218d5875a7c5597648 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:22.479609 1198 ts_tablet_manager.cc:1905] T 3d3a6d9f687e45218d5875a7c5597648 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:22.480284 885 tablet_service.cc:1467] Processing CreateTablet for tablet bc533518faa24dffa3306ff092e8d507 (DEFAULT_TABLE table=default.Table_03babf194cc448f296512a09b2e3fe3e [id=64b42698782f41a589577af20b33501e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:22.480803 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc533518faa24dffa3306ff092e8d507. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:22.481065 1192 tablet_bootstrap.cc:492] T 53cc528d282c47788d5d558569f074c1 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:22.481227 1192 ts_tablet_manager.cc:1397] T 53cc528d282c47788d5d558569f074c1 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.040s user 0.006s sys 0.010s
I20250113 02:51:22.482086 1192 raft_consensus.cc:357] T 53cc528d282c47788d5d558569f074c1 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:22.482334 1192 raft_consensus.cc:383] T 53cc528d282c47788d5d558569f074c1 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:22.482403 1192 raft_consensus.cc:738] T 53cc528d282c47788d5d558569f074c1 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:22.482649 1192 consensus_queue.cc:260] T 53cc528d282c47788d5d558569f074c1 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:22.483201 1192 ts_tablet_manager.cc:1428] T 53cc528d282c47788d5d558569f074c1 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250113 02:51:22.483435 1192 tablet_bootstrap.cc:492] T c71511a100f14322b5091a9804720fa0 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:22.484409 1518 raft_consensus.cc:491] T 6c8ca30720664f3a819c4bff2493934d P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:22.484622 1196 pending_rounds.cc:62] T 3d3a6d9f687e45218d5875a7c5597648 P df69f48656b44bb890f76587ee69cc0f: Trying to abort 1 pending ops.
I20250113 02:51:22.484705 899 tablet_service.cc:1514] Processing DeleteTablet for tablet f129bb3fbbfd492d98822be0a6626e44 with delete_type TABLET_DATA_DELETED (replaced by tablet 67baaab1e36e415381579d7385d4ab5a) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:22.484580 1518 raft_consensus.cc:513] T 6c8ca30720664f3a819c4bff2493934d P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:22.484954 890 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a26e2b5e1054aa6afe7f44bf8d43108 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:21 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:22.485060 1192 tablet_bootstrap.cc:654] T c71511a100f14322b5091a9804720fa0 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:22.485185 893 tablet_service.cc:1514] Processing DeleteTablet for tablet 05bf5ab0df874dd5b6559f6d82069184 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:21 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:22.484709 1196 pending_rounds.cc:69] T 3d3a6d9f687e45218d5875a7c5597648 P df69f48656b44bb890f76587ee69cc0f: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113673450194202624 op_type: NO_OP noop_request { }
I20250113 02:51:22.485517 1518 leader_election.cc:290] T 6c8ca30720664f3a819c4bff2493934d P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:22.485584 1196 raft_consensus.cc:2883] T 3d3a6d9f687e45218d5875a7c5597648 P df69f48656b44bb890f76587ee69cc0f [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250113 02:51:22.484469 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c518f791b4d74c54ab0c0364d29b4a64. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:22.485697 1196 raft_consensus.cc:2267] T 3d3a6d9f687e45218d5875a7c5597648 P df69f48656b44bb890f76587ee69cc0f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:22.486074 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8ca30720664f3a819c4bff2493934d" candidate_uuid: "df69f48656b44bb890f76587ee69cc0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:22.486701 1196 ts_tablet_manager.cc:1905] T 3d3a6d9f687e45218d5875a7c5597648 P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250113 02:51:22.486701 852 leader_election.cc:343] T 6c8ca30720664f3a819c4bff2493934d P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:22.491042 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc6a7322c0c34adb85457750526d8955. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:22.495012 1198 ts_tablet_manager.cc:1918] T 3d3a6d9f687e45218d5875a7c5597648 P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:22.495131 1198 log.cc:1198] T 3d3a6d9f687e45218d5875a7c5597648 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/3d3a6d9f687e45218d5875a7c5597648
I20250113 02:51:22.495469 1198 ts_tablet_manager.cc:1939] T 3d3a6d9f687e45218d5875a7c5597648 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:22.496323 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee240d838d4a4bbba97f8ed61f69de78. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:22.496613 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet 3d3a6d9f687e45218d5875a7c5597648 (table default.Table_21dd1d15c2bb4849af81bb4d8e1a5f04 [id=219903e6141d465bbf5396197210c420]) successfully deleted
I20250113 02:51:22.498021 1516 raft_consensus.cc:491] T 4efa63b8be794e1db4b340f4c0c4cd5a P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:22.498220 1516 raft_consensus.cc:513] T 4efa63b8be794e1db4b340f4c0c4cd5a P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
W20250113 02:51:22.498973 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 2c399784af46426b8327d175c7a6cce8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2c399784af46426b8327d175c7a6cce8 already in progress: creating tablet
I20250113 02:51:22.499245 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4efa63b8be794e1db4b340f4c0c4cd5a" candidate_uuid: "df69f48656b44bb890f76587ee69cc0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
W20250113 02:51:22.499809 852 leader_election.cc:343] T 4efa63b8be794e1db4b340f4c0c4cd5a P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
W20250113 02:51:22.499823 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet eebad045e7bb43fa9ca9c9af8b3b7ea5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet eebad045e7bb43fa9ca9c9af8b3b7ea5 already in progress: creating tablet
I20250113 02:51:22.500173 1516 leader_election.cc:290] T 4efa63b8be794e1db4b340f4c0c4cd5a P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587), 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
I20250113 02:51:22.500362 892 tablet_service.cc:1514] Processing DeleteTablet for tablet f56230a2c75d4763a54671c79358f4d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:22 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:22.500567 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c7842d6f2c074e3b9d1a3378db9856be. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:22.500847 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 7d3562a48c6e4badbbcf37640e91b5ba because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7d3562a48c6e4badbbcf37640e91b5ba already in progress: creating tablet
I20250113 02:51:22.501262 1198 tablet_replica.cc:331] T d35ec7e4a5084a74bea172ef8d9990b5 P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:22.501673 1198 raft_consensus.cc:2238] T d35ec7e4a5084a74bea172ef8d9990b5 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:22.502167 1198 raft_consensus.cc:2267] T d35ec7e4a5084a74bea172ef8d9990b5 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:22.503600 1196 ts_tablet_manager.cc:1918] T 3d3a6d9f687e45218d5875a7c5597648 P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250113 02:51:22.503765 1196 log.cc:1198] T 3d3a6d9f687e45218d5875a7c5597648 P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/3d3a6d9f687e45218d5875a7c5597648
I20250113 02:51:22.504341 1196 ts_tablet_manager.cc:1939] T 3d3a6d9f687e45218d5875a7c5597648 P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:22.504532 1198 ts_tablet_manager.cc:1905] T d35ec7e4a5084a74bea172ef8d9990b5 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:22.504889 1196 tablet_replica.cc:331] T d101e98b72544c30a3a4bc60d1576aa3 P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:22.508325 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet 3d3a6d9f687e45218d5875a7c5597648 (table default.Table_21dd1d15c2bb4849af81bb4d8e1a5f04 [id=219903e6141d465bbf5396197210c420]) successfully deleted
I20250113 02:51:22.512173 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 788ab82626424b6e833e31761a15b9b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:22.519407 1196 raft_consensus.cc:2238] T d101e98b72544c30a3a4bc60d1576aa3 P df69f48656b44bb890f76587ee69cc0f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:22.519678 1196 raft_consensus.cc:2267] T d101e98b72544c30a3a4bc60d1576aa3 P df69f48656b44bb890f76587ee69cc0f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:22.520637 1196 ts_tablet_manager.cc:1905] T d101e98b72544c30a3a4bc60d1576aa3 P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:22.526935 1196 ts_tablet_manager.cc:1918] T d101e98b72544c30a3a4bc60d1576aa3 P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:22.527098 1196 log.cc:1198] T d101e98b72544c30a3a4bc60d1576aa3 P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/d101e98b72544c30a3a4bc60d1576aa3
I20250113 02:51:22.527274 1198 ts_tablet_manager.cc:1918] T d35ec7e4a5084a74bea172ef8d9990b5 P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:22.527442 1198 log.cc:1198] T d35ec7e4a5084a74bea172ef8d9990b5 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/d35ec7e4a5084a74bea172ef8d9990b5
I20250113 02:51:22.527483 1196 ts_tablet_manager.cc:1939] T d101e98b72544c30a3a4bc60d1576aa3 P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:22.527794 1198 ts_tablet_manager.cc:1939] T d35ec7e4a5084a74bea172ef8d9990b5 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:22.527944 1196 tablet_replica.cc:331] T 34ab7ee573284550aaac9f814e8bd7ab P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:22.528067 1438 raft_consensus.cc:491] T 8547b63c40ec4ccebd962b5a3b2c5e0f P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:22.528175 1196 raft_consensus.cc:2238] T 34ab7ee573284550aaac9f814e8bd7ab P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:22.528347 1196 raft_consensus.cc:2267] T 34ab7ee573284550aaac9f814e8bd7ab P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:22.528364 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet d101e98b72544c30a3a4bc60d1576aa3 (table default.Table_ce0f77e091564c29bf54791dda458d78 [id=86a46eed92bc4bddad870e392232cb51]) successfully deleted
I20250113 02:51:22.528316 1438 raft_consensus.cc:513] T 8547b63c40ec4ccebd962b5a3b2c5e0f P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:22.535838 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8ca30720664f3a819c4bff2493934d" candidate_uuid: "df69f48656b44bb890f76587ee69cc0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
W20250113 02:51:22.536851 851 leader_election.cc:343] T 6c8ca30720664f3a819c4bff2493934d P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:22.536860 1196 ts_tablet_manager.cc:1905] T 34ab7ee573284550aaac9f814e8bd7ab P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:22.537050 851 leader_election.cc:304] T 6c8ca30720664f3a819c4bff2493934d P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df69f48656b44bb890f76587ee69cc0f; no voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca
I20250113 02:51:22.537374 1516 raft_consensus.cc:2743] T 6c8ca30720664f3a819c4bff2493934d P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:22.538342 745 tablet_service.cc:1514] Processing DeleteTablet for tablet 076ed7744da54ddebd8fc99686beb6ae with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:17 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:22.538828 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet d35ec7e4a5084a74bea172ef8d9990b5 (table default.Table_4369a143c72643b084df58d25314c6dd [id=36ff239c08a54be99ca307354810fff3]) successfully deleted
W20250113 02:51:22.544826 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet f56230a2c75d4763a54671c79358f4d9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f56230a2c75d4763a54671c79358f4d9 already in progress: creating tablet
W20250113 02:51:22.545809 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet f129bb3fbbfd492d98822be0a6626e44 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f129bb3fbbfd492d98822be0a6626e44 already in progress: creating tablet
W20250113 02:51:22.546728 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet bbc7832c50114d6f890fec45a3d994e9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bbc7832c50114d6f890fec45a3d994e9 already in progress: creating tablet
I20250113 02:51:22.550303 1196 ts_tablet_manager.cc:1918] T 34ab7ee573284550aaac9f814e8bd7ab P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:22.550459 1196 log.cc:1198] T 34ab7ee573284550aaac9f814e8bd7ab P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/34ab7ee573284550aaac9f814e8bd7ab
I20250113 02:51:22.550870 1196 ts_tablet_manager.cc:1939] T 34ab7ee573284550aaac9f814e8bd7ab P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
W20250113 02:51:22.551434 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 0a7e7713bc8648378b668d8b769ac927 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0a7e7713bc8648378b668d8b769ac927 already in progress: creating tablet
W20250113 02:51:22.551676 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 05bf5ab0df874dd5b6559f6d82069184 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 05bf5ab0df874dd5b6559f6d82069184 already in progress: creating tablet
W20250113 02:51:22.551896 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 1a26e2b5e1054aa6afe7f44bf8d43108 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1a26e2b5e1054aa6afe7f44bf8d43108 already in progress: creating tablet
I20250113 02:51:22.552170 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet 34ab7ee573284550aaac9f814e8bd7ab (table default.Table_c59cac1b4e0249129c7671a4919283c8 [id=1cd50765f4964c15a341d307ff2d449f]) successfully deleted
W20250113 02:51:22.557090 442 catalog_manager.cc:4839] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet bbc7832c50114d6f890fec45a3d994e9 because the tablet was not found. No further retry: Not found: Tablet not found: bbc7832c50114d6f890fec45a3d994e9
W20250113 02:51:22.557402 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 0a7e7713bc8648378b668d8b769ac927 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0a7e7713bc8648378b668d8b769ac927 already in progress: creating tablet
W20250113 02:51:22.557890 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet f129bb3fbbfd492d98822be0a6626e44 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f129bb3fbbfd492d98822be0a6626e44 already in progress: creating tablet
W20250113 02:51:22.558660 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 1a26e2b5e1054aa6afe7f44bf8d43108 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1a26e2b5e1054aa6afe7f44bf8d43108 already in progress: creating tablet
W20250113 02:51:22.559723 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 05bf5ab0df874dd5b6559f6d82069184 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 05bf5ab0df874dd5b6559f6d82069184 already in progress: creating tablet
W20250113 02:51:22.560370 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet f56230a2c75d4763a54671c79358f4d9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f56230a2c75d4763a54671c79358f4d9 already in progress: creating tablet
I20250113 02:51:22.593130 1516 raft_consensus.cc:491] T 4fe54131aafb40ee9d8bc5ea299af193 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:22.593397 1516 raft_consensus.cc:513] T 4fe54131aafb40ee9d8bc5ea299af193 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:22.594261 1516 leader_election.cc:290] T 4fe54131aafb40ee9d8bc5ea299af193 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587), 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
I20250113 02:51:22.597465 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe54131aafb40ee9d8bc5ea299af193" candidate_uuid: "df69f48656b44bb890f76587ee69cc0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:22.601172 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet 076ed7744da54ddebd8fc99686beb6ae with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:17 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:22.601258 744 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a26e2b5e1054aa6afe7f44bf8d43108 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:21 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:22.601459 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet 68023c30a7f644eaa5b4ddd60d6c5ef5 with delete_type TABLET_DATA_DELETED (replaced by tablet 32351d6de1574353a6e8073d333da477) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:22.601635 1198 tablet_replica.cc:331] T 076ed7744da54ddebd8fc99686beb6ae P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:22.601667 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet 18e2d79d731142f89b9089869bc736e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:18 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:22.601824 1198 raft_consensus.cc:2238] T 076ed7744da54ddebd8fc99686beb6ae P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:22.602003 1198 raft_consensus.cc:2267] T 076ed7744da54ddebd8fc99686beb6ae P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:22.602542 1198 ts_tablet_manager.cc:1905] T 076ed7744da54ddebd8fc99686beb6ae P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:22.604063 1368 consensus_queue.cc:237] T d101e98b72544c30a3a4bc60d1576aa3 P 1186fa869eb14812832fa6562531f196 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:22.609328 1198 ts_tablet_manager.cc:1918] T 076ed7744da54ddebd8fc99686beb6ae P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:22.609449 1198 log.cc:1198] T 076ed7744da54ddebd8fc99686beb6ae P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/076ed7744da54ddebd8fc99686beb6ae
I20250113 02:51:22.609848 1198 ts_tablet_manager.cc:1939] T 076ed7744da54ddebd8fc99686beb6ae P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:22.610158 1525 raft_consensus.cc:491] T 8547b63c40ec4ccebd962b5a3b2c5e0f P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:22.610323 1525 raft_consensus.cc:513] T 8547b63c40ec4ccebd962b5a3b2c5e0f P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:22.610795 1198 tablet_replica.cc:331] T 68023c30a7f644eaa5b4ddd60d6c5ef5 P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:22.610999 1198 raft_consensus.cc:2238] T 68023c30a7f644eaa5b4ddd60d6c5ef5 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:22.611027 1525 leader_election.cc:290] T 8547b63c40ec4ccebd962b5a3b2c5e0f P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:22.611160 1198 raft_consensus.cc:2267] T 68023c30a7f644eaa5b4ddd60d6c5ef5 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:22.611176 1525 raft_consensus.cc:491] T 1d81eec487dd4016b3b804451da31c49 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:22.611303 1525 raft_consensus.cc:513] T 1d81eec487dd4016b3b804451da31c49 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
W20250113 02:51:22.612908 852 leader_election.cc:343] T 4fe54131aafb40ee9d8bc5ea299af193 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:22.614061 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4efa63b8be794e1db4b340f4c0c4cd5a" candidate_uuid: "df69f48656b44bb890f76587ee69cc0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
I20250113 02:51:22.614699 1525 leader_election.cc:290] T 1d81eec487dd4016b3b804451da31c49 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:22.614595 748 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d81eec487dd4016b3b804451da31c49 with delete_type TABLET_DATA_DELETED (replaced by tablet 438d1d2300e044f39a15586f335935ba) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:22.614957 1535 consensus_queue.cc:1035] T ad9b95c5450a4c9c89a0930862348d49 P 1186fa869eb14812832fa6562531f196 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.047s
I20250113 02:51:22.621802 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet 076ed7744da54ddebd8fc99686beb6ae (table default.Table_8b98cd7fbceb4d3f8b138e4da9bb0e05 [id=4100a37b30624d0bab2ec027b783be00]) successfully deleted
I20250113 02:51:22.628135 1198 ts_tablet_manager.cc:1905] T 68023c30a7f644eaa5b4ddd60d6c5ef5 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:22.632993 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8547b63c40ec4ccebd962b5a3b2c5e0f" candidate_uuid: "df69f48656b44bb890f76587ee69cc0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:22.633270 1061 raft_consensus.cc:2463] T 8547b63c40ec4ccebd962b5a3b2c5e0f P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df69f48656b44bb890f76587ee69cc0f in term 0.
W20250113 02:51:22.635134 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1014e51f1c184d729552a5b34292f232 consensus state has no leader
I20250113 02:51:22.632345 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8547b63c40ec4ccebd962b5a3b2c5e0f" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:22.638289 919 raft_consensus.cc:2463] T 8547b63c40ec4ccebd962b5a3b2c5e0f P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1186fa869eb14812832fa6562531f196 in term 0.
I20250113 02:51:22.639004 1191 tablet_bootstrap.cc:492] T d35ec7e4a5084a74bea172ef8d9990b5 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:22.639067 1438 leader_election.cc:290] T 8547b63c40ec4ccebd962b5a3b2c5e0f P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587), 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
I20250113 02:51:22.639169 1191 ts_tablet_manager.cc:1397] T d35ec7e4a5084a74bea172ef8d9990b5 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.444s user 0.003s sys 0.000s
I20250113 02:51:22.639904 1191 raft_consensus.cc:357] T d35ec7e4a5084a74bea172ef8d9990b5 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:22.640170 1191 raft_consensus.cc:383] T d35ec7e4a5084a74bea172ef8d9990b5 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:22.640242 1191 raft_consensus.cc:738] T d35ec7e4a5084a74bea172ef8d9990b5 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:22.640461 1191 consensus_queue.cc:260] T d35ec7e4a5084a74bea172ef8d9990b5 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:22.643800 1191 ts_tablet_manager.cc:1428] T d35ec7e4a5084a74bea172ef8d9990b5 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250113 02:51:22.647516 1191 tablet_bootstrap.cc:492] T 18e2d79d731142f89b9089869bc736e5 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:22.648348 1195 ts_tablet_manager.cc:1918] T 68023c30a7f644eaa5b4ddd60d6c5ef5 P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:22.648541 1195 log.cc:1198] T 68023c30a7f644eaa5b4ddd60d6c5ef5 P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/68023c30a7f644eaa5b4ddd60d6c5ef5
I20250113 02:51:22.649021 1195 ts_tablet_manager.cc:1939] T 68023c30a7f644eaa5b4ddd60d6c5ef5 P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:22.650274 1191 tablet_bootstrap.cc:654] T 18e2d79d731142f89b9089869bc736e5 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:22.650826 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8547b63c40ec4ccebd962b5a3b2c5e0f" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:22.651113 1059 raft_consensus.cc:2463] T 8547b63c40ec4ccebd962b5a3b2c5e0f P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1186fa869eb14812832fa6562531f196 in term 0.
I20250113 02:51:22.651187 1062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d81eec487dd4016b3b804451da31c49" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:22.651439 1062 raft_consensus.cc:2463] T 1d81eec487dd4016b3b804451da31c49 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1186fa869eb14812832fa6562531f196 in term 0.
I20250113 02:51:22.653733 1194 tablet_bootstrap.cc:492] T d7c4c7b0cfc44ac1952bc73cbe1bced8 P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:22.653918 1194 ts_tablet_manager.cc:1397] T d7c4c7b0cfc44ac1952bc73cbe1bced8 P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.394s user 0.062s sys 0.124s
I20250113 02:51:22.654181 709 leader_election.cc:304] T 8547b63c40ec4ccebd962b5a3b2c5e0f P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters:
W20250113 02:51:22.654295 448 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8547b63c40ec4ccebd962b5a3b2c5e0f consensus state has no leader
I20250113 02:51:22.654521 709 leader_election.cc:304] T 1d81eec487dd4016b3b804451da31c49 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters:
I20250113 02:51:22.654752 1535 raft_consensus.cc:2798] T 8547b63c40ec4ccebd962b5a3b2c5e0f P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 02:51:22.654860 1535 raft_consensus.cc:491] T 8547b63c40ec4ccebd962b5a3b2c5e0f P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 02:51:22.654879 1368 raft_consensus.cc:2798] T 1d81eec487dd4016b3b804451da31c49 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 02:51:22.654982 1535 raft_consensus.cc:3054] T 8547b63c40ec4ccebd962b5a3b2c5e0f P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:22.655005 1368 raft_consensus.cc:491] T 1d81eec487dd4016b3b804451da31c49 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 02:51:22.654876 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_44352829b8da471388d4d10f849b0cd7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:22.655086 1368 raft_consensus.cc:3054] T 1d81eec487dd4016b3b804451da31c49 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Advancing to term 1
W20250113 02:51:22.655571 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_44352829b8da471388d4d10f849b0cd7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250113 02:51:22.656139 454 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 42fad91fe1b64a21a2c8412a2585af47 consensus state has no leader
I20250113 02:51:22.656534 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_4ef3edc5b07c4e0881456bfe23741c90"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:22.657137 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4ef3edc5b07c4e0881456bfe23741c90 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:22.657863 1535 raft_consensus.cc:513] T 8547b63c40ec4ccebd962b5a3b2c5e0f P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
W20250113 02:51:22.658313 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1d2fdc75d00f4e749d322ac6cd5fc410 has no consensus state
I20250113 02:51:22.658473 1535 leader_election.cc:290] T 8547b63c40ec4ccebd962b5a3b2c5e0f P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 election: Requested vote from peers 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:22.658677 455 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_54963a1a7ff24d5a99644441a755f5de" } modify_external_catalogs: true
I20250113 02:51:22.658847 455 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_54963a1a7ff24d5a99644441a755f5de" } modify_external_catalogs: true
I20250113 02:51:22.657923 1368 raft_consensus.cc:513] T 1d81eec487dd4016b3b804451da31c49 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:22.659498 1368 leader_election.cc:290] T 1d81eec487dd4016b3b804451da31c49 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 election: Requested vote from peers 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:22.659824 1062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8547b63c40ec4ccebd962b5a3b2c5e0f" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca"
I20250113 02:51:22.660017 1062 raft_consensus.cc:3054] T 8547b63c40ec4ccebd962b5a3b2c5e0f P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:22.659961 1194 raft_consensus.cc:357] T d7c4c7b0cfc44ac1952bc73cbe1bced8 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:22.660247 1194 raft_consensus.cc:383] T d7c4c7b0cfc44ac1952bc73cbe1bced8 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:22.660321 1194 raft_consensus.cc:738] T d7c4c7b0cfc44ac1952bc73cbe1bced8 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:22.660576 1194 consensus_queue.cc:260] T d7c4c7b0cfc44ac1952bc73cbe1bced8 P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:22.661074 1194 ts_tablet_manager.cc:1428] T d7c4c7b0cfc44ac1952bc73cbe1bced8 P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20250113 02:51:22.661365 1194 tablet_bootstrap.cc:492] T 41bcb2eafff14fa8a040982c723f99c1 P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:22.662020 449 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_cd499f4a8487400690a9ab957c606b7f" } modify_external_catalogs: true
I20250113 02:51:22.662221 449 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_cd499f4a8487400690a9ab957c606b7f" } modify_external_catalogs: true
I20250113 02:51:22.662777 1062 raft_consensus.cc:2463] T 8547b63c40ec4ccebd962b5a3b2c5e0f P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1186fa869eb14812832fa6562531f196 in term 1.
I20250113 02:51:22.662956 1194 tablet_bootstrap.cc:654] T 41bcb2eafff14fa8a040982c723f99c1 P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:22.663862 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_03fafac4f68e4eb3be2ea41493d9e006" } modify_external_catalogs: true
I20250113 02:51:22.664093 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_03fafac4f68e4eb3be2ea41493d9e006" } modify_external_catalogs: true
I20250113 02:51:22.664378 1062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d81eec487dd4016b3b804451da31c49" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca"
I20250113 02:51:22.664589 1062 raft_consensus.cc:3054] T 1d81eec487dd4016b3b804451da31c49 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:22.666045 852 leader_election.cc:304] T 8547b63c40ec4ccebd962b5a3b2c5e0f P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f; no voters:
I20250113 02:51:22.668713 1062 raft_consensus.cc:2463] T 1d81eec487dd4016b3b804451da31c49 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1186fa869eb14812832fa6562531f196 in term 1.
I20250113 02:51:22.669373 1368 consensus_queue.cc:1035] T ad9b95c5450a4c9c89a0930862348d49 P 1186fa869eb14812832fa6562531f196 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 02:51:22.669373 709 leader_election.cc:304] T 8547b63c40ec4ccebd962b5a3b2c5e0f P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters:
I20250113 02:51:22.672410 1368 raft_consensus.cc:2798] T 8547b63c40ec4ccebd962b5a3b2c5e0f P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Leader election won for term 1
I20250113 02:51:22.673002 1368 raft_consensus.cc:695] T 8547b63c40ec4ccebd962b5a3b2c5e0f P 1186fa869eb14812832fa6562531f196 [term 1 LEADER]: Becoming Leader. State: Replica: 1186fa869eb14812832fa6562531f196, State: Running, Role: LEADER
I20250113 02:51:22.672966 709 leader_election.cc:304] T 1d81eec487dd4016b3b804451da31c49 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters:
I20250113 02:51:22.674027 1535 raft_consensus.cc:2798] T 1d81eec487dd4016b3b804451da31c49 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Leader election won for term 1
I20250113 02:51:22.674016 1516 raft_consensus.cc:2798] T 8547b63c40ec4ccebd962b5a3b2c5e0f P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 02:51:22.673913 1368 consensus_queue.cc:237] T 8547b63c40ec4ccebd962b5a3b2c5e0f P 1186fa869eb14812832fa6562531f196 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:22.674142 1516 raft_consensus.cc:491] T 8547b63c40ec4ccebd962b5a3b2c5e0f P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 02:51:22.674232 1516 raft_consensus.cc:3054] T 8547b63c40ec4ccebd962b5a3b2c5e0f P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:22.674284 1535 raft_consensus.cc:695] T 1d81eec487dd4016b3b804451da31c49 P 1186fa869eb14812832fa6562531f196 [term 1 LEADER]: Becoming Leader. State: Replica: 1186fa869eb14812832fa6562531f196, State: Running, Role: LEADER
I20250113 02:51:22.674515 1535 consensus_queue.cc:237] T 1d81eec487dd4016b3b804451da31c49 P 1186fa869eb14812832fa6562531f196 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
W20250113 02:51:22.683830 851 leader_election.cc:343] T 4efa63b8be794e1db4b340f4c0c4cd5a P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:22.684028 851 leader_election.cc:304] T 4efa63b8be794e1db4b340f4c0c4cd5a P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df69f48656b44bb890f76587ee69cc0f; no voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca
I20250113 02:51:22.684304 1518 raft_consensus.cc:2743] T 4efa63b8be794e1db4b340f4c0c4cd5a P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:22.687314 1516 raft_consensus.cc:513] T 8547b63c40ec4ccebd962b5a3b2c5e0f P df69f48656b44bb890f76587ee69cc0f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:22.687518 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_f2da084a4aee4d19b45e784e56f926a6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:22.688024 1516 leader_election.cc:290] T 8547b63c40ec4ccebd962b5a3b2c5e0f P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 election: Requested vote from peers 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587), 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
I20250113 02:51:22.688905 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8547b63c40ec4ccebd962b5a3b2c5e0f" candidate_uuid: "df69f48656b44bb890f76587ee69cc0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca"
I20250113 02:51:22.688985 446 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_2d201bcde7f04c76b66dd5c6a09ddbff" } modify_external_catalogs: true
I20250113 02:51:22.689157 1061 raft_consensus.cc:2388] T 8547b63c40ec4ccebd962b5a3b2c5e0f P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate df69f48656b44bb890f76587ee69cc0f in current term 1: Already voted for candidate 1186fa869eb14812832fa6562531f196 in this term.
I20250113 02:51:22.689157 446 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_2d201bcde7f04c76b66dd5c6a09ddbff" } modify_external_catalogs: true
W20250113 02:51:22.689795 711 leader_election.cc:343] T 1d81eec487dd4016b3b804451da31c49 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Not found: Tablet not found: 1d81eec487dd4016b3b804451da31c49
I20250113 02:51:22.690100 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8547b63c40ec4ccebd962b5a3b2c5e0f" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f"
I20250113 02:51:22.690317 919 raft_consensus.cc:2388] T 8547b63c40ec4ccebd962b5a3b2c5e0f P df69f48656b44bb890f76587ee69cc0f [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 1186fa869eb14812832fa6562531f196 in current term 1: Already voted for candidate df69f48656b44bb890f76587ee69cc0f in this term.
W20250113 02:51:22.692046 711 leader_election.cc:343] T 1d81eec487dd4016b3b804451da31c49 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Not found: Tablet not found: 1d81eec487dd4016b3b804451da31c49
I20250113 02:51:22.704620 452 catalog_manager.cc:5618] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587) does not have the latest schema for tablet 8547b63c40ec4ccebd962b5a3b2c5e0f (table default.Table_3d5fd97116b343b0a6c798e1f22327c2 [id=1396c4482ee646d18330d200fd3f9816]). Expected version 1 got 0
I20250113 02:51:22.706111 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_c604c45a4d3a49f0b6fd1ec85582d495" } new_table_name: "default.Table_eaa71690c1224509a1b527415d09f340" modify_external_catalogs: true
I20250113 02:51:22.707017 1034 tablet_service.cc:1467] Processing CreateTablet for tablet b815003a27f64809baaa2d43d6293cef (DEFAULT_TABLE table=default.Table_b21bac46382e43c4bb99fc94778daeeb [id=42e19138c08f422a92ba9621e9609957]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:22.707484 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b815003a27f64809baaa2d43d6293cef. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:22.707754 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet 53cc528d282c47788d5d558569f074c1 with delete_type TABLET_DATA_DELETED (replaced by tablet 7d3562a48c6e4badbbcf37640e91b5ba) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:22.707707 1038 tablet_service.cc:1467] Processing CreateTablet for tablet b33025a462e74f2aa26cc3779f36615b (DEFAULT_TABLE table=default.Table_719ed613a9354add97b036faad401762 [id=a8e428598ca94e15b04f1df8112a4abc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:22.707963 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d81eec487dd4016b3b804451da31c49 with delete_type TABLET_DATA_DELETED (replaced by tablet 438d1d2300e044f39a15586f335935ba) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:22.708204 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b33025a462e74f2aa26cc3779f36615b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:22.708249 1040 tablet_service.cc:1514] Processing DeleteTablet for tablet 107e6e9e2d84496f8fce92b9ba377189 with delete_type TABLET_DATA_DELETED (replaced by tablet 7ae2f30bf92742008e53b1ec94ebd0cf) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:22.710407 889 tablet_service.cc:1467] Processing CreateTablet for tablet b815003a27f64809baaa2d43d6293cef (DEFAULT_TABLE table=default.Table_b21bac46382e43c4bb99fc94778daeeb [id=42e19138c08f422a92ba9621e9609957]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:22.711017 888 tablet_service.cc:1467] Processing CreateTablet for tablet 42dbd37e20fa47318bc6497423dbe565 (DEFAULT_TABLE table=default.Table_37b52d3cfe0f4b0fb1ef1468117512dd [id=45aa63632c914a4e99eb69759b56c842]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:22.711206 1029 tablet_service.cc:1467] Processing CreateTablet for tablet ba8980ad3a364c888a825affbcbc9b1f (DEFAULT_TABLE table=default.Table_6716ec8ca07b476fbaa9480c4d821568 [id=a1d8c84954c747adb2aa43726d7b5334]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:22.711076 1035 tablet_service.cc:1467] Processing CreateTablet for tablet 42dbd37e20fa47318bc6497423dbe565 (DEFAULT_TABLE table=default.Table_37b52d3cfe0f4b0fb1ef1468117512dd [id=45aa63632c914a4e99eb69759b56c842]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:22.711578 897 tablet_service.cc:1467] Processing CreateTablet for tablet ba8980ad3a364c888a825affbcbc9b1f (DEFAULT_TABLE table=default.Table_6716ec8ca07b476fbaa9480c4d821568 [id=a1d8c84954c747adb2aa43726d7b5334]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:22.704444 453 catalog_manager.cc:5618] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467) does not have the latest schema for tablet 4fe54131aafb40ee9d8bc5ea299af193 (table default.Table_7c37f90c233c4b79b3e2f866b8198d0c [id=70089f757a1c45f6b983f216fb167478]). Expected version 2 got 0
I20250113 02:51:22.712117 881 tablet_service.cc:1467] Processing CreateTablet for tablet b33025a462e74f2aa26cc3779f36615b (DEFAULT_TABLE table=default.Table_719ed613a9354add97b036faad401762 [id=a8e428598ca94e15b04f1df8112a4abc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:22.712018 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba8980ad3a364c888a825affbcbc9b1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:22.712469 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b33025a462e74f2aa26cc3779f36615b. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:22.712636 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f2da084a4aee4d19b45e784e56f926a6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:22.714012 453 catalog_manager.cc:5618] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587) does not have the latest schema for tablet ad9b95c5450a4c9c89a0930862348d49 (table default.Table_936e73f0b76046d285567ac0778e36a9 [id=219c739889d04a97aafa4fe1b55b0a24]). Expected version 2 got 0
I20250113 02:51:22.714720 453 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_d5c2b805dfbf4227bb1ac190525b7dee" } modify_external_catalogs: true
I20250113 02:51:22.714906 453 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_d5c2b805dfbf4227bb1ac190525b7dee" } modify_external_catalogs: true
I20250113 02:51:22.716135 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba8980ad3a364c888a825affbcbc9b1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:22.716147 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 42dbd37e20fa47318bc6497423dbe565. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:22.720227 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 42dbd37e20fa47318bc6497423dbe565. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:22.724108 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b815003a27f64809baaa2d43d6293cef. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:22.727903 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_74ce8ff3e2c64e088ac82c48ec722b39"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:22.728506 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_74ce8ff3e2c64e088ac82c48ec722b39 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:22.731252 752 tablet_service.cc:1467] Processing CreateTablet for tablet b33025a462e74f2aa26cc3779f36615b (DEFAULT_TABLE table=default.Table_719ed613a9354add97b036faad401762 [id=a8e428598ca94e15b04f1df8112a4abc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:22.731261 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_bfda83bbe8bb4025b01edf1e38b1b73c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:22.732411 748 tablet_service.cc:1467] Processing CreateTablet for tablet b815003a27f64809baaa2d43d6293cef (DEFAULT_TABLE table=default.Table_b21bac46382e43c4bb99fc94778daeeb [id=42e19138c08f422a92ba9621e9609957]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:22.733414 744 tablet_service.cc:1467] Processing CreateTablet for tablet 42dbd37e20fa47318bc6497423dbe565 (DEFAULT_TABLE table=default.Table_37b52d3cfe0f4b0fb1ef1468117512dd [id=45aa63632c914a4e99eb69759b56c842]), partition=RANGE (key) PARTITION UNBOUNDED
W20250113 02:51:22.733101 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bfda83bbe8bb4025b01edf1e38b1b73c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:22.736222 743 tablet_service.cc:1467] Processing CreateTablet for tablet 3d85539cf0074a03b167d3279e940079 (DEFAULT_TABLE table=default.Table_3a73b4672b154b99af1a24e962007ea0 [id=f742014cd0d44db99c4c1a031630d827]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:22.745333 754 tablet_service.cc:1467] Processing CreateTablet for tablet dc680b5680b84abbab219296bc7ee707 (DEFAULT_TABLE table=default.Table_25a2c280183748df92b064c40756f5ac [id=a1ac1dc79bce47258f1dbf33837b782c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:22.747148 746 tablet_service.cc:1467] Processing CreateTablet for tablet 82505f1389184d0ca12ec0e2d6084d22 (DEFAULT_TABLE table=default.Table_437ebdf134da491e947f52a1bcc89f01 [id=210a731382514eb098fb9442e6c290ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:22.747736 737 tablet_service.cc:1467] Processing CreateTablet for tablet 153b8f90b5a847b88944f683d306f579 (DEFAULT_TABLE table=default.Table_a7c3697e25a94815ac03a3cb1a57bc0c [id=39d8412c30d24c44a01c24b3c9645a8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:22.748934 1194 tablet_bootstrap.cc:492] T 41bcb2eafff14fa8a040982c723f99c1 P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:22.749104 1194 ts_tablet_manager.cc:1397] T 41bcb2eafff14fa8a040982c723f99c1 P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.088s user 0.003s sys 0.002s
I20250113 02:51:22.749107 1038 tablet_service.cc:1467] Processing CreateTablet for tablet 3d85539cf0074a03b167d3279e940079 (DEFAULT_TABLE table=default.Table_3a73b4672b154b99af1a24e962007ea0 [id=f742014cd0d44db99c4c1a031630d827]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:22.749552 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3d85539cf0074a03b167d3279e940079. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:22.749797 1028 tablet_service.cc:1467] Processing CreateTablet for tablet 82505f1389184d0ca12ec0e2d6084d22 (DEFAULT_TABLE table=default.Table_437ebdf134da491e947f52a1bcc89f01 [id=210a731382514eb098fb9442e6c290ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:22.750186 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82505f1389184d0ca12ec0e2d6084d22. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:22.750032 1194 raft_consensus.cc:357] T 41bcb2eafff14fa8a040982c723f99c1 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:22.750259 1194 raft_consensus.cc:383] T 41bcb2eafff14fa8a040982c723f99c1 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:22.750341 1194 raft_consensus.cc:738] T 41bcb2eafff14fa8a040982c723f99c1 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:22.750877 1024 tablet_service.cc:1467] Processing CreateTablet for tablet 153b8f90b5a847b88944f683d306f579 (DEFAULT_TABLE table=default.Table_a7c3697e25a94815ac03a3cb1a57bc0c [id=39d8412c30d24c44a01c24b3c9645a8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:22.751235 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 153b8f90b5a847b88944f683d306f579. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:22.752323 1033 tablet_service.cc:1467] Processing CreateTablet for tablet 438604e8859f422f9caccb61c5b5b125 (DEFAULT_TABLE table=default.Table_059d10dd368b428db10509d40437287b [id=c6e65e4bab5040cca1f0f55721639d2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:22.749547 1040 tablet_service.cc:1467] Processing CreateTablet for tablet dc680b5680b84abbab219296bc7ee707 (DEFAULT_TABLE table=default.Table_25a2c280183748df92b064c40756f5ac [id=a1ac1dc79bce47258f1dbf33837b782c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:22.752923 1040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dc680b5680b84abbab219296bc7ee707. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:22.753719 740 tablet_service.cc:1467] Processing CreateTablet for tablet 64aa8390b1b1475c8743b6491d40674c (DEFAULT_TABLE table=default.Table_b89481a4a7be4e49b13f3260d8a31d5d [id=6c751a49967d448ba1432934782b08d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:22.754393 742 tablet_service.cc:1467] Processing CreateTablet for tablet 0c408921a09a463eb464e08edcf042b3 (DEFAULT_TABLE table=default.Table_08def01561d44ec5a88f5b5a82d47e3b [id=c24af26d1a734152af598a5aa258cb06]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:22.755200 882 tablet_service.cc:1467] Processing CreateTablet for tablet 0c408921a09a463eb464e08edcf042b3 (DEFAULT_TABLE table=default.Table_08def01561d44ec5a88f5b5a82d47e3b [id=c24af26d1a734152af598a5aa258cb06]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:22.755617 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c408921a09a463eb464e08edcf042b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:22.759573 1198 ts_tablet_manager.cc:1918] T 68023c30a7f644eaa5b4ddd60d6c5ef5 P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:22.760354 749 tablet_service.cc:1467] Processing CreateTablet for tablet 438604e8859f422f9caccb61c5b5b125 (DEFAULT_TABLE table=default.Table_059d10dd368b428db10509d40437287b [id=c6e65e4bab5040cca1f0f55721639d2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:22.760612 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 438604e8859f422f9caccb61c5b5b125. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:22.761835 739 tablet_service.cc:1467] Processing CreateTablet for tablet ba8980ad3a364c888a825affbcbc9b1f (DEFAULT_TABLE table=default.Table_6716ec8ca07b476fbaa9480c4d821568 [id=a1d8c84954c747adb2aa43726d7b5334]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:22.763304 1023 tablet_service.cc:1467] Processing CreateTablet for tablet 0c408921a09a463eb464e08edcf042b3 (DEFAULT_TABLE table=default.Table_08def01561d44ec5a88f5b5a82d47e3b [id=c24af26d1a734152af598a5aa258cb06]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:22.763669 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c408921a09a463eb464e08edcf042b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:22.765745 888 tablet_service.cc:1467] Processing CreateTablet for tablet 82505f1389184d0ca12ec0e2d6084d22 (DEFAULT_TABLE table=default.Table_437ebdf134da491e947f52a1bcc89f01 [id=210a731382514eb098fb9442e6c290ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:22.766361 895 tablet_service.cc:1467] Processing CreateTablet for tablet 153b8f90b5a847b88944f683d306f579 (DEFAULT_TABLE table=default.Table_a7c3697e25a94815ac03a3cb1a57bc0c [id=39d8412c30d24c44a01c24b3c9645a8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:22.768055 880 tablet_service.cc:1467] Processing CreateTablet for tablet 64aa8390b1b1475c8743b6491d40674c (DEFAULT_TABLE table=default.Table_b89481a4a7be4e49b13f3260d8a31d5d [id=6c751a49967d448ba1432934782b08d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:22.768083 885 tablet_service.cc:1467] Processing CreateTablet for tablet 438604e8859f422f9caccb61c5b5b125 (DEFAULT_TABLE table=default.Table_059d10dd368b428db10509d40437287b [id=c6e65e4bab5040cca1f0f55721639d2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:22.768251 898 tablet_service.cc:1514] Processing DeleteTablet for tablet 4efa63b8be794e1db4b340f4c0c4cd5a with delete_type TABLET_DATA_DELETED (replaced by tablet eceed0f52c9f46bbaa78ca773a7afde5) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:22.768144 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 153b8f90b5a847b88944f683d306f579. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:22.769239 1191 tablet_bootstrap.cc:492] T 18e2d79d731142f89b9089869bc736e5 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:22.769438 1191 ts_tablet_manager.cc:1397] T 18e2d79d731142f89b9089869bc736e5 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.122s user 0.005s sys 0.005s
I20250113 02:51:22.771227 1191 raft_consensus.cc:357] T 18e2d79d731142f89b9089869bc736e5 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:22.771502 1191 raft_consensus.cc:383] T 18e2d79d731142f89b9089869bc736e5 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:22.771636 1191 raft_consensus.cc:738] T 18e2d79d731142f89b9089869bc736e5 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:22.771999 1191 consensus_queue.cc:260] T 18e2d79d731142f89b9089869bc736e5 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:22.772266 753 tablet_service.cc:1467] Processing CreateTablet for tablet 1d2fdc75d00f4e749d322ac6cd5fc410 (DEFAULT_TABLE table=default.Table_b09c6cee53c64995a167959c15f5e288 [id=0ab8e0c80dda42fcb07f2fd6fbf01609]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:22.777181 1191 ts_tablet_manager.cc:1428] T 18e2d79d731142f89b9089869bc736e5 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.008s user 0.000s sys 0.002s
I20250113 02:51:22.779139 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe54131aafb40ee9d8bc5ea299af193" candidate_uuid: "df69f48656b44bb890f76587ee69cc0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
I20250113 02:51:22.779501 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8547b63c40ec4ccebd962b5a3b2c5e0f" candidate_uuid: "df69f48656b44bb890f76587ee69cc0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
I20250113 02:51:22.779793 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8547b63c40ec4ccebd962b5a3b2c5e0f" candidate_uuid: "df69f48656b44bb890f76587ee69cc0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196"
I20250113 02:51:22.787830 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet c1fb2b36c0504cb4ba32671ae4c620d1
I20250113 02:51:22.788506 447 catalog_manager.cc:6596] ReplaceTablet: tablet c1fb2b36c0504cb4ba32671ae4c620d1 deleted and replaced by tablet fb2458e2044d49029ed2d6851db9de25
I20250113 02:51:22.788661 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 0644f5600c5b47e29f3fa73060ed96e3
I20250113 02:51:22.788885 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_82f5a202f8d04a2680202b3ab06994c4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:22.789167 449 master_service.cc:946] ReplaceTablet: received request to replace tablet 7d53de6379b84103ba905effb0835676 from {username='slave'} at 127.0.0.1:49258
W20250113 02:51:22.789463 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_82f5a202f8d04a2680202b3ab06994c4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:22.790341 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 42cf1e708644427f9d4dcd41ae2bcb37
I20250113 02:51:22.790825 455 master_service.cc:946] ReplaceTablet: received request to replace tablet 23e04961f072440fa3e3f9485e40f79c from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:22.791437 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 092892a36e0e47cb8d2184c2bbde95d7
I20250113 02:51:22.792279 450 master_service.cc:946] ReplaceTablet: received request to replace tablet 23e04961f072440fa3e3f9485e40f79c from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:22.792448 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64aa8390b1b1475c8743b6491d40674c. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:22.792614 851 leader_election.cc:343] T 4fe54131aafb40ee9d8bc5ea299af193 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:22.792444 1196 tablet_replica.cc:331] T 4efa63b8be794e1db4b340f4c0c4cd5a P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:22.794134 851 leader_election.cc:304] T 4fe54131aafb40ee9d8bc5ea299af193 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df69f48656b44bb890f76587ee69cc0f; no voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca
I20250113 02:51:22.794220 1194 consensus_queue.cc:260] T 41bcb2eafff14fa8a040982c723f99c1 P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:22.792771 891 tablet_service.cc:1467] Processing CreateTablet for tablet 3d85539cf0074a03b167d3279e940079 (DEFAULT_TABLE table=default.Table_3a73b4672b154b99af1a24e962007ea0 [id=f742014cd0d44db99c4c1a031630d827]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:22.796684 750 tablet_service.cc:1514] Processing DeleteTablet for tablet 0644f5600c5b47e29f3fa73060ed96e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:22 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:22.797098 750 tablet_service.cc:1514] Processing DeleteTablet for tablet 42cf1e708644427f9d4dcd41ae2bcb37 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:22 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:22.797430 750 tablet_service.cc:1514] Processing DeleteTablet for tablet 092892a36e0e47cb8d2184c2bbde95d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:22 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:22.801296 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c408921a09a463eb464e08edcf042b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:22.806777 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet c1fb2b36c0504cb4ba32671ae4c620d1 with delete_type TABLET_DATA_DELETED (replaced by tablet fb2458e2044d49029ed2d6851db9de25) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:22.807411 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet b0113b16110e4787ac1c4676b2c8bd1e
I20250113 02:51:22.808032 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 438604e8859f422f9caccb61c5b5b125. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:22.808552 886 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c8ca30720664f3a819c4bff2493934d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:18 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:22.808821 851 leader_election.cc:304] T 8547b63c40ec4ccebd962b5a3b2c5e0f P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df69f48656b44bb890f76587ee69cc0f; no voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca
I20250113 02:51:22.809243 1196 raft_consensus.cc:2238] T 4efa63b8be794e1db4b340f4c0c4cd5a P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
W20250113 02:51:22.810794 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b0b267b0d9c54ba496c89f6cb287dffd consensus state has no leader
I20250113 02:51:22.812656 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 7d53de6379b84103ba905effb0835676
I20250113 02:51:22.813128 449 catalog_manager.cc:6596] ReplaceTablet: tablet 7d53de6379b84103ba905effb0835676 deleted and replaced by tablet fd0b8592997d4a61b8ca854ecbb67c55
I20250113 02:51:22.813922 897 tablet_service.cc:1467] Processing CreateTablet for tablet dc680b5680b84abbab219296bc7ee707 (DEFAULT_TABLE table=default.Table_25a2c280183748df92b064c40756f5ac [id=a1ac1dc79bce47258f1dbf33837b782c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:22.814430 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 80d31d5ead804ddaa032969e401a8f7e
I20250113 02:51:22.817755 742 tablet_service.cc:1514] Processing DeleteTablet for tablet d35ec7e4a5084a74bea172ef8d9990b5 with delete_type TABLET_DATA_DELETED (replaced by tablet b4806b18a4194f9c95350d89ba62e13a) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:22.818512 750 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d53de6379b84103ba905effb0835676 with delete_type TABLET_DATA_DELETED (replaced by tablet fd0b8592997d4a61b8ca854ecbb67c55) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:22.819010 750 tablet_service.cc:1514] Processing DeleteTablet for tablet 80d31d5ead804ddaa032969e401a8f7e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:22 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:22.819295 742 tablet_service.cc:1514] Processing DeleteTablet for tablet b0113b16110e4787ac1c4676b2c8bd1e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:22 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:22.819720 446 catalog_manager.cc:5526] T 8547b63c40ec4ccebd962b5a3b2c5e0f P 1186fa869eb14812832fa6562531f196 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1186fa869eb14812832fa6562531f196 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "1186fa869eb14812832fa6562531f196" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } health_report { overall_health: HEALTHY } } }
I20250113 02:51:22.820376 446 catalog_manager.cc:5618] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095) does not have the latest schema for tablet 8547b63c40ec4ccebd962b5a3b2c5e0f (table default.Table_3d5fd97116b343b0a6c798e1f22327c2 [id=1396c4482ee646d18330d200fd3f9816]). Expected version 1 got 0
I20250113 02:51:22.810514 1034 tablet_service.cc:1467] Processing CreateTablet for tablet 64aa8390b1b1475c8743b6491d40674c (DEFAULT_TABLE table=default.Table_b89481a4a7be4e49b13f3260d8a31d5d [id=6c751a49967d448ba1432934782b08d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:22.822952 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64aa8390b1b1475c8743b6491d40674c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:22.828714 881 tablet_service.cc:1467] Processing CreateTablet for tablet 1d2fdc75d00f4e749d322ac6cd5fc410 (DEFAULT_TABLE table=default.Table_b09c6cee53c64995a167959c15f5e288 [id=0ab8e0c80dda42fcb07f2fd6fbf01609]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:22.829496 1196 raft_consensus.cc:2267] T 4efa63b8be794e1db4b340f4c0c4cd5a P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:22.830094 1196 ts_tablet_manager.cc:1905] T 4efa63b8be794e1db4b340f4c0c4cd5a P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:22.830372 889 tablet_service.cc:1514] Processing DeleteTablet for tablet c1fb2b36c0504cb4ba32671ae4c620d1 with delete_type TABLET_DATA_DELETED (replaced by tablet fb2458e2044d49029ed2d6851db9de25) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:22.830399 1516 raft_consensus.cc:2743] T 4fe54131aafb40ee9d8bc5ea299af193 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:22.831362 1040 tablet_service.cc:1514] Processing DeleteTablet for tablet 80d31d5ead804ddaa032969e401a8f7e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:22 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:22.831506 741 tablet_service.cc:1514] Processing DeleteTablet for tablet c1fb2b36c0504cb4ba32671ae4c620d1 with delete_type TABLET_DATA_DELETED (replaced by tablet fb2458e2044d49029ed2d6851db9de25) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:22.831790 1033 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d53de6379b84103ba905effb0835676 with delete_type TABLET_DATA_DELETED (replaced by tablet fd0b8592997d4a61b8ca854ecbb67c55) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:22.832007 742 tablet_service.cc:1514] Processing DeleteTablet for tablet 18e2d79d731142f89b9089869bc736e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:18 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:22.832067 1198 log.cc:1198] T 68023c30a7f644eaa5b4ddd60d6c5ef5 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/68023c30a7f644eaa5b4ddd60d6c5ef5
I20250113 02:51:22.832105 1035 tablet_service.cc:1467] Processing CreateTablet for tablet 1d2fdc75d00f4e749d322ac6cd5fc410 (DEFAULT_TABLE table=default.Table_b09c6cee53c64995a167959c15f5e288 [id=0ab8e0c80dda42fcb07f2fd6fbf01609]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:22.832507 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d2fdc75d00f4e749d322ac6cd5fc410. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:22.832566 1198 ts_tablet_manager.cc:1939] T 68023c30a7f644eaa5b4ddd60d6c5ef5 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:22.832659 886 tablet_service.cc:1514] Processing DeleteTablet for tablet 0644f5600c5b47e29f3fa73060ed96e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:22 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:22.832801 1438 raft_consensus.cc:2743] T 8547b63c40ec4ccebd962b5a3b2c5e0f P df69f48656b44bb890f76587ee69cc0f [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250113 02:51:22.833051 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64aa8390b1b1475c8743b6491d40674c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:22.833551 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet 68023c30a7f644eaa5b4ddd60d6c5ef5 (table default.Table_78c0bcf1e2d64abc81253301920cf566 [id=e5831758ca184c1998f551fe0285d010]) successfully deleted
I20250113 02:51:22.834962 1198 tablet_replica.cc:331] T 18e2d79d731142f89b9089869bc736e5 P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:22.835139 1198 raft_consensus.cc:2238] T 18e2d79d731142f89b9089869bc736e5 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:22.835811 1198 raft_consensus.cc:2267] T 18e2d79d731142f89b9089869bc736e5 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:22.836375 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d2fdc75d00f4e749d322ac6cd5fc410. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:22.836421 1198 ts_tablet_manager.cc:1905] T 18e2d79d731142f89b9089869bc736e5 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:22.841003 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet 0644f5600c5b47e29f3fa73060ed96e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:22 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:22.841190 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet 42cf1e708644427f9d4dcd41ae2bcb37 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:22 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:22.841341 1039 tablet_service.cc:1514] Processing DeleteTablet for tablet 092892a36e0e47cb8d2184c2bbde95d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:22 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:22.841758 449 catalog_manager.cc:5618] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587) does not have the latest schema for tablet ad9b95c5450a4c9c89a0930862348d49 (table default.Table_936e73f0b76046d285567ac0778e36a9 [id=219c739889d04a97aafa4fe1b55b0a24]). Expected version 2 got 0
I20250113 02:51:22.842600 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet b0113b16110e4787ac1c4676b2c8bd1e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:22 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:22.843326 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 438604e8859f422f9caccb61c5b5b125. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:22.844175 892 tablet_service.cc:1514] Processing DeleteTablet for tablet 80d31d5ead804ddaa032969e401a8f7e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:22 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:22.844358 449 catalog_manager.cc:5618] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467) does not have the latest schema for tablet ad9b95c5450a4c9c89a0930862348d49 (table default.Table_936e73f0b76046d285567ac0778e36a9 [id=219c739889d04a97aafa4fe1b55b0a24]). Expected version 2 got 0
I20250113 02:51:22.857333 1198 ts_tablet_manager.cc:1918] T 18e2d79d731142f89b9089869bc736e5 P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:22.859212 1198 log.cc:1198] T 18e2d79d731142f89b9089869bc736e5 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/18e2d79d731142f89b9089869bc736e5
I20250113 02:51:22.859608 737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 153b8f90b5a847b88944f683d306f579. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:22.859745 1198 ts_tablet_manager.cc:1939] T 18e2d79d731142f89b9089869bc736e5 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:22.861155 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dc680b5680b84abbab219296bc7ee707. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:22.863597 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet 18e2d79d731142f89b9089869bc736e5 (table default.Table_f9aa5df6ba3d43999ebd6570df3108d1 [id=fa5a1cda58f54103ac637c12d99e4e38]) successfully deleted
I20250113 02:51:22.864023 1198 tablet_replica.cc:331] T 53cc528d282c47788d5d558569f074c1 P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:22.867271 1198 raft_consensus.cc:2238] T 53cc528d282c47788d5d558569f074c1 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:22.867504 1198 raft_consensus.cc:2267] T 53cc528d282c47788d5d558569f074c1 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:22.868245 1368 raft_consensus.cc:491] T c71511a100f14322b5091a9804720fa0 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:22.868409 1368 raft_consensus.cc:513] T c71511a100f14322b5091a9804720fa0 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:22.869691 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b33025a462e74f2aa26cc3779f36615b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:22.870595 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 42dbd37e20fa47318bc6497423dbe565. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:22.871012 1062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c71511a100f14322b5091a9804720fa0" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:22.871316 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b815003a27f64809baaa2d43d6293cef. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:22.871659 709 leader_election.cc:343] T c71511a100f14322b5091a9804720fa0 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:22.871991 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82505f1389184d0ca12ec0e2d6084d22. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:22.872700 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3d85539cf0074a03b167d3279e940079. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:22.876147 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba8980ad3a364c888a825affbcbc9b1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:22.877377 1368 leader_election.cc:290] T c71511a100f14322b5091a9804720fa0 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
W20250113 02:51:22.877843 711 leader_election.cc:343] T c71511a100f14322b5091a9804720fa0 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Not found: Tablet not found: c71511a100f14322b5091a9804720fa0
I20250113 02:51:22.878118 711 leader_election.cc:304] T c71511a100f14322b5091a9804720fa0 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:22.878387 1368 raft_consensus.cc:2743] T c71511a100f14322b5091a9804720fa0 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:22.881526 886 tablet_service.cc:1514] Processing DeleteTablet for tablet 42cf1e708644427f9d4dcd41ae2bcb37 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:22 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:22.882371 889 tablet_service.cc:1514] Processing DeleteTablet for tablet 092892a36e0e47cb8d2184c2bbde95d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:22 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:22.882568 882 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d53de6379b84103ba905effb0835676 with delete_type TABLET_DATA_DELETED (replaced by tablet fd0b8592997d4a61b8ca854ecbb67c55) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:22.882741 892 tablet_service.cc:1514] Processing DeleteTablet for tablet d7c4c7b0cfc44ac1952bc73cbe1bced8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:18 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:22.882938 898 tablet_service.cc:1514] Processing DeleteTablet for tablet b0113b16110e4787ac1c4676b2c8bd1e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:22 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:22.883172 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82505f1389184d0ca12ec0e2d6084d22. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:22.884857 1062 raft_consensus.cc:1270] T 8547b63c40ec4ccebd962b5a3b2c5e0f P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Refusing update from remote peer 1186fa869eb14812832fa6562531f196: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250113 02:51:22.885521 918 raft_consensus.cc:1270] T 8547b63c40ec4ccebd962b5a3b2c5e0f P df69f48656b44bb890f76587ee69cc0f [term 1 FOLLOWER]: Refusing update from remote peer 1186fa869eb14812832fa6562531f196: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250113 02:51:22.886121 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d2fdc75d00f4e749d322ac6cd5fc410. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:22.888142 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dc680b5680b84abbab219296bc7ee707. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:22.893307 1368 consensus_queue.cc:1035] T 8547b63c40ec4ccebd962b5a3b2c5e0f P 1186fa869eb14812832fa6562531f196 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.008s
I20250113 02:51:22.899132 449 catalog_manager.cc:5618] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467) does not have the latest schema for tablet ad9b95c5450a4c9c89a0930862348d49 (table default.Table_936e73f0b76046d285567ac0778e36a9 [id=219c739889d04a97aafa4fe1b55b0a24]). Expected version 2 got 0
I20250113 02:51:22.902716 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3d85539cf0074a03b167d3279e940079. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:22.913008 1444 mvcc.cc:204] Tried to move back new op lower bound from 7113673441484201984 to 7113673424588984320. Current Snapshot: MvccSnapshot[applied={T|T < 7113673441484201984 or (T in {7113673441484201984})}]
I20250113 02:51:22.913151 1528 consensus_queue.cc:1035] T 8547b63c40ec4ccebd962b5a3b2c5e0f P 1186fa869eb14812832fa6562531f196 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.019s
I20250113 02:51:22.914068 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_7fc5d3f1e17c47c9938bc53627275ae0" } new_table_name: "default.Table_82a014a9f48c43b4b14d63522bac7124" modify_external_catalogs: true
I20250113 02:51:22.914595 448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_01a7a0bb02fd483ebe043578768a48e6" } new_table_name: "default.Table_276aa0166db04badb3d250a05e6e7f50" modify_external_catalogs: true
I20250113 02:51:22.915761 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_addfcf0ba42e4d218b80a9748c704587" } new_table_name: "default.Table_9659cbc5ffdb400b8415dc81aa211f89" modify_external_catalogs: true
I20250113 02:51:22.916489 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_e81c865deb634a57bf1906b18b31a60a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:22.917022 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e81c865deb634a57bf1906b18b31a60a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:22.917017 454 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_437ebdf134da491e947f52a1bcc89f01" } new_table_name: "default.Table_855f1069eae9472998eb347f20b003b8" modify_external_catalogs: true
W20250113 02:51:22.924810 448 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet be4b2a8a3927410eaf12e7592646e4af consensus state has no leader
W20250113 02:51:22.926980 454 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 82505f1389184d0ca12ec0e2d6084d22 consensus state has no leader
W20250113 02:51:22.927358 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 572e4eed1e924381bada27f8389821a7 consensus state has no leader
W20250113 02:51:22.928828 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6a4c6dc08df744e6856bdf99ab65c575 has no consensus state
I20250113 02:51:22.931859 449 catalog_manager.cc:5618] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587) does not have the latest schema for tablet 8547b63c40ec4ccebd962b5a3b2c5e0f (table default.Table_3d5fd97116b343b0a6c798e1f22327c2 [id=1396c4482ee646d18330d200fd3f9816]). Expected version 1 got 0
I20250113 02:51:22.956193 1198 ts_tablet_manager.cc:1905] T 53cc528d282c47788d5d558569f074c1 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:22.959542 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_2087b79408c949cc88238ba68aabcaf1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:22.960716 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2087b79408c949cc88238ba68aabcaf1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:22.962085 447 master_service.cc:946] ReplaceTablet: received request to replace tablet 55ebe9c044d944bc813396c699ba9aa1 from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:22.961987 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_eef7fe6c891b4fe984f8fb4a1c667a8e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:22.962327 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_d018642ee8b5413b90b86e9e114353a3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:22.962920 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_eef7fe6c891b4fe984f8fb4a1c667a8e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:22.969579 1550 tablet.cc:1721] T 8547b63c40ec4ccebd962b5a3b2c5e0f P 4216a7f853d642b7bc3e5cff458f3aca: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250113 02:51:22.971256 1194 ts_tablet_manager.cc:1428] T 41bcb2eafff14fa8a040982c723f99c1 P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.199s user 0.001s sys 0.002s
I20250113 02:51:22.972302 1194 tablet_bootstrap.cc:492] T fc92f19390fd48a4a45ba8fb839020fd P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:22.983793 1191 tablet_bootstrap.cc:492] T d5bd09a1ede644c59e400e49c8125793 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:22.986565 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet 68023c30a7f644eaa5b4ddd60d6c5ef5 (table default.Table_78c0bcf1e2d64abc81253301920cf566 [id=e5831758ca184c1998f551fe0285d010]) successfully deleted
I20250113 02:51:22.987900 1195 tablet_replica.cc:331] T c71511a100f14322b5091a9804720fa0 P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:22.990003 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_6b241846ad39435097591810fa57f13d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:22.990216 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_4f36a72976b7408299b97e28dce161b8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:22.990279 1192 thread.cc:635] log-alloc [worker] (thread pool) Time spent creating pthread: real 0.502s user 0.069s sys 0.091s
W20250113 02:51:22.990645 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6b241846ad39435097591810fa57f13d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250113 02:51:22.990772 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4f36a72976b7408299b97e28dce161b8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250113 02:51:22.990689 1192 thread.cc:601] log-alloc [worker] (thread pool) Time spent starting thread: real 0.505s user 0.069s sys 0.091s
W20250113 02:51:22.991561 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d018642ee8b5413b90b86e9e114353a3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:22.997460 1192 tablet_bootstrap.cc:492] T c71511a100f14322b5091a9804720fa0 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:22.988448 1195 raft_consensus.cc:2238] T c71511a100f14322b5091a9804720fa0 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:23.001427 452 catalog_manager.cc:5618] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587) does not have the latest schema for tablet 8547b63c40ec4ccebd962b5a3b2c5e0f (table default.Table_3d5fd97116b343b0a6c798e1f22327c2 [id=1396c4482ee646d18330d200fd3f9816]). Expected version 1 got 0
I20250113 02:51:22.998836 1192 ts_tablet_manager.cc:1397] T c71511a100f14322b5091a9804720fa0 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.515s user 0.071s sys 0.091s
I20250113 02:51:23.006989 1192 raft_consensus.cc:357] T c71511a100f14322b5091a9804720fa0 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:23.008523 1192 raft_consensus.cc:383] T c71511a100f14322b5091a9804720fa0 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:23.008838 1192 raft_consensus.cc:738] T c71511a100f14322b5091a9804720fa0 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:23.009348 1192 consensus_queue.cc:260] T c71511a100f14322b5091a9804720fa0 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:23.010736 1192 ts_tablet_manager.cc:1428] T c71511a100f14322b5091a9804720fa0 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250113 02:51:23.011569 1192 tablet_bootstrap.cc:492] T d7c4c7b0cfc44ac1952bc73cbe1bced8 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:23.013253 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_2caed919541e43c78a858513a2128f50" } modify_external_catalogs: true
I20250113 02:51:23.013356 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_2565c0551f9943a084ace8f7e463e59d" } modify_external_catalogs: true
I20250113 02:51:23.013463 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_2caed919541e43c78a858513a2128f50" } modify_external_catalogs: true
I20250113 02:51:23.013561 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_2565c0551f9943a084ace8f7e463e59d" } modify_external_catalogs: true
I20250113 02:51:23.015144 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_c15dbcc7e55d4e3c80fb41bc4a950c81"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:23.015698 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c15dbcc7e55d4e3c80fb41bc4a950c81 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:23.016117 455 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_af3ebf543c9448b7ae473d721a23d47e" } modify_external_catalogs: true
I20250113 02:51:23.016315 455 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_af3ebf543c9448b7ae473d721a23d47e" } modify_external_catalogs: true
I20250113 02:51:23.024003 1195 raft_consensus.cc:2267] T c71511a100f14322b5091a9804720fa0 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:23.027456 1192 tablet_bootstrap.cc:654] T d7c4c7b0cfc44ac1952bc73cbe1bced8 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:23.031632 447 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_f5342954ee0548ec90d482ed62198240" } modify_external_catalogs: true
I20250113 02:51:23.032017 447 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_f5342954ee0548ec90d482ed62198240" } modify_external_catalogs: true
I20250113 02:51:23.032424 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 41fcde988f72496b824271fa36c1a226
I20250113 02:51:23.032560 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet e70658c90555463498d3fc2e2f13ea7f
I20250113 02:51:23.033123 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_547022f75adf480fb21bc92b5b1c0e2d" } new_table_name: "default.Table_7a4628807cc44ebdadf927ff709de3a2" modify_external_catalogs: true
I20250113 02:51:23.033241 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_a627be2353214265addee118a16ade57"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:23.033818 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a627be2353214265addee118a16ade57 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:23.034030 891 tablet_service.cc:1514] Processing DeleteTablet for tablet 41fcde988f72496b824271fa36c1a226 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:23 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:23.034575 891 tablet_service.cc:1514] Processing DeleteTablet for tablet e70658c90555463498d3fc2e2f13ea7f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:23 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:23.034751 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 6d63f0e5835540a78c82020b20a07dac
I20250113 02:51:23.035447 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_68c383ddd5194ab0ba962868f50a690f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:23.035833 740 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d63f0e5835540a78c82020b20a07dac with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:23 UTC) from {username='slave'} at 127.0.0.1:55398
W20250113 02:51:23.035921 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_68c383ddd5194ab0ba962868f50a690f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:23.036177 753 tablet_service.cc:1514] Processing DeleteTablet for tablet e70658c90555463498d3fc2e2f13ea7f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:23 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:23.036742 891 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d63f0e5835540a78c82020b20a07dac with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:23 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:23.035398 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_df6d561e8e21480ba865be6710c85dc6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:23.037184 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_df6d561e8e21480ba865be6710c85dc6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:23.037212 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet 41fcde988f72496b824271fa36c1a226 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:23 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:23.037465 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet e70658c90555463498d3fc2e2f13ea7f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:23 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:23.037679 1039 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d63f0e5835540a78c82020b20a07dac with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:23 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:23.037897 754 tablet_service.cc:1514] Processing DeleteTablet for tablet 41fcde988f72496b824271fa36c1a226 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:23 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:23.038628 453 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_ccefabb7a6024c17897743e268c1ce49" } modify_external_catalogs: true
I20250113 02:51:23.038817 1551 tablet.cc:1721] T 8547b63c40ec4ccebd962b5a3b2c5e0f P 1186fa869eb14812832fa6562531f196: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250113 02:51:23.038894 453 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_ccefabb7a6024c17897743e268c1ce49" } modify_external_catalogs: true
I20250113 02:51:23.041976 1196 ts_tablet_manager.cc:1918] T 4efa63b8be794e1db4b340f4c0c4cd5a P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:23.042124 1196 log.cc:1198] T 4efa63b8be794e1db4b340f4c0c4cd5a P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/4efa63b8be794e1db4b340f4c0c4cd5a
I20250113 02:51:23.043207 1196 ts_tablet_manager.cc:1939] T 4efa63b8be794e1db4b340f4c0c4cd5a P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:23.043642 1196 tablet_replica.cc:331] T 6c8ca30720664f3a819c4bff2493934d P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:23.043820 1196 raft_consensus.cc:2238] T 6c8ca30720664f3a819c4bff2493934d P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:23.043836 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet 4efa63b8be794e1db4b340f4c0c4cd5a (table default.Table_6a0195b054a043269e133f6ed8dd5979 [id=4b2bd999368c4b628c70cb3ee8c2b1f3]) successfully deleted
I20250113 02:51:23.044003 1196 raft_consensus.cc:2267] T 6c8ca30720664f3a819c4bff2493934d P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:23.048110 1194 tablet_bootstrap.cc:654] T fc92f19390fd48a4a45ba8fb839020fd P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:23.049681 1196 ts_tablet_manager.cc:1905] T 6c8ca30720664f3a819c4bff2493934d P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:23.063679 448 catalog_manager.cc:6071] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: default.Table_3d5fd97116b343b0a6c798e1f22327c2 [id=1396c4482ee646d18330d200fd3f9816] alter complete (version 1)
W20250113 02:51:23.065210 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fb18c16fb3ba44ec8a190b88d3bbd1b1 consensus state has no leader
I20250113 02:51:23.065394 453 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet 52fc731ccf964086a253684f86f0cc78
I20250113 02:51:23.066967 1039 tablet_service.cc:1514] Processing DeleteTablet for tablet c71511a100f14322b5091a9804720fa0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:20 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:23.068701 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet ddf79eec41214693859f526265a55492
I20250113 02:51:23.069711 1039 tablet_service.cc:1514] Processing DeleteTablet for tablet ddf79eec41214693859f526265a55492 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:23 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:23.070040 895 tablet_service.cc:1514] Processing DeleteTablet for tablet ddf79eec41214693859f526265a55492 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:23 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:23.073112 447 catalog_manager.cc:5618] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095) does not have the latest schema for tablet 8547b63c40ec4ccebd962b5a3b2c5e0f (table default.Table_3d5fd97116b343b0a6c798e1f22327c2 [id=1396c4482ee646d18330d200fd3f9816]). Expected version 1 got 0
I20250113 02:51:23.074271 754 tablet_service.cc:1514] Processing DeleteTablet for tablet ddf79eec41214693859f526265a55492 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:23 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:23.078225 1195 ts_tablet_manager.cc:1905] T c71511a100f14322b5091a9804720fa0 P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:23.095746 446 master_service.cc:946] ReplaceTablet: received request to replace tablet f68f9f4076d9475d938e795e72bdd468 from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:23.101926 1191 tablet_bootstrap.cc:654] T d5bd09a1ede644c59e400e49c8125793 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:23.113756 447 master_service.cc:946] ReplaceTablet: received request to replace tablet d7c4c7b0cfc44ac1952bc73cbe1bced8 from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:23.116286 1550 tablet.cc:1721] T ad9b95c5450a4c9c89a0930862348d49 P 4216a7f853d642b7bc3e5cff458f3aca: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250113 02:51:23.116806 447 catalog_manager.cc:5618] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095) does not have the latest schema for tablet ad9b95c5450a4c9c89a0930862348d49 (table default.Table_936e73f0b76046d285567ac0778e36a9 [id=219c739889d04a97aafa4fe1b55b0a24]). Expected version 2 got 0
I20250113 02:51:23.128692 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet f68f9f4076d9475d938e795e72bdd468
I20250113 02:51:23.129398 1039 tablet_service.cc:1514] Processing DeleteTablet for tablet f68f9f4076d9475d938e795e72bdd468 with delete_type TABLET_DATA_DELETED (replaced by tablet f2c6c8c17d6e499d84ad7c0c23889cc4) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:23.130008 446 catalog_manager.cc:6596] ReplaceTablet: tablet f68f9f4076d9475d938e795e72bdd468 deleted and replaced by tablet f2c6c8c17d6e499d84ad7c0c23889cc4
I20250113 02:51:23.130373 895 tablet_service.cc:1514] Processing DeleteTablet for tablet f68f9f4076d9475d938e795e72bdd468 with delete_type TABLET_DATA_DELETED (replaced by tablet f2c6c8c17d6e499d84ad7c0c23889cc4) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:23.131871 447 catalog_manager.cc:6071] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: default.Table_936e73f0b76046d285567ac0778e36a9 [id=219c739889d04a97aafa4fe1b55b0a24] alter complete (version 2)
I20250113 02:51:23.133194 1198 ts_tablet_manager.cc:1918] T 53cc528d282c47788d5d558569f074c1 P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:23.133289 1535 raft_consensus.cc:491] T 076ed7744da54ddebd8fc99686beb6ae P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:23.133463 1198 log.cc:1198] T 53cc528d282c47788d5d558569f074c1 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/53cc528d282c47788d5d558569f074c1
I20250113 02:51:23.134094 1198 ts_tablet_manager.cc:1939] T 53cc528d282c47788d5d558569f074c1 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:23.133906 1535 raft_consensus.cc:513] T 076ed7744da54ddebd8fc99686beb6ae P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:23.135248 1198 tablet_replica.cc:331] T 1d81eec487dd4016b3b804451da31c49 P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:23.135318 1535 leader_election.cc:290] T 076ed7744da54ddebd8fc99686beb6ae P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:23.135432 1198 raft_consensus.cc:2238] T 1d81eec487dd4016b3b804451da31c49 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:23.135609 1198 raft_consensus.cc:2267] T 1d81eec487dd4016b3b804451da31c49 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:23.136334 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet 53cc528d282c47788d5d558569f074c1 (table default.Table_06f81a97cb014a3bad594e88c8cf4733 [id=7c2bc6d4232f4585b84d71e3e1f85794]) successfully deleted
I20250113 02:51:23.136181 1198 ts_tablet_manager.cc:1905] T 1d81eec487dd4016b3b804451da31c49 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250113 02:51:23.137192 709 leader_election.cc:343] T 076ed7744da54ddebd8fc99686beb6ae P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Not found: Tablet not found: 076ed7744da54ddebd8fc99686beb6ae
W20250113 02:51:23.137336 711 leader_election.cc:343] T 076ed7744da54ddebd8fc99686beb6ae P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Not found: Tablet not found: 076ed7744da54ddebd8fc99686beb6ae
I20250113 02:51:23.137463 739 tablet_service.cc:1514] Processing DeleteTablet for tablet f68f9f4076d9475d938e795e72bdd468 with delete_type TABLET_DATA_DELETED (replaced by tablet f2c6c8c17d6e499d84ad7c0c23889cc4) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:23.137807 711 leader_election.cc:304] T 076ed7744da54ddebd8fc99686beb6ae P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:23.138337 1535 raft_consensus.cc:2743] T 076ed7744da54ddebd8fc99686beb6ae P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250113 02:51:23.139582 441 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet f8a6eebb6b6d4b63903e68750e181707 (table default.Table_4484fe40b4454c5ea4943a4d38f49c5e [id=a0e91da6e0ee449b92b7afbc6f68087d], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet f8a6eebb6b6d4b63903e68750e181707 consensus state has no leader
I20250113 02:51:23.147935 1551 tablet.cc:1721] T ad9b95c5450a4c9c89a0930862348d49 P 1186fa869eb14812832fa6562531f196: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250113 02:51:23.150817 447 master_service.cc:946] ReplaceTablet: received request to replace tablet c215ffe61b9b426286cc5e808463078d from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:23.152130 1192 tablet_bootstrap.cc:492] T d7c4c7b0cfc44ac1952bc73cbe1bced8 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:23.152343 1192 ts_tablet_manager.cc:1397] T d7c4c7b0cfc44ac1952bc73cbe1bced8 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.141s user 0.016s sys 0.025s
I20250113 02:51:23.153151 1192 raft_consensus.cc:357] T d7c4c7b0cfc44ac1952bc73cbe1bced8 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:23.153383 1192 raft_consensus.cc:383] T d7c4c7b0cfc44ac1952bc73cbe1bced8 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:23.153460 1192 raft_consensus.cc:738] T d7c4c7b0cfc44ac1952bc73cbe1bced8 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:23.153730 1192 consensus_queue.cc:260] T d7c4c7b0cfc44ac1952bc73cbe1bced8 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:23.154405 1192 ts_tablet_manager.cc:1428] T d7c4c7b0cfc44ac1952bc73cbe1bced8 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250113 02:51:23.156831 1192 tablet_bootstrap.cc:492] T 1c26074fdd804bd79cb0c295622d11ff P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:23.157778 1194 tablet_bootstrap.cc:492] T fc92f19390fd48a4a45ba8fb839020fd P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:23.158964 1548 mvcc.cc:204] Tried to move back new op lower bound from 7113673453087428608 to 7113673452235517952. Current Snapshot: MvccSnapshot[applied={T|T < 7113673453087428608}]
I20250113 02:51:23.159444 1194 ts_tablet_manager.cc:1397] T fc92f19390fd48a4a45ba8fb839020fd P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.188s user 0.004s sys 0.004s
I20250113 02:51:23.160317 1194 raft_consensus.cc:357] T fc92f19390fd48a4a45ba8fb839020fd P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:23.160485 1039 tablet_service.cc:1514] Processing DeleteTablet for tablet d7c4c7b0cfc44ac1952bc73cbe1bced8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:18 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:23.161667 1194 raft_consensus.cc:383] T fc92f19390fd48a4a45ba8fb839020fd P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:23.161753 1194 raft_consensus.cc:738] T fc92f19390fd48a4a45ba8fb839020fd P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:23.162338 1194 consensus_queue.cc:260] T fc92f19390fd48a4a45ba8fb839020fd P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:23.163321 1194 ts_tablet_manager.cc:1428] T fc92f19390fd48a4a45ba8fb839020fd P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20250113 02:51:23.163367 1558 tablet.cc:1721] T 8547b63c40ec4ccebd962b5a3b2c5e0f P df69f48656b44bb890f76587ee69cc0f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250113 02:51:23.147671 1195 ts_tablet_manager.cc:1918] T c71511a100f14322b5091a9804720fa0 P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:23.163940 1195 log.cc:1198] T c71511a100f14322b5091a9804720fa0 P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/c71511a100f14322b5091a9804720fa0
I20250113 02:51:23.164544 1195 ts_tablet_manager.cc:1939] T c71511a100f14322b5091a9804720fa0 P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:23.169098 447 catalog_manager.cc:5618] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095) does not have the latest schema for tablet ad9b95c5450a4c9c89a0930862348d49 (table default.Table_936e73f0b76046d285567ac0778e36a9 [id=219c739889d04a97aafa4fe1b55b0a24]). Expected version 2 got 0
I20250113 02:51:23.173874 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 332) took 1065 ms. Trace:
I20250113 02:51:23.178439 1558 tablet.cc:1721] T ad9b95c5450a4c9c89a0930862348d49 P df69f48656b44bb890f76587ee69cc0f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250113 02:51:23.180848 1192 tablet_bootstrap.cc:654] T 1c26074fdd804bd79cb0c295622d11ff P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:23.184156 1195 rpcz_store.cc:276] 0113 02:51:22.108882 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:22.109033 (+ 151us) service_pool.cc:225] Handling call
0113 02:51:22.986459 (+877426us) ts_tablet_manager.cc:1145] Deleting tablet c71511a100f14322b5091a9804720fa0
0113 02:51:22.986470 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:23.138041 (+151571us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:23.147488 (+ 9447us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:23.173836 (+ 26348us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":877189}
I20250113 02:51:23.185766 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet c71511a100f14322b5091a9804720fa0 (table default.Table_8606b4caf08b43b18b25ee8849eebd07 [id=85bca730e1274dbebb2f7c2e6eaea7c7]) successfully deleted
I20250113 02:51:23.186251 1195 tablet_replica.cc:331] T d101e98b72544c30a3a4bc60d1576aa3 P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:23.186687 1195 raft_consensus.cc:2238] T d101e98b72544c30a3a4bc60d1576aa3 P 1186fa869eb14812832fa6562531f196 [term 1 LEADER]: Raft consensus shutting down.
I20250113 02:51:23.191150 1195 pending_rounds.cc:62] T d101e98b72544c30a3a4bc60d1576aa3 P 1186fa869eb14812832fa6562531f196: Trying to abort 1 pending ops.
I20250113 02:51:23.191992 1195 pending_rounds.cc:69] T d101e98b72544c30a3a4bc60d1576aa3 P 1186fa869eb14812832fa6562531f196: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113673451949232128 op_type: NO_OP noop_request { }
I20250113 02:51:23.192170 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_db2a6ff23fc94f67b8883df9728be422" } new_table_name: "default.Table_0e85f1c98fbc41c29ac21fbc44f01c93" modify_external_catalogs: true
I20250113 02:51:23.192615 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_15d236fec5f24590b15d5d21f67e3941"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:23.192744 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_1d46c338f1ed42c9a61f4aa130290e22" } new_table_name: "default.Table_5e5c2e147f764a3389263a89c310f968" modify_external_catalogs: true
W20250113 02:51:23.193131 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_15d236fec5f24590b15d5d21f67e3941 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:23.193568 454 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_9659cbc5ffdb400b8415dc81aa211f89" } new_table_name: "default.Table_c7fb9512ad3a4a698261ea6b58a542db" modify_external_catalogs: true
I20250113 02:51:23.194617 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_c6b6ad0e811844518a1f5802b983d3d5" } new_table_name: "default.Table_53bb556171424601915f305c106a2dc4" modify_external_catalogs: true
W20250113 02:51:23.198233 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b5c1882125b5491ea79a29a6dac6771e consensus state has no leader
I20250113 02:51:23.198771 1195 raft_consensus.cc:2883] T d101e98b72544c30a3a4bc60d1576aa3 P 1186fa869eb14812832fa6562531f196 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250113 02:51:23.199090 1195 raft_consensus.cc:2267] T d101e98b72544c30a3a4bc60d1576aa3 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250113 02:51:23.203708 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6d4d8b5cc8ad4c9e935d8afaf5766e66 consensus state has no leader
I20250113 02:51:23.204250 1195 ts_tablet_manager.cc:1905] T d101e98b72544c30a3a4bc60d1576aa3 P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:23.220567 1191 tablet_bootstrap.cc:492] T d5bd09a1ede644c59e400e49c8125793 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:23.220890 1191 ts_tablet_manager.cc:1397] T d5bd09a1ede644c59e400e49c8125793 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.265s user 0.007s sys 0.004s
I20250113 02:51:23.221952 1191 raft_consensus.cc:357] T d5bd09a1ede644c59e400e49c8125793 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:23.222277 1191 raft_consensus.cc:383] T d5bd09a1ede644c59e400e49c8125793 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:23.222411 1191 raft_consensus.cc:738] T d5bd09a1ede644c59e400e49c8125793 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:23.222741 1191 consensus_queue.cc:260] T d5bd09a1ede644c59e400e49c8125793 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:23.223357 1191 ts_tablet_manager.cc:1428] T d5bd09a1ede644c59e400e49c8125793 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250113 02:51:23.224035 1191 tablet_bootstrap.cc:492] T 53cc528d282c47788d5d558569f074c1 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:23.227074 1191 tablet_bootstrap.cc:654] T 53cc528d282c47788d5d558569f074c1 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:23.228955 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_17489ca6f295469eb97e9a3a5c7cb1aa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:23.229585 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_17489ca6f295469eb97e9a3a5c7cb1aa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:23.232769 1192 tablet_bootstrap.cc:492] T 1c26074fdd804bd79cb0c295622d11ff P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:23.232955 1192 ts_tablet_manager.cc:1397] T 1c26074fdd804bd79cb0c295622d11ff P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.076s user 0.016s sys 0.011s
I20250113 02:51:23.233804 1192 raft_consensus.cc:357] T 1c26074fdd804bd79cb0c295622d11ff P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:23.234061 1192 raft_consensus.cc:383] T 1c26074fdd804bd79cb0c295622d11ff P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:23.234169 1192 raft_consensus.cc:738] T 1c26074fdd804bd79cb0c295622d11ff P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:23.234397 1192 consensus_queue.cc:260] T 1c26074fdd804bd79cb0c295622d11ff P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:23.236985 453 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_157d1350cce74502a990462ac9bd0e5f" } modify_external_catalogs: true
I20250113 02:51:23.237228 453 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_157d1350cce74502a990462ac9bd0e5f" } modify_external_catalogs: true
I20250113 02:51:23.239204 455 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_7805d72c367b4ef8aa585de05f3d9b90" } modify_external_catalogs: true
I20250113 02:51:23.239364 455 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_7805d72c367b4ef8aa585de05f3d9b90" } modify_external_catalogs: true
I20250113 02:51:23.239320 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_2b71ec43b2ee485e92d81d55f56a44fc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:23.239472 1192 ts_tablet_manager.cc:1428] T 1c26074fdd804bd79cb0c295622d11ff P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
W20250113 02:51:23.239871 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2b71ec43b2ee485e92d81d55f56a44fc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:23.241521 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_7b2714b0471c4a3eb2a9f0590a27562d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:23.241777 1192 tablet_bootstrap.cc:492] T be80b631ccd747f1b9355a00787a7b55 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
W20250113 02:51:23.242009 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7b2714b0471c4a3eb2a9f0590a27562d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:23.241997 1039 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c26074fdd804bd79cb0c295622d11ff with delete_type TABLET_DATA_DELETED (replaced by tablet 9fff52e99c64462a8a73fccc3c22ca63) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:23.244112 1192 tablet_bootstrap.cc:654] T be80b631ccd747f1b9355a00787a7b55 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
W20250113 02:51:23.244660 454 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 572e4eed1e924381bada27f8389821a7 consensus state has no leader
W20250113 02:51:23.245076 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4b07a985897641c3ac48021c2a4daedb consensus state has no leader
I20250113 02:51:23.248874 446 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_6b241846ad39435097591810fa57f13d" } modify_external_catalogs: true
I20250113 02:51:23.249188 446 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_6b241846ad39435097591810fa57f13d" } modify_external_catalogs: true
I20250113 02:51:23.251368 1194 tablet_bootstrap.cc:492] T 93a2729c5a604a27a9bbc451ccb4e128 P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:23.252657 1192 tablet_bootstrap.cc:492] T be80b631ccd747f1b9355a00787a7b55 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:23.252833 1192 ts_tablet_manager.cc:1397] T be80b631ccd747f1b9355a00787a7b55 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.004s
I20250113 02:51:23.253510 1194 tablet_bootstrap.cc:654] T 93a2729c5a604a27a9bbc451ccb4e128 P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:23.253619 1192 raft_consensus.cc:357] T be80b631ccd747f1b9355a00787a7b55 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:23.253839 1192 raft_consensus.cc:383] T be80b631ccd747f1b9355a00787a7b55 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:23.253923 1192 raft_consensus.cc:738] T be80b631ccd747f1b9355a00787a7b55 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:23.254169 1192 consensus_queue.cc:260] T be80b631ccd747f1b9355a00787a7b55 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:23.254784 1192 ts_tablet_manager.cc:1428] T be80b631ccd747f1b9355a00787a7b55 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250113 02:51:23.255332 1192 tablet_bootstrap.cc:492] T 317c6e8013b745c290bf3190dc237ab8 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:23.255934 1442 raft_consensus.cc:491] T 107e6e9e2d84496f8fce92b9ba377189 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:23.256078 1442 raft_consensus.cc:513] T 107e6e9e2d84496f8fce92b9ba377189 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:23.256692 1442 leader_election.cc:290] T 107e6e9e2d84496f8fce92b9ba377189 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:23.257360 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "107e6e9e2d84496f8fce92b9ba377189" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
W20250113 02:51:23.257439 998 leader_election.cc:343] T 107e6e9e2d84496f8fce92b9ba377189 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Not found: Tablet not found: 107e6e9e2d84496f8fce92b9ba377189
W20250113 02:51:23.258411 995 leader_election.cc:343] T 107e6e9e2d84496f8fce92b9ba377189 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:23.258533 995 leader_election.cc:304] T 107e6e9e2d84496f8fce92b9ba377189 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4216a7f853d642b7bc3e5cff458f3aca; no voters: 1186fa869eb14812832fa6562531f196, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:23.262055 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_25480b2b4ab44c05b96c1c77cec60f61"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:23.262655 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_25480b2b4ab44c05b96c1c77cec60f61 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:23.262957 1192 tablet_bootstrap.cc:654] T 317c6e8013b745c290bf3190dc237ab8 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:23.264233 1442 raft_consensus.cc:2743] T 107e6e9e2d84496f8fce92b9ba377189 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:23.265559 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_9e44dff41935489aa1306e2e5d02bf85"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:23.266088 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9e44dff41935489aa1306e2e5d02bf85 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:23.266781 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_1df1ad89ec084bd4928ccfa8376b0bc6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:23.267136 1191 tablet_bootstrap.cc:492] T 53cc528d282c47788d5d558569f074c1 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
W20250113 02:51:23.267238 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1df1ad89ec084bd4928ccfa8376b0bc6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:23.267287 1191 ts_tablet_manager.cc:1397] T 53cc528d282c47788d5d558569f074c1 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.043s user 0.003s sys 0.001s
I20250113 02:51:23.268043 1191 raft_consensus.cc:357] T 53cc528d282c47788d5d558569f074c1 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:23.268234 1191 raft_consensus.cc:383] T 53cc528d282c47788d5d558569f074c1 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:23.268316 1191 raft_consensus.cc:738] T 53cc528d282c47788d5d558569f074c1 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:23.269533 1442 raft_consensus.cc:491] T c71511a100f14322b5091a9804720fa0 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:23.269732 1442 raft_consensus.cc:513] T c71511a100f14322b5091a9804720fa0 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:23.270220 1191 consensus_queue.cc:260] T 53cc528d282c47788d5d558569f074c1 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:23.270486 1442 leader_election.cc:290] T c71511a100f14322b5091a9804720fa0 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
I20250113 02:51:23.271312 1191 ts_tablet_manager.cc:1428] T 53cc528d282c47788d5d558569f074c1 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
W20250113 02:51:23.271327 998 leader_election.cc:343] T c71511a100f14322b5091a9804720fa0 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Not found: Tablet not found: c71511a100f14322b5091a9804720fa0
I20250113 02:51:23.275378 773 raft_consensus.cc:3054] T d5bd09a1ede644c59e400e49c8125793 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Advancing to term 1
W20250113 02:51:23.275554 995 leader_election.cc:343] T c71511a100f14322b5091a9804720fa0 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Not found: Tablet not found: c71511a100f14322b5091a9804720fa0
I20250113 02:51:23.275686 995 leader_election.cc:304] T c71511a100f14322b5091a9804720fa0 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4216a7f853d642b7bc3e5cff458f3aca; no voters: 1186fa869eb14812832fa6562531f196, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:23.276114 1564 raft_consensus.cc:2743] T c71511a100f14322b5091a9804720fa0 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:23.276360 1191 tablet_bootstrap.cc:492] T 107e6e9e2d84496f8fce92b9ba377189 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:23.278889 1195 ts_tablet_manager.cc:1918] T d101e98b72544c30a3a4bc60d1576aa3 P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250113 02:51:23.279268 1195 log.cc:1198] T d101e98b72544c30a3a4bc60d1576aa3 P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/d101e98b72544c30a3a4bc60d1576aa3
I20250113 02:51:23.279915 1195 ts_tablet_manager.cc:1939] T d101e98b72544c30a3a4bc60d1576aa3 P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:23.282483 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 333) took 1140 ms. Trace:
I20250113 02:51:23.283077 1564 consensus_queue.cc:1035] T d5bd09a1ede644c59e400e49c8125793 P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 02:51:23.291108 1564 consensus_queue.cc:1035] T d5bd09a1ede644c59e400e49c8125793 P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 02:51:23.287961 1525 raft_consensus.cc:491] T d5bd09a1ede644c59e400e49c8125793 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:23.296423 1525 raft_consensus.cc:513] T d5bd09a1ede644c59e400e49c8125793 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:23.297173 1525 leader_election.cc:290] T d5bd09a1ede644c59e400e49c8125793 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:23.297791 1062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bd09a1ede644c59e400e49c8125793" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:23.281169 1191 tablet_bootstrap.cc:654] T 107e6e9e2d84496f8fce92b9ba377189 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:23.299522 1195 rpcz_store.cc:276] 0113 02:51:22.141720 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:22.141775 (+ 55us) service_pool.cc:225] Handling call
0113 02:51:23.186185 (+1044410us) ts_tablet_manager.cc:1145] Deleting tablet d101e98b72544c30a3a4bc60d1576aa3
0113 02:51:23.186197 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:23.233956 (+ 47759us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:23.257577 (+ 23621us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:23.282450 (+ 24873us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1044216}
I20250113 02:51:23.300246 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 355) took 1023 ms. Trace:
I20250113 02:51:23.300356 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet d101e98b72544c30a3a4bc60d1576aa3 (table default.Table_ce0f77e091564c29bf54791dda458d78 [id=86a46eed92bc4bddad870e392232cb51]) successfully deleted
I20250113 02:51:23.300782 1195 rpcz_store.cc:276] 0113 02:51:22.276418 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:22.343751 (+ 67333us) service_pool.cc:225] Handling call
0113 02:51:23.300101 (+956350us) ts_tablet_manager.cc:1145] Deleting tablet f56230a2c75d4763a54671c79358f4d9
0113 02:51:23.300112 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:23.300215 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":943574}
I20250113 02:51:23.304543 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 356) took 1024 ms. Trace:
W20250113 02:51:23.304700 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet f56230a2c75d4763a54671c79358f4d9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f56230a2c75d4763a54671c79358f4d9 already in progress: creating tablet
I20250113 02:51:23.305327 1191 tablet_bootstrap.cc:492] T 107e6e9e2d84496f8fce92b9ba377189 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:23.320056 1195 rpcz_store.cc:276] 0113 02:51:22.280508 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:22.356577 (+ 76069us) service_pool.cc:225] Handling call
0113 02:51:23.304396 (+947819us) ts_tablet_manager.cc:1145] Deleting tablet f129bb3fbbfd492d98822be0a6626e44
0113 02:51:23.304406 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:23.304501 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":947082}
I20250113 02:51:23.320711 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 357) took 1040 ms. Trace:
W20250113 02:51:23.320865 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet f129bb3fbbfd492d98822be0a6626e44 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f129bb3fbbfd492d98822be0a6626e44 already in progress: creating tablet
I20250113 02:51:23.320917 1195 rpcz_store.cc:276] 0113 02:51:22.280647 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:22.357339 (+ 76692us) service_pool.cc:225] Handling call
0113 02:51:23.320522 (+963183us) ts_tablet_manager.cc:1145] Deleting tablet bbc7832c50114d6f890fec45a3d994e9
0113 02:51:23.320533 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:23.320674 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":962048}
I20250113 02:51:23.321552 1195 tablet_replica.cc:331] T 076ed7744da54ddebd8fc99686beb6ae P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:23.321789 1195 raft_consensus.cc:2238] T 076ed7744da54ddebd8fc99686beb6ae P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:23.321972 1195 raft_consensus.cc:2267] T 076ed7744da54ddebd8fc99686beb6ae P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250113 02:51:23.321995 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet bbc7832c50114d6f890fec45a3d994e9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bbc7832c50114d6f890fec45a3d994e9 already in progress: creating tablet
W20250113 02:51:23.322256 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 0a7e7713bc8648378b668d8b769ac927 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0a7e7713bc8648378b668d8b769ac927 already in progress: creating tablet
W20250113 02:51:23.322474 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 05bf5ab0df874dd5b6559f6d82069184 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 05bf5ab0df874dd5b6559f6d82069184 already in progress: creating tablet
I20250113 02:51:23.323043 1195 ts_tablet_manager.cc:1905] T 076ed7744da54ddebd8fc99686beb6ae P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:23.320077 1191 ts_tablet_manager.cc:1397] T 107e6e9e2d84496f8fce92b9ba377189 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.033s user 0.002s sys 0.002s
I20250113 02:51:23.326714 1191 raft_consensus.cc:357] T 107e6e9e2d84496f8fce92b9ba377189 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:23.327031 1191 raft_consensus.cc:383] T 107e6e9e2d84496f8fce92b9ba377189 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:23.327158 1191 raft_consensus.cc:738] T 107e6e9e2d84496f8fce92b9ba377189 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:23.327473 1191 consensus_queue.cc:260] T 107e6e9e2d84496f8fce92b9ba377189 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:23.328181 1191 ts_tablet_manager.cc:1428] T 107e6e9e2d84496f8fce92b9ba377189 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250113 02:51:23.328550 1191 tablet_bootstrap.cc:492] T 21c4b419bf2148e0b140301fa80622bb P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:23.330359 1191 tablet_bootstrap.cc:654] T 21c4b419bf2148e0b140301fa80622bb P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:23.337232 1564 raft_consensus.cc:491] T d7c4c7b0cfc44ac1952bc73cbe1bced8 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:23.337476 1195 ts_tablet_manager.cc:1918] T 076ed7744da54ddebd8fc99686beb6ae P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:23.337606 1195 log.cc:1198] T 076ed7744da54ddebd8fc99686beb6ae P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/076ed7744da54ddebd8fc99686beb6ae
I20250113 02:51:23.337487 1564 raft_consensus.cc:513] T d7c4c7b0cfc44ac1952bc73cbe1bced8 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:23.337950 1195 ts_tablet_manager.cc:1939] T 076ed7744da54ddebd8fc99686beb6ae P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:23.338413 1564 leader_election.cc:290] T d7c4c7b0cfc44ac1952bc73cbe1bced8 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
I20250113 02:51:23.338871 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet 076ed7744da54ddebd8fc99686beb6ae (table default.Table_8b98cd7fbceb4d3f8b138e4da9bb0e05 [id=4100a37b30624d0bab2ec027b783be00]) successfully deleted
I20250113 02:51:23.340467 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7c4c7b0cfc44ac1952bc73cbe1bced8" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
W20250113 02:51:23.341951 995 leader_election.cc:343] T d7c4c7b0cfc44ac1952bc73cbe1bced8 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:23.349835 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_f4fbf20403334a2c99de02125a36de82"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:23.350065 739 tablet_service.cc:1514] Processing DeleteTablet for tablet 53cc528d282c47788d5d558569f074c1 with delete_type TABLET_DATA_DELETED (replaced by tablet 7d3562a48c6e4badbbcf37640e91b5ba) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:23.350160 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 359) took 1006 ms. Trace:
I20250113 02:51:23.350245 1195 rpcz_store.cc:276] 0113 02:51:22.342630 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:22.359486 (+ 16856us) service_pool.cc:225] Handling call
0113 02:51:23.349226 (+989740us) ts_tablet_manager.cc:1145] Deleting tablet 1a26e2b5e1054aa6afe7f44bf8d43108
0113 02:51:23.349236 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:23.349347 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":747822}
W20250113 02:51:23.350790 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f4fbf20403334a2c99de02125a36de82 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:23.351444 1195 tablet_replica.cc:331] T 1d81eec487dd4016b3b804451da31c49 P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:23.351626 1039 tablet_service.cc:1514] Processing DeleteTablet for tablet be80b631ccd747f1b9355a00787a7b55 with delete_type TABLET_DATA_DELETED (replaced by tablet e70658c90555463498d3fc2e2f13ea7f) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:23.351862 1195 raft_consensus.cc:2238] T 1d81eec487dd4016b3b804451da31c49 P 1186fa869eb14812832fa6562531f196 [term 1 LEADER]: Raft consensus shutting down.
I20250113 02:51:23.353516 1195 pending_rounds.cc:62] T 1d81eec487dd4016b3b804451da31c49 P 1186fa869eb14812832fa6562531f196: Trying to abort 1 pending ops.
I20250113 02:51:23.353911 1195 pending_rounds.cc:69] T 1d81eec487dd4016b3b804451da31c49 P 1186fa869eb14812832fa6562531f196: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113673452237500416 op_type: NO_OP noop_request { }
I20250113 02:51:23.354220 1195 raft_consensus.cc:2883] T 1d81eec487dd4016b3b804451da31c49 P 1186fa869eb14812832fa6562531f196 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250113 02:51:23.354467 1195 raft_consensus.cc:2267] T 1d81eec487dd4016b3b804451da31c49 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:23.355464 1195 ts_tablet_manager.cc:1905] T 1d81eec487dd4016b3b804451da31c49 P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250113 02:51:23.351061 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 1a26e2b5e1054aa6afe7f44bf8d43108 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1a26e2b5e1054aa6afe7f44bf8d43108 already in progress: creating tablet
I20250113 02:51:23.298393 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bd09a1ede644c59e400e49c8125793" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:23.339444 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7c4c7b0cfc44ac1952bc73cbe1bced8" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:23.357836 917 raft_consensus.cc:2463] T d7c4c7b0cfc44ac1952bc73cbe1bced8 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 0.
I20250113 02:51:23.358363 998 leader_election.cc:304] T d7c4c7b0cfc44ac1952bc73cbe1bced8 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f; no voters: 1186fa869eb14812832fa6562531f196
I20250113 02:51:23.358755 1564 raft_consensus.cc:2798] T d7c4c7b0cfc44ac1952bc73cbe1bced8 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 02:51:23.358889 1564 raft_consensus.cc:491] T d7c4c7b0cfc44ac1952bc73cbe1bced8 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 02:51:23.359019 1564 raft_consensus.cc:3054] T d7c4c7b0cfc44ac1952bc73cbe1bced8 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:23.359117 711 leader_election.cc:304] T d5bd09a1ede644c59e400e49c8125793 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:23.362629 1564 raft_consensus.cc:513] T d7c4c7b0cfc44ac1952bc73cbe1bced8 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:23.363497 1564 leader_election.cc:290] T d7c4c7b0cfc44ac1952bc73cbe1bced8 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Requested vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
I20250113 02:51:23.363811 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7c4c7b0cfc44ac1952bc73cbe1bced8" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f"
I20250113 02:51:23.364105 917 raft_consensus.cc:3054] T d7c4c7b0cfc44ac1952bc73cbe1bced8 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:23.366900 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7c4c7b0cfc44ac1952bc73cbe1bced8" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196"
I20250113 02:51:23.367792 917 raft_consensus.cc:2463] T d7c4c7b0cfc44ac1952bc73cbe1bced8 P df69f48656b44bb890f76587ee69cc0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 1.
I20250113 02:51:23.368252 998 leader_election.cc:304] T d7c4c7b0cfc44ac1952bc73cbe1bced8 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f; no voters:
I20250113 02:51:23.368554 1564 raft_consensus.cc:2798] T d7c4c7b0cfc44ac1952bc73cbe1bced8 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Leader election won for term 1
W20250113 02:51:23.369120 995 leader_election.cc:343] T d7c4c7b0cfc44ac1952bc73cbe1bced8 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:23.369972 1195 ts_tablet_manager.cc:1918] T 1d81eec487dd4016b3b804451da31c49 P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250113 02:51:23.369720 1564 raft_consensus.cc:695] T d7c4c7b0cfc44ac1952bc73cbe1bced8 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: Becoming Leader. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Running, Role: LEADER
I20250113 02:51:23.370105 1195 log.cc:1198] T 1d81eec487dd4016b3b804451da31c49 P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/1d81eec487dd4016b3b804451da31c49
I20250113 02:51:23.370473 1195 ts_tablet_manager.cc:1939] T 1d81eec487dd4016b3b804451da31c49 P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:23.370314 1564 consensus_queue.cc:237] T d7c4c7b0cfc44ac1952bc73cbe1bced8 P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:23.371253 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 361) took 1013 ms. Trace:
I20250113 02:51:23.371361 1195 rpcz_store.cc:276] 0113 02:51:22.357547 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:22.397835 (+ 40288us) service_pool.cc:225] Handling call
0113 02:51:23.351369 (+953534us) ts_tablet_manager.cc:1145] Deleting tablet 1d81eec487dd4016b3b804451da31c49
0113 02:51:23.351378 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:23.365910 (+ 14532us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:23.369886 (+ 3976us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:23.371193 (+ 1307us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":736494}
I20250113 02:51:23.371531 1368 raft_consensus.cc:491] T d35ec7e4a5084a74bea172ef8d9990b5 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:23.375810 1368 raft_consensus.cc:513] T d35ec7e4a5084a74bea172ef8d9990b5 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:23.376435 1368 leader_election.cc:290] T d35ec7e4a5084a74bea172ef8d9990b5 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
W20250113 02:51:23.377367 711 leader_election.cc:343] T d35ec7e4a5084a74bea172ef8d9990b5 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Not found: Tablet not found: d35ec7e4a5084a74bea172ef8d9990b5
W20250113 02:51:23.377486 709 leader_election.cc:343] T d35ec7e4a5084a74bea172ef8d9990b5 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Not found: Tablet not found: d35ec7e4a5084a74bea172ef8d9990b5
I20250113 02:51:23.377566 711 leader_election.cc:304] T d35ec7e4a5084a74bea172ef8d9990b5 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:23.377990 1368 raft_consensus.cc:2743] T d35ec7e4a5084a74bea172ef8d9990b5 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:23.379576 1195 tablet_replica.cc:331] T d35ec7e4a5084a74bea172ef8d9990b5 P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:23.380033 1195 raft_consensus.cc:2238] T d35ec7e4a5084a74bea172ef8d9990b5 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:23.380527 1195 raft_consensus.cc:2267] T d35ec7e4a5084a74bea172ef8d9990b5 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:23.381248 1195 ts_tablet_manager.cc:1905] T d35ec7e4a5084a74bea172ef8d9990b5 P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:23.390798 1195 ts_tablet_manager.cc:1918] T d35ec7e4a5084a74bea172ef8d9990b5 P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:23.393079 1196 ts_tablet_manager.cc:1918] T 6c8ca30720664f3a819c4bff2493934d P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:23.393354 1196 log.cc:1198] T 6c8ca30720664f3a819c4bff2493934d P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/6c8ca30720664f3a819c4bff2493934d
I20250113 02:51:23.393869 1196 ts_tablet_manager.cc:1939] T 6c8ca30720664f3a819c4bff2493934d P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:23.395222 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet 6c8ca30720664f3a819c4bff2493934d (table default.Table_8aa63512c7bb4bdb86f37c9ee24df209 [id=aadd55280b654ceb96c290786a0bfce0]) successfully deleted
I20250113 02:51:23.395586 1195 log.cc:1198] T d35ec7e4a5084a74bea172ef8d9990b5 P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/d35ec7e4a5084a74bea172ef8d9990b5
I20250113 02:51:23.396217 1195 ts_tablet_manager.cc:1939] T d35ec7e4a5084a74bea172ef8d9990b5 P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:23.400246 1195 tablet_replica.cc:331] T 18e2d79d731142f89b9089869bc736e5 P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:23.400676 1195 raft_consensus.cc:2238] T 18e2d79d731142f89b9089869bc736e5 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:23.401177 1195 raft_consensus.cc:2267] T 18e2d79d731142f89b9089869bc736e5 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:23.401935 1195 ts_tablet_manager.cc:1905] T 18e2d79d731142f89b9089869bc736e5 P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250113 02:51:23.404265 421 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.104.2:38467, user_credentials={real_user=slave}} blocked reactor thread for 259660us
I20250113 02:51:23.408432 1525 raft_consensus.cc:2743] T d5bd09a1ede644c59e400e49c8125793 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250113 02:51:23.415066 439 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.60:42067, user_credentials={real_user=slave}} blocked reactor thread for 48382.3us
I20250113 02:51:23.416206 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet 1d81eec487dd4016b3b804451da31c49 (table default.Table_d7d1f5a34e5f4c05b474b0b61f912cd9 [id=2c7925ec0d8d4cd7b8761268d3962f15]) successfully deleted
W20250113 02:51:23.416481 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 0644f5600c5b47e29f3fa73060ed96e3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0644f5600c5b47e29f3fa73060ed96e3 already in progress: creating tablet
W20250113 02:51:23.416707 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 42cf1e708644427f9d4dcd41ae2bcb37 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 42cf1e708644427f9d4dcd41ae2bcb37 already in progress: creating tablet
W20250113 02:51:23.416889 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 092892a36e0e47cb8d2184c2bbde95d7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 092892a36e0e47cb8d2184c2bbde95d7 already in progress: creating tablet
I20250113 02:51:23.417064 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet d35ec7e4a5084a74bea172ef8d9990b5 (table default.Table_4369a143c72643b084df58d25314c6dd [id=36ff239c08a54be99ca307354810fff3]) successfully deleted
W20250113 02:51:23.417265 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 7d53de6379b84103ba905effb0835676 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7d53de6379b84103ba905effb0835676 already in progress: creating tablet
W20250113 02:51:23.417476 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 80d31d5ead804ddaa032969e401a8f7e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 80d31d5ead804ddaa032969e401a8f7e already in progress: creating tablet
W20250113 02:51:23.417687 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet b0113b16110e4787ac1c4676b2c8bd1e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b0113b16110e4787ac1c4676b2c8bd1e already in progress: creating tablet
W20250113 02:51:23.417917 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet c1fb2b36c0504cb4ba32671ae4c620d1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c1fb2b36c0504cb4ba32671ae4c620d1 already in progress: creating tablet
I20250113 02:51:23.423600 1195 ts_tablet_manager.cc:1918] T 18e2d79d731142f89b9089869bc736e5 P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:23.425110 1195 log.cc:1198] T 18e2d79d731142f89b9089869bc736e5 P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/18e2d79d731142f89b9089869bc736e5
I20250113 02:51:23.425751 1195 ts_tablet_manager.cc:1939] T 18e2d79d731142f89b9089869bc736e5 P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:23.427263 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet 18e2d79d731142f89b9089869bc736e5 (table default.Table_f9aa5df6ba3d43999ebd6570df3108d1 [id=fa5a1cda58f54103ac637c12d99e4e38]) successfully deleted
W20250113 02:51:23.428288 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 6d63f0e5835540a78c82020b20a07dac because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6d63f0e5835540a78c82020b20a07dac already in progress: creating tablet
W20250113 02:51:23.429105 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet e70658c90555463498d3fc2e2f13ea7f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e70658c90555463498d3fc2e2f13ea7f already in progress: creating tablet
W20250113 02:51:23.429845 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 41fcde988f72496b824271fa36c1a226 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 41fcde988f72496b824271fa36c1a226 already in progress: creating tablet
W20250113 02:51:23.432008 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet ddf79eec41214693859f526265a55492 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ddf79eec41214693859f526265a55492 already in progress: creating tablet
W20250113 02:51:23.432720 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet f68f9f4076d9475d938e795e72bdd468 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f68f9f4076d9475d938e795e72bdd468 already in progress: creating tablet
I20250113 02:51:23.433351 1195 tablet_replica.cc:331] T 53cc528d282c47788d5d558569f074c1 P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:23.433538 1195 raft_consensus.cc:2238] T 53cc528d282c47788d5d558569f074c1 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:23.433727 1195 raft_consensus.cc:2267] T 53cc528d282c47788d5d558569f074c1 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:23.435050 1195 ts_tablet_manager.cc:1905] T 53cc528d282c47788d5d558569f074c1 P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:23.435621 739 tablet_service.cc:1514] Processing DeleteTablet for tablet 107e6e9e2d84496f8fce92b9ba377189 with delete_type TABLET_DATA_DELETED (replaced by tablet 7ae2f30bf92742008e53b1ec94ebd0cf) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:23.435621 448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_8a2c850672684fb0aef23bd1edfc8cd8" } new_table_name: "default.Table_15bd9f6352964b8ba8ba9f4f021011b9" modify_external_catalogs: true
I20250113 02:51:23.447330 1195 ts_tablet_manager.cc:1918] T 53cc528d282c47788d5d558569f074c1 P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:23.451798 1195 log.cc:1198] T 53cc528d282c47788d5d558569f074c1 P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/53cc528d282c47788d5d558569f074c1
I20250113 02:51:23.451932 1192 tablet_bootstrap.cc:492] T 317c6e8013b745c290bf3190dc237ab8 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:23.452157 1192 ts_tablet_manager.cc:1397] T 317c6e8013b745c290bf3190dc237ab8 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.197s user 0.049s sys 0.092s
I20250113 02:51:23.452591 1195 ts_tablet_manager.cc:1939] T 53cc528d282c47788d5d558569f074c1 P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:23.453229 1192 raft_consensus.cc:357] T 317c6e8013b745c290bf3190dc237ab8 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:23.453495 1192 raft_consensus.cc:383] T 317c6e8013b745c290bf3190dc237ab8 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:23.453604 1192 raft_consensus.cc:738] T 317c6e8013b745c290bf3190dc237ab8 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:23.453866 1192 consensus_queue.cc:260] T 317c6e8013b745c290bf3190dc237ab8 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:23.454509 1192 ts_tablet_manager.cc:1428] T 317c6e8013b745c290bf3190dc237ab8 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250113 02:51:23.454825 1192 tablet_bootstrap.cc:492] T 4efa63b8be794e1db4b340f4c0c4cd5a P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:23.455130 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet 53cc528d282c47788d5d558569f074c1 (table default.Table_06f81a97cb014a3bad594e88c8cf4733 [id=7c2bc6d4232f4585b84d71e3e1f85794]) successfully deleted
I20250113 02:51:23.455842 1195 tablet_replica.cc:331] T 107e6e9e2d84496f8fce92b9ba377189 P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:23.456784 1195 raft_consensus.cc:2238] T 107e6e9e2d84496f8fce92b9ba377189 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:23.457082 1194 tablet_bootstrap.cc:492] T 93a2729c5a604a27a9bbc451ccb4e128 P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:23.457222 1195 raft_consensus.cc:2267] T 107e6e9e2d84496f8fce92b9ba377189 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:23.457309 1194 ts_tablet_manager.cc:1397] T 93a2729c5a604a27a9bbc451ccb4e128 P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.206s user 0.004s sys 0.000s
I20250113 02:51:23.458257 1195 ts_tablet_manager.cc:1905] T 107e6e9e2d84496f8fce92b9ba377189 P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:23.458343 1194 raft_consensus.cc:357] T 93a2729c5a604a27a9bbc451ccb4e128 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:23.458709 1194 raft_consensus.cc:383] T 93a2729c5a604a27a9bbc451ccb4e128 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:23.458866 1194 raft_consensus.cc:738] T 93a2729c5a604a27a9bbc451ccb4e128 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:23.459208 1194 consensus_queue.cc:260] T 93a2729c5a604a27a9bbc451ccb4e128 P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:23.459878 1194 ts_tablet_manager.cc:1428] T 93a2729c5a604a27a9bbc451ccb4e128 P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250113 02:51:23.460007 1192 tablet_bootstrap.cc:654] T 4efa63b8be794e1db4b340f4c0c4cd5a P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:23.465363 449 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_991d8c72ce924153ad119a440b54fb1c" } modify_external_catalogs: true
I20250113 02:51:23.465590 449 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_991d8c72ce924153ad119a440b54fb1c" } modify_external_catalogs: true
I20250113 02:51:23.481997 1192 tablet_bootstrap.cc:492] T 4efa63b8be794e1db4b340f4c0c4cd5a P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:23.482190 1192 ts_tablet_manager.cc:1397] T 4efa63b8be794e1db4b340f4c0c4cd5a P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.027s user 0.002s sys 0.004s
I20250113 02:51:23.482281 1195 ts_tablet_manager.cc:1918] T 107e6e9e2d84496f8fce92b9ba377189 P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:23.482424 1195 log.cc:1198] T 107e6e9e2d84496f8fce92b9ba377189 P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/107e6e9e2d84496f8fce92b9ba377189
I20250113 02:51:23.482784 1195 ts_tablet_manager.cc:1939] T 107e6e9e2d84496f8fce92b9ba377189 P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:23.483016 1192 raft_consensus.cc:357] T 4efa63b8be794e1db4b340f4c0c4cd5a P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:23.483237 1192 raft_consensus.cc:383] T 4efa63b8be794e1db4b340f4c0c4cd5a P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:23.483301 1192 raft_consensus.cc:738] T 4efa63b8be794e1db4b340f4c0c4cd5a P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:23.483537 1192 consensus_queue.cc:260] T 4efa63b8be794e1db4b340f4c0c4cd5a P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:23.484292 1191 tablet_bootstrap.cc:492] T 21c4b419bf2148e0b140301fa80622bb P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:23.484436 1191 ts_tablet_manager.cc:1397] T 21c4b419bf2148e0b140301fa80622bb P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.156s user 0.020s sys 0.032s
I20250113 02:51:23.485433 1191 raft_consensus.cc:357] T 21c4b419bf2148e0b140301fa80622bb P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:23.485647 1191 raft_consensus.cc:383] T 21c4b419bf2148e0b140301fa80622bb P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:23.485741 1191 raft_consensus.cc:738] T 21c4b419bf2148e0b140301fa80622bb P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:23.489957 451 catalog_manager.cc:5618] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467) does not have the latest schema for tablet 8547b63c40ec4ccebd962b5a3b2c5e0f (table default.Table_3d5fd97116b343b0a6c798e1f22327c2 [id=1396c4482ee646d18330d200fd3f9816]). Expected version 1 got 0
I20250113 02:51:23.490027 1192 ts_tablet_manager.cc:1428] T 4efa63b8be794e1db4b340f4c0c4cd5a P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20250113 02:51:23.490629 1194 tablet_bootstrap.cc:492] T 1c26074fdd804bd79cb0c295622d11ff P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:23.492498 1192 tablet_bootstrap.cc:492] T d7920c0544c0466bad6bd8c09aae1a7d P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:23.494519 1192 tablet_bootstrap.cc:654] T d7920c0544c0466bad6bd8c09aae1a7d P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:23.496155 897 tablet_service.cc:1467] Processing CreateTablet for tablet acd83024592e431898e9675c17f50d4f (DEFAULT_TABLE table=default.Table_e81c865deb634a57bf1906b18b31a60a [id=044304e1c4b44cc28b8990d6f28145a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:23.496555 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet acd83024592e431898e9675c17f50d4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:23.502833 1194 tablet_bootstrap.cc:654] T 1c26074fdd804bd79cb0c295622d11ff P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:23.504271 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_b19190849e5245a380b5349a7abce2d5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:23.504841 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b19190849e5245a380b5349a7abce2d5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:23.507547 1198 ts_tablet_manager.cc:1918] T 1d81eec487dd4016b3b804451da31c49 P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:23.511251 1039 tablet_service.cc:1467] Processing CreateTablet for tablet acd83024592e431898e9675c17f50d4f (DEFAULT_TABLE table=default.Table_e81c865deb634a57bf1906b18b31a60a [id=044304e1c4b44cc28b8990d6f28145a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:23.511749 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet acd83024592e431898e9675c17f50d4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:23.511960 1023 tablet_service.cc:1467] Processing CreateTablet for tablet c13e9a51c1e74b44b352bade78390ba5 (DEFAULT_TABLE table=default.Table_82f5a202f8d04a2680202b3ab06994c4 [id=caf098e2b57849a997bd7c8eaf1f9326]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:23.512302 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c13e9a51c1e74b44b352bade78390ba5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:23.514202 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_bfda83bbe8bb4025b01edf1e38b1b73c" } modify_external_catalogs: true
I20250113 02:51:23.514475 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_bfda83bbe8bb4025b01edf1e38b1b73c" } modify_external_catalogs: true
I20250113 02:51:23.516327 891 tablet_service.cc:1467] Processing CreateTablet for tablet 938cc1ccd8dd449996836188ba9e64ff (DEFAULT_TABLE table=default.Table_44352829b8da471388d4d10f849b0cd7 [id=5824acd778ed47d09f087bbf8e7822d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:23.516606 895 tablet_service.cc:1467] Processing CreateTablet for tablet 1968c7e9357c4c6e901d334ace0467fb (DEFAULT_TABLE table=default.Table_f2da084a4aee4d19b45e784e56f926a6 [id=fa9bcd8c66c34d65b43561572abb8890]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:23.516731 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 938cc1ccd8dd449996836188ba9e64ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:23.516932 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1968c7e9357c4c6e901d334ace0467fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:23.516321 888 tablet_service.cc:1467] Processing CreateTablet for tablet c13e9a51c1e74b44b352bade78390ba5 (DEFAULT_TABLE table=default.Table_82f5a202f8d04a2680202b3ab06994c4 [id=caf098e2b57849a997bd7c8eaf1f9326]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:23.517390 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c13e9a51c1e74b44b352bade78390ba5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:23.518234 881 tablet_service.cc:1514] Processing DeleteTablet for tablet fc92f19390fd48a4a45ba8fb839020fd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:17 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:23.519040 881 tablet_service.cc:1514] Processing DeleteTablet for tablet 93a2729c5a604a27a9bbc451ccb4e128 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:20 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:23.519560 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_ed883bfc96094be9bb6e07bc510b6042"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:23.520130 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ed883bfc96094be9bb6e07bc510b6042 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:23.521127 752 tablet_service.cc:1467] Processing CreateTablet for tablet c13e9a51c1e74b44b352bade78390ba5 (DEFAULT_TABLE table=default.Table_82f5a202f8d04a2680202b3ab06994c4 [id=caf098e2b57849a997bd7c8eaf1f9326]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:23.521530 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c13e9a51c1e74b44b352bade78390ba5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:23.523206 1198 log.cc:1198] T 1d81eec487dd4016b3b804451da31c49 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/1d81eec487dd4016b3b804451da31c49
I20250113 02:51:23.523665 1198 ts_tablet_manager.cc:1939] T 1d81eec487dd4016b3b804451da31c49 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:23.524276 744 tablet_service.cc:1467] Processing CreateTablet for tablet 1968c7e9357c4c6e901d334ace0467fb (DEFAULT_TABLE table=default.Table_f2da084a4aee4d19b45e784e56f926a6 [id=fa9bcd8c66c34d65b43561572abb8890]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:23.524628 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1968c7e9357c4c6e901d334ace0467fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:23.528285 446 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet 37c7be0795f045e8b8f20d6aaa945b6f
I20250113 02:51:23.528287 743 tablet_service.cc:1467] Processing CreateTablet for tablet 938cc1ccd8dd449996836188ba9e64ff (DEFAULT_TABLE table=default.Table_44352829b8da471388d4d10f849b0cd7 [id=5824acd778ed47d09f087bbf8e7822d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:23.528733 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 938cc1ccd8dd449996836188ba9e64ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:23.528806 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_ba72b5bfcbe6447196e4d8640ab0ff65" } new_table_name: "default.Table_ba98ed25e93a45a288784d4e5bdee029" modify_external_catalogs: true
I20250113 02:51:23.532430 739 tablet_service.cc:1467] Processing CreateTablet for tablet acd83024592e431898e9675c17f50d4f (DEFAULT_TABLE table=default.Table_e81c865deb634a57bf1906b18b31a60a [id=044304e1c4b44cc28b8990d6f28145a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:23.532773 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet acd83024592e431898e9675c17f50d4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:23.533280 748 tablet_service.cc:1467] Processing CreateTablet for tablet b38cd34d67834a3f9bf5b5636bb79217 (DEFAULT_TABLE table=default.Table_5a11b1fe188b4b4ba215476c50f5f974 [id=3aa20a5d180b4cc98ac6044843927ab2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:23.533833 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b38cd34d67834a3f9bf5b5636bb79217. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:23.534960 1198 tablet_replica.cc:331] T 107e6e9e2d84496f8fce92b9ba377189 P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:23.535189 1198 raft_consensus.cc:2238] T 107e6e9e2d84496f8fce92b9ba377189 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:23.535367 1198 raft_consensus.cc:2267] T 107e6e9e2d84496f8fce92b9ba377189 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:23.535998 1198 ts_tablet_manager.cc:1905] T 107e6e9e2d84496f8fce92b9ba377189 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:23.536690 895 tablet_service.cc:1467] Processing CreateTablet for tablet b38cd34d67834a3f9bf5b5636bb79217 (DEFAULT_TABLE table=default.Table_5a11b1fe188b4b4ba215476c50f5f974 [id=3aa20a5d180b4cc98ac6044843927ab2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:23.537089 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b38cd34d67834a3f9bf5b5636bb79217. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:23.543836 448 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 51d9483ecacb413796d0fd9e7560eac1 consensus state has no leader
I20250113 02:51:23.544422 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 999fecb3fc65466cad21cc1ff5ef2d44
I20250113 02:51:23.546787 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 99c8046f5d124ca49899df365114c69f
I20250113 02:51:23.547482 448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_5e5c2e147f764a3389263a89c310f968" } new_table_name: "default.Table_54119752ebc147b8b34429f1c38b71d9" modify_external_catalogs: true
I20250113 02:51:23.547533 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_14ab7cebe87e40b4bf54f68ee6a1ab80"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:23.548164 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_14ab7cebe87e40b4bf54f68ee6a1ab80 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:23.548943 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_22552ae21cee41b9b15cd6344572f085"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:23.549470 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_22552ae21cee41b9b15cd6344572f085 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:23.550673 881 tablet_service.cc:1514] Processing DeleteTablet for tablet 99c8046f5d124ca49899df365114c69f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:23 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:23.552250 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_6c21e014853e4ab1996e23f0822d2b85" } new_table_name: "default.Table_e3775884d0c7400ab769befa718e8fc7" modify_external_catalogs: true
I20250113 02:51:23.553766 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet 999fecb3fc65466cad21cc1ff5ef2d44 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:23 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:23.553817 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_38394a92f7d84b02998b70e0ab85e35f" } new_table_name: "default.Table_9ef18ee71ad843c5921750a2f8478855" modify_external_catalogs: true
I20250113 02:51:23.554004 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet 99c8046f5d124ca49899df365114c69f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:23 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:23.554728 1024 tablet_service.cc:1467] Processing CreateTablet for tablet b38cd34d67834a3f9bf5b5636bb79217 (DEFAULT_TABLE table=default.Table_5a11b1fe188b4b4ba215476c50f5f974 [id=3aa20a5d180b4cc98ac6044843927ab2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:23.555130 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b38cd34d67834a3f9bf5b5636bb79217. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:23.556458 881 tablet_service.cc:1467] Processing CreateTablet for tablet 3f62f344c5a648328ca2de8ca1069c19 (DEFAULT_TABLE table=default.Table_d84fa5147d8a430ca92f49e98b2f7839 [id=7783df86a6ee404f87055036508adb20]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:23.556820 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3f62f344c5a648328ca2de8ca1069c19. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:23.560355 1039 tablet_service.cc:1467] Processing CreateTablet for tablet 938cc1ccd8dd449996836188ba9e64ff (DEFAULT_TABLE table=default.Table_44352829b8da471388d4d10f849b0cd7 [id=5824acd778ed47d09f087bbf8e7822d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:23.560727 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet 1d81eec487dd4016b3b804451da31c49 (table default.Table_d7d1f5a34e5f4c05b474b0b61f912cd9 [id=2c7925ec0d8d4cd7b8761268d3962f15]) successfully deleted
I20250113 02:51:23.560745 1035 tablet_service.cc:1467] Processing CreateTablet for tablet 6a4c6dc08df744e6856bdf99ab65c575 (DEFAULT_TABLE table=default.Table_82a014a9f48c43b4b14d63522bac7124 [id=c489a295f02b4f00ac7ea90e62d8ed04]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:23.560935 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 938cc1ccd8dd449996836188ba9e64ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:23.561267 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a4c6dc08df744e6856bdf99ab65c575. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:23.561473 1029 tablet_service.cc:1467] Processing CreateTablet for tablet 3f62f344c5a648328ca2de8ca1069c19 (DEFAULT_TABLE table=default.Table_d84fa5147d8a430ca92f49e98b2f7839 [id=7783df86a6ee404f87055036508adb20]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:23.561883 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3f62f344c5a648328ca2de8ca1069c19. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:23.562209 898 tablet_service.cc:1467] Processing CreateTablet for tablet e570465a3d4046b1ac19a3a034fe91f1 (DEFAULT_TABLE table=default.Table_766a63106d8a4fd29bb338e76bd96089 [id=3042221d269a4bd88c04c1b6bb1bcd26]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:23.562573 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e570465a3d4046b1ac19a3a034fe91f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:23.564229 892 tablet_service.cc:1467] Processing CreateTablet for tablet 9f8eb43a39774d2c8e9cb4e88a405b0f (DEFAULT_TABLE table=default.Table_bfda83bbe8bb4025b01edf1e38b1b73c [id=e900f4da41bf46bcace34b132f3ba977]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:23.564554 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f8eb43a39774d2c8e9cb4e88a405b0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:23.568181 746 tablet_service.cc:1514] Processing DeleteTablet for tablet 999fecb3fc65466cad21cc1ff5ef2d44 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:23 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:23.571895 746 tablet_service.cc:1514] Processing DeleteTablet for tablet 99c8046f5d124ca49899df365114c69f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:23 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:23.572463 754 tablet_service.cc:1467] Processing CreateTablet for tablet 6a4c6dc08df744e6856bdf99ab65c575 (DEFAULT_TABLE table=default.Table_82a014a9f48c43b4b14d63522bac7124 [id=c489a295f02b4f00ac7ea90e62d8ed04]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:23.573310 1023 tablet_service.cc:1467] Processing CreateTablet for tablet 1968c7e9357c4c6e901d334ace0467fb (DEFAULT_TABLE table=default.Table_f2da084a4aee4d19b45e784e56f926a6 [id=fa9bcd8c66c34d65b43561572abb8890]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:23.573762 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1968c7e9357c4c6e901d334ace0467fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:23.573887 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a4c6dc08df744e6856bdf99ab65c575. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:23.576443 882 tablet_service.cc:1467] Processing CreateTablet for tablet 67baaab1e36e415381579d7385d4ab5a (DEFAULT_TABLE table=default.Table_5182a1c5c7a2462ba930940ae04b505f [id=6666a614063942c99a78b95d35add037]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:23.576841 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 67baaab1e36e415381579d7385d4ab5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:23.577962 746 tablet_service.cc:1467] Processing CreateTablet for tablet 3f62f344c5a648328ca2de8ca1069c19 (DEFAULT_TABLE table=default.Table_d84fa5147d8a430ca92f49e98b2f7839 [id=7783df86a6ee404f87055036508adb20]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:23.578423 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3f62f344c5a648328ca2de8ca1069c19. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:23.579834 753 tablet_service.cc:1467] Processing CreateTablet for tablet fd0b8592997d4a61b8ca854ecbb67c55 (DEFAULT_TABLE table=default.Table_6f2972dcacb44ccb9e4f3d828ee536ec [id=d1b0bb18a7b84473803cd1294b0829ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:23.581375 1438 raft_consensus.cc:491] T 41bcb2eafff14fa8a040982c723f99c1 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:23.581600 1438 raft_consensus.cc:513] T 41bcb2eafff14fa8a040982c723f99c1 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:23.582497 1438 leader_election.cc:290] T 41bcb2eafff14fa8a040982c723f99c1 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587), 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
I20250113 02:51:23.582957 1062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41bcb2eafff14fa8a040982c723f99c1" candidate_uuid: "df69f48656b44bb890f76587ee69cc0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:23.583745 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd0b8592997d4a61b8ca854ecbb67c55. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:23.583734 852 leader_election.cc:343] T 41bcb2eafff14fa8a040982c723f99c1 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:23.584112 749 tablet_service.cc:1467] Processing CreateTablet for tablet 9f8eb43a39774d2c8e9cb4e88a405b0f (DEFAULT_TABLE table=default.Table_bfda83bbe8bb4025b01edf1e38b1b73c [id=e900f4da41bf46bcace34b132f3ba977]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:23.585306 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f8eb43a39774d2c8e9cb4e88a405b0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:23.586414 748 tablet_service.cc:1467] Processing CreateTablet for tablet aa1ebdb92a5f4c92a5195b614689c100 (DEFAULT_TABLE table=default.Table_74ce8ff3e2c64e088ac82c48ec722b39 [id=4a5048f6224642deafdc335e17679294]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:23.589885 740 tablet_service.cc:1467] Processing CreateTablet for tablet e570465a3d4046b1ac19a3a034fe91f1 (DEFAULT_TABLE table=default.Table_766a63106d8a4fd29bb338e76bd96089 [id=3042221d269a4bd88c04c1b6bb1bcd26]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:23.590910 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41bcb2eafff14fa8a040982c723f99c1" candidate_uuid: "df69f48656b44bb890f76587ee69cc0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
I20250113 02:51:23.590905 737 tablet_service.cc:1467] Processing CreateTablet for tablet 67baaab1e36e415381579d7385d4ab5a (DEFAULT_TABLE table=default.Table_5182a1c5c7a2462ba930940ae04b505f [id=6666a614063942c99a78b95d35add037]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:23.591413 742 tablet_service.cc:1467] Processing CreateTablet for tablet 283fc7f1c74e480c8a63c74d3ae9eb7d (DEFAULT_TABLE table=default.Table_1505767419694e5ba79a90c46780bbb7 [id=7128c928ad5b40aa910aa297049c4944]), partition=RANGE (key) PARTITION UNBOUNDED
W20250113 02:51:23.592031 851 leader_election.cc:343] T 41bcb2eafff14fa8a040982c723f99c1 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:23.592206 851 leader_election.cc:304] T 41bcb2eafff14fa8a040982c723f99c1 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df69f48656b44bb890f76587ee69cc0f; no voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca
I20250113 02:51:23.592840 1438 raft_consensus.cc:2743] T 41bcb2eafff14fa8a040982c723f99c1 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:23.593098 1442 raft_consensus.cc:491] T 1c26074fdd804bd79cb0c295622d11ff P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:23.593302 1442 raft_consensus.cc:513] T 1c26074fdd804bd79cb0c295622d11ff P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:23.594198 1442 leader_election.cc:290] T 1c26074fdd804bd79cb0c295622d11ff P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
I20250113 02:51:23.595952 882 tablet_service.cc:1467] Processing CreateTablet for tablet aa1ebdb92a5f4c92a5195b614689c100 (DEFAULT_TABLE table=default.Table_74ce8ff3e2c64e088ac82c48ec722b39 [id=4a5048f6224642deafdc335e17679294]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:23.596745 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa1ebdb92a5f4c92a5195b614689c100. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:23.597681 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c26074fdd804bd79cb0c295622d11ff" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:23.596330 891 tablet_service.cc:1514] Processing DeleteTablet for tablet 999fecb3fc65466cad21cc1ff5ef2d44 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:23 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:23.597079 888 tablet_service.cc:1467] Processing CreateTablet for tablet fd0b8592997d4a61b8ca854ecbb67c55 (DEFAULT_TABLE table=default.Table_6f2972dcacb44ccb9e4f3d828ee536ec [id=d1b0bb18a7b84473803cd1294b0829ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:23.598068 1034 tablet_service.cc:1467] Processing CreateTablet for tablet aa1ebdb92a5f4c92a5195b614689c100 (DEFAULT_TABLE table=default.Table_74ce8ff3e2c64e088ac82c48ec722b39 [id=4a5048f6224642deafdc335e17679294]), partition=RANGE (key) PARTITION UNBOUNDED
W20250113 02:51:23.598383 998 leader_election.cc:343] T 1c26074fdd804bd79cb0c295622d11ff P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:23.598467 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa1ebdb92a5f4c92a5195b614689c100. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:23.598676 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd0b8592997d4a61b8ca854ecbb67c55. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:23.598790 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa1ebdb92a5f4c92a5195b614689c100. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:23.598687 1024 tablet_service.cc:1467] Processing CreateTablet for tablet fd0b8592997d4a61b8ca854ecbb67c55 (DEFAULT_TABLE table=default.Table_6f2972dcacb44ccb9e4f3d828ee536ec [id=d1b0bb18a7b84473803cd1294b0829ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:23.599105 737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 67baaab1e36e415381579d7385d4ab5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:23.599686 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 283fc7f1c74e480c8a63c74d3ae9eb7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:23.599952 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e570465a3d4046b1ac19a3a034fe91f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:23.600920 1191 consensus_queue.cc:260] T 21c4b419bf2148e0b140301fa80622bb P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:23.601219 897 tablet_service.cc:1467] Processing CreateTablet for tablet 6a4c6dc08df744e6856bdf99ab65c575 (DEFAULT_TABLE table=default.Table_82a014a9f48c43b4b14d63522bac7124 [id=c489a295f02b4f00ac7ea90e62d8ed04]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:23.603040 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd0b8592997d4a61b8ca854ecbb67c55. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:23.603365 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a4c6dc08df744e6856bdf99ab65c575. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:23.604228 1191 ts_tablet_manager.cc:1428] T 21c4b419bf2148e0b140301fa80622bb P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.120s user 0.002s sys 0.000s
I20250113 02:51:23.605261 891 tablet_service.cc:1467] Processing CreateTablet for tablet 283fc7f1c74e480c8a63c74d3ae9eb7d (DEFAULT_TABLE table=default.Table_1505767419694e5ba79a90c46780bbb7 [id=7128c928ad5b40aa910aa297049c4944]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:23.605738 1191 tablet_bootstrap.cc:492] T 6c8ca30720664f3a819c4bff2493934d P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:23.605939 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 283fc7f1c74e480c8a63c74d3ae9eb7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:23.608757 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet 107e6e9e2d84496f8fce92b9ba377189 (table default.Table_74627d276edf4d75a2fcaa627a7f921b [id=de6b4835f6c04c20b6a5ef8404dc78f7]) successfully deleted
I20250113 02:51:23.608794 1029 tablet_service.cc:1467] Processing CreateTablet for tablet 67baaab1e36e415381579d7385d4ab5a (DEFAULT_TABLE table=default.Table_5182a1c5c7a2462ba930940ae04b505f [id=6666a614063942c99a78b95d35add037]), partition=RANGE (key) PARTITION UNBOUNDED
W20250113 02:51:23.609104 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 999fecb3fc65466cad21cc1ff5ef2d44 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 999fecb3fc65466cad21cc1ff5ef2d44 already in progress: creating tablet
I20250113 02:51:23.609227 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 67baaab1e36e415381579d7385d4ab5a. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:23.609392 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 99c8046f5d124ca49899df365114c69f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 99c8046f5d124ca49899df365114c69f already in progress: creating tablet
I20250113 02:51:23.610548 741 tablet_service.cc:1467] Processing CreateTablet for tablet 21bcdd741104474bad7023426856d18f (DEFAULT_TABLE table=default.Table_4ef3edc5b07c4e0881456bfe23741c90 [id=46a732d6c22e472d82163d1295b34dcd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:23.611013 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 21bcdd741104474bad7023426856d18f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:23.612289 1035 tablet_service.cc:1467] Processing CreateTablet for tablet e570465a3d4046b1ac19a3a034fe91f1 (DEFAULT_TABLE table=default.Table_766a63106d8a4fd29bb338e76bd96089 [id=3042221d269a4bd88c04c1b6bb1bcd26]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:23.612380 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c26074fdd804bd79cb0c295622d11ff" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
I20250113 02:51:23.612691 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e570465a3d4046b1ac19a3a034fe91f1. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:23.615525 995 leader_election.cc:343] T 1c26074fdd804bd79cb0c295622d11ff P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:23.615723 995 leader_election.cc:304] T 1c26074fdd804bd79cb0c295622d11ff P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4216a7f853d642b7bc3e5cff458f3aca; no voters: 1186fa869eb14812832fa6562531f196, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:23.616298 1442 raft_consensus.cc:2743] T 1c26074fdd804bd79cb0c295622d11ff P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250113 02:51:23.619624 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet c1fb2b36c0504cb4ba32671ae4c620d1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c1fb2b36c0504cb4ba32671ae4c620d1 already in progress: creating tablet
W20250113 02:51:23.620260 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 0644f5600c5b47e29f3fa73060ed96e3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0644f5600c5b47e29f3fa73060ed96e3 already in progress: creating tablet
I20250113 02:51:23.620337 1196 tablet_replica.cc:331] T d7c4c7b0cfc44ac1952bc73cbe1bced8 P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
W20250113 02:51:23.620471 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 80d31d5ead804ddaa032969e401a8f7e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 80d31d5ead804ddaa032969e401a8f7e already in progress: creating tablet
I20250113 02:51:23.620523 1196 raft_consensus.cc:2238] T d7c4c7b0cfc44ac1952bc73cbe1bced8 P df69f48656b44bb890f76587ee69cc0f [term 1 FOLLOWER]: Raft consensus shutting down.
W20250113 02:51:23.620678 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 42cf1e708644427f9d4dcd41ae2bcb37 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 42cf1e708644427f9d4dcd41ae2bcb37 already in progress: creating tablet
I20250113 02:51:23.620702 1196 raft_consensus.cc:2267] T d7c4c7b0cfc44ac1952bc73cbe1bced8 P df69f48656b44bb890f76587ee69cc0f [term 1 FOLLOWER]: Raft consensus is shut down!
W20250113 02:51:23.620893 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 092892a36e0e47cb8d2184c2bbde95d7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 092892a36e0e47cb8d2184c2bbde95d7 already in progress: creating tablet
W20250113 02:51:23.621237 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 7d53de6379b84103ba905effb0835676 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7d53de6379b84103ba905effb0835676 already in progress: creating tablet
I20250113 02:51:23.621343 1196 ts_tablet_manager.cc:1905] T d7c4c7b0cfc44ac1952bc73cbe1bced8 P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:23.643932 1039 tablet_service.cc:1467] Processing CreateTablet for tablet 9f8eb43a39774d2c8e9cb4e88a405b0f (DEFAULT_TABLE table=default.Table_bfda83bbe8bb4025b01edf1e38b1b73c [id=e900f4da41bf46bcace34b132f3ba977]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:23.645113 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f8eb43a39774d2c8e9cb4e88a405b0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:23.652379 895 tablet_service.cc:1467] Processing CreateTablet for tablet 21bcdd741104474bad7023426856d18f (DEFAULT_TABLE table=default.Table_4ef3edc5b07c4e0881456bfe23741c90 [id=46a732d6c22e472d82163d1295b34dcd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:23.652830 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 21bcdd741104474bad7023426856d18f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:23.660977 898 tablet_service.cc:1467] Processing CreateTablet for tablet fe8f075d4c9a4f9abe372bfdb81a829c (DEFAULT_TABLE table=default.Table_e1f19e9072094459a8aba69a14179bda [id=d37981b613bd494685de3e48d382f485]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:23.661418 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe8f075d4c9a4f9abe372bfdb81a829c. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:23.664953 440 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.61:35271, user_credentials={real_user=slave}} blocked reactor thread for 66808.7us
I20250113 02:51:23.677793 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet c518f791b4d74c54ab0c0364d29b4a64
I20250113 02:51:23.681564 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_345955ac31bd403292668ee24ce8eed3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:23.682124 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_345955ac31bd403292668ee24ce8eed3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:23.684613 453 catalog_manager.cc:5618] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587) does not have the latest schema for tablet 317c6e8013b745c290bf3190dc237ab8 (table default.Table_13fd4df9c207466cb85cd8ed9062722c [id=6bfb67a96b2b47849f7fa12c57a2f4a1]). Expected version 1 got 0
I20250113 02:51:23.685427 891 tablet_service.cc:1514] Processing DeleteTablet for tablet c518f791b4d74c54ab0c0364d29b4a64 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:23 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:23.688719 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 9f8eb43a39774d2c8e9cb4e88a405b0f
I20250113 02:51:23.689438 451 master_service.cc:946] ReplaceTablet: received request to replace tablet e70658c90555463498d3fc2e2f13ea7f from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:23.690033 882 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f8eb43a39774d2c8e9cb4e88a405b0f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:23 UTC) from {username='slave'} at 127.0.0.1:37366
W20250113 02:51:23.692572 448 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6d4d8b5cc8ad4c9e935d8afaf5766e66 consensus state has no leader
I20250113 02:51:23.693495 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_349bd0d1105d42e4889232adbacc4d3a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:23.694240 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_349bd0d1105d42e4889232adbacc4d3a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:23.696372 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_7f15d522310c4e8a8afc5d1b77bcaeec"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:23.696877 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7f15d522310c4e8a8afc5d1b77bcaeec in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:23.702646 753 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f8eb43a39774d2c8e9cb4e88a405b0f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:23 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:23.702592 744 tablet_service.cc:1467] Processing CreateTablet for tablet fe8f075d4c9a4f9abe372bfdb81a829c (DEFAULT_TABLE table=default.Table_e1f19e9072094459a8aba69a14179bda [id=d37981b613bd494685de3e48d382f485]), partition=RANGE (key) PARTITION UNBOUNDED
W20250113 02:51:23.703549 439 catalog_manager.cc:4839] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 9f8eb43a39774d2c8e9cb4e88a405b0f because the tablet was not found. No further retry: Not found: Tablet not found: 9f8eb43a39774d2c8e9cb4e88a405b0f
W20250113 02:51:23.705687 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d8bb60fdd06946739b75b255303d9d24 consensus state has no leader
I20250113 02:51:23.706354 739 tablet_service.cc:1467] Processing CreateTablet for tablet 4c759831773849f2bc52e484ffa0cf49 (DEFAULT_TABLE table=default.Table_0fcdb63a92324044972d1e69630f524d [id=cc7909c1a59e460a8db262923636f9f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:23.706939 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe8f075d4c9a4f9abe372bfdb81a829c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:23.707994 1194 tablet_bootstrap.cc:492] T 1c26074fdd804bd79cb0c295622d11ff P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:23.708176 1194 ts_tablet_manager.cc:1397] T 1c26074fdd804bd79cb0c295622d11ff P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.218s user 0.002s sys 0.002s
I20250113 02:51:23.708977 1194 raft_consensus.cc:357] T 1c26074fdd804bd79cb0c295622d11ff P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:23.709218 1194 raft_consensus.cc:383] T 1c26074fdd804bd79cb0c295622d11ff P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:23.709286 1194 raft_consensus.cc:738] T 1c26074fdd804bd79cb0c295622d11ff P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:23.709555 1194 consensus_queue.cc:260] T 1c26074fdd804bd79cb0c295622d11ff P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:23.709925 1034 tablet_service.cc:1467] Processing CreateTablet for tablet 283fc7f1c74e480c8a63c74d3ae9eb7d (DEFAULT_TABLE table=default.Table_1505767419694e5ba79a90c46780bbb7 [id=7128c928ad5b40aa910aa297049c4944]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:23.709867 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_518b2b6cde4441c88c49806de09f0aea"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:23.710366 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 283fc7f1c74e480c8a63c74d3ae9eb7d. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:23.710530 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_518b2b6cde4441c88c49806de09f0aea in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:23.711503 1194 ts_tablet_manager.cc:1428] T 1c26074fdd804bd79cb0c295622d11ff P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250113 02:51:23.714848 1198 ts_tablet_manager.cc:1918] T 107e6e9e2d84496f8fce92b9ba377189 P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:23.721973 1194 tablet_bootstrap.cc:492] T be80b631ccd747f1b9355a00787a7b55 P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:23.723706 1194 tablet_bootstrap.cc:654] T be80b631ccd747f1b9355a00787a7b55 P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:23.724229 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c759831773849f2bc52e484ffa0cf49. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:23.728360 1198 log.cc:1198] T 107e6e9e2d84496f8fce92b9ba377189 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/107e6e9e2d84496f8fce92b9ba377189
I20250113 02:51:23.729303 1028 tablet_service.cc:1467] Processing CreateTablet for tablet 21bcdd741104474bad7023426856d18f (DEFAULT_TABLE table=default.Table_4ef3edc5b07c4e0881456bfe23741c90 [id=46a732d6c22e472d82163d1295b34dcd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:23.733952 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 21bcdd741104474bad7023426856d18f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:23.737145 1033 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f8eb43a39774d2c8e9cb4e88a405b0f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:23 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:23.737455 1198 ts_tablet_manager.cc:1939] T 107e6e9e2d84496f8fce92b9ba377189 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
W20250113 02:51:23.738781 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 32cba8f3efc54e78b4df41c15df82eea consensus state has no leader
I20250113 02:51:23.740074 743 tablet_service.cc:1514] Processing DeleteTablet for tablet c518f791b4d74c54ab0c0364d29b4a64 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:23 UTC) from {username='slave'} at 127.0.0.1:55398
W20250113 02:51:23.743542 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet c518f791b4d74c54ab0c0364d29b4a64 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c518f791b4d74c54ab0c0364d29b4a64 already in progress: creating tablet
I20250113 02:51:23.743758 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet c518f791b4d74c54ab0c0364d29b4a64 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:23 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:23.744035 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet 4efa63b8be794e1db4b340f4c0c4cd5a with delete_type TABLET_DATA_DELETED (replaced by tablet eceed0f52c9f46bbaa78ca773a7afde5) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:23.744557 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 365) took 1037 ms. Trace:
I20250113 02:51:23.749153 1035 tablet_service.cc:1467] Processing CreateTablet for tablet fe8f075d4c9a4f9abe372bfdb81a829c (DEFAULT_TABLE table=default.Table_e1f19e9072094459a8aba69a14179bda [id=d37981b613bd494685de3e48d382f485]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:23.749691 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe8f075d4c9a4f9abe372bfdb81a829c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:23.750236 449 master_service.cc:946] ReplaceTablet: received request to replace tablet a0ca2e3a3ce94f3aa20df64c89c7cf95 from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:23.751389 742 tablet_service.cc:1514] Processing DeleteTablet for tablet 21c4b419bf2148e0b140301fa80622bb with delete_type TABLET_DATA_DELETED (replaced by tablet 23e04961f072440fa3e3f9485e40f79c) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:23.744655 1198 rpcz_store.cc:276] 0113 02:51:22.707190 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:22.708132 (+ 942us) service_pool.cc:225] Handling call
0113 02:51:23.534350 (+826218us) ts_tablet_manager.cc:1145] Deleting tablet 107e6e9e2d84496f8fce92b9ba377189
0113 02:51:23.534362 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:23.656477 (+122115us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:23.659882 (+ 3405us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:23.744499 (+ 84617us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":825987}
I20250113 02:51:23.751752 1195 tablet_replica.cc:331] T 21c4b419bf2148e0b140301fa80622bb P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:23.751938 1195 raft_consensus.cc:2238] T 21c4b419bf2148e0b140301fa80622bb P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:23.752163 1195 raft_consensus.cc:2267] T 21c4b419bf2148e0b140301fa80622bb P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:23.753389 1198 tablet_replica.cc:331] T c71511a100f14322b5091a9804720fa0 P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:23.753569 1198 raft_consensus.cc:2238] T c71511a100f14322b5091a9804720fa0 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:23.753737 1198 raft_consensus.cc:2267] T c71511a100f14322b5091a9804720fa0 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:23.754253 1198 ts_tablet_manager.cc:1905] T c71511a100f14322b5091a9804720fa0 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:23.755012 1196 ts_tablet_manager.cc:1918] T d7c4c7b0cfc44ac1952bc73cbe1bced8 P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:23.756156 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_37b1d9d091924f8d9a8a2cbf1d06af1a" } modify_external_catalogs: true
I20250113 02:51:23.756426 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_37b1d9d091924f8d9a8a2cbf1d06af1a" } modify_external_catalogs: true
I20250113 02:51:23.755157 1196 log.cc:1198] T d7c4c7b0cfc44ac1952bc73cbe1bced8 P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/d7c4c7b0cfc44ac1952bc73cbe1bced8
I20250113 02:51:23.758869 1191 tablet_bootstrap.cc:654] T 6c8ca30720664f3a819c4bff2493934d P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:23.759267 1196 ts_tablet_manager.cc:1939] T d7c4c7b0cfc44ac1952bc73cbe1bced8 P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:23.760385 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet d7c4c7b0cfc44ac1952bc73cbe1bced8 (table default.Table_5727b5d8bf51442e8ae5f7c97f4bfbc8 [id=0c69597e8c5b4ec89179169a7a84e048]) successfully deleted
I20250113 02:51:23.760859 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_2c38174be1e04983a19a30e0076b466c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:23.761031 1195 ts_tablet_manager.cc:1905] T 21c4b419bf2148e0b140301fa80622bb P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250113 02:51:23.761507 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2c38174be1e04983a19a30e0076b466c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:23.764106 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet 107e6e9e2d84496f8fce92b9ba377189 (table default.Table_74627d276edf4d75a2fcaa627a7f921b [id=de6b4835f6c04c20b6a5ef8404dc78f7]) successfully deleted
I20250113 02:51:23.764432 1194 tablet_bootstrap.cc:492] T be80b631ccd747f1b9355a00787a7b55 P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:23.764604 1194 ts_tablet_manager.cc:1397] T be80b631ccd747f1b9355a00787a7b55 P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.043s user 0.003s sys 0.000s
W20250113 02:51:23.765484 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9fff52e99c64462a8a73fccc3c22ca63 consensus state has no leader
I20250113 02:51:23.765437 1194 raft_consensus.cc:357] T be80b631ccd747f1b9355a00787a7b55 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:23.765641 1194 raft_consensus.cc:383] T be80b631ccd747f1b9355a00787a7b55 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:23.765733 1194 raft_consensus.cc:738] T be80b631ccd747f1b9355a00787a7b55 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:23.765913 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_3ed64b1d864b4b79bdaef850c30c486b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:23.765954 1194 consensus_queue.cc:260] T be80b631ccd747f1b9355a00787a7b55 P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:23.766049 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_22552ae21cee41b9b15cd6344572f085" } new_table_name: "default.Table_6d2c74a667ab4954a8380355f54d648f" modify_external_catalogs: true
W20250113 02:51:23.766459 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3ed64b1d864b4b79bdaef850c30c486b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:23.766508 1194 ts_tablet_manager.cc:1428] T be80b631ccd747f1b9355a00787a7b55 P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250113 02:51:23.766906 1194 tablet_bootstrap.cc:492] T 317c6e8013b745c290bf3190dc237ab8 P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:23.767009 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_25a2c280183748df92b064c40756f5ac" } modify_external_catalogs: true
I20250113 02:51:23.767187 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_25a2c280183748df92b064c40756f5ac" } modify_external_catalogs: true
I20250113 02:51:23.768101 453 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_f4fbf20403334a2c99de02125a36de82" } modify_external_catalogs: true
I20250113 02:51:23.768275 453 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_f4fbf20403334a2c99de02125a36de82" } modify_external_catalogs: true
I20250113 02:51:23.768572 1194 tablet_bootstrap.cc:654] T 317c6e8013b745c290bf3190dc237ab8 P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
W20250113 02:51:23.781515 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet b0113b16110e4787ac1c4676b2c8bd1e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b0113b16110e4787ac1c4676b2c8bd1e already in progress: creating tablet
I20250113 02:51:23.782455 895 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c26074fdd804bd79cb0c295622d11ff with delete_type TABLET_DATA_DELETED (replaced by tablet 9fff52e99c64462a8a73fccc3c22ca63) from {username='slave'} at 127.0.0.1:37366
W20250113 02:51:23.784329 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 80d31d5ead804ddaa032969e401a8f7e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 80d31d5ead804ddaa032969e401a8f7e already in progress: creating tablet
W20250113 02:51:23.784710 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 7d53de6379b84103ba905effb0835676 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7d53de6379b84103ba905effb0835676 already in progress: creating tablet
W20250113 02:51:23.785017 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet c1fb2b36c0504cb4ba32671ae4c620d1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c1fb2b36c0504cb4ba32671ae4c620d1 already in progress: creating tablet
W20250113 02:51:23.785331 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 0644f5600c5b47e29f3fa73060ed96e3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0644f5600c5b47e29f3fa73060ed96e3 already in progress: creating tablet
W20250113 02:51:23.785614 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 42cf1e708644427f9d4dcd41ae2bcb37 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 42cf1e708644427f9d4dcd41ae2bcb37 already in progress: creating tablet
I20250113 02:51:23.785624 882 tablet_service.cc:1514] Processing DeleteTablet for tablet be80b631ccd747f1b9355a00787a7b55 with delete_type TABLET_DATA_DELETED (replaced by tablet e70658c90555463498d3fc2e2f13ea7f) from {username='slave'} at 127.0.0.1:37366
W20250113 02:51:23.785902 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 092892a36e0e47cb8d2184c2bbde95d7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 092892a36e0e47cb8d2184c2bbde95d7 already in progress: creating tablet
W20250113 02:51:23.786206 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet b0113b16110e4787ac1c4676b2c8bd1e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b0113b16110e4787ac1c4676b2c8bd1e already in progress: creating tablet
W20250113 02:51:23.786485 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 41fcde988f72496b824271fa36c1a226 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 41fcde988f72496b824271fa36c1a226 already in progress: creating tablet
W20250113 02:51:23.786788 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet e70658c90555463498d3fc2e2f13ea7f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e70658c90555463498d3fc2e2f13ea7f already in progress: creating tablet
W20250113 02:51:23.787084 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 6d63f0e5835540a78c82020b20a07dac because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6d63f0e5835540a78c82020b20a07dac already in progress: creating tablet
I20250113 02:51:23.788628 1198 ts_tablet_manager.cc:1918] T c71511a100f14322b5091a9804720fa0 P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:23.788852 1198 log.cc:1198] T c71511a100f14322b5091a9804720fa0 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/c71511a100f14322b5091a9804720fa0
I20250113 02:51:23.789292 1198 ts_tablet_manager.cc:1939] T c71511a100f14322b5091a9804720fa0 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:23.790202 1192 tablet_bootstrap.cc:492] T d7920c0544c0466bad6bd8c09aae1a7d P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:23.790354 1192 ts_tablet_manager.cc:1397] T d7920c0544c0466bad6bd8c09aae1a7d P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.298s user 0.075s sys 0.122s
I20250113 02:51:23.791127 1192 raft_consensus.cc:357] T d7920c0544c0466bad6bd8c09aae1a7d P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:23.791332 1192 raft_consensus.cc:383] T d7920c0544c0466bad6bd8c09aae1a7d P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:23.791440 1192 raft_consensus.cc:738] T d7920c0544c0466bad6bd8c09aae1a7d P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:23.791695 882 tablet_service.cc:1467] Processing CreateTablet for tablet 4c759831773849f2bc52e484ffa0cf49 (DEFAULT_TABLE table=default.Table_0fcdb63a92324044972d1e69630f524d [id=cc7909c1a59e460a8db262923636f9f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:23.791664 1192 consensus_queue.cc:260] T d7920c0544c0466bad6bd8c09aae1a7d P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:23.792268 1192 ts_tablet_manager.cc:1428] T d7920c0544c0466bad6bd8c09aae1a7d P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
W20250113 02:51:23.792313 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 88a776aa820c44b3a3854ca5022bd353 has no consensus state
I20250113 02:51:23.792611 1192 tablet_bootstrap.cc:492] T 4fe54131aafb40ee9d8bc5ea299af193 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:23.792748 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet dc680b5680b84abbab219296bc7ee707
I20250113 02:51:23.793460 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet a0ca2e3a3ce94f3aa20df64c89c7cf95
I20250113 02:51:23.793886 449 catalog_manager.cc:6596] ReplaceTablet: tablet a0ca2e3a3ce94f3aa20df64c89c7cf95 deleted and replaced by tablet 2470b010b571444eba3f132323038f33
I20250113 02:51:23.795053 895 tablet_service.cc:1514] Processing DeleteTablet for tablet dc680b5680b84abbab219296bc7ee707 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:23 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:23.795436 453 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet c980ce5f4946427cad3e3cb601318ac4
I20250113 02:51:23.795826 895 tablet_service.cc:1514] Processing DeleteTablet for tablet a0ca2e3a3ce94f3aa20df64c89c7cf95 with delete_type TABLET_DATA_DELETED (replaced by tablet 2470b010b571444eba3f132323038f33) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:23.796414 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet f31dd916336a4d8fb3ab015af7de8ccb
I20250113 02:51:23.801038 895 tablet_service.cc:1514] Processing DeleteTablet for tablet f31dd916336a4d8fb3ab015af7de8ccb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:23 UTC) from {username='slave'} at 127.0.0.1:37366
W20250113 02:51:23.801528 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 41fcde988f72496b824271fa36c1a226 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 41fcde988f72496b824271fa36c1a226 already in progress: creating tablet
I20250113 02:51:23.805334 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet c71511a100f14322b5091a9804720fa0 (table default.Table_8606b4caf08b43b18b25ee8849eebd07 [id=85bca730e1274dbebb2f7c2e6eaea7c7]) successfully deleted
I20250113 02:51:23.805580 737 tablet_service.cc:1514] Processing DeleteTablet for tablet a0ca2e3a3ce94f3aa20df64c89c7cf95 with delete_type TABLET_DATA_DELETED (replaced by tablet 2470b010b571444eba3f132323038f33) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:23.805877 1198 tablet_replica.cc:331] T d7c4c7b0cfc44ac1952bc73cbe1bced8 P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:23.805891 742 tablet_service.cc:1514] Processing DeleteTablet for tablet f31dd916336a4d8fb3ab015af7de8ccb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:23 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:23.806084 741 tablet_service.cc:1514] Processing DeleteTablet for tablet dc680b5680b84abbab219296bc7ee707 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:23 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:23.807708 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c759831773849f2bc52e484ffa0cf49. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:23.805356 451 catalog_manager.cc:5618] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587) does not have the latest schema for tablet d7920c0544c0466bad6bd8c09aae1a7d (table default.Table_c9a2c2b1c62849babefee677197da844 [id=05755562cfc24370a3e6fb245a573dd4]). Expected version 1 got 0
W20250113 02:51:23.810910 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet ddf79eec41214693859f526265a55492 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ddf79eec41214693859f526265a55492 already in progress: creating tablet
W20250113 02:51:23.811169 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet f68f9f4076d9475d938e795e72bdd468 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f68f9f4076d9475d938e795e72bdd468 already in progress: creating tablet
I20250113 02:51:23.816919 451 master_service.cc:946] ReplaceTablet: received request to replace tablet c1fb2b36c0504cb4ba32671ae4c620d1 from {username='slave'} at 127.0.0.1:49258
W20250113 02:51:23.821130 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet e70658c90555463498d3fc2e2f13ea7f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e70658c90555463498d3fc2e2f13ea7f already in progress: creating tablet
W20250113 02:51:23.821902 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 6d63f0e5835540a78c82020b20a07dac because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6d63f0e5835540a78c82020b20a07dac already in progress: creating tablet
I20250113 02:51:23.822542 1196 tablet_replica.cc:331] T fc92f19390fd48a4a45ba8fb839020fd P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
W20250113 02:51:23.822829 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet ddf79eec41214693859f526265a55492 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ddf79eec41214693859f526265a55492 already in progress: creating tablet
W20250113 02:51:23.823114 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet f68f9f4076d9475d938e795e72bdd468 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f68f9f4076d9475d938e795e72bdd468 already in progress: creating tablet
I20250113 02:51:23.823508 1196 raft_consensus.cc:2238] T fc92f19390fd48a4a45ba8fb839020fd P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:23.823755 1196 raft_consensus.cc:2267] T fc92f19390fd48a4a45ba8fb839020fd P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:23.826143 741 tablet_service.cc:1467] Processing CreateTablet for tablet fb2458e2044d49029ed2d6851db9de25 (DEFAULT_TABLE table=default.Table_53a85211f49241c4af531e479ecdc0cb [id=ff8b884b04954ff48a7c8a72df232e8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:23.826642 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb2458e2044d49029ed2d6851db9de25. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:23.828809 895 tablet_service.cc:1467] Processing CreateTablet for tablet fb2458e2044d49029ed2d6851db9de25 (DEFAULT_TABLE table=default.Table_53a85211f49241c4af531e479ecdc0cb [id=ff8b884b04954ff48a7c8a72df232e8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:23.829294 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb2458e2044d49029ed2d6851db9de25. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:23.832373 451 master_service.cc:946] ReplaceTablet: received request to replace tablet dc680b5680b84abbab219296bc7ee707 from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:23.837026 1198 raft_consensus.cc:2238] T d7c4c7b0cfc44ac1952bc73cbe1bced8 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: Raft consensus shutting down.
I20250113 02:51:23.839831 1039 tablet_service.cc:1514] Processing DeleteTablet for tablet dc680b5680b84abbab219296bc7ee707 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:23 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:23.840034 1192 tablet_bootstrap.cc:654] T 4fe54131aafb40ee9d8bc5ea299af193 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:23.840373 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet a0ca2e3a3ce94f3aa20df64c89c7cf95 with delete_type TABLET_DATA_DELETED (replaced by tablet 2470b010b571444eba3f132323038f33) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:23.841368 1196 ts_tablet_manager.cc:1905] T fc92f19390fd48a4a45ba8fb839020fd P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:23.844699 1198 pending_rounds.cc:62] T d7c4c7b0cfc44ac1952bc73cbe1bced8 P 4216a7f853d642b7bc3e5cff458f3aca: Trying to abort 1 pending ops.
I20250113 02:51:23.844844 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet f31dd916336a4d8fb3ab015af7de8ccb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:23 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:23.845577 1028 tablet_service.cc:1467] Processing CreateTablet for tablet fb2458e2044d49029ed2d6851db9de25 (DEFAULT_TABLE table=default.Table_53a85211f49241c4af531e479ecdc0cb [id=ff8b884b04954ff48a7c8a72df232e8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:23.846460 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb2458e2044d49029ed2d6851db9de25. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:23.848953 451 master_service.cc:946] ReplaceTablet: received request to replace tablet ee240d838d4a4bbba97f8ed61f69de78 from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:23.849823 1198 pending_rounds.cc:69] T d7c4c7b0cfc44ac1952bc73cbe1bced8 P 4216a7f853d642b7bc3e5cff458f3aca: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113673455088713728 op_type: NO_OP noop_request { }
I20250113 02:51:23.849982 1198 raft_consensus.cc:2883] T d7c4c7b0cfc44ac1952bc73cbe1bced8 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250113 02:51:23.850342 1198 raft_consensus.cc:2267] T d7c4c7b0cfc44ac1952bc73cbe1bced8 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:23.851025 1034 tablet_service.cc:1467] Processing CreateTablet for tablet 4c759831773849f2bc52e484ffa0cf49 (DEFAULT_TABLE table=default.Table_0fcdb63a92324044972d1e69630f524d [id=cc7909c1a59e460a8db262923636f9f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:23.851855 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c759831773849f2bc52e484ffa0cf49. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:23.853575 1191 tablet_bootstrap.cc:492] T 6c8ca30720664f3a819c4bff2493934d P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:23.853817 1191 ts_tablet_manager.cc:1397] T 6c8ca30720664f3a819c4bff2493934d P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.248s user 0.003s sys 0.001s
I20250113 02:51:23.854574 1191 raft_consensus.cc:357] T 6c8ca30720664f3a819c4bff2493934d P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:23.854902 1191 raft_consensus.cc:383] T 6c8ca30720664f3a819c4bff2493934d P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:23.855033 1191 raft_consensus.cc:738] T 6c8ca30720664f3a819c4bff2493934d P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:23.855327 1191 consensus_queue.cc:260] T 6c8ca30720664f3a819c4bff2493934d P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:23.856220 1198 ts_tablet_manager.cc:1905] T d7c4c7b0cfc44ac1952bc73cbe1bced8 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:23.858287 1191 ts_tablet_manager.cc:1428] T 6c8ca30720664f3a819c4bff2493934d P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250113 02:51:23.860814 1191 tablet_bootstrap.cc:492] T fc92f19390fd48a4a45ba8fb839020fd P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:23.862677 1191 tablet_bootstrap.cc:654] T fc92f19390fd48a4a45ba8fb839020fd P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:23.865546 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet ee240d838d4a4bbba97f8ed61f69de78
I20250113 02:51:23.866047 451 catalog_manager.cc:6596] ReplaceTablet: tablet ee240d838d4a4bbba97f8ed61f69de78 deleted and replaced by tablet 95e47db8a4d74615800ba6c049ec7131
I20250113 02:51:23.869156 882 tablet_service.cc:1514] Processing DeleteTablet for tablet ee240d838d4a4bbba97f8ed61f69de78 with delete_type TABLET_DATA_DELETED (replaced by tablet 95e47db8a4d74615800ba6c049ec7131) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:23.869451 1198 ts_tablet_manager.cc:1918] T d7c4c7b0cfc44ac1952bc73cbe1bced8 P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250113 02:51:23.869575 1198 log.cc:1198] T d7c4c7b0cfc44ac1952bc73cbe1bced8 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/d7c4c7b0cfc44ac1952bc73cbe1bced8
I20250113 02:51:23.869992 1198 ts_tablet_manager.cc:1939] T d7c4c7b0cfc44ac1952bc73cbe1bced8 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:23.869447 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet ee240d838d4a4bbba97f8ed61f69de78 with delete_type TABLET_DATA_DELETED (replaced by tablet 95e47db8a4d74615800ba6c049ec7131) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:23.872884 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet d7c4c7b0cfc44ac1952bc73cbe1bced8 (table default.Table_5727b5d8bf51442e8ae5f7c97f4bfbc8 [id=0c69597e8c5b4ec89179169a7a84e048]) successfully deleted
I20250113 02:51:23.874286 1198 tablet_replica.cc:331] T 1c26074fdd804bd79cb0c295622d11ff P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:23.874501 1198 raft_consensus.cc:2238] T 1c26074fdd804bd79cb0c295622d11ff P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:23.874655 1198 raft_consensus.cc:2267] T 1c26074fdd804bd79cb0c295622d11ff P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:23.875182 1198 ts_tablet_manager.cc:1905] T 1c26074fdd804bd79cb0c295622d11ff P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:23.879866 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_b77c072e465c474db9349f1a3b31c323" } modify_external_catalogs: true
I20250113 02:51:23.880080 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_b77c072e465c474db9349f1a3b31c323" } modify_external_catalogs: true
I20250113 02:51:23.883833 1196 ts_tablet_manager.cc:1918] T fc92f19390fd48a4a45ba8fb839020fd P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:23.867187 741 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c8ca30720664f3a819c4bff2493934d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:18 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:23.868934 748 tablet_service.cc:1514] Processing DeleteTablet for tablet ee240d838d4a4bbba97f8ed61f69de78 with delete_type TABLET_DATA_DELETED (replaced by tablet 95e47db8a4d74615800ba6c049ec7131) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:23.886755 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_50f21224ee254e3bb13748a44e257783"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:23.887334 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_50f21224ee254e3bb13748a44e257783 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:23.890599 1192 tablet_bootstrap.cc:492] T 4fe54131aafb40ee9d8bc5ea299af193 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:23.890798 1192 ts_tablet_manager.cc:1397] T 4fe54131aafb40ee9d8bc5ea299af193 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.098s user 0.003s sys 0.007s
I20250113 02:51:23.891609 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet b8b6ffe690fc459db78b2b3bc12671dd
I20250113 02:51:23.892716 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet b8b6ffe690fc459db78b2b3bc12671dd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:23 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:23.893224 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_8be7ebd7e1984b979fce6e7aac1f576e" } modify_external_catalogs: true
I20250113 02:51:23.893433 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_8be7ebd7e1984b979fce6e7aac1f576e" } modify_external_catalogs: true
I20250113 02:51:23.893513 748 tablet_service.cc:1514] Processing DeleteTablet for tablet b8b6ffe690fc459db78b2b3bc12671dd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:23 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:23.893997 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_0b855cf933544dce95621708e777674f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:23.894215 1192 raft_consensus.cc:357] T 4fe54131aafb40ee9d8bc5ea299af193 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:23.894423 1192 raft_consensus.cc:383] T 4fe54131aafb40ee9d8bc5ea299af193 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:23.894481 1192 raft_consensus.cc:738] T 4fe54131aafb40ee9d8bc5ea299af193 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
W20250113 02:51:23.894505 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0b855cf933544dce95621708e777674f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:23.894701 1192 consensus_queue.cc:260] T 4fe54131aafb40ee9d8bc5ea299af193 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:23.895308 1192 ts_tablet_manager.cc:1428] T 4fe54131aafb40ee9d8bc5ea299af193 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250113 02:51:23.895586 1192 tablet_bootstrap.cc:492] T 93a2729c5a604a27a9bbc451ccb4e128 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:23.899684 1192 tablet_bootstrap.cc:654] T 93a2729c5a604a27a9bbc451ccb4e128 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:23.893754 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_96651718e3f643fcacec8ef593a9b30e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:23.900573 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_96651718e3f643fcacec8ef593a9b30e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:23.877792 1195 ts_tablet_manager.cc:1918] T 21c4b419bf2148e0b140301fa80622bb P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:23.900754 1195 log.cc:1198] T 21c4b419bf2148e0b140301fa80622bb P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/21c4b419bf2148e0b140301fa80622bb
I20250113 02:51:23.901157 1195 ts_tablet_manager.cc:1939] T 21c4b419bf2148e0b140301fa80622bb P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:23.902179 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet 21c4b419bf2148e0b140301fa80622bb (table default.Table_4a0258998d4f413b80d701c74526b86d [id=6fd9215b11274a76b06813227889927b]) successfully deleted
W20250113 02:51:23.903874 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet a0ca2e3a3ce94f3aa20df64c89c7cf95 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a0ca2e3a3ce94f3aa20df64c89c7cf95 already in progress: creating tablet
W20250113 02:51:23.905354 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet f31dd916336a4d8fb3ab015af7de8ccb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f31dd916336a4d8fb3ab015af7de8ccb already in progress: creating tablet
I20250113 02:51:23.907153 1198 ts_tablet_manager.cc:1918] T 1c26074fdd804bd79cb0c295622d11ff P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250113 02:51:23.907585 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet dc680b5680b84abbab219296bc7ee707 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dc680b5680b84abbab219296bc7ee707 already in progress: creating tablet
I20250113 02:51:23.908115 1195 tablet_replica.cc:331] T 6c8ca30720664f3a819c4bff2493934d P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:23.908360 1195 raft_consensus.cc:2238] T 6c8ca30720664f3a819c4bff2493934d P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:23.908571 1195 raft_consensus.cc:2267] T 6c8ca30720664f3a819c4bff2493934d P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:23.909405 1195 ts_tablet_manager.cc:1905] T 6c8ca30720664f3a819c4bff2493934d P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:23.909950 1198 log.cc:1198] T 1c26074fdd804bd79cb0c295622d11ff P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/1c26074fdd804bd79cb0c295622d11ff
I20250113 02:51:23.910538 1198 ts_tablet_manager.cc:1939] T 1c26074fdd804bd79cb0c295622d11ff P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:23.910784 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_2c38174be1e04983a19a30e0076b466c" } new_table_name: "default.Table_a7a5a9569d064b488facac137ec35a85" modify_external_catalogs: true
I20250113 02:51:23.912184 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_059d10dd368b428db10509d40437287b" } new_table_name: "default.Table_2cb862a84f3f44c5bcfb646461a5af1c" modify_external_catalogs: true
I20250113 02:51:23.917361 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet 1c26074fdd804bd79cb0c295622d11ff (table default.Table_9ef18ee71ad843c5921750a2f8478855 [id=bd8bd572ed524f2cae01f820f07e3c36]) successfully deleted
W20250113 02:51:23.918011 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 883027967e0c442a91ac820210b9c6c7 has no consensus state
I20250113 02:51:23.918548 1198 tablet_replica.cc:331] T be80b631ccd747f1b9355a00787a7b55 P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:23.919064 1198 raft_consensus.cc:2238] T be80b631ccd747f1b9355a00787a7b55 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:23.919373 1198 raft_consensus.cc:2267] T be80b631ccd747f1b9355a00787a7b55 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:23.919997 1198 ts_tablet_manager.cc:1905] T be80b631ccd747f1b9355a00787a7b55 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:23.893780 882 tablet_service.cc:1514] Processing DeleteTablet for tablet b8b6ffe690fc459db78b2b3bc12671dd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:23 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:23.920270 1196 log.cc:1198] T fc92f19390fd48a4a45ba8fb839020fd P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/fc92f19390fd48a4a45ba8fb839020fd
I20250113 02:51:23.918259 1194 tablet_bootstrap.cc:492] T 317c6e8013b745c290bf3190dc237ab8 P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:23.920558 1194 ts_tablet_manager.cc:1397] T 317c6e8013b745c290bf3190dc237ab8 P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.154s user 0.027s sys 0.042s
W20250113 02:51:23.921131 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 438604e8859f422f9caccb61c5b5b125 consensus state has no leader
I20250113 02:51:23.921591 1194 raft_consensus.cc:357] T 317c6e8013b745c290bf3190dc237ab8 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:23.921890 1194 raft_consensus.cc:383] T 317c6e8013b745c290bf3190dc237ab8 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:23.921972 1194 raft_consensus.cc:738] T 317c6e8013b745c290bf3190dc237ab8 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:23.922240 1194 consensus_queue.cc:260] T 317c6e8013b745c290bf3190dc237ab8 P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:23.922772 1194 ts_tablet_manager.cc:1428] T 317c6e8013b745c290bf3190dc237ab8 P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250113 02:51:23.923213 1194 tablet_bootstrap.cc:492] T d3b695ac20ba4205b2d8732db4740531 P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:23.926683 1194 tablet_bootstrap.cc:654] T d3b695ac20ba4205b2d8732db4740531 P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:23.926667 1198 ts_tablet_manager.cc:1918] T be80b631ccd747f1b9355a00787a7b55 P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:23.926795 1198 log.cc:1198] T be80b631ccd747f1b9355a00787a7b55 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/be80b631ccd747f1b9355a00787a7b55
I20250113 02:51:23.927160 1198 ts_tablet_manager.cc:1939] T be80b631ccd747f1b9355a00787a7b55 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:23.928187 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet be80b631ccd747f1b9355a00787a7b55 (table default.Table_2565c0551f9943a084ace8f7e463e59d [id=fe6f8d302c3e489d8bb832503fd05642]) successfully deleted
I20250113 02:51:23.928746 1192 tablet_bootstrap.cc:492] T 93a2729c5a604a27a9bbc451ccb4e128 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:23.928907 1192 ts_tablet_manager.cc:1397] T 93a2729c5a604a27a9bbc451ccb4e128 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.033s user 0.002s sys 0.001s
I20250113 02:51:23.929741 1192 raft_consensus.cc:357] T 93a2729c5a604a27a9bbc451ccb4e128 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:23.929946 1192 raft_consensus.cc:383] T 93a2729c5a604a27a9bbc451ccb4e128 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:23.930033 1192 raft_consensus.cc:738] T 93a2729c5a604a27a9bbc451ccb4e128 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:23.933161 1191 tablet_bootstrap.cc:492] T fc92f19390fd48a4a45ba8fb839020fd P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:23.933418 1191 ts_tablet_manager.cc:1397] T fc92f19390fd48a4a45ba8fb839020fd P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.073s user 0.002s sys 0.002s
I20250113 02:51:23.934505 1191 raft_consensus.cc:357] T fc92f19390fd48a4a45ba8fb839020fd P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:23.934825 1191 raft_consensus.cc:383] T fc92f19390fd48a4a45ba8fb839020fd P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:23.934973 1191 raft_consensus.cc:738] T fc92f19390fd48a4a45ba8fb839020fd P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:23.935321 1191 consensus_queue.cc:260] T fc92f19390fd48a4a45ba8fb839020fd P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:23.938490 1191 ts_tablet_manager.cc:1428] T fc92f19390fd48a4a45ba8fb839020fd P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250113 02:51:23.938946 1196 ts_tablet_manager.cc:1939] T fc92f19390fd48a4a45ba8fb839020fd P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:23.939294 1191 tablet_bootstrap.cc:492] T 317c6e8013b745c290bf3190dc237ab8 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:23.940512 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet fc92f19390fd48a4a45ba8fb839020fd (table default.Table_c94783b201764702b6de4f3219090efe [id=285c6d11c414470f8c40eefdf6a8a9e8]) successfully deleted
I20250113 02:51:23.943195 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_53bb556171424601915f305c106a2dc4" } new_table_name: "default.Table_066f136aea8141fa91862f7285fbaac2" modify_external_catalogs: true
I20250113 02:51:23.942924 1196 tablet_replica.cc:331] T 93a2729c5a604a27a9bbc451ccb4e128 P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:23.945118 1196 raft_consensus.cc:2238] T 93a2729c5a604a27a9bbc451ccb4e128 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:23.945158 1192 consensus_queue.cc:260] T 93a2729c5a604a27a9bbc451ccb4e128 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:23.944804 1194 tablet_bootstrap.cc:492] T d3b695ac20ba4205b2d8732db4740531 P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:23.945804 1194 ts_tablet_manager.cc:1397] T d3b695ac20ba4205b2d8732db4740531 P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.023s user 0.003s sys 0.002s
I20250113 02:51:23.946692 1194 raft_consensus.cc:357] T d3b695ac20ba4205b2d8732db4740531 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:23.946985 1194 raft_consensus.cc:383] T d3b695ac20ba4205b2d8732db4740531 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:23.947126 1194 raft_consensus.cc:738] T d3b695ac20ba4205b2d8732db4740531 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:23.948858 1194 consensus_queue.cc:260] T d3b695ac20ba4205b2d8732db4740531 P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:23.949334 1192 ts_tablet_manager.cc:1428] T 93a2729c5a604a27a9bbc451ccb4e128 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.020s user 0.001s sys 0.001s
I20250113 02:51:23.949620 1194 ts_tablet_manager.cc:1428] T d3b695ac20ba4205b2d8732db4740531 P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
W20250113 02:51:23.949875 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4b07a985897641c3ac48021c2a4daedb consensus state has no leader
I20250113 02:51:23.949975 1194 tablet_bootstrap.cc:492] T 21c4b419bf2148e0b140301fa80622bb P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:23.952054 1196 raft_consensus.cc:2267] T 93a2729c5a604a27a9bbc451ccb4e128 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:23.952833 1191 tablet_bootstrap.cc:654] T 317c6e8013b745c290bf3190dc237ab8 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:23.953119 1192 tablet_bootstrap.cc:492] T f1a02c37cb4f43f8a9de6484f4431958 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:23.953433 1196 ts_tablet_manager.cc:1905] T 93a2729c5a604a27a9bbc451ccb4e128 P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250113 02:51:23.956811 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 999fecb3fc65466cad21cc1ff5ef2d44 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 999fecb3fc65466cad21cc1ff5ef2d44 already in progress: creating tablet
I20250113 02:51:23.956241 1192 tablet_bootstrap.cc:654] T f1a02c37cb4f43f8a9de6484f4431958 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:23.957886 748 tablet_service.cc:1514] Processing DeleteTablet for tablet fc92f19390fd48a4a45ba8fb839020fd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:17 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:23.958158 446 master_service.cc:946] ReplaceTablet: received request to replace tablet ebc5f1ea1e1648d49904b17603d3cd2a from {username='slave'} at 127.0.0.1:49258
W20250113 02:51:23.958130 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 99c8046f5d124ca49899df365114c69f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 99c8046f5d124ca49899df365114c69f already in progress: creating tablet
W20250113 02:51:23.958817 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 9f8eb43a39774d2c8e9cb4e88a405b0f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9f8eb43a39774d2c8e9cb4e88a405b0f already in progress: creating tablet
W20250113 02:51:23.960987 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet c518f791b4d74c54ab0c0364d29b4a64 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c518f791b4d74c54ab0c0364d29b4a64 already in progress: creating tablet
I20250113 02:51:23.961346 1198 tablet_replica.cc:331] T 4efa63b8be794e1db4b340f4c0c4cd5a P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:23.961609 1198 raft_consensus.cc:2238] T 4efa63b8be794e1db4b340f4c0c4cd5a P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:23.961815 1198 raft_consensus.cc:2267] T 4efa63b8be794e1db4b340f4c0c4cd5a P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:23.962692 1198 ts_tablet_manager.cc:1905] T 4efa63b8be794e1db4b340f4c0c4cd5a P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:23.971061 1196 ts_tablet_manager.cc:1918] T 93a2729c5a604a27a9bbc451ccb4e128 P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:23.971050 1198 ts_tablet_manager.cc:1918] T 4efa63b8be794e1db4b340f4c0c4cd5a P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:23.971215 1198 log.cc:1198] T 4efa63b8be794e1db4b340f4c0c4cd5a P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/4efa63b8be794e1db4b340f4c0c4cd5a
I20250113 02:51:23.976804 1198 ts_tablet_manager.cc:1939] T 4efa63b8be794e1db4b340f4c0c4cd5a P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:23.978917 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet 4efa63b8be794e1db4b340f4c0c4cd5a (table default.Table_6a0195b054a043269e133f6ed8dd5979 [id=4b2bd999368c4b628c70cb3ee8c2b1f3]) successfully deleted
I20250113 02:51:23.979861 1191 tablet_bootstrap.cc:492] T 317c6e8013b745c290bf3190dc237ab8 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:23.980187 1191 ts_tablet_manager.cc:1397] T 317c6e8013b745c290bf3190dc237ab8 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.041s user 0.002s sys 0.002s
W20250113 02:51:23.980933 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet dc680b5680b84abbab219296bc7ee707 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dc680b5680b84abbab219296bc7ee707 already in progress: creating tablet
I20250113 02:51:23.981210 1191 raft_consensus.cc:357] T 317c6e8013b745c290bf3190dc237ab8 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:23.981659 1191 raft_consensus.cc:383] T 317c6e8013b745c290bf3190dc237ab8 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:23.981801 1191 raft_consensus.cc:738] T 317c6e8013b745c290bf3190dc237ab8 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
W20250113 02:51:23.982056 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet a0ca2e3a3ce94f3aa20df64c89c7cf95 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a0ca2e3a3ce94f3aa20df64c89c7cf95 already in progress: creating tablet
I20250113 02:51:23.982228 1191 consensus_queue.cc:260] T 317c6e8013b745c290bf3190dc237ab8 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:23.982336 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_4dae6fdf9a1e4400b85b31bf8fcc172c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:23.982942 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4dae6fdf9a1e4400b85b31bf8fcc172c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250113 02:51:23.982976 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet f31dd916336a4d8fb3ab015af7de8ccb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f31dd916336a4d8fb3ab015af7de8ccb already in progress: creating tablet
I20250113 02:51:23.983533 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_4f36a72976b7408299b97e28dce161b8" } new_table_name: "default.Table_b2598c65e9d04d8aa594a9f3d61d1e5a" modify_external_catalogs: true
W20250113 02:51:23.984154 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet ee240d838d4a4bbba97f8ed61f69de78 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ee240d838d4a4bbba97f8ed61f69de78 already in progress: creating tablet
I20250113 02:51:23.984324 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_5181e98c2c214ebcab54c33615482528"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:23.984791 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5181e98c2c214ebcab54c33615482528 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250113 02:51:23.985172 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet b8b6ffe690fc459db78b2b3bc12671dd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b8b6ffe690fc459db78b2b3bc12671dd already in progress: creating tablet
I20250113 02:51:23.985297 1191 ts_tablet_manager.cc:1428] T 317c6e8013b745c290bf3190dc237ab8 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250113 02:51:23.985724 1191 tablet_bootstrap.cc:492] T d7c4c7b0cfc44ac1952bc73cbe1bced8 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:23.986387 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_41d844ed7a144e59bd9e14d543ad0573"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:23.986886 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_41d844ed7a144e59bd9e14d543ad0573 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:23.990160 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_2d2546458fac45b79308ff4e7f043292"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:23.990666 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2d2546458fac45b79308ff4e7f043292 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:23.990875 1195 ts_tablet_manager.cc:1918] T 6c8ca30720664f3a819c4bff2493934d P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:23.991029 1195 log.cc:1198] T 6c8ca30720664f3a819c4bff2493934d P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/6c8ca30720664f3a819c4bff2493934d
I20250113 02:51:23.991451 1195 ts_tablet_manager.cc:1939] T 6c8ca30720664f3a819c4bff2493934d P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
W20250113 02:51:23.998452 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 542df00229f645788f1b821c0ce00c46 has no consensus state
I20250113 02:51:23.998955 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_064659c197854029a5d7af1798def6d9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:23.999449 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_064659c197854029a5d7af1798def6d9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:23.999792 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_e3775884d0c7400ab769befa718e8fc7" } new_table_name: "default.Table_c1b46fe79c2e4aabbb2a97139f4078c3" modify_external_catalogs: true
I20250113 02:51:24.005306 1192 tablet_bootstrap.cc:492] T f1a02c37cb4f43f8a9de6484f4431958 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:24.005513 1192 ts_tablet_manager.cc:1397] T f1a02c37cb4f43f8a9de6484f4431958 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.052s user 0.015s sys 0.005s
I20250113 02:51:24.005921 446 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_a7c3697e25a94815ac03a3cb1a57bc0c" } modify_external_catalogs: true
I20250113 02:51:24.006117 446 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_a7c3697e25a94815ac03a3cb1a57bc0c" } modify_external_catalogs: true
I20250113 02:51:24.006410 1192 raft_consensus.cc:357] T f1a02c37cb4f43f8a9de6484f4431958 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:24.006670 1192 raft_consensus.cc:383] T f1a02c37cb4f43f8a9de6484f4431958 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:24.006749 1192 raft_consensus.cc:738] T f1a02c37cb4f43f8a9de6484f4431958 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:24.007023 1192 consensus_queue.cc:260] T f1a02c37cb4f43f8a9de6484f4431958 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:24.009865 1192 ts_tablet_manager.cc:1428] T f1a02c37cb4f43f8a9de6484f4431958 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.004s user 0.001s sys 0.002s
I20250113 02:51:24.010227 1192 tablet_bootstrap.cc:492] T 21c4b419bf2148e0b140301fa80622bb P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:24.012127 1192 tablet_bootstrap.cc:654] T 21c4b419bf2148e0b140301fa80622bb P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:24.012521 1191 tablet_bootstrap.cc:654] T d7c4c7b0cfc44ac1952bc73cbe1bced8 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:24.014279 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet 6c8ca30720664f3a819c4bff2493934d (table default.Table_8aa63512c7bb4bdb86f37c9ee24df209 [id=aadd55280b654ceb96c290786a0bfce0]) successfully deleted
I20250113 02:51:23.972023 1194 tablet_bootstrap.cc:654] T 21c4b419bf2148e0b140301fa80622bb P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:24.016602 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_59401ddb940e4e789a5bd54f96548ba5" } modify_external_catalogs: true
I20250113 02:51:24.016783 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_59401ddb940e4e789a5bd54f96548ba5" } modify_external_catalogs: true
I20250113 02:51:24.017809 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_96fc1675456247f2a1c6d67ce524f203"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:24.018064 1196 log.cc:1198] T 93a2729c5a604a27a9bbc451ccb4e128 P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/93a2729c5a604a27a9bbc451ccb4e128
W20250113 02:51:24.018347 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_96fc1675456247f2a1c6d67ce524f203 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:24.018610 1196 ts_tablet_manager.cc:1939] T 93a2729c5a604a27a9bbc451ccb4e128 P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:24.020081 1196 tablet_replica.cc:331] T 1c26074fdd804bd79cb0c295622d11ff P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:24.020421 1196 raft_consensus.cc:2238] T 1c26074fdd804bd79cb0c295622d11ff P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:24.021445 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet 93a2729c5a604a27a9bbc451ccb4e128 (table default.Table_4f1f306815b145218e24c93821376721 [id=5d1feb17ef344c9287e571496b2b405f]) successfully deleted
W20250113 02:51:24.021692 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 99c8046f5d124ca49899df365114c69f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 99c8046f5d124ca49899df365114c69f already in progress: creating tablet
W20250113 02:51:24.021884 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 999fecb3fc65466cad21cc1ff5ef2d44 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 999fecb3fc65466cad21cc1ff5ef2d44 already in progress: creating tablet
W20250113 02:51:24.022078 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet c518f791b4d74c54ab0c0364d29b4a64 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c518f791b4d74c54ab0c0364d29b4a64 already in progress: creating tablet
W20250113 02:51:24.022270 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 9f8eb43a39774d2c8e9cb4e88a405b0f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9f8eb43a39774d2c8e9cb4e88a405b0f already in progress: creating tablet
I20250113 02:51:24.024247 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_02655512f28b47429ebb70e1676844ea"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:24.024745 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_02655512f28b47429ebb70e1676844ea in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250113 02:51:24.025946 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet ee240d838d4a4bbba97f8ed61f69de78 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ee240d838d4a4bbba97f8ed61f69de78 already in progress: creating tablet
W20250113 02:51:24.029426 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet b8b6ffe690fc459db78b2b3bc12671dd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b8b6ffe690fc459db78b2b3bc12671dd already in progress: creating tablet
I20250113 02:51:24.032810 1195 tablet_replica.cc:331] T fc92f19390fd48a4a45ba8fb839020fd P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:24.033105 1195 raft_consensus.cc:2238] T fc92f19390fd48a4a45ba8fb839020fd P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus shutting down.
W20250113 02:51:24.033169 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 32cba8f3efc54e78b4df41c15df82eea consensus state has no leader
I20250113 02:51:24.033345 1195 raft_consensus.cc:2267] T fc92f19390fd48a4a45ba8fb839020fd P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:24.033499 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_c6c64517b49e4843ba63567a871d22bd" } modify_external_catalogs: true
I20250113 02:51:24.033675 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_c6c64517b49e4843ba63567a871d22bd" } modify_external_catalogs: true
I20250113 02:51:24.034322 1195 ts_tablet_manager.cc:1905] T fc92f19390fd48a4a45ba8fb839020fd P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:24.035138 1442 raft_consensus.cc:491] T f1a02c37cb4f43f8a9de6484f4431958 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:24.035321 1442 raft_consensus.cc:513] T f1a02c37cb4f43f8a9de6484f4431958 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:24.036060 1442 leader_election.cc:290] T f1a02c37cb4f43f8a9de6484f4431958 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
I20250113 02:51:24.037456 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1a02c37cb4f43f8a9de6484f4431958" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
W20250113 02:51:24.038327 995 leader_election.cc:343] T f1a02c37cb4f43f8a9de6484f4431958 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:24.038885 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1a02c37cb4f43f8a9de6484f4431958" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:24.039548 1192 tablet_bootstrap.cc:492] T 21c4b419bf2148e0b140301fa80622bb P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
W20250113 02:51:24.039582 998 leader_election.cc:343] T f1a02c37cb4f43f8a9de6484f4431958 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:24.039736 998 leader_election.cc:304] T f1a02c37cb4f43f8a9de6484f4431958 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4216a7f853d642b7bc3e5cff458f3aca; no voters: 1186fa869eb14812832fa6562531f196, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:24.039805 1194 tablet_bootstrap.cc:492] T 21c4b419bf2148e0b140301fa80622bb P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:24.040004 1194 ts_tablet_manager.cc:1397] T 21c4b419bf2148e0b140301fa80622bb P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.090s user 0.002s sys 0.005s
I20250113 02:51:24.040176 1442 raft_consensus.cc:2743] T f1a02c37cb4f43f8a9de6484f4431958 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:24.041424 1194 raft_consensus.cc:357] T 21c4b419bf2148e0b140301fa80622bb P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:24.041697 1194 raft_consensus.cc:383] T 21c4b419bf2148e0b140301fa80622bb P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:24.041793 1194 raft_consensus.cc:738] T 21c4b419bf2148e0b140301fa80622bb P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:24.042099 1194 consensus_queue.cc:260] T 21c4b419bf2148e0b140301fa80622bb P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:24.039733 1192 ts_tablet_manager.cc:1397] T 21c4b419bf2148e0b140301fa80622bb P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.030s user 0.004s sys 0.016s
I20250113 02:51:24.042742 1194 ts_tablet_manager.cc:1428] T 21c4b419bf2148e0b140301fa80622bb P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.003s user 0.000s sys 0.000s
I20250113 02:51:24.043166 1194 tablet_bootstrap.cc:492] T b0361f11d0974ed19e7d4eb1c32a2f47 P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:24.044524 1192 raft_consensus.cc:357] T 21c4b419bf2148e0b140301fa80622bb P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:24.044771 1192 raft_consensus.cc:383] T 21c4b419bf2148e0b140301fa80622bb P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:24.044869 1192 raft_consensus.cc:738] T 21c4b419bf2148e0b140301fa80622bb P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:24.045054 1194 tablet_bootstrap.cc:654] T b0361f11d0974ed19e7d4eb1c32a2f47 P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:24.045115 1192 consensus_queue.cc:260] T 21c4b419bf2148e0b140301fa80622bb P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:24.046742 1191 tablet_bootstrap.cc:492] T d7c4c7b0cfc44ac1952bc73cbe1bced8 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:24.046953 1191 ts_tablet_manager.cc:1397] T d7c4c7b0cfc44ac1952bc73cbe1bced8 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.061s user 0.002s sys 0.003s
I20250113 02:51:24.047736 1192 ts_tablet_manager.cc:1428] T 21c4b419bf2148e0b140301fa80622bb P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250113 02:51:24.048101 1192 tablet_bootstrap.cc:492] T b834f0acf5314bcb9d81c73de1b31842 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:24.048048 1191 raft_consensus.cc:357] T d7c4c7b0cfc44ac1952bc73cbe1bced8 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:24.048409 1191 raft_consensus.cc:383] T d7c4c7b0cfc44ac1952bc73cbe1bced8 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:24.048578 1191 raft_consensus.cc:738] T d7c4c7b0cfc44ac1952bc73cbe1bced8 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:24.048926 1191 consensus_queue.cc:260] T d7c4c7b0cfc44ac1952bc73cbe1bced8 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:24.051440 1192 tablet_bootstrap.cc:654] T b834f0acf5314bcb9d81c73de1b31842 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:24.052264 1191 ts_tablet_manager.cc:1428] T d7c4c7b0cfc44ac1952bc73cbe1bced8 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20250113 02:51:24.054816 1191 tablet_bootstrap.cc:492] T 34ab7ee573284550aaac9f814e8bd7ab P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:24.055521 1195 ts_tablet_manager.cc:1918] T fc92f19390fd48a4a45ba8fb839020fd P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:24.055661 1195 log.cc:1198] T fc92f19390fd48a4a45ba8fb839020fd P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/fc92f19390fd48a4a45ba8fb839020fd
I20250113 02:51:24.056121 1195 ts_tablet_manager.cc:1939] T fc92f19390fd48a4a45ba8fb839020fd P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:24.057271 1191 tablet_bootstrap.cc:654] T 34ab7ee573284550aaac9f814e8bd7ab P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:24.057497 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet fc92f19390fd48a4a45ba8fb839020fd (table default.Table_c94783b201764702b6de4f3219090efe [id=285c6d11c414470f8c40eefdf6a8a9e8]) successfully deleted
I20250113 02:51:24.060206 1196 raft_consensus.cc:2267] T 1c26074fdd804bd79cb0c295622d11ff P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:24.061241 1196 ts_tablet_manager.cc:1905] T 1c26074fdd804bd79cb0c295622d11ff P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:24.061576 1192 tablet_bootstrap.cc:492] T b834f0acf5314bcb9d81c73de1b31842 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:24.061759 1192 ts_tablet_manager.cc:1397] T b834f0acf5314bcb9d81c73de1b31842 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.000s
I20250113 02:51:24.062686 1192 raft_consensus.cc:357] T b834f0acf5314bcb9d81c73de1b31842 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:24.062956 1192 raft_consensus.cc:383] T b834f0acf5314bcb9d81c73de1b31842 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:24.063045 1192 raft_consensus.cc:738] T b834f0acf5314bcb9d81c73de1b31842 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:24.063328 1192 consensus_queue.cc:260] T b834f0acf5314bcb9d81c73de1b31842 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:24.064059 1192 ts_tablet_manager.cc:1428] T b834f0acf5314bcb9d81c73de1b31842 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250113 02:51:24.064414 1192 tablet_bootstrap.cc:492] T fc92f19390fd48a4a45ba8fb839020fd P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:24.066352 1192 tablet_bootstrap.cc:654] T fc92f19390fd48a4a45ba8fb839020fd P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:24.074395 1192 tablet_bootstrap.cc:492] T fc92f19390fd48a4a45ba8fb839020fd P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:24.074558 1192 ts_tablet_manager.cc:1397] T fc92f19390fd48a4a45ba8fb839020fd P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.005s
I20250113 02:51:24.075392 1192 raft_consensus.cc:357] T fc92f19390fd48a4a45ba8fb839020fd P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:24.075595 1192 raft_consensus.cc:383] T fc92f19390fd48a4a45ba8fb839020fd P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:24.075690 1192 raft_consensus.cc:738] T fc92f19390fd48a4a45ba8fb839020fd P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:24.075901 1192 consensus_queue.cc:260] T fc92f19390fd48a4a45ba8fb839020fd P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:24.076545 1192 ts_tablet_manager.cc:1428] T fc92f19390fd48a4a45ba8fb839020fd P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250113 02:51:24.076804 1192 tablet_bootstrap.cc:492] T 157905316fc545b6a7864ae1c4a2a584 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:24.079615 1194 tablet_bootstrap.cc:492] T b0361f11d0974ed19e7d4eb1c32a2f47 P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:24.079814 1194 ts_tablet_manager.cc:1397] T b0361f11d0974ed19e7d4eb1c32a2f47 P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.037s user 0.000s sys 0.004s
I20250113 02:51:24.080304 1192 tablet_bootstrap.cc:654] T 157905316fc545b6a7864ae1c4a2a584 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:24.080741 1194 raft_consensus.cc:357] T b0361f11d0974ed19e7d4eb1c32a2f47 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:24.086239 1194 raft_consensus.cc:383] T b0361f11d0974ed19e7d4eb1c32a2f47 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:24.086444 1194 raft_consensus.cc:738] T b0361f11d0974ed19e7d4eb1c32a2f47 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:24.086803 1194 consensus_queue.cc:260] T b0361f11d0974ed19e7d4eb1c32a2f47 P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:24.087575 1194 ts_tablet_manager.cc:1428] T b0361f11d0974ed19e7d4eb1c32a2f47 P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20250113 02:51:24.087947 1194 tablet_bootstrap.cc:492] T 438d1d2300e044f39a15586f335935ba P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:24.089767 1194 tablet_bootstrap.cc:654] T 438d1d2300e044f39a15586f335935ba P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:24.090844 1192 tablet_bootstrap.cc:492] T 157905316fc545b6a7864ae1c4a2a584 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:24.091043 1192 ts_tablet_manager.cc:1397] T 157905316fc545b6a7864ae1c4a2a584 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.004s
I20250113 02:51:24.091779 1191 tablet_bootstrap.cc:492] T 34ab7ee573284550aaac9f814e8bd7ab P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:24.091823 1192 raft_consensus.cc:357] T 157905316fc545b6a7864ae1c4a2a584 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:24.092068 1192 raft_consensus.cc:383] T 157905316fc545b6a7864ae1c4a2a584 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:24.092154 1192 raft_consensus.cc:738] T 157905316fc545b6a7864ae1c4a2a584 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:24.092116 1191 ts_tablet_manager.cc:1397] T 34ab7ee573284550aaac9f814e8bd7ab P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.037s user 0.001s sys 0.003s
I20250113 02:51:24.092401 1192 consensus_queue.cc:260] T 157905316fc545b6a7864ae1c4a2a584 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:24.093061 1192 ts_tablet_manager.cc:1428] T 157905316fc545b6a7864ae1c4a2a584 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250113 02:51:24.093343 1192 tablet_bootstrap.cc:492] T 41bcb2eafff14fa8a040982c723f99c1 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:24.093365 1191 raft_consensus.cc:357] T 34ab7ee573284550aaac9f814e8bd7ab P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:24.093739 1191 raft_consensus.cc:383] T 34ab7ee573284550aaac9f814e8bd7ab P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:24.093863 1191 raft_consensus.cc:738] T 34ab7ee573284550aaac9f814e8bd7ab P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:24.094233 1191 consensus_queue.cc:260] T 34ab7ee573284550aaac9f814e8bd7ab P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:24.095817 1192 tablet_bootstrap.cc:654] T 41bcb2eafff14fa8a040982c723f99c1 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:24.095806 449 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_ca04ad8d0d8b40d1ba6b147731579526" } modify_external_catalogs: true
I20250113 02:51:24.096064 449 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_ca04ad8d0d8b40d1ba6b147731579526" } modify_external_catalogs: true
I20250113 02:51:24.097718 454 master_service.cc:946] ReplaceTablet: received request to replace tablet 42dbd37e20fa47318bc6497423dbe565 from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:24.099725 1191 ts_tablet_manager.cc:1428] T 34ab7ee573284550aaac9f814e8bd7ab P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20250113 02:51:24.101300 1191 tablet_bootstrap.cc:492] T 41bcb2eafff14fa8a040982c723f99c1 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:24.101828 1192 tablet_bootstrap.cc:492] T 41bcb2eafff14fa8a040982c723f99c1 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:24.101991 1192 ts_tablet_manager.cc:1397] T 41bcb2eafff14fa8a040982c723f99c1 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20250113 02:51:24.102685 1192 raft_consensus.cc:357] T 41bcb2eafff14fa8a040982c723f99c1 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:24.102874 1192 raft_consensus.cc:383] T 41bcb2eafff14fa8a040982c723f99c1 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:24.102985 1192 raft_consensus.cc:738] T 41bcb2eafff14fa8a040982c723f99c1 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:24.103217 1192 consensus_queue.cc:260] T 41bcb2eafff14fa8a040982c723f99c1 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:24.103827 1192 ts_tablet_manager.cc:1428] T 41bcb2eafff14fa8a040982c723f99c1 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250113 02:51:24.104125 1192 tablet_bootstrap.cc:492] T ebc5f1ea1e1648d49904b17603d3cd2a P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:24.086099 1196 ts_tablet_manager.cc:1918] T 1c26074fdd804bd79cb0c295622d11ff P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:24.104372 1196 log.cc:1198] T 1c26074fdd804bd79cb0c295622d11ff P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/1c26074fdd804bd79cb0c295622d11ff
I20250113 02:51:24.104878 1191 tablet_bootstrap.cc:654] T 41bcb2eafff14fa8a040982c723f99c1 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:24.106410 1192 tablet_bootstrap.cc:654] T ebc5f1ea1e1648d49904b17603d3cd2a P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:24.106927 1196 ts_tablet_manager.cc:1939] T 1c26074fdd804bd79cb0c295622d11ff P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:24.108548 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet 1c26074fdd804bd79cb0c295622d11ff (table default.Table_9ef18ee71ad843c5921750a2f8478855 [id=bd8bd572ed524f2cae01f820f07e3c36]) successfully deleted
I20250113 02:51:24.109064 1192 tablet_bootstrap.cc:492] T ebc5f1ea1e1648d49904b17603d3cd2a P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:24.109222 1192 ts_tablet_manager.cc:1397] T ebc5f1ea1e1648d49904b17603d3cd2a P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20250113 02:51:24.109484 1194 tablet_bootstrap.cc:492] T 438d1d2300e044f39a15586f335935ba P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:24.109686 1194 ts_tablet_manager.cc:1397] T 438d1d2300e044f39a15586f335935ba P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.022s user 0.000s sys 0.003s
I20250113 02:51:24.109993 1192 raft_consensus.cc:357] T ebc5f1ea1e1648d49904b17603d3cd2a P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:24.110194 1192 raft_consensus.cc:383] T ebc5f1ea1e1648d49904b17603d3cd2a P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:24.110271 1192 raft_consensus.cc:738] T ebc5f1ea1e1648d49904b17603d3cd2a P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:24.110509 1192 consensus_queue.cc:260] T ebc5f1ea1e1648d49904b17603d3cd2a P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:24.110800 1194 raft_consensus.cc:357] T 438d1d2300e044f39a15586f335935ba P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:24.111061 1194 raft_consensus.cc:383] T 438d1d2300e044f39a15586f335935ba P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:24.111121 1192 ts_tablet_manager.cc:1428] T ebc5f1ea1e1648d49904b17603d3cd2a P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250113 02:51:24.111395 1196 tablet_replica.cc:331] T be80b631ccd747f1b9355a00787a7b55 P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:24.111411 1192 tablet_bootstrap.cc:492] T 23e9321f08ee4ce7b2252ad5c76c0a08 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:24.111721 1196 raft_consensus.cc:2238] T be80b631ccd747f1b9355a00787a7b55 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:24.111595 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_ed91fb5c64de42bea271f05b8c284edc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:24.112083 1196 raft_consensus.cc:2267] T be80b631ccd747f1b9355a00787a7b55 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
W20250113 02:51:24.112178 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ed91fb5c64de42bea271f05b8c284edc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:24.113005 1192 tablet_bootstrap.cc:654] T 23e9321f08ee4ce7b2252ad5c76c0a08 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:24.119403 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_bb7ff740ee724b3999eeec918a03b1ae"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:24.119704 1192 tablet_bootstrap.cc:492] T 23e9321f08ee4ce7b2252ad5c76c0a08 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:24.119840 1192 ts_tablet_manager.cc:1397] T 23e9321f08ee4ce7b2252ad5c76c0a08 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.002s
I20250113 02:51:24.111507 1194 raft_consensus.cc:738] T 438d1d2300e044f39a15586f335935ba P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
W20250113 02:51:24.120134 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bb7ff740ee724b3999eeec918a03b1ae in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:24.120327 1194 consensus_queue.cc:260] T 438d1d2300e044f39a15586f335935ba P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:24.120594 1192 raft_consensus.cc:357] T 23e9321f08ee4ce7b2252ad5c76c0a08 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:24.120787 1192 raft_consensus.cc:383] T 23e9321f08ee4ce7b2252ad5c76c0a08 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:24.120875 1192 raft_consensus.cc:738] T 23e9321f08ee4ce7b2252ad5c76c0a08 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:24.120921 1194 ts_tablet_manager.cc:1428] T 438d1d2300e044f39a15586f335935ba P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.011s user 0.000s sys 0.001s
I20250113 02:51:24.121225 1194 tablet_bootstrap.cc:492] T 157905316fc545b6a7864ae1c4a2a584 P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:24.121121 1192 consensus_queue.cc:260] T 23e9321f08ee4ce7b2252ad5c76c0a08 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:24.121722 1192 ts_tablet_manager.cc:1428] T 23e9321f08ee4ce7b2252ad5c76c0a08 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250113 02:51:24.122045 1192 tablet_bootstrap.cc:492] T b0361f11d0974ed19e7d4eb1c32a2f47 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:24.123649 1192 tablet_bootstrap.cc:654] T b0361f11d0974ed19e7d4eb1c32a2f47 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:24.123486 1196 ts_tablet_manager.cc:1905] T be80b631ccd747f1b9355a00787a7b55 P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:24.128873 416 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskXJrxug/build/asan/bin/kudu pid 563
I20250113 02:51:24.130077 1192 tablet_bootstrap.cc:492] T b0361f11d0974ed19e7d4eb1c32a2f47 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:24.130195 1194 tablet_bootstrap.cc:654] T 157905316fc545b6a7864ae1c4a2a584 P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:24.130247 1192 ts_tablet_manager.cc:1397] T b0361f11d0974ed19e7d4eb1c32a2f47 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20250113 02:51:24.131024 1192 raft_consensus.cc:357] T b0361f11d0974ed19e7d4eb1c32a2f47 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:24.131270 1192 raft_consensus.cc:383] T b0361f11d0974ed19e7d4eb1c32a2f47 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:24.131364 1192 raft_consensus.cc:738] T b0361f11d0974ed19e7d4eb1c32a2f47 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:24.131289 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_d3474542a47c407ea91c69d7601c4af9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:24.131614 1192 consensus_queue.cc:260] T b0361f11d0974ed19e7d4eb1c32a2f47 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:24.132081 1191 tablet_bootstrap.cc:492] T 41bcb2eafff14fa8a040982c723f99c1 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:24.132237 1192 ts_tablet_manager.cc:1428] T b0361f11d0974ed19e7d4eb1c32a2f47 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250113 02:51:24.132284 1191 ts_tablet_manager.cc:1397] T 41bcb2eafff14fa8a040982c723f99c1 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.031s user 0.004s sys 0.000s
I20250113 02:51:24.132535 1192 tablet_bootstrap.cc:492] T 6c8ca30720664f3a819c4bff2493934d P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:24.133184 1191 raft_consensus.cc:357] T 41bcb2eafff14fa8a040982c723f99c1 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
W20250113 02:51:24.133339 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d3474542a47c407ea91c69d7601c4af9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:24.133523 1191 raft_consensus.cc:383] T 41bcb2eafff14fa8a040982c723f99c1 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:24.133652 1191 raft_consensus.cc:738] T 41bcb2eafff14fa8a040982c723f99c1 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:24.133961 1191 consensus_queue.cc:260] T 41bcb2eafff14fa8a040982c723f99c1 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:24.134493 1192 tablet_bootstrap.cc:654] T 6c8ca30720664f3a819c4bff2493934d P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:24.134786 1191 ts_tablet_manager.cc:1428] T 41bcb2eafff14fa8a040982c723f99c1 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250113 02:51:24.135538 1191 tablet_bootstrap.cc:492] T 1c26074fdd804bd79cb0c295622d11ff P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:24.137743 1191 tablet_bootstrap.cc:654] T 1c26074fdd804bd79cb0c295622d11ff P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
W20250113 02:51:24.140362 442 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 883027967e0c442a91ac820210b9c6c7 (table default.Table_a7a5a9569d064b488facac137ec35a85 [id=af62d77b2fb74001b86f0d760325eaa8], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 883027967e0c442a91ac820210b9c6c7 has no consensus state
I20250113 02:51:24.144387 1438 raft_consensus.cc:491] T 317c6e8013b745c290bf3190dc237ab8 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:24.144610 1438 raft_consensus.cc:513] T 317c6e8013b745c290bf3190dc237ab8 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:24.145381 1438 leader_election.cc:290] T 317c6e8013b745c290bf3190dc237ab8 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587), 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
I20250113 02:51:24.146031 1192 tablet_bootstrap.cc:492] T 6c8ca30720664f3a819c4bff2493934d P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:24.146196 1192 ts_tablet_manager.cc:1397] T 6c8ca30720664f3a819c4bff2493934d P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.002s
I20250113 02:51:24.146394 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "317c6e8013b745c290bf3190dc237ab8" candidate_uuid: "df69f48656b44bb890f76587ee69cc0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
I20250113 02:51:24.146608 773 raft_consensus.cc:2463] T 317c6e8013b745c290bf3190dc237ab8 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df69f48656b44bb890f76587ee69cc0f in term 0.
I20250113 02:51:24.147076 1192 raft_consensus.cc:357] T 6c8ca30720664f3a819c4bff2493934d P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:24.147236 851 leader_election.cc:304] T 317c6e8013b745c290bf3190dc237ab8 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1186fa869eb14812832fa6562531f196, df69f48656b44bb890f76587ee69cc0f; no voters:
I20250113 02:51:24.147313 1192 raft_consensus.cc:383] T 6c8ca30720664f3a819c4bff2493934d P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:24.147408 1192 raft_consensus.cc:738] T 6c8ca30720664f3a819c4bff2493934d P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:24.147648 1438 raft_consensus.cc:2798] T 317c6e8013b745c290bf3190dc237ab8 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 02:51:24.147639 1192 consensus_queue.cc:260] T 6c8ca30720664f3a819c4bff2493934d P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:24.147764 1438 raft_consensus.cc:491] T 317c6e8013b745c290bf3190dc237ab8 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 02:51:24.147845 1438 raft_consensus.cc:3054] T 317c6e8013b745c290bf3190dc237ab8 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:24.148335 1192 ts_tablet_manager.cc:1428] T 6c8ca30720664f3a819c4bff2493934d P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250113 02:51:24.150189 1438 raft_consensus.cc:513] T 317c6e8013b745c290bf3190dc237ab8 P df69f48656b44bb890f76587ee69cc0f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:24.150367 1192 tablet_bootstrap.cc:492] T d304b65eba8c40c6922333506a60540c P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:24.151146 627 generic_service.cc:196] Checking for leaks (request via RPC)
I20250113 02:51:24.151099 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_49892af4efbf4bdb8230eef4f9a2056e" } new_table_name: "default.Table_ee202b8a323d448887683f55980e6004" modify_external_catalogs: true
I20250113 02:51:24.152334 1062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "317c6e8013b745c290bf3190dc237ab8" candidate_uuid: "df69f48656b44bb890f76587ee69cc0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:24.153358 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "317c6e8013b745c290bf3190dc237ab8" candidate_uuid: "df69f48656b44bb890f76587ee69cc0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca"
I20250113 02:51:24.155426 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "317c6e8013b745c290bf3190dc237ab8" candidate_uuid: "df69f48656b44bb890f76587ee69cc0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196"
I20250113 02:51:24.155616 773 raft_consensus.cc:3054] T 317c6e8013b745c290bf3190dc237ab8 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:24.156013 448 catalog_manager.cc:5618] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587) does not have the latest schema for tablet 4fe54131aafb40ee9d8bc5ea299af193 (table default.Table_7c37f90c233c4b79b3e2f866b8198d0c [id=70089f757a1c45f6b983f216fb167478]). Expected version 2 got 0
I20250113 02:51:24.156775 448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_6a9cc0903b674ac49fd6ffaa542a5595" } new_table_name: "default.Table_182ee8e8d8854eeabef569bd48babd04" modify_external_catalogs: true
I20250113 02:51:24.157889 773 raft_consensus.cc:2463] T 317c6e8013b745c290bf3190dc237ab8 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df69f48656b44bb890f76587ee69cc0f in term 1.
I20250113 02:51:24.158313 447 catalog_manager.cc:5618] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467) does not have the latest schema for tablet 317c6e8013b745c290bf3190dc237ab8 (table default.Table_13fd4df9c207466cb85cd8ed9062722c [id=6bfb67a96b2b47849f7fa12c57a2f4a1]). Expected version 1 got 0
I20250113 02:51:24.158917 447 master_service.cc:946] ReplaceTablet: received request to replace tablet 7d3562a48c6e4badbbcf37640e91b5ba from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:24.159410 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_a55ba304b6a04e2492d6aedba6aad86e" } new_table_name: "default.Table_bf689f8eb8884e88a72e9b50598fc389" modify_external_catalogs: true
I20250113 02:51:24.160535 455 catalog_manager.cc:5618] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095) does not have the latest schema for tablet 317c6e8013b745c290bf3190dc237ab8 (table default.Table_13fd4df9c207466cb85cd8ed9062722c [id=6bfb67a96b2b47849f7fa12c57a2f4a1]). Expected version 1 got 0
I20250113 02:51:24.161883 851 leader_election.cc:304] T 317c6e8013b745c290bf3190dc237ab8 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1186fa869eb14812832fa6562531f196, df69f48656b44bb890f76587ee69cc0f; no voters: 4216a7f853d642b7bc3e5cff458f3aca
I20250113 02:51:24.164419 455 catalog_manager.cc:5618] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467) does not have the latest schema for tablet d3b695ac20ba4205b2d8732db4740531 (table default.Table_8b87c8469fbb465e80375aa0446de6c2 [id=f85f0c435ffa4fdca81d86de4aa4fd5f]). Expected version 3 got 0
I20250113 02:51:24.164685 455 catalog_manager.cc:5618] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467) does not have the latest schema for tablet 438d1d2300e044f39a15586f335935ba (table default.Table_d7d1f5a34e5f4c05b474b0b61f912cd9 [id=2c7925ec0d8d4cd7b8761268d3962f15]). Expected version 3 got 0
I20250113 02:51:24.166270 882 tablet_service.cc:1514] Processing DeleteTablet for tablet 21c4b419bf2148e0b140301fa80622bb with delete_type TABLET_DATA_DELETED (replaced by tablet 23e04961f072440fa3e3f9485e40f79c) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:24.167311 882 tablet_service.cc:1514] Processing DeleteTablet for tablet b0361f11d0974ed19e7d4eb1c32a2f47 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:17 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:24.168782 1438 leader_election.cc:290] T 317c6e8013b745c290bf3190dc237ab8 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 election: Requested vote from peers 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587), 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
I20250113 02:51:24.169078 1438 raft_consensus.cc:2798] T 317c6e8013b745c290bf3190dc237ab8 P df69f48656b44bb890f76587ee69cc0f [term 1 FOLLOWER]: Leader election won for term 1
I20250113 02:51:24.169250 1438 raft_consensus.cc:695] T 317c6e8013b745c290bf3190dc237ab8 P df69f48656b44bb890f76587ee69cc0f [term 1 LEADER]: Becoming Leader. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Running, Role: LEADER
I20250113 02:51:24.169632 1438 consensus_queue.cc:237] T 317c6e8013b745c290bf3190dc237ab8 P df69f48656b44bb890f76587ee69cc0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:24.172398 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 0916b41c171a419791797559fbc2807d
I20250113 02:51:24.173417 455 catalog_manager.cc:5618] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095) does not have the latest schema for tablet 317c6e8013b745c290bf3190dc237ab8 (table default.Table_13fd4df9c207466cb85cd8ed9062722c [id=6bfb67a96b2b47849f7fa12c57a2f4a1]). Expected version 1 got 0
W20250113 02:51:24.175848 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6e1284c8aaba44b5810b0d06d36757eb consensus state has no leader
I20250113 02:51:24.177436 748 tablet_service.cc:1514] Processing DeleteTablet for tablet 0916b41c171a419791797559fbc2807d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:23 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:24.177697 741 tablet_service.cc:1514] Processing DeleteTablet for tablet d7c4c7b0cfc44ac1952bc73cbe1bced8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:18 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:24.177886 744 tablet_service.cc:1514] Processing DeleteTablet for tablet 34ab7ee573284550aaac9f814e8bd7ab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:17 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:24.178320 739 tablet_service.cc:1467] Processing CreateTablet for tablet cfa963a6df094ce0b9c54408f75ca6b2 (DEFAULT_TABLE table=default.Table_518b2b6cde4441c88c49806de09f0aea [id=9dcef6fe27f247ff868d8e2ab2a37ead]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:24.178768 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cfa963a6df094ce0b9c54408f75ca6b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:24.178874 744 tablet_service.cc:1467] Processing CreateTablet for tablet 79aaab73797c474bae06831fa308b066 (DEFAULT_TABLE table=default.Table_7f15d522310c4e8a8afc5d1b77bcaeec [id=89b7f6f853ca4e3c87cbccd41698009a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:24.179314 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 79aaab73797c474bae06831fa308b066. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:24.181496 1195 tablet_replica.cc:331] T d7c4c7b0cfc44ac1952bc73cbe1bced8 P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:24.181864 1195 raft_consensus.cc:2238] T d7c4c7b0cfc44ac1952bc73cbe1bced8 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:24.182072 1195 raft_consensus.cc:2267] T d7c4c7b0cfc44ac1952bc73cbe1bced8 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:24.183219 1195 ts_tablet_manager.cc:1905] T d7c4c7b0cfc44ac1952bc73cbe1bced8 P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:24.184422 1062 raft_consensus.cc:2463] T 317c6e8013b745c290bf3190dc237ab8 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df69f48656b44bb890f76587ee69cc0f in term 0.
I20250113 02:51:24.185882 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet 0916b41c171a419791797559fbc2807d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:23 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:24.186280 1028 tablet_service.cc:1467] Processing CreateTablet for tablet cfa963a6df094ce0b9c54408f75ca6b2 (DEFAULT_TABLE table=default.Table_518b2b6cde4441c88c49806de09f0aea [id=9dcef6fe27f247ff868d8e2ab2a37ead]), partition=RANGE (key) PARTITION UNBOUNDED
W20250113 02:51:24.187302 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 0916b41c171a419791797559fbc2807d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0916b41c171a419791797559fbc2807d already in progress: creating tablet
I20250113 02:51:24.188071 453 catalog_manager.cc:5526] T 317c6e8013b745c290bf3190dc237ab8 P df69f48656b44bb890f76587ee69cc0f reported cstate change: term changed from 0 to 1, leader changed from <none> to df69f48656b44bb890f76587ee69cc0f (127.0.104.2). New cstate: current_term: 1 leader_uuid: "df69f48656b44bb890f76587ee69cc0f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } health_report { overall_health: UNKNOWN } } }
I20250113 02:51:24.188588 453 catalog_manager.cc:5618] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467) does not have the latest schema for tablet 317c6e8013b745c290bf3190dc237ab8 (table default.Table_13fd4df9c207466cb85cd8ed9062722c [id=6bfb67a96b2b47849f7fa12c57a2f4a1]). Expected version 1 got 0
I20250113 02:51:24.189371 1442 raft_consensus.cc:491] T ebc5f1ea1e1648d49904b17603d3cd2a P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:24.190371 882 tablet_service.cc:1514] Processing DeleteTablet for tablet 0916b41c171a419791797559fbc2807d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:23 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:24.191283 882 tablet_service.cc:1467] Processing CreateTablet for tablet cfa963a6df094ce0b9c54408f75ca6b2 (DEFAULT_TABLE table=default.Table_518b2b6cde4441c88c49806de09f0aea [id=9dcef6fe27f247ff868d8e2ab2a37ead]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:24.191648 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cfa963a6df094ce0b9c54408f75ca6b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:24.192634 741 tablet_service.cc:1467] Processing CreateTablet for tablet f1237f7b04af4ebbb39b03aa3c78a499 (DEFAULT_TABLE table=default.Table_14ab7cebe87e40b4bf54f68ee6a1ab80 [id=0817677719d045fd83bb85890f11df25]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:24.193042 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1237f7b04af4ebbb39b03aa3c78a499. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:24.196619 895 tablet_service.cc:1467] Processing CreateTablet for tablet 79aaab73797c474bae06831fa308b066 (DEFAULT_TABLE table=default.Table_7f15d522310c4e8a8afc5d1b77bcaeec [id=89b7f6f853ca4e3c87cbccd41698009a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:24.196988 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 79aaab73797c474bae06831fa308b066. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:24.200291 891 tablet_service.cc:1467] Processing CreateTablet for tablet ca75e7cae4f6423191a17bb5a8f150a9 (DEFAULT_TABLE table=default.Table_345955ac31bd403292668ee24ce8eed3 [id=8d1df898666f414e986c292222dbf2fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:24.200699 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca75e7cae4f6423191a17bb5a8f150a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:24.201355 888 tablet_service.cc:1467] Processing CreateTablet for tablet f1237f7b04af4ebbb39b03aa3c78a499 (DEFAULT_TABLE table=default.Table_14ab7cebe87e40b4bf54f68ee6a1ab80 [id=0817677719d045fd83bb85890f11df25]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:24.201751 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1237f7b04af4ebbb39b03aa3c78a499. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:24.204419 455 master_service.cc:946] ReplaceTablet: received request to replace tablet b4806b18a4194f9c95350d89ba62e13a from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:24.204560 1195 ts_tablet_manager.cc:1918] T d7c4c7b0cfc44ac1952bc73cbe1bced8 P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:24.204710 1195 log.cc:1198] T d7c4c7b0cfc44ac1952bc73cbe1bced8 P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/d7c4c7b0cfc44ac1952bc73cbe1bced8
I20250113 02:51:24.205090 1195 ts_tablet_manager.cc:1939] T d7c4c7b0cfc44ac1952bc73cbe1bced8 P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:24.205744 1442 raft_consensus.cc:513] T ebc5f1ea1e1648d49904b17603d3cd2a P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:24.206387 744 tablet_service.cc:1467] Processing CreateTablet for tablet ca75e7cae4f6423191a17bb5a8f150a9 (DEFAULT_TABLE table=default.Table_345955ac31bd403292668ee24ce8eed3 [id=8d1df898666f414e986c292222dbf2fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:24.207010 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca75e7cae4f6423191a17bb5a8f150a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:24.207734 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_7e49ba949e8949fa9924acdef98c3947"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:24.208315 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7e49ba949e8949fa9924acdef98c3947 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250113 02:51:24.210286 851 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:40095, user_credentials={real_user=slave}} blocked reactor thread for 48399us
I20250113 02:51:24.212656 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cfa963a6df094ce0b9c54408f75ca6b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:24.214991 1035 tablet_service.cc:1467] Processing CreateTablet for tablet ca75e7cae4f6423191a17bb5a8f150a9 (DEFAULT_TABLE table=default.Table_345955ac31bd403292668ee24ce8eed3 [id=8d1df898666f414e986c292222dbf2fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:24.215219 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebc5f1ea1e1648d49904b17603d3cd2a" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:24.215423 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca75e7cae4f6423191a17bb5a8f150a9. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:24.215741 998 leader_election.cc:343] T ebc5f1ea1e1648d49904b17603d3cd2a P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:24.216658 1039 tablet_service.cc:1467] Processing CreateTablet for tablet f1237f7b04af4ebbb39b03aa3c78a499 (DEFAULT_TABLE table=default.Table_14ab7cebe87e40b4bf54f68ee6a1ab80 [id=0817677719d045fd83bb85890f11df25]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:24.217278 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1237f7b04af4ebbb39b03aa3c78a499. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:24.218236 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 0916b41c171a419791797559fbc2807d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0916b41c171a419791797559fbc2807d already in progress: creating tablet
I20250113 02:51:24.219326 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebc5f1ea1e1648d49904b17603d3cd2a" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
I20250113 02:51:24.219488 1196 ts_tablet_manager.cc:1918] T be80b631ccd747f1b9355a00787a7b55 P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:24.219622 1196 log.cc:1198] T be80b631ccd747f1b9355a00787a7b55 P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/be80b631ccd747f1b9355a00787a7b55
I20250113 02:51:24.220027 1196 ts_tablet_manager.cc:1939] T be80b631ccd747f1b9355a00787a7b55 P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:24.220232 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 42dbd37e20fa47318bc6497423dbe565
I20250113 02:51:24.220311 1192 tablet_bootstrap.cc:654] T d304b65eba8c40c6922333506a60540c P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:24.220669 454 catalog_manager.cc:6596] ReplaceTablet: tablet 42dbd37e20fa47318bc6497423dbe565 deleted and replaced by tablet 9174df876fe64adc805fba0a36fb4d38
I20250113 02:51:24.220932 739 tablet_service.cc:1514] Processing DeleteTablet for tablet 42dbd37e20fa47318bc6497423dbe565 with delete_type TABLET_DATA_DELETED (replaced by tablet 9174df876fe64adc805fba0a36fb4d38) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:24.221021 454 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_b21bac46382e43c4bb99fc94778daeeb" } new_table_name: "default.Table_c2e8fd7d2cc14fa39c0abfece783c6c9" modify_external_catalogs: true
I20250113 02:51:24.221189 1196 tablet_replica.cc:331] T 21c4b419bf2148e0b140301fa80622bb P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:24.221386 1196 raft_consensus.cc:2238] T 21c4b419bf2148e0b140301fa80622bb P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:24.221535 1196 raft_consensus.cc:2267] T 21c4b419bf2148e0b140301fa80622bb P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:24.221848 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet d7c4c7b0cfc44ac1952bc73cbe1bced8 (table default.Table_5727b5d8bf51442e8ae5f7c97f4bfbc8 [id=0c69597e8c5b4ec89179169a7a84e048]) successfully deleted
I20250113 02:51:24.222508 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 153b8f90b5a847b88944f683d306f579
I20250113 02:51:24.222640 1034 tablet_service.cc:1467] Processing CreateTablet for tablet 79aaab73797c474bae06831fa308b066 (DEFAULT_TABLE table=default.Table_7f15d522310c4e8a8afc5d1b77bcaeec [id=89b7f6f853ca4e3c87cbccd41698009a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:24.223078 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 79aaab73797c474bae06831fa308b066. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:24.223057 446 master_service.cc:946] ReplaceTablet: received request to replace tablet c7243d27a1c642269a3d0e67632e458c from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:24.224097 739 tablet_service.cc:1514] Processing DeleteTablet for tablet 153b8f90b5a847b88944f683d306f579 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:24 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:24.224730 1442 leader_election.cc:290] T ebc5f1ea1e1648d49904b17603d3cd2a P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
I20250113 02:51:24.225250 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 0fe5421dc3ea4be684cab084f13d8df0
W20250113 02:51:24.226209 995 leader_election.cc:343] T ebc5f1ea1e1648d49904b17603d3cd2a P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:24.228055 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet be80b631ccd747f1b9355a00787a7b55 (table default.Table_2565c0551f9943a084ace8f7e463e59d [id=fe6f8d302c3e489d8bb832503fd05642]) successfully deleted
W20250113 02:51:24.228307 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet dc680b5680b84abbab219296bc7ee707 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dc680b5680b84abbab219296bc7ee707 already in progress: creating tablet
W20250113 02:51:24.228576 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet a0ca2e3a3ce94f3aa20df64c89c7cf95 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a0ca2e3a3ce94f3aa20df64c89c7cf95 already in progress: creating tablet
W20250113 02:51:24.228792 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet f31dd916336a4d8fb3ab015af7de8ccb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f31dd916336a4d8fb3ab015af7de8ccb already in progress: creating tablet
W20250113 02:51:24.229060 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet ee240d838d4a4bbba97f8ed61f69de78 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ee240d838d4a4bbba97f8ed61f69de78 already in progress: creating tablet
W20250113 02:51:24.229243 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet b8b6ffe690fc459db78b2b3bc12671dd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b8b6ffe690fc459db78b2b3bc12671dd already in progress: creating tablet
I20250113 02:51:24.230122 882 tablet_service.cc:1514] Processing DeleteTablet for tablet 0fe5421dc3ea4be684cab084f13d8df0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:24 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:24.230998 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_915411462fb84dd7b917caa2f3a31498"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:24.231595 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_915411462fb84dd7b917caa2f3a31498 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:24.232652 995 leader_election.cc:304] T ebc5f1ea1e1648d49904b17603d3cd2a P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4216a7f853d642b7bc3e5cff458f3aca; no voters: 1186fa869eb14812832fa6562531f196, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:24.233165 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet 42dbd37e20fa47318bc6497423dbe565 with delete_type TABLET_DATA_DELETED (replaced by tablet 9174df876fe64adc805fba0a36fb4d38) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:24.233218 891 tablet_service.cc:1514] Processing DeleteTablet for tablet 153b8f90b5a847b88944f683d306f579 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:24 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:24.233466 888 tablet_service.cc:1514] Processing DeleteTablet for tablet 42dbd37e20fa47318bc6497423dbe565 with delete_type TABLET_DATA_DELETED (replaced by tablet 9174df876fe64adc805fba0a36fb4d38) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:24.234061 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet 0fe5421dc3ea4be684cab084f13d8df0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:24 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:24.234649 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet 153b8f90b5a847b88944f683d306f579 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:24 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:24.235405 1442 raft_consensus.cc:2743] T ebc5f1ea1e1648d49904b17603d3cd2a P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:24.235903 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 655b63ff61ff4f4aa08e0449df39352a
I20250113 02:51:24.236708 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_77b0b19284064a118a50b9bcfe72765f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:24.237246 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_77b0b19284064a118a50b9bcfe72765f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:24.237568 888 tablet_service.cc:1514] Processing DeleteTablet for tablet 655b63ff61ff4f4aa08e0449df39352a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:24 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:24.239055 748 tablet_service.cc:1467] Processing CreateTablet for tablet cf7b8bd6801743f49e9b9473747447c5 (DEFAULT_TABLE table=default.Table_ed883bfc96094be9bb6e07bc510b6042 [id=3e7c139aef3847df965d0400bd2ffa45]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:24.239533 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cf7b8bd6801743f49e9b9473747447c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:24.239837 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet c7842d6f2c074e3b9d1a3378db9856be
I20250113 02:51:24.240618 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_6754736c569e4f55848afc146fe41b74"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:24.241119 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6754736c569e4f55848afc146fe41b74 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250113 02:51:24.242316 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4c615b0b83d74fcfae0c51047dbfbbf4 consensus state has no leader
I20250113 02:51:24.242638 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_deb7c4bd8a954e5081cdbee12274dbcf" } new_table_name: "default.Table_1b71e9ea2fca48daa60e8d692eab8e49" modify_external_catalogs: true
W20250113 02:51:24.243870 448 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 56b022c94847411785a7e9db3757a2d9 consensus state has no leader
I20250113 02:51:24.244220 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet b4806b18a4194f9c95350d89ba62e13a
I20250113 02:51:24.244427 742 tablet_service.cc:1514] Processing DeleteTablet for tablet c7842d6f2c074e3b9d1a3378db9856be with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:24 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:24.244666 455 catalog_manager.cc:6596] ReplaceTablet: tablet b4806b18a4194f9c95350d89ba62e13a deleted and replaced by tablet 146dad5c977f464496f7d21611030dad
I20250113 02:51:24.245023 742 tablet_service.cc:1514] Processing DeleteTablet for tablet b4806b18a4194f9c95350d89ba62e13a with delete_type TABLET_DATA_DELETED (replaced by tablet 146dad5c977f464496f7d21611030dad) from {username='slave'} at 127.0.0.1:55398
W20250113 02:51:24.247051 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 42dbd37e20fa47318bc6497423dbe565 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 42dbd37e20fa47318bc6497423dbe565 already in progress: creating tablet
W20250113 02:51:24.247316 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 0fe5421dc3ea4be684cab084f13d8df0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0fe5421dc3ea4be684cab084f13d8df0 already in progress: creating tablet
W20250113 02:51:24.247519 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 153b8f90b5a847b88944f683d306f579 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 153b8f90b5a847b88944f683d306f579 already in progress: creating tablet
I20250113 02:51:24.250385 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet b4806b18a4194f9c95350d89ba62e13a with delete_type TABLET_DATA_DELETED (replaced by tablet 146dad5c977f464496f7d21611030dad) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:24.250385 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet c7842d6f2c074e3b9d1a3378db9856be with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:24 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:24.257459 741 tablet_service.cc:1514] Processing DeleteTablet for tablet 655b63ff61ff4f4aa08e0449df39352a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:24 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:24.258917 888 tablet_service.cc:1514] Processing DeleteTablet for tablet c7842d6f2c074e3b9d1a3378db9856be with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:24 UTC) from {username='slave'} at 127.0.0.1:37366
W20250113 02:51:24.258949 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet b4806b18a4194f9c95350d89ba62e13a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b4806b18a4194f9c95350d89ba62e13a already in progress: creating tablet
W20250113 02:51:24.259299 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet c7842d6f2c074e3b9d1a3378db9856be because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c7842d6f2c074e3b9d1a3378db9856be already in progress: creating tablet
I20250113 02:51:24.259392 739 tablet_service.cc:1514] Processing DeleteTablet for tablet 0fe5421dc3ea4be684cab084f13d8df0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:24 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:24.259752 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet 655b63ff61ff4f4aa08e0449df39352a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:24 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:24.260334 888 tablet_service.cc:1514] Processing DeleteTablet for tablet b4806b18a4194f9c95350d89ba62e13a with delete_type TABLET_DATA_DELETED (replaced by tablet 146dad5c977f464496f7d21611030dad) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:24.260812 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet b834f0acf5314bcb9d81c73de1b31842 with delete_type TABLET_DATA_DELETED (replaced by tablet be4b2a8a3927410eaf12e7592646e4af) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:24.260995 1039 tablet_service.cc:1514] Processing DeleteTablet for tablet 21c4b419bf2148e0b140301fa80622bb with delete_type TABLET_DATA_DELETED (replaced by tablet 23e04961f072440fa3e3f9485e40f79c) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:24.261066 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c8ca30720664f3a819c4bff2493934d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:18 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:24.261255 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet f1a02c37cb4f43f8a9de6484f4431958 with delete_type TABLET_DATA_DELETED (replaced by tablet 2c399784af46426b8327d175c7a6cce8) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:24.261283 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet ebc5f1ea1e1648d49904b17603d3cd2a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:19 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:24.261453 1039 tablet_service.cc:1514] Processing DeleteTablet for tablet 23e9321f08ee4ce7b2252ad5c76c0a08 with delete_type TABLET_DATA_DELETED (replaced by tablet 6666922d677a4c678f80d7c6c47fd2a5) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:24.261498 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet 157905316fc545b6a7864ae1c4a2a584 with delete_type TABLET_DATA_DELETED (replaced by tablet de4a58bf63054890808e2e2fe46e43d2) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:24.261634 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet fc92f19390fd48a4a45ba8fb839020fd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:17 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:24.261731 1198 tablet_replica.cc:331] T b834f0acf5314bcb9d81c73de1b31842 P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:24.261819 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet b0361f11d0974ed19e7d4eb1c32a2f47 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:17 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:24.261942 1198 raft_consensus.cc:2238] T b834f0acf5314bcb9d81c73de1b31842 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:24.262940 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_67d32aa180f44d0e9614b4e367747129"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:24.263629 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_67d32aa180f44d0e9614b4e367747129 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:24.264329 1198 raft_consensus.cc:2267] T b834f0acf5314bcb9d81c73de1b31842 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:24.265260 1198 ts_tablet_manager.cc:1905] T b834f0acf5314bcb9d81c73de1b31842 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:24.265388 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_72783655bdee4fcf8c2fff2186fe0a60"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:24.265887 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_72783655bdee4fcf8c2fff2186fe0a60 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:24.267153 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_c59be71e676f4fbc8dee9729b5555abb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:24.267598 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c59be71e676f4fbc8dee9729b5555abb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250113 02:51:24.269788 454 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b815003a27f64809baaa2d43d6293cef consensus state has no leader
I20250113 02:51:24.270301 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet c7243d27a1c642269a3d0e67632e458c
I20250113 02:51:24.270430 1195 tablet_replica.cc:331] T 34ab7ee573284550aaac9f814e8bd7ab P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:24.270673 1195 raft_consensus.cc:2238] T 34ab7ee573284550aaac9f814e8bd7ab P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:24.270759 446 catalog_manager.cc:6596] ReplaceTablet: tablet c7243d27a1c642269a3d0e67632e458c deleted and replaced by tablet ed68a2b7ebcf4841bef54798dd3ed80d
I20250113 02:51:24.270862 1195 raft_consensus.cc:2267] T 34ab7ee573284550aaac9f814e8bd7ab P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250113 02:51:24.272243 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 655b63ff61ff4f4aa08e0449df39352a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 655b63ff61ff4f4aa08e0449df39352a already in progress: creating tablet
I20250113 02:51:24.273211 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet c7243d27a1c642269a3d0e67632e458c with delete_type TABLET_DATA_DELETED (replaced by tablet ed68a2b7ebcf4841bef54798dd3ed80d) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:24.273928 1198 ts_tablet_manager.cc:1918] T b834f0acf5314bcb9d81c73de1b31842 P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:24.273922 1195 ts_tablet_manager.cc:1905] T 34ab7ee573284550aaac9f814e8bd7ab P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:24.274063 1198 log.cc:1198] T b834f0acf5314bcb9d81c73de1b31842 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/b834f0acf5314bcb9d81c73de1b31842
I20250113 02:51:24.274436 1198 ts_tablet_manager.cc:1939] T b834f0acf5314bcb9d81c73de1b31842 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:24.275415 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet b834f0acf5314bcb9d81c73de1b31842 (table default.Table_276aa0166db04badb3d250a05e6e7f50 [id=124669ccadaf49e19d2660b1495f9647]) successfully deleted
I20250113 02:51:24.275748 739 tablet_service.cc:1514] Processing DeleteTablet for tablet c7243d27a1c642269a3d0e67632e458c with delete_type TABLET_DATA_DELETED (replaced by tablet ed68a2b7ebcf4841bef54798dd3ed80d) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:24.277365 888 tablet_service.cc:1514] Processing DeleteTablet for tablet c7243d27a1c642269a3d0e67632e458c with delete_type TABLET_DATA_DELETED (replaced by tablet ed68a2b7ebcf4841bef54798dd3ed80d) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:24.277617 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet 93a2729c5a604a27a9bbc451ccb4e128 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:20 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:24.278791 1192 tablet_bootstrap.cc:492] T d304b65eba8c40c6922333506a60540c P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:24.279074 1192 ts_tablet_manager.cc:1397] T d304b65eba8c40c6922333506a60540c P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.129s user 0.000s sys 0.004s
I20250113 02:51:24.280084 1192 raft_consensus.cc:357] T d304b65eba8c40c6922333506a60540c P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:24.280562 1192 raft_consensus.cc:383] T d304b65eba8c40c6922333506a60540c P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:24.280699 1192 raft_consensus.cc:738] T d304b65eba8c40c6922333506a60540c P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:24.281106 1192 consensus_queue.cc:260] T d304b65eba8c40c6922333506a60540c P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:24.283710 739 tablet_service.cc:1467] Processing CreateTablet for tablet 019eb21c661547cfa0b619d119eb024e (DEFAULT_TABLE table=default.Table_25480b2b4ab44c05b96c1c77cec60f61 [id=97bfbd46eecd4e57b7f9757c413f6795]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:24.284715 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 019eb21c661547cfa0b619d119eb024e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:24.284853 1035 tablet_service.cc:1467] Processing CreateTablet for tablet cf7b8bd6801743f49e9b9473747447c5 (DEFAULT_TABLE table=default.Table_ed883bfc96094be9bb6e07bc510b6042 [id=3e7c139aef3847df965d0400bd2ffa45]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:24.285269 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cf7b8bd6801743f49e9b9473747447c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:24.285559 1191 tablet_bootstrap.cc:492] T 1c26074fdd804bd79cb0c295622d11ff P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:24.285811 1191 ts_tablet_manager.cc:1397] T 1c26074fdd804bd79cb0c295622d11ff P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.151s user 0.009s sys 0.019s
I20250113 02:51:24.286675 1191 raft_consensus.cc:357] T 1c26074fdd804bd79cb0c295622d11ff P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:24.286976 1191 raft_consensus.cc:383] T 1c26074fdd804bd79cb0c295622d11ff P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:24.287097 1191 raft_consensus.cc:738] T 1c26074fdd804bd79cb0c295622d11ff P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:24.287410 1191 consensus_queue.cc:260] T 1c26074fdd804bd79cb0c295622d11ff P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:24.288893 1191 ts_tablet_manager.cc:1428] T 1c26074fdd804bd79cb0c295622d11ff P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250113 02:51:24.289240 1191 tablet_bootstrap.cc:492] T 157905316fc545b6a7864ae1c4a2a584 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:24.290861 1191 tablet_bootstrap.cc:654] T 157905316fc545b6a7864ae1c4a2a584 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:24.291774 1194 tablet_bootstrap.cc:492] T 157905316fc545b6a7864ae1c4a2a584 P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:24.292805 1028 tablet_service.cc:1467] Processing CreateTablet for tablet 019eb21c661547cfa0b619d119eb024e (DEFAULT_TABLE table=default.Table_25480b2b4ab44c05b96c1c77cec60f61 [id=97bfbd46eecd4e57b7f9757c413f6795]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:24.292997 1194 ts_tablet_manager.cc:1397] T 157905316fc545b6a7864ae1c4a2a584 P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.172s user 0.008s sys 0.022s
I20250113 02:51:24.293284 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 019eb21c661547cfa0b619d119eb024e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:24.293866 1194 raft_consensus.cc:357] T 157905316fc545b6a7864ae1c4a2a584 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:24.294083 1194 raft_consensus.cc:383] T 157905316fc545b6a7864ae1c4a2a584 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:24.294183 1194 raft_consensus.cc:738] T 157905316fc545b6a7864ae1c4a2a584 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:24.294418 1194 consensus_queue.cc:260] T 157905316fc545b6a7864ae1c4a2a584 P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:24.296816 1198 tablet_replica.cc:331] T 21c4b419bf2148e0b140301fa80622bb P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:24.297081 1198 raft_consensus.cc:2238] T 21c4b419bf2148e0b140301fa80622bb P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:24.297276 1198 raft_consensus.cc:2267] T 21c4b419bf2148e0b140301fa80622bb P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:24.299780 1192 ts_tablet_manager.cc:1428] T d304b65eba8c40c6922333506a60540c P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.021s user 0.002s sys 0.000s
I20250113 02:51:24.300208 1192 tablet_bootstrap.cc:492] T d3b695ac20ba4205b2d8732db4740531 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:24.302492 1192 tablet_bootstrap.cc:654] T d3b695ac20ba4205b2d8732db4740531 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:24.303622 748 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c26074fdd804bd79cb0c295622d11ff with delete_type TABLET_DATA_DELETED (replaced by tablet 9fff52e99c64462a8a73fccc3c22ca63) from {username='slave'} at 127.0.0.1:55398
W20250113 02:51:24.304893 421 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.104.2:38467, user_credentials={real_user=slave}} blocked reactor thread for 108774us
I20250113 02:51:24.305392 453 catalog_manager.cc:5618] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587) does not have the latest schema for tablet d304b65eba8c40c6922333506a60540c (table default.Table_e03d954cc20048e4b0d598282689643d [id=e62f8e4c51b04729903b1943275c5f7f]). Expected version 1 got 0
I20250113 02:51:24.306640 1198 ts_tablet_manager.cc:1905] T 21c4b419bf2148e0b140301fa80622bb P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:24.312804 748 tablet_service.cc:1467] Processing CreateTablet for tablet e01e7f8db83d461bbfbb2e901419ce0b (DEFAULT_TABLE table=default.Table_7b2714b0471c4a3eb2a9f0590a27562d [id=efc4ebba97f74c9ba12ab1dcc6577da0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:24.313303 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e01e7f8db83d461bbfbb2e901419ce0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:24.321174 453 catalog_manager.cc:5618] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587) does not have the latest schema for tablet d304b65eba8c40c6922333506a60540c (table default.Table_e03d954cc20048e4b0d598282689643d [id=e62f8e4c51b04729903b1943275c5f7f]). Expected version 1 got 0
I20250113 02:51:24.321424 1195 ts_tablet_manager.cc:1918] T 34ab7ee573284550aaac9f814e8bd7ab P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:24.321588 1195 log.cc:1198] T 34ab7ee573284550aaac9f814e8bd7ab P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/34ab7ee573284550aaac9f814e8bd7ab
W20250113 02:51:24.321880 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 95e47db8a4d74615800ba6c049ec7131 has no consensus state
I20250113 02:51:24.322068 1195 ts_tablet_manager.cc:1939] T 34ab7ee573284550aaac9f814e8bd7ab P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:24.323335 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet 34ab7ee573284550aaac9f814e8bd7ab (table default.Table_c59cac1b4e0249129c7671a4919283c8 [id=1cd50765f4964c15a341d307ff2d449f]) successfully deleted
I20250113 02:51:24.325033 1192 tablet_bootstrap.cc:492] T d3b695ac20ba4205b2d8732db4740531 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:24.325227 1192 ts_tablet_manager.cc:1397] T d3b695ac20ba4205b2d8732db4740531 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.025s user 0.004s sys 0.000s
I20250113 02:51:24.326237 1192 raft_consensus.cc:357] T d3b695ac20ba4205b2d8732db4740531 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:24.326512 1192 raft_consensus.cc:383] T d3b695ac20ba4205b2d8732db4740531 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:24.326642 1192 raft_consensus.cc:738] T d3b695ac20ba4205b2d8732db4740531 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:24.326937 1192 consensus_queue.cc:260] T d3b695ac20ba4205b2d8732db4740531 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:24.327171 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_eba6422342a742499b1c54520d991881"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:24.327700 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_eba6422342a742499b1c54520d991881 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:24.329290 1192 ts_tablet_manager.cc:1428] T d3b695ac20ba4205b2d8732db4740531 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20250113 02:51:24.329634 1192 tablet_bootstrap.cc:492] T 34ab7ee573284550aaac9f814e8bd7ab P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:24.330322 1035 tablet_service.cc:1467] Processing CreateTablet for tablet e01e7f8db83d461bbfbb2e901419ce0b (DEFAULT_TABLE table=default.Table_7b2714b0471c4a3eb2a9f0590a27562d [id=efc4ebba97f74c9ba12ab1dcc6577da0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:24.330771 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e01e7f8db83d461bbfbb2e901419ce0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:24.332034 1192 tablet_bootstrap.cc:654] T 34ab7ee573284550aaac9f814e8bd7ab P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
W20250113 02:51:24.333994 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 42dbd37e20fa47318bc6497423dbe565 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 42dbd37e20fa47318bc6497423dbe565 already in progress: creating tablet
I20250113 02:51:24.334668 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_503d2ce0d8d74387bf59e25670b61bcb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:24.334821 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 153b8f90b5a847b88944f683d306f579 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 153b8f90b5a847b88944f683d306f579 already in progress: creating tablet
W20250113 02:51:24.335251 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_503d2ce0d8d74387bf59e25670b61bcb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:24.336194 741 tablet_service.cc:1467] Processing CreateTablet for tablet f105f8b32a654e4ca154e8fc0cb96091 (DEFAULT_TABLE table=default.Table_15d236fec5f24590b15d5d21f67e3941 [id=505134ec091a46c9a1dc599ae2514a3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:24.336696 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f105f8b32a654e4ca154e8fc0cb96091. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:24.338986 450 catalog_manager.cc:5618] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587) does not have the latest schema for tablet d3b695ac20ba4205b2d8732db4740531 (table default.Table_8b87c8469fbb465e80375aa0446de6c2 [id=f85f0c435ffa4fdca81d86de4aa4fd5f]). Expected version 3 got 0
I20250113 02:51:24.340489 1024 tablet_service.cc:1467] Processing CreateTablet for tablet 0839a35fcab04f3c9d03e256553e01c8 (DEFAULT_TABLE table=default.Table_68c383ddd5194ab0ba962868f50a690f [id=9a463335fc8b4d5788887042fb926bb7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:24.340801 742 tablet_service.cc:1467] Processing CreateTablet for tablet 0839a35fcab04f3c9d03e256553e01c8 (DEFAULT_TABLE table=default.Table_68c383ddd5194ab0ba962868f50a690f [id=9a463335fc8b4d5788887042fb926bb7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:24.341284 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0839a35fcab04f3c9d03e256553e01c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:24.341346 1028 tablet_service.cc:1467] Processing CreateTablet for tablet 8cc78bf7d6324b7ba10a1b1045923f2b (DEFAULT_TABLE table=default.Table_2b71ec43b2ee485e92d81d55f56a44fc [id=925246ca69314b9490c9f968f3597229]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:24.341823 1034 tablet_service.cc:1467] Processing CreateTablet for tablet f105f8b32a654e4ca154e8fc0cb96091 (DEFAULT_TABLE table=default.Table_15d236fec5f24590b15d5d21f67e3941 [id=505134ec091a46c9a1dc599ae2514a3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:24.344266 737 tablet_service.cc:1467] Processing CreateTablet for tablet 16a132384b0e4264a73d09a3ea133893 (DEFAULT_TABLE table=default.Table_eef7fe6c891b4fe984f8fb4a1c667a8e [id=ada38f49d3074fd49727e4cbe8b576d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:24.344686 737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 16a132384b0e4264a73d09a3ea133893. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:24.346832 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0839a35fcab04f3c9d03e256553e01c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:24.348176 882 tablet_service.cc:1467] Processing CreateTablet for tablet 019eb21c661547cfa0b619d119eb024e (DEFAULT_TABLE table=default.Table_25480b2b4ab44c05b96c1c77cec60f61 [id=97bfbd46eecd4e57b7f9757c413f6795]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:24.348484 1039 tablet_service.cc:1467] Processing CreateTablet for tablet 16a132384b0e4264a73d09a3ea133893 (DEFAULT_TABLE table=default.Table_eef7fe6c891b4fe984f8fb4a1c667a8e [id=ada38f49d3074fd49727e4cbe8b576d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:24.348771 898 tablet_service.cc:1467] Processing CreateTablet for tablet 8cc78bf7d6324b7ba10a1b1045923f2b (DEFAULT_TABLE table=default.Table_2b71ec43b2ee485e92d81d55f56a44fc [id=925246ca69314b9490c9f968f3597229]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:24.348868 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 16a132384b0e4264a73d09a3ea133893. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:24.349371 897 tablet_service.cc:1467] Processing CreateTablet for tablet f105f8b32a654e4ca154e8fc0cb96091 (DEFAULT_TABLE table=default.Table_15d236fec5f24590b15d5d21f67e3941 [id=505134ec091a46c9a1dc599ae2514a3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:24.349879 892 tablet_service.cc:1467] Processing CreateTablet for tablet 0839a35fcab04f3c9d03e256553e01c8 (DEFAULT_TABLE table=default.Table_68c383ddd5194ab0ba962868f50a690f [id=9a463335fc8b4d5788887042fb926bb7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:24.350500 1442 raft_consensus.cc:491] T 157905316fc545b6a7864ae1c4a2a584 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:24.350666 1442 raft_consensus.cc:513] T 157905316fc545b6a7864ae1c4a2a584 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:24.351446 1442 leader_election.cc:290] T 157905316fc545b6a7864ae1c4a2a584 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
I20250113 02:51:24.351635 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8cc78bf7d6324b7ba10a1b1045923f2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:24.352609 891 tablet_service.cc:1514] Processing DeleteTablet for tablet 157905316fc545b6a7864ae1c4a2a584 with delete_type TABLET_DATA_DELETED (replaced by tablet de4a58bf63054890808e2e2fe46e43d2) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:24.354702 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "157905316fc545b6a7864ae1c4a2a584" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
I20250113 02:51:24.354861 1194 ts_tablet_manager.cc:1428] T 157905316fc545b6a7864ae1c4a2a584 P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.051s user 0.001s sys 0.001s
I20250113 02:51:24.355196 1194 tablet_bootstrap.cc:492] T f1a02c37cb4f43f8a9de6484f4431958 P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
W20250113 02:51:24.355405 995 leader_election.cc:343] T 157905316fc545b6a7864ae1c4a2a584 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:24.355768 895 tablet_service.cc:1467] Processing CreateTablet for tablet e01e7f8db83d461bbfbb2e901419ce0b (DEFAULT_TABLE table=default.Table_7b2714b0471c4a3eb2a9f0590a27562d [id=efc4ebba97f74c9ba12ab1dcc6577da0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:24.355799 744 tablet_service.cc:1467] Processing CreateTablet for tablet 8cc78bf7d6324b7ba10a1b1045923f2b (DEFAULT_TABLE table=default.Table_2b71ec43b2ee485e92d81d55f56a44fc [id=925246ca69314b9490c9f968f3597229]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:24.355943 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f105f8b32a654e4ca154e8fc0cb96091. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:24.356148 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8cc78bf7d6324b7ba10a1b1045923f2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:24.356318 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8cc78bf7d6324b7ba10a1b1045923f2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:24.356760 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "157905316fc545b6a7864ae1c4a2a584" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:24.357018 918 raft_consensus.cc:2463] T 157905316fc545b6a7864ae1c4a2a584 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 0.
I20250113 02:51:24.357383 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f105f8b32a654e4ca154e8fc0cb96091. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:24.360162 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e01e7f8db83d461bbfbb2e901419ce0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:24.361647 998 leader_election.cc:304] T 157905316fc545b6a7864ae1c4a2a584 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f; no voters: 1186fa869eb14812832fa6562531f196
I20250113 02:51:24.362380 1442 raft_consensus.cc:2798] T 157905316fc545b6a7864ae1c4a2a584 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 02:51:24.362529 1442 raft_consensus.cc:491] T 157905316fc545b6a7864ae1c4a2a584 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 02:51:24.362648 1442 raft_consensus.cc:3054] T 157905316fc545b6a7864ae1c4a2a584 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:24.362689 1196 ts_tablet_manager.cc:1905] T 21c4b419bf2148e0b140301fa80622bb P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:24.364888 1194 tablet_bootstrap.cc:654] T f1a02c37cb4f43f8a9de6484f4431958 P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:24.365232 1442 raft_consensus.cc:513] T 157905316fc545b6a7864ae1c4a2a584 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:24.366400 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "157905316fc545b6a7864ae1c4a2a584" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196"
I20250113 02:51:24.366497 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "157905316fc545b6a7864ae1c4a2a584" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f"
I20250113 02:51:24.366691 918 raft_consensus.cc:3054] T 157905316fc545b6a7864ae1c4a2a584 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:24.370096 918 raft_consensus.cc:2463] T 157905316fc545b6a7864ae1c4a2a584 P df69f48656b44bb890f76587ee69cc0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 1.
I20250113 02:51:24.370416 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0839a35fcab04f3c9d03e256553e01c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:24.372510 891 tablet_service.cc:1467] Processing CreateTablet for tablet 16a132384b0e4264a73d09a3ea133893 (DEFAULT_TABLE table=default.Table_eef7fe6c891b4fe984f8fb4a1c667a8e [id=ada38f49d3074fd49727e4cbe8b576d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:24.372887 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 16a132384b0e4264a73d09a3ea133893. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:24.376550 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 019eb21c661547cfa0b619d119eb024e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:24.378361 737 tablet_service.cc:1467] Processing CreateTablet for tablet 542df00229f645788f1b821c0ce00c46 (DEFAULT_TABLE table=default.Table_b2598c65e9d04d8aa594a9f3d61d1e5a [id=e4f6f4ec007c454eba94263a37690f56]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:24.378690 450 catalog_manager.cc:5618] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587) does not have the latest schema for tablet d3b695ac20ba4205b2d8732db4740531 (table default.Table_8b87c8469fbb465e80375aa0446de6c2 [id=f85f0c435ffa4fdca81d86de4aa4fd5f]). Expected version 3 got 0
I20250113 02:51:24.378896 737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 542df00229f645788f1b821c0ce00c46. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:24.379284 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet c7842d6f2c074e3b9d1a3378db9856be because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c7842d6f2c074e3b9d1a3378db9856be already in progress: creating tablet
I20250113 02:51:24.379312 1442 leader_election.cc:290] T 157905316fc545b6a7864ae1c4a2a584 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Requested vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
W20250113 02:51:24.379535 995 leader_election.cc:343] T 157905316fc545b6a7864ae1c4a2a584 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:24.380113 998 leader_election.cc:304] T 157905316fc545b6a7864ae1c4a2a584 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f; no voters: 1186fa869eb14812832fa6562531f196
W20250113 02:51:24.380723 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet b4806b18a4194f9c95350d89ba62e13a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b4806b18a4194f9c95350d89ba62e13a already in progress: creating tablet
I20250113 02:51:24.381002 1442 raft_consensus.cc:491] T 93a2729c5a604a27a9bbc451ccb4e128 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250113 02:51:24.381045 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 655b63ff61ff4f4aa08e0449df39352a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 655b63ff61ff4f4aa08e0449df39352a already in progress: creating tablet
I20250113 02:51:24.381187 1442 raft_consensus.cc:513] T 93a2729c5a604a27a9bbc451ccb4e128 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
W20250113 02:51:24.381618 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 0fe5421dc3ea4be684cab084f13d8df0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0fe5421dc3ea4be684cab084f13d8df0 already in progress: creating tablet
I20250113 02:51:24.381922 1442 leader_election.cc:290] T 93a2729c5a604a27a9bbc451ccb4e128 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
I20250113 02:51:24.382086 1442 raft_consensus.cc:2798] T 157905316fc545b6a7864ae1c4a2a584 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Leader election won for term 1
I20250113 02:51:24.382138 1195 tablet_replica.cc:331] T 1c26074fdd804bd79cb0c295622d11ff P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:24.382196 1442 raft_consensus.cc:695] T 157905316fc545b6a7864ae1c4a2a584 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: Becoming Leader. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Running, Role: LEADER
I20250113 02:51:24.382335 1195 raft_consensus.cc:2238] T 1c26074fdd804bd79cb0c295622d11ff P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:24.382501 1195 raft_consensus.cc:2267] T 1c26074fdd804bd79cb0c295622d11ff P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:24.382477 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93a2729c5a604a27a9bbc451ccb4e128" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
I20250113 02:51:24.382467 1442 consensus_queue.cc:237] T 157905316fc545b6a7864ae1c4a2a584 P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
W20250113 02:51:24.382974 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet c7243d27a1c642269a3d0e67632e458c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c7243d27a1c642269a3d0e67632e458c already in progress: creating tablet
W20250113 02:51:24.383397 995 leader_election.cc:343] T 93a2729c5a604a27a9bbc451ccb4e128 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:24.386566 888 tablet_service.cc:1467] Processing CreateTablet for tablet cf7b8bd6801743f49e9b9473747447c5 (DEFAULT_TABLE table=default.Table_ed883bfc96094be9bb6e07bc510b6042 [id=3e7c139aef3847df965d0400bd2ffa45]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:24.387004 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cf7b8bd6801743f49e9b9473747447c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:24.388332 1195 ts_tablet_manager.cc:1905] T 1c26074fdd804bd79cb0c295622d11ff P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:24.389783 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_82d6d0a936524f35beab62f451e5eb99"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:24.393321 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_82d6d0a936524f35beab62f451e5eb99 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:24.397799 1198 ts_tablet_manager.cc:1918] T 21c4b419bf2148e0b140301fa80622bb P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:24.401016 1196 ts_tablet_manager.cc:1918] T 21c4b419bf2148e0b140301fa80622bb P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:24.401906 1191 tablet_bootstrap.cc:492] T 157905316fc545b6a7864ae1c4a2a584 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:24.402072 1191 ts_tablet_manager.cc:1397] T 157905316fc545b6a7864ae1c4a2a584 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.113s user 0.027s sys 0.027s
I20250113 02:51:24.402978 1191 raft_consensus.cc:357] T 157905316fc545b6a7864ae1c4a2a584 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:24.403185 1191 raft_consensus.cc:383] T 157905316fc545b6a7864ae1c4a2a584 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:24.403277 1191 raft_consensus.cc:738] T 157905316fc545b6a7864ae1c4a2a584 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:24.403512 1191 consensus_queue.cc:260] T 157905316fc545b6a7864ae1c4a2a584 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:24.403654 748 tablet_service.cc:1467] Processing CreateTablet for tablet 88a776aa820c44b3a3854ca5022bd353 (DEFAULT_TABLE table=default.Table_6d2c74a667ab4954a8380355f54d648f [id=5f6159d685c54a29b39e171bb1b2b24a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:24.409451 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 88a776aa820c44b3a3854ca5022bd353. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:24.412472 1198 log.cc:1198] T 21c4b419bf2148e0b140301fa80622bb P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/21c4b419bf2148e0b140301fa80622bb
I20250113 02:51:24.413164 882 tablet_service.cc:1467] Processing CreateTablet for tablet b217cb8c09fb41cab4f53e605c37efe8 (DEFAULT_TABLE table=default.Table_3ed64b1d864b4b79bdaef850c30c486b [id=f8180eb6217846b6be7819aab2edaf99]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:24.414040 452 master_service.cc:946] ReplaceTablet: received request to replace tablet b33025a462e74f2aa26cc3779f36615b from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:24.414000 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_ee202b8a323d448887683f55980e6004" } new_table_name: "default.Table_bfc5031f68f64006aceefe2ee986223a" modify_external_catalogs: true
I20250113 02:51:24.414661 892 tablet_service.cc:1467] Processing CreateTablet for tablet 883027967e0c442a91ac820210b9c6c7 (DEFAULT_TABLE table=default.Table_a7a5a9569d064b488facac137ec35a85 [id=af62d77b2fb74001b86f0d760325eaa8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:24.413898 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_f08525e2a08e48ec85ef55287d0bac49" } modify_external_catalogs: true
I20250113 02:51:24.415253 898 tablet_service.cc:1467] Processing CreateTablet for tablet 88a776aa820c44b3a3854ca5022bd353 (DEFAULT_TABLE table=default.Table_6d2c74a667ab4954a8380355f54d648f [id=5f6159d685c54a29b39e171bb1b2b24a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:24.415493 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_f08525e2a08e48ec85ef55287d0bac49" } modify_external_catalogs: true
I20250113 02:51:24.418640 1035 tablet_service.cc:1467] Processing CreateTablet for tablet 542df00229f645788f1b821c0ce00c46 (DEFAULT_TABLE table=default.Table_b2598c65e9d04d8aa594a9f3d61d1e5a [id=e4f6f4ec007c454eba94263a37690f56]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:24.419157 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 542df00229f645788f1b821c0ce00c46. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:24.421999 1198 ts_tablet_manager.cc:1939] T 21c4b419bf2148e0b140301fa80622bb P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:24.424345 1198 tablet_replica.cc:331] T 6c8ca30720664f3a819c4bff2493934d P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:24.424724 1039 tablet_service.cc:1467] Processing CreateTablet for tablet 88a776aa820c44b3a3854ca5022bd353 (DEFAULT_TABLE table=default.Table_6d2c74a667ab4954a8380355f54d648f [id=5f6159d685c54a29b39e171bb1b2b24a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:24.426374 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 88a776aa820c44b3a3854ca5022bd353. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:24.426440 1028 tablet_service.cc:1467] Processing CreateTablet for tablet 883027967e0c442a91ac820210b9c6c7 (DEFAULT_TABLE table=default.Table_a7a5a9569d064b488facac137ec35a85 [id=af62d77b2fb74001b86f0d760325eaa8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:24.426811 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 883027967e0c442a91ac820210b9c6c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:24.425897 1024 tablet_service.cc:1467] Processing CreateTablet for tablet b217cb8c09fb41cab4f53e605c37efe8 (DEFAULT_TABLE table=default.Table_3ed64b1d864b4b79bdaef850c30c486b [id=f8180eb6217846b6be7819aab2edaf99]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:24.425346 1034 tablet_service.cc:1467] Processing CreateTablet for tablet f2c6c8c17d6e499d84ad7c0c23889cc4 (DEFAULT_TABLE table=default.Table_df5d17a7c90c4a5a9e0d75558749d6a8 [id=9e58548d1a184cc0b362f148c44005ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:24.429194 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b217cb8c09fb41cab4f53e605c37efe8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:24.429296 897 tablet_service.cc:1467] Processing CreateTablet for tablet 569d666c128a45b7a526c13be5090ae3 (DEFAULT_TABLE table=default.Table_349bd0d1105d42e4889232adbacc4d3a [id=f10700c3143d4a6c9f4f45508db7fa93]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:24.429961 1198 raft_consensus.cc:2238] T 6c8ca30720664f3a819c4bff2493934d P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:24.421646 749 tablet_service.cc:1467] Processing CreateTablet for tablet b3ce6d851f564b849796f174860aacd5 (DEFAULT_TABLE table=default.Table_b19190849e5245a380b5349a7abce2d5 [id=7084032f38d34a358b50e973136bd134]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:24.430559 1194 tablet_bootstrap.cc:492] T f1a02c37cb4f43f8a9de6484f4431958 P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:24.431171 742 tablet_service.cc:1467] Processing CreateTablet for tablet b217cb8c09fb41cab4f53e605c37efe8 (DEFAULT_TABLE table=default.Table_3ed64b1d864b4b79bdaef850c30c486b [id=f8180eb6217846b6be7819aab2edaf99]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:24.431555 744 tablet_service.cc:1467] Processing CreateTablet for tablet f2c6c8c17d6e499d84ad7c0c23889cc4 (DEFAULT_TABLE table=default.Table_df5d17a7c90c4a5a9e0d75558749d6a8 [id=9e58548d1a184cc0b362f148c44005ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:24.432032 889 tablet_service.cc:1467] Processing CreateTablet for tablet 247af1a17df04b27898f86642336d51e (DEFAULT_TABLE table=default.Table_2087b79408c949cc88238ba68aabcaf1 [id=43940b76df2c4fb5bc58e5e2975cac8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:24.432515 1611 raft_consensus.cc:491] T b0361f11d0974ed19e7d4eb1c32a2f47 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:24.432693 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3ce6d851f564b849796f174860aacd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:24.432662 1611 raft_consensus.cc:513] T b0361f11d0974ed19e7d4eb1c32a2f47 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:24.433486 1611 leader_election.cc:290] T b0361f11d0974ed19e7d4eb1c32a2f47 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587), 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
I20250113 02:51:24.433596 753 tablet_service.cc:1467] Processing CreateTablet for tablet 247af1a17df04b27898f86642336d51e (DEFAULT_TABLE table=default.Table_2087b79408c949cc88238ba68aabcaf1 [id=43940b76df2c4fb5bc58e5e2975cac8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:24.434484 1196 log.cc:1198] T 21c4b419bf2148e0b140301fa80622bb P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/21c4b419bf2148e0b140301fa80622bb
I20250113 02:51:24.435423 754 tablet_service.cc:1467] Processing CreateTablet for tablet 68c4a7e8c0fb4feb85498709b00d3258 (DEFAULT_TABLE table=default.Table_df6d561e8e21480ba865be6710c85dc6 [id=8d2f84cd670a4e8d9fa17b8313d3fab4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:24.436244 746 tablet_service.cc:1467] Processing CreateTablet for tablet f4379f7fc6b54e6785f27975d9030cce (DEFAULT_TABLE table=default.Table_c15dbcc7e55d4e3c80fb41bc4a950c81 [id=2c1508d711f44d26a4359ab9c7645cb1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:24.436537 740 tablet_service.cc:1467] Processing CreateTablet for tablet 59fadc693f554854bc2874f8db7c64c8 (DEFAULT_TABLE table=default.Table_17489ca6f295469eb97e9a3a5c7cb1aa [id=0e77d79275d44f37b11395f03d737c59]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:24.436903 752 tablet_service.cc:1467] Processing CreateTablet for tablet 58053845187f466599750f2a62f4c3e1 (DEFAULT_TABLE table=default.Table_9e44dff41935489aa1306e2e5d02bf85 [id=2c51fbdb672d4b01a5932fade2beb3d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:24.436610 1196 ts_tablet_manager.cc:1939] T 21c4b419bf2148e0b140301fa80622bb P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:24.437381 739 tablet_service.cc:1514] Processing DeleteTablet for tablet 157905316fc545b6a7864ae1c4a2a584 with delete_type TABLET_DATA_DELETED (replaced by tablet de4a58bf63054890808e2e2fe46e43d2) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:24.438688 750 tablet_service.cc:1467] Processing CreateTablet for tablet 65d9bbd8c15f40a4a50c1695fb5e648e (DEFAULT_TABLE table=default.Table_1df1ad89ec084bd4928ccfa8376b0bc6 [id=32ea1ffb81004a9db8e67189bea55237]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:24.439040 1196 tablet_replica.cc:331] T b0361f11d0974ed19e7d4eb1c32a2f47 P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:24.439532 1196 raft_consensus.cc:2238] T b0361f11d0974ed19e7d4eb1c32a2f47 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:24.440171 1196 raft_consensus.cc:2267] T b0361f11d0974ed19e7d4eb1c32a2f47 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:24.441708 886 tablet_service.cc:1467] Processing CreateTablet for tablet f4379f7fc6b54e6785f27975d9030cce (DEFAULT_TABLE table=default.Table_c15dbcc7e55d4e3c80fb41bc4a950c81 [id=2c1508d711f44d26a4359ab9c7645cb1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:24.442778 880 tablet_service.cc:1467] Processing CreateTablet for tablet 68c4a7e8c0fb4feb85498709b00d3258 (DEFAULT_TABLE table=default.Table_df6d561e8e21480ba865be6710c85dc6 [id=8d2f84cd670a4e8d9fa17b8313d3fab4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:24.443107 747 tablet_service.cc:1467] Processing CreateTablet for tablet 9b2d0aa6f0e9456d95f4e48c360ab6e8 (DEFAULT_TABLE table=default.Table_a627be2353214265addee118a16ade57 [id=8b48305e2d1b4a0fa1c1fab70d570cf6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:24.443926 891 tablet_service.cc:1467] Processing CreateTablet for tablet 542df00229f645788f1b821c0ce00c46 (DEFAULT_TABLE table=default.Table_b2598c65e9d04d8aa594a9f3d61d1e5a [id=e4f6f4ec007c454eba94263a37690f56]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:24.444139 885 tablet_service.cc:1467] Processing CreateTablet for tablet 58053845187f466599750f2a62f4c3e1 (DEFAULT_TABLE table=default.Table_9e44dff41935489aa1306e2e5d02bf85 [id=2c51fbdb672d4b01a5932fade2beb3d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:24.444334 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2c6c8c17d6e499d84ad7c0c23889cc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:24.444243 741 tablet_service.cc:1467] Processing CreateTablet for tablet 883027967e0c442a91ac820210b9c6c7 (DEFAULT_TABLE table=default.Table_a7a5a9569d064b488facac137ec35a85 [id=af62d77b2fb74001b86f0d760325eaa8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:24.444448 1062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0361f11d0974ed19e7d4eb1c32a2f47" candidate_uuid: "df69f48656b44bb890f76587ee69cc0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:24.444656 1062 raft_consensus.cc:2463] T b0361f11d0974ed19e7d4eb1c32a2f47 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df69f48656b44bb890f76587ee69cc0f in term 0.
I20250113 02:51:24.444983 743 tablet_service.cc:1467] Processing CreateTablet for tablet 569d666c128a45b7a526c13be5090ae3 (DEFAULT_TABLE table=default.Table_349bd0d1105d42e4889232adbacc4d3a [id=f10700c3143d4a6c9f4f45508db7fa93]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:24.445194 852 leader_election.cc:304] T b0361f11d0974ed19e7d4eb1c32a2f47 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f; no voters:
I20250113 02:51:24.445318 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b217cb8c09fb41cab4f53e605c37efe8. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:24.445391 852 raft_consensus.cc:2681] T b0361f11d0974ed19e7d4eb1c32a2f47 P df69f48656b44bb890f76587ee69cc0f: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250113 02:51:24.445502 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f4379f7fc6b54e6785f27975d9030cce. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:24.446082 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 58053845187f466599750f2a62f4c3e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:24.447057 893 tablet_service.cc:1467] Processing CreateTablet for tablet 65d9bbd8c15f40a4a50c1695fb5e648e (DEFAULT_TABLE table=default.Table_1df1ad89ec084bd4928ccfa8376b0bc6 [id=32ea1ffb81004a9db8e67189bea55237]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:24.447221 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b217cb8c09fb41cab4f53e605c37efe8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:24.447154 895 tablet_service.cc:1467] Processing CreateTablet for tablet f2c6c8c17d6e499d84ad7c0c23889cc4 (DEFAULT_TABLE table=default.Table_df5d17a7c90c4a5a9e0d75558749d6a8 [id=9e58548d1a184cc0b362f148c44005ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:24.447494 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65d9bbd8c15f40a4a50c1695fb5e648e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:24.447633 881 tablet_service.cc:1467] Processing CreateTablet for tablet 59fadc693f554854bc2874f8db7c64c8 (DEFAULT_TABLE table=default.Table_17489ca6f295469eb97e9a3a5c7cb1aa [id=0e77d79275d44f37b11395f03d737c59]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:24.447705 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 58053845187f466599750f2a62f4c3e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:24.448019 1198 raft_consensus.cc:2267] T 6c8ca30720664f3a819c4bff2493934d P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:24.447955 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59fadc693f554854bc2874f8db7c64c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:24.448309 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 68c4a7e8c0fb4feb85498709b00d3258. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:24.448786 1198 ts_tablet_manager.cc:1905] T 6c8ca30720664f3a819c4bff2493934d P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:24.449434 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f4379f7fc6b54e6785f27975d9030cce. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:24.452662 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 88a776aa820c44b3a3854ca5022bd353. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:24.453266 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 247af1a17df04b27898f86642336d51e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:24.453789 1442 raft_consensus.cc:491] T d304b65eba8c40c6922333506a60540c P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:24.453871 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet 21c4b419bf2148e0b140301fa80622bb (table default.Table_4a0258998d4f413b80d701c74526b86d [id=6fd9215b11274a76b06813227889927b]) successfully deleted
I20250113 02:51:24.453966 1442 raft_consensus.cc:513] T d304b65eba8c40c6922333506a60540c P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:24.454625 1442 leader_election.cc:290] T d304b65eba8c40c6922333506a60540c P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
I20250113 02:51:24.455029 449 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_1df1ad89ec084bd4928ccfa8376b0bc6" } modify_external_catalogs: true
I20250113 02:51:24.455240 449 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_1df1ad89ec084bd4928ccfa8376b0bc6" } modify_external_catalogs: true
I20250113 02:51:24.455628 1029 tablet_service.cc:1467] Processing CreateTablet for tablet f4379f7fc6b54e6785f27975d9030cce (DEFAULT_TABLE table=default.Table_c15dbcc7e55d4e3c80fb41bc4a950c81 [id=2c1508d711f44d26a4359ab9c7645cb1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:24.455694 1040 tablet_service.cc:1467] Processing CreateTablet for tablet 65d9bbd8c15f40a4a50c1695fb5e648e (DEFAULT_TABLE table=default.Table_1df1ad89ec084bd4928ccfa8376b0bc6 [id=32ea1ffb81004a9db8e67189bea55237]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:24.456068 1040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65d9bbd8c15f40a4a50c1695fb5e648e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:24.456221 1023 tablet_service.cc:1467] Processing CreateTablet for tablet 68c4a7e8c0fb4feb85498709b00d3258 (DEFAULT_TABLE table=default.Table_df6d561e8e21480ba865be6710c85dc6 [id=8d2f84cd670a4e8d9fa17b8313d3fab4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:24.456540 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 68c4a7e8c0fb4feb85498709b00d3258. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:24.457360 1033 tablet_service.cc:1467] Processing CreateTablet for tablet 58053845187f466599750f2a62f4c3e1 (DEFAULT_TABLE table=default.Table_9e44dff41935489aa1306e2e5d02bf85 [id=2c51fbdb672d4b01a5932fade2beb3d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:24.457718 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 58053845187f466599750f2a62f4c3e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:24.458432 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0361f11d0974ed19e7d4eb1c32a2f47" candidate_uuid: "df69f48656b44bb890f76587ee69cc0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
I20250113 02:51:24.459401 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b2d0aa6f0e9456d95f4e48c360ab6e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:24.459810 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 68c4a7e8c0fb4feb85498709b00d3258. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:24.460218 454 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_f8af5d3e0dc64d3fae054f82a7b33ccf" } modify_external_catalogs: true
I20250113 02:51:24.460390 454 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_f8af5d3e0dc64d3fae054f82a7b33ccf" } modify_external_catalogs: true
I20250113 02:51:24.461405 1192 tablet_bootstrap.cc:492] T 34ab7ee573284550aaac9f814e8bd7ab P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:24.461669 1192 ts_tablet_manager.cc:1397] T 34ab7ee573284550aaac9f814e8bd7ab P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.132s user 0.002s sys 0.003s
I20250113 02:51:24.461637 736 tablet_service.cc:1467] Processing CreateTablet for tablet 2470b010b571444eba3f132323038f33 (DEFAULT_TABLE table=default.Table_f08525e2a08e48ec85ef55287d0bac49 [id=96ff1ee9327f41a5a1bb2757b188254a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:24.461858 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 247af1a17df04b27898f86642336d51e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:24.462033 736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2470b010b571444eba3f132323038f33. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:24.462200 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59fadc693f554854bc2874f8db7c64c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:24.463635 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65d9bbd8c15f40a4a50c1695fb5e648e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:24.463660 1024 tablet_service.cc:1467] Processing CreateTablet for tablet 569d666c128a45b7a526c13be5090ae3 (DEFAULT_TABLE table=default.Table_349bd0d1105d42e4889232adbacc4d3a [id=f10700c3143d4a6c9f4f45508db7fa93]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:24.464048 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 569d666c128a45b7a526c13be5090ae3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:24.464128 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 569d666c128a45b7a526c13be5090ae3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:24.464294 1442 raft_consensus.cc:491] T d3b695ac20ba4205b2d8732db4740531 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:24.464439 1442 raft_consensus.cc:513] T d3b695ac20ba4205b2d8732db4740531 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:24.465170 1442 leader_election.cc:290] T d3b695ac20ba4205b2d8732db4740531 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:24.465147 1192 raft_consensus.cc:357] T 34ab7ee573284550aaac9f814e8bd7ab P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:24.466495 1192 raft_consensus.cc:383] T 34ab7ee573284550aaac9f814e8bd7ab P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:24.466895 1192 raft_consensus.cc:738] T 34ab7ee573284550aaac9f814e8bd7ab P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:24.468617 1039 tablet_service.cc:1467] Processing CreateTablet for tablet 247af1a17df04b27898f86642336d51e (DEFAULT_TABLE table=default.Table_2087b79408c949cc88238ba68aabcaf1 [id=43940b76df2c4fb5bc58e5e2975cac8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:24.468322 1028 tablet_service.cc:1467] Processing CreateTablet for tablet 59fadc693f554854bc2874f8db7c64c8 (DEFAULT_TABLE table=default.Table_17489ca6f295469eb97e9a3a5c7cb1aa [id=0e77d79275d44f37b11395f03d737c59]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:24.469013 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 247af1a17df04b27898f86642336d51e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:24.469280 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 883027967e0c442a91ac820210b9c6c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:24.468290 1192 consensus_queue.cc:260] T 34ab7ee573284550aaac9f814e8bd7ab P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:24.469998 1034 tablet_service.cc:1467] Processing CreateTablet for tablet b3ce6d851f564b849796f174860aacd5 (DEFAULT_TABLE table=default.Table_b19190849e5245a380b5349a7abce2d5 [id=7084032f38d34a358b50e973136bd134]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:24.470959 1040 tablet_service.cc:1467] Processing CreateTablet for tablet 2470b010b571444eba3f132323038f33 (DEFAULT_TABLE table=default.Table_f08525e2a08e48ec85ef55287d0bac49 [id=96ff1ee9327f41a5a1bb2757b188254a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:24.471669 1038 tablet_service.cc:1467] Processing CreateTablet for tablet 9b2d0aa6f0e9456d95f4e48c360ab6e8 (DEFAULT_TABLE table=default.Table_a627be2353214265addee118a16ade57 [id=8b48305e2d1b4a0fa1c1fab70d570cf6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:24.470211 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59fadc693f554854bc2874f8db7c64c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:24.472646 1040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2470b010b571444eba3f132323038f33. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:24.472685 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d304b65eba8c40c6922333506a60540c" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
I20250113 02:51:24.472707 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3b695ac20ba4205b2d8732db4740531" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
I20250113 02:51:24.473309 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b2d0aa6f0e9456d95f4e48c360ab6e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:24.473810 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2c6c8c17d6e499d84ad7c0c23889cc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:24.479056 1192 ts_tablet_manager.cc:1428] T 34ab7ee573284550aaac9f814e8bd7ab P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.016s user 0.002s sys 0.001s
W20250113 02:51:24.480633 995 leader_election.cc:343] T d304b65eba8c40c6922333506a60540c P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:24.481011 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet 21c4b419bf2148e0b140301fa80622bb (table default.Table_4a0258998d4f413b80d701c74526b86d [id=6fd9215b11274a76b06813227889927b]) successfully deleted
I20250113 02:51:24.483777 1191 ts_tablet_manager.cc:1428] T 157905316fc545b6a7864ae1c4a2a584 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.031s user 0.003s sys 0.000s
I20250113 02:51:24.484215 1191 tablet_bootstrap.cc:492] T be80b631ccd747f1b9355a00787a7b55 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:24.488610 446 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_c59be71e676f4fbc8dee9729b5555abb" } modify_external_catalogs: true
I20250113 02:51:24.488806 446 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_c59be71e676f4fbc8dee9729b5555abb" } modify_external_catalogs: true
W20250113 02:51:24.494035 995 leader_election.cc:343] T d3b695ac20ba4205b2d8732db4740531 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:24.494122 1192 tablet_bootstrap.cc:492] T 32351d6de1574353a6e8073d333da477 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:24.494066 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f4379f7fc6b54e6785f27975d9030cce. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:24.503298 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 569d666c128a45b7a526c13be5090ae3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:24.503649 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2c6c8c17d6e499d84ad7c0c23889cc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:24.506318 1194 ts_tablet_manager.cc:1397] T f1a02c37cb4f43f8a9de6484f4431958 P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.076s user 0.006s sys 0.004s
I20250113 02:51:24.507063 1196 ts_tablet_manager.cc:1905] T b0361f11d0974ed19e7d4eb1c32a2f47 P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:24.507591 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet b33025a462e74f2aa26cc3779f36615b
I20250113 02:51:24.508102 452 catalog_manager.cc:6596] ReplaceTablet: tablet b33025a462e74f2aa26cc3779f36615b deleted and replaced by tablet f9269967dea94fc3a337546ad0842e11
W20250113 02:51:24.508486 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6e1284c8aaba44b5810b0d06d36757eb consensus state has no leader
I20250113 02:51:24.508883 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 2470b010b571444eba3f132323038f33
I20250113 02:51:24.509572 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 65d9bbd8c15f40a4a50c1695fb5e648e
W20250113 02:51:24.509527 851 leader_election.cc:343] T b0361f11d0974ed19e7d4eb1c32a2f47 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:24.510681 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_15bd9f6352964b8ba8ba9f4f021011b9" } new_table_name: "default.Table_9417f16fb6384fde8e191ca02808aa78" modify_external_catalogs: true
I20250113 02:51:24.510911 448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_df5d17a7c90c4a5a9e0d75558749d6a8" } new_table_name: "default.Table_92ea541283614df981d96c5b74362f1d" modify_external_catalogs: true
I20250113 02:51:24.511164 890 tablet_service.cc:1467] Processing CreateTablet for tablet 2470b010b571444eba3f132323038f33 (DEFAULT_TABLE table=default.Table_f08525e2a08e48ec85ef55287d0bac49 [id=96ff1ee9327f41a5a1bb2757b188254a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:24.512156 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_8fb00fc64f6c46fdb32a6c8159bd63eb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:24.512648 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8fb00fc64f6c46fdb32a6c8159bd63eb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:24.513024 449 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_2cb862a84f3f44c5bcfb646461a5af1c" } modify_external_catalogs: true
I20250113 02:51:24.513337 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_276aa0166db04badb3d250a05e6e7f50" } new_table_name: "default.Table_24d9af3d0b0740d69c791fc96916ef0a" modify_external_catalogs: true
I20250113 02:51:24.514045 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_9ecf36f09ace4d59b57e75e0967b2956"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:24.514533 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9ecf36f09ace4d59b57e75e0967b2956 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:24.515815 449 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_2cb862a84f3f44c5bcfb646461a5af1c" } modify_external_catalogs: true
I20250113 02:51:24.516155 888 tablet_service.cc:1467] Processing CreateTablet for tablet b3ce6d851f564b849796f174860aacd5 (DEFAULT_TABLE table=default.Table_b19190849e5245a380b5349a7abce2d5 [id=7084032f38d34a358b50e973136bd134]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:24.518340 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_d1a29205746d47a6bcf4719cdaeaf285"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:24.518509 750 tablet_service.cc:1514] Processing DeleteTablet for tablet 65d9bbd8c15f40a4a50c1695fb5e648e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:24 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:24.518659 744 tablet_service.cc:1514] Processing DeleteTablet for tablet b33025a462e74f2aa26cc3779f36615b with delete_type TABLET_DATA_DELETED (replaced by tablet f9269967dea94fc3a337546ad0842e11) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:24.518776 1191 tablet_bootstrap.cc:654] T be80b631ccd747f1b9355a00787a7b55 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
W20250113 02:51:24.518975 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d1a29205746d47a6bcf4719cdaeaf285 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:24.520020 899 tablet_service.cc:1467] Processing CreateTablet for tablet 9b2d0aa6f0e9456d95f4e48c360ab6e8 (DEFAULT_TABLE table=default.Table_a627be2353214265addee118a16ade57 [id=8b48305e2d1b4a0fa1c1fab70d570cf6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:24.524533 1194 raft_consensus.cc:357] T f1a02c37cb4f43f8a9de6484f4431958 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:24.526072 737 tablet_service.cc:1514] Processing DeleteTablet for tablet 2470b010b571444eba3f132323038f33 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:24 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:24.526373 1194 raft_consensus.cc:383] T f1a02c37cb4f43f8a9de6484f4431958 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:24.526474 1194 raft_consensus.cc:738] T f1a02c37cb4f43f8a9de6484f4431958 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:24.527325 1194 consensus_queue.cc:260] T f1a02c37cb4f43f8a9de6484f4431958 P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:24.527426 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_67d32aa180f44d0e9614b4e367747129" } new_table_name: "default.Table_bb3ef05cbb904c14a9c2347246fa2fab" modify_external_catalogs: true
I20250113 02:51:24.527896 1194 ts_tablet_manager.cc:1428] T f1a02c37cb4f43f8a9de6484f4431958 P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.021s user 0.001s sys 0.001s
I20250113 02:51:24.529201 1194 tablet_bootstrap.cc:492] T d7920c0544c0466bad6bd8c09aae1a7d P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:24.529644 744 tablet_service.cc:1467] Processing CreateTablet for tablet afcb5b1977c447d3a51de9cb72d7196a (DEFAULT_TABLE table=default.Table_d018642ee8b5413b90b86e9e114353a3 [id=d0ffa26e65894ab89facfccd75985d51]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:24.530031 896 tablet_service.cc:1514] Processing DeleteTablet for tablet 2470b010b571444eba3f132323038f33 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:24 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:24.530081 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet afcb5b1977c447d3a51de9cb72d7196a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:24.534585 894 tablet_service.cc:1514] Processing DeleteTablet for tablet b33025a462e74f2aa26cc3779f36615b with delete_type TABLET_DATA_DELETED (replaced by tablet f9269967dea94fc3a337546ad0842e11) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:24.536360 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2470b010b571444eba3f132323038f33. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:24.536675 1191 tablet_bootstrap.cc:492] T be80b631ccd747f1b9355a00787a7b55 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:24.536840 1191 ts_tablet_manager.cc:1397] T be80b631ccd747f1b9355a00787a7b55 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.053s user 0.000s sys 0.003s
I20250113 02:51:24.537451 894 tablet_service.cc:1514] Processing DeleteTablet for tablet 65d9bbd8c15f40a4a50c1695fb5e648e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:24 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:24.537578 1191 raft_consensus.cc:357] T be80b631ccd747f1b9355a00787a7b55 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:24.540036 1191 raft_consensus.cc:383] T be80b631ccd747f1b9355a00787a7b55 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:24.540132 1191 raft_consensus.cc:738] T be80b631ccd747f1b9355a00787a7b55 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:24.540359 1191 consensus_queue.cc:260] T be80b631ccd747f1b9355a00787a7b55 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:24.540848 1191 ts_tablet_manager.cc:1428] T be80b631ccd747f1b9355a00787a7b55 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250113 02:51:24.542273 1191 tablet_bootstrap.cc:492] T b0361f11d0974ed19e7d4eb1c32a2f47 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:24.544539 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3ce6d851f564b849796f174860aacd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:24.554188 894 tablet_service.cc:1467] Processing CreateTablet for tablet afcb5b1977c447d3a51de9cb72d7196a (DEFAULT_TABLE table=default.Table_d018642ee8b5413b90b86e9e114353a3 [id=d0ffa26e65894ab89facfccd75985d51]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:24.555325 1191 tablet_bootstrap.cc:654] T b0361f11d0974ed19e7d4eb1c32a2f47 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:24.556638 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 883027967e0c442a91ac820210b9c6c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:24.558620 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 9e736211eb3c4536b9fd1d90d2d83838
I20250113 02:51:24.559293 744 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e736211eb3c4536b9fd1d90d2d83838 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:24 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:24.560259 446 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet 2c2465b5467143cf9af9362baaad64ac
I20250113 02:51:24.562315 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3ce6d851f564b849796f174860aacd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:24.562251 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_aebf098beedb49a4acfdb767bf5ac454"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:24.562520 744 tablet_service.cc:1514] Processing DeleteTablet for tablet be80b631ccd747f1b9355a00787a7b55 with delete_type TABLET_DATA_DELETED (replaced by tablet e70658c90555463498d3fc2e2f13ea7f) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:24.562561 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet afcb5b1977c447d3a51de9cb72d7196a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:24.564447 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 542df00229f645788f1b821c0ce00c46. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:24.568323 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b2d0aa6f0e9456d95f4e48c360ab6e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:24.568815 896 tablet_service.cc:1514] Processing DeleteTablet for tablet f1a02c37cb4f43f8a9de6484f4431958 with delete_type TABLET_DATA_DELETED (replaced by tablet 2c399784af46426b8327d175c7a6cce8) from {username='slave'} at 127.0.0.1:37366
W20250113 02:51:24.570552 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_aebf098beedb49a4acfdb767bf5ac454 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:24.572676 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_44413b1d27e14bff98f1a50e0d1a4dd4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:24.573997 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_44413b1d27e14bff98f1a50e0d1a4dd4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:24.581591 1195 ts_tablet_manager.cc:1918] T 1c26074fdd804bd79cb0c295622d11ff P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:24.583837 880 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e736211eb3c4536b9fd1d90d2d83838 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:24 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:24.582386 1525 raft_consensus.cc:491] T 41bcb2eafff14fa8a040982c723f99c1 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:24.584233 1525 raft_consensus.cc:513] T 41bcb2eafff14fa8a040982c723f99c1 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:24.585201 1525 leader_election.cc:290] T 41bcb2eafff14fa8a040982c723f99c1 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:24.586287 1195 log.cc:1198] T 1c26074fdd804bd79cb0c295622d11ff P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/1c26074fdd804bd79cb0c295622d11ff
I20250113 02:51:24.586273 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41bcb2eafff14fa8a040982c723f99c1" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:24.586514 918 raft_consensus.cc:2463] T 41bcb2eafff14fa8a040982c723f99c1 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1186fa869eb14812832fa6562531f196 in term 0.
I20250113 02:51:24.587088 711 leader_election.cc:304] T 41bcb2eafff14fa8a040982c723f99c1 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1186fa869eb14812832fa6562531f196, df69f48656b44bb890f76587ee69cc0f; no voters:
I20250113 02:51:24.587553 1525 raft_consensus.cc:2798] T 41bcb2eafff14fa8a040982c723f99c1 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 02:51:24.587684 1525 raft_consensus.cc:491] T 41bcb2eafff14fa8a040982c723f99c1 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 02:51:24.587802 1525 raft_consensus.cc:3054] T 41bcb2eafff14fa8a040982c723f99c1 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:24.591734 1525 raft_consensus.cc:513] T 41bcb2eafff14fa8a040982c723f99c1 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:24.593408 1525 leader_election.cc:290] T 41bcb2eafff14fa8a040982c723f99c1 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 election: Requested vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:24.593868 1195 ts_tablet_manager.cc:1939] T 1c26074fdd804bd79cb0c295622d11ff P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:24.595259 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet 1c26074fdd804bd79cb0c295622d11ff (table default.Table_9ef18ee71ad843c5921750a2f8478855 [id=bd8bd572ed524f2cae01f820f07e3c36]) successfully deleted
I20250113 02:51:24.596230 1195 tablet_replica.cc:331] T 157905316fc545b6a7864ae1c4a2a584 P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:24.596477 1195 raft_consensus.cc:2238] T 157905316fc545b6a7864ae1c4a2a584 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:24.596657 1191 tablet_bootstrap.cc:492] T b0361f11d0974ed19e7d4eb1c32a2f47 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:24.598892 1191 ts_tablet_manager.cc:1397] T b0361f11d0974ed19e7d4eb1c32a2f47 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.057s user 0.024s sys 0.015s
I20250113 02:51:24.598930 1195 raft_consensus.cc:2267] T 157905316fc545b6a7864ae1c4a2a584 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:24.599741 1191 raft_consensus.cc:357] T b0361f11d0974ed19e7d4eb1c32a2f47 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:24.599946 1191 raft_consensus.cc:383] T b0361f11d0974ed19e7d4eb1c32a2f47 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:24.600066 1191 raft_consensus.cc:738] T b0361f11d0974ed19e7d4eb1c32a2f47 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:24.600124 1031 tablet_service.cc:1514] Processing DeleteTablet for tablet b33025a462e74f2aa26cc3779f36615b with delete_type TABLET_DATA_DELETED (replaced by tablet f9269967dea94fc3a337546ad0842e11) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:24.600311 1191 consensus_queue.cc:260] T b0361f11d0974ed19e7d4eb1c32a2f47 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:24.600854 1191 ts_tablet_manager.cc:1428] T b0361f11d0974ed19e7d4eb1c32a2f47 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250113 02:51:24.601087 1030 tablet_service.cc:1514] Processing DeleteTablet for tablet 2470b010b571444eba3f132323038f33 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:24 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:24.601146 1191 tablet_bootstrap.cc:492] T f1a02c37cb4f43f8a9de6484f4431958 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:24.601279 1042 tablet_service.cc:1514] Processing DeleteTablet for tablet 65d9bbd8c15f40a4a50c1695fb5e648e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:24 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:24.602831 1191 tablet_bootstrap.cc:654] T f1a02c37cb4f43f8a9de6484f4431958 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:24.601147 1195 ts_tablet_manager.cc:1905] T 157905316fc545b6a7864ae1c4a2a584 P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:24.604523 1525 raft_consensus.cc:491] T be80b631ccd747f1b9355a00787a7b55 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:24.604745 1525 raft_consensus.cc:513] T be80b631ccd747f1b9355a00787a7b55 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:24.605986 1525 leader_election.cc:290] T be80b631ccd747f1b9355a00787a7b55 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:24.608552 1042 tablet_service.cc:1467] Processing CreateTablet for tablet afcb5b1977c447d3a51de9cb72d7196a (DEFAULT_TABLE table=default.Table_d018642ee8b5413b90b86e9e114353a3 [id=d0ffa26e65894ab89facfccd75985d51]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:24.609112 1042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet afcb5b1977c447d3a51de9cb72d7196a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:24.616392 1030 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e736211eb3c4536b9fd1d90d2d83838 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:24 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:24.617789 1611 raft_consensus.cc:491] T 438d1d2300e044f39a15586f335935ba P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:24.621483 1611 raft_consensus.cc:513] T 438d1d2300e044f39a15586f335935ba P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:24.622352 1611 leader_election.cc:290] T 438d1d2300e044f39a15586f335935ba P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:24.623823 1062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "438d1d2300e044f39a15586f335935ba" candidate_uuid: "df69f48656b44bb890f76587ee69cc0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:24.624161 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "438d1d2300e044f39a15586f335935ba" candidate_uuid: "df69f48656b44bb890f76587ee69cc0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
I20250113 02:51:24.624855 1438 raft_consensus.cc:491] T d3b695ac20ba4205b2d8732db4740531 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:24.628552 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41bcb2eafff14fa8a040982c723f99c1" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f"
W20250113 02:51:24.630116 711 leader_election.cc:343] T be80b631ccd747f1b9355a00787a7b55 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Not found: Tablet not found: be80b631ccd747f1b9355a00787a7b55
I20250113 02:51:24.630976 744 tablet_service.cc:1514] Processing DeleteTablet for tablet b0361f11d0974ed19e7d4eb1c32a2f47 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:17 UTC) from {username='slave'} at 127.0.0.1:55398
W20250113 02:51:24.631855 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 51d9483ecacb413796d0fd9e7560eac1 consensus state has no leader
W20250113 02:51:24.632201 451 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d59134b029a143038837366f51f00d28 has no consensus state
I20250113 02:51:24.632897 918 raft_consensus.cc:3054] T 41bcb2eafff14fa8a040982c723f99c1 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Advancing to term 1
W20250113 02:51:24.633061 448 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f2c6c8c17d6e499d84ad7c0c23889cc4 consensus state has no leader
W20250113 02:51:24.633514 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet be4b2a8a3927410eaf12e7592646e4af consensus state has no leader
I20250113 02:51:24.633803 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 438604e8859f422f9caccb61c5b5b125
I20250113 02:51:24.634938 744 tablet_service.cc:1514] Processing DeleteTablet for tablet 438604e8859f422f9caccb61c5b5b125 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:24 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:24.634999 1192 tablet_bootstrap.cc:654] T 32351d6de1574353a6e8073d333da477 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:24.636085 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_677784013a07442b9963695510a0a055"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:24.636816 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_677784013a07442b9963695510a0a055 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250113 02:51:24.637939 998 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:38467, user_credentials={real_user=slave}} blocked reactor thread for 267362us
I20250113 02:51:24.639647 1438 raft_consensus.cc:513] T d3b695ac20ba4205b2d8732db4740531 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
W20250113 02:51:24.641184 852 leader_election.cc:343] T 438d1d2300e044f39a15586f335935ba P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:24.643904 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet 438604e8859f422f9caccb61c5b5b125 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:24 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:24.645022 1030 tablet_service.cc:1514] Processing DeleteTablet for tablet 34ab7ee573284550aaac9f814e8bd7ab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:17 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:24.646751 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d304b65eba8c40c6922333506a60540c" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:24.648031 1062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3b695ac20ba4205b2d8732db4740531" candidate_uuid: "df69f48656b44bb890f76587ee69cc0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:24.648280 1062 raft_consensus.cc:2463] T d3b695ac20ba4205b2d8732db4740531 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df69f48656b44bb890f76587ee69cc0f in term 0.
I20250113 02:51:24.650453 453 master_service.cc:946] ReplaceTablet: received request to replace tablet 9f8eb43a39774d2c8e9cb4e88a405b0f from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:24.650732 852 leader_election.cc:304] T d3b695ac20ba4205b2d8732db4740531 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f; no voters:
I20250113 02:51:24.651033 1438 leader_election.cc:290] T d3b695ac20ba4205b2d8732db4740531 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:24.652674 1438 raft_consensus.cc:2798] T d3b695ac20ba4205b2d8732db4740531 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 02:51:24.652791 1438 raft_consensus.cc:491] T d3b695ac20ba4205b2d8732db4740531 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 02:51:24.653870 1438 raft_consensus.cc:3054] T d3b695ac20ba4205b2d8732db4740531 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Advancing to term 1
W20250113 02:51:24.654413 851 leader_election.cc:343] T 438d1d2300e044f39a15586f335935ba P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:24.665961 851 leader_election.cc:304] T 438d1d2300e044f39a15586f335935ba P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df69f48656b44bb890f76587ee69cc0f; no voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca
I20250113 02:51:24.671330 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3b695ac20ba4205b2d8732db4740531" candidate_uuid: "df69f48656b44bb890f76587ee69cc0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
I20250113 02:51:24.672652 1195 ts_tablet_manager.cc:1918] T 157905316fc545b6a7864ae1c4a2a584 P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:24.672845 1195 log.cc:1198] T 157905316fc545b6a7864ae1c4a2a584 P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/157905316fc545b6a7864ae1c4a2a584
I20250113 02:51:24.673295 1195 ts_tablet_manager.cc:1939] T 157905316fc545b6a7864ae1c4a2a584 P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:24.674654 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet 157905316fc545b6a7864ae1c4a2a584 (table default.Table_5b0e60572cb54fcca8b3b5b94326ae3a [id=9f5add219a7c48da9a8afa966e22d1bd]) successfully deleted
I20250113 02:51:24.675087 1195 tablet_replica.cc:331] T be80b631ccd747f1b9355a00787a7b55 P 1186fa869eb14812832fa6562531f196: stopping tablet replica
W20250113 02:51:24.675225 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 65d9bbd8c15f40a4a50c1695fb5e648e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 65d9bbd8c15f40a4a50c1695fb5e648e already in progress: creating tablet
I20250113 02:51:24.675288 1195 raft_consensus.cc:2238] T be80b631ccd747f1b9355a00787a7b55 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:24.675454 1195 raft_consensus.cc:2267] T be80b631ccd747f1b9355a00787a7b55 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250113 02:51:24.675459 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet b33025a462e74f2aa26cc3779f36615b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b33025a462e74f2aa26cc3779f36615b already in progress: creating tablet
W20250113 02:51:24.675650 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 2470b010b571444eba3f132323038f33 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2470b010b571444eba3f132323038f33 already in progress: creating tablet
W20250113 02:51:24.675873 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 9e736211eb3c4536b9fd1d90d2d83838 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9e736211eb3c4536b9fd1d90d2d83838 already in progress: creating tablet
I20250113 02:51:24.676432 1191 tablet_bootstrap.cc:492] T f1a02c37cb4f43f8a9de6484f4431958 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:24.676599 1191 ts_tablet_manager.cc:1397] T f1a02c37cb4f43f8a9de6484f4431958 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.076s user 0.021s sys 0.032s
I20250113 02:51:24.677312 1191 raft_consensus.cc:357] T f1a02c37cb4f43f8a9de6484f4431958 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:24.677531 1191 raft_consensus.cc:383] T f1a02c37cb4f43f8a9de6484f4431958 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:24.677599 1191 raft_consensus.cc:738] T f1a02c37cb4f43f8a9de6484f4431958 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:24.677834 1191 consensus_queue.cc:260] T f1a02c37cb4f43f8a9de6484f4431958 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:24.678409 1191 ts_tablet_manager.cc:1428] T f1a02c37cb4f43f8a9de6484f4431958 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250113 02:51:24.680678 1191 tablet_bootstrap.cc:492] T d7920c0544c0466bad6bd8c09aae1a7d P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:24.680795 1195 ts_tablet_manager.cc:1905] T be80b631ccd747f1b9355a00787a7b55 P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:24.687435 744 tablet_service.cc:1514] Processing DeleteTablet for tablet f1a02c37cb4f43f8a9de6484f4431958 with delete_type TABLET_DATA_DELETED (replaced by tablet 2c399784af46426b8327d175c7a6cce8) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:24.688918 1191 tablet_bootstrap.cc:654] T d7920c0544c0466bad6bd8c09aae1a7d P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:24.689939 1438 raft_consensus.cc:513] T d3b695ac20ba4205b2d8732db4740531 P df69f48656b44bb890f76587ee69cc0f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:24.676067 1618 raft_consensus.cc:491] T 4fe54131aafb40ee9d8bc5ea299af193 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:24.687312 918 raft_consensus.cc:2463] T 41bcb2eafff14fa8a040982c723f99c1 P df69f48656b44bb890f76587ee69cc0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1186fa869eb14812832fa6562531f196 in term 1.
I20250113 02:51:24.694664 711 leader_election.cc:304] T 41bcb2eafff14fa8a040982c723f99c1 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1186fa869eb14812832fa6562531f196, df69f48656b44bb890f76587ee69cc0f; no voters:
I20250113 02:51:24.694994 1525 raft_consensus.cc:2798] T 41bcb2eafff14fa8a040982c723f99c1 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Leader election won for term 1
I20250113 02:51:24.695647 1442 raft_consensus.cc:491] T d7920c0544c0466bad6bd8c09aae1a7d P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:24.693356 1618 raft_consensus.cc:513] T 4fe54131aafb40ee9d8bc5ea299af193 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:24.701855 1195 ts_tablet_manager.cc:1918] T be80b631ccd747f1b9355a00787a7b55 P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:24.703120 1195 log.cc:1198] T be80b631ccd747f1b9355a00787a7b55 P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/be80b631ccd747f1b9355a00787a7b55
I20250113 02:51:24.703684 1195 ts_tablet_manager.cc:1939] T be80b631ccd747f1b9355a00787a7b55 P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:24.703686 1611 raft_consensus.cc:2743] T 438d1d2300e044f39a15586f335935ba P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:24.704839 1442 raft_consensus.cc:513] T d7920c0544c0466bad6bd8c09aae1a7d P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:24.705124 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet be80b631ccd747f1b9355a00787a7b55 (table default.Table_2565c0551f9943a084ace8f7e463e59d [id=fe6f8d302c3e489d8bb832503fd05642]) successfully deleted
I20250113 02:51:24.705185 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe54131aafb40ee9d8bc5ea299af193" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
I20250113 02:51:24.705521 1442 leader_election.cc:290] T d7920c0544c0466bad6bd8c09aae1a7d P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
I20250113 02:51:24.706063 1195 tablet_replica.cc:331] T b0361f11d0974ed19e7d4eb1c32a2f47 P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:24.706511 1195 raft_consensus.cc:2238] T b0361f11d0974ed19e7d4eb1c32a2f47 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:24.706642 1442 raft_consensus.cc:491] T 41bcb2eafff14fa8a040982c723f99c1 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:24.706825 1442 raft_consensus.cc:513] T 41bcb2eafff14fa8a040982c723f99c1 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:24.706979 1195 raft_consensus.cc:2267] T b0361f11d0974ed19e7d4eb1c32a2f47 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:24.706604 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7920c0544c0466bad6bd8c09aae1a7d" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
I20250113 02:51:24.708114 1195 ts_tablet_manager.cc:1905] T b0361f11d0974ed19e7d4eb1c32a2f47 P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:24.708283 1194 tablet_bootstrap.cc:654] T d7920c0544c0466bad6bd8c09aae1a7d P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:24.710716 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_5e92e31ef2494f34a1fab99e7c6bd9f7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:24.710778 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_a5f105b8a4ad4933b499ebaa8ecca921"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:24.711086 881 tablet_service.cc:1514] Processing DeleteTablet for tablet 438604e8859f422f9caccb61c5b5b125 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:24 UTC) from {username='slave'} at 127.0.0.1:37366
W20250113 02:51:24.711338 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5e92e31ef2494f34a1fab99e7c6bd9f7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250113 02:51:24.711369 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a5f105b8a4ad4933b499ebaa8ecca921 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:24.712262 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_7843edf8e2e14e51b6ab66031605949e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:24.712684 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7843edf8e2e14e51b6ab66031605949e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:24.713853 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_5030a9f5589a4f85865ee2ba6562ee1d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:24.714344 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5030a9f5589a4f85865ee2ba6562ee1d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:24.715137 1525 raft_consensus.cc:695] T 41bcb2eafff14fa8a040982c723f99c1 P 1186fa869eb14812832fa6562531f196 [term 1 LEADER]: Becoming Leader. State: Replica: 1186fa869eb14812832fa6562531f196, State: Running, Role: LEADER
I20250113 02:51:24.715441 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_8fb00fc64f6c46fdb32a6c8159bd63eb" } new_table_name: "default.Table_59b60c21ea364f89967c58758807a6a4" modify_external_catalogs: true
I20250113 02:51:24.715415 1525 consensus_queue.cc:237] T 41bcb2eafff14fa8a040982c723f99c1 P 1186fa869eb14812832fa6562531f196 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:24.716627 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_1505767419694e5ba79a90c46780bbb7" } modify_external_catalogs: true
I20250113 02:51:24.717036 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_1505767419694e5ba79a90c46780bbb7" } modify_external_catalogs: true
I20250113 02:51:24.717576 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_41d844ed7a144e59bd9e14d543ad0573" } modify_external_catalogs: true
I20250113 02:51:24.717741 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_41d844ed7a144e59bd9e14d543ad0573" } modify_external_catalogs: true
W20250113 02:51:24.719511 995 leader_election.cc:343] T 4fe54131aafb40ee9d8bc5ea299af193 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Illegal state: must be running to vote when last-logged opid is not known
W20250113 02:51:24.719808 995 leader_election.cc:343] T d7920c0544c0466bad6bd8c09aae1a7d P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:24.720299 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3b695ac20ba4205b2d8732db4740531" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:24.707698 1062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3b695ac20ba4205b2d8732db4740531" candidate_uuid: "df69f48656b44bb890f76587ee69cc0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca"
I20250113 02:51:24.720551 917 raft_consensus.cc:2388] T d3b695ac20ba4205b2d8732db4740531 P df69f48656b44bb890f76587ee69cc0f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4216a7f853d642b7bc3e5cff458f3aca in current term 1: Already voted for candidate df69f48656b44bb890f76587ee69cc0f in this term.
W20250113 02:51:24.720929 851 leader_election.cc:343] T d3b695ac20ba4205b2d8732db4740531 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Illegal state: must be running to vote when last-logged opid is not known
W20250113 02:51:24.721160 851 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:40095, user_credentials={real_user=slave}} blocked reactor thread for 49003.2us
I20250113 02:51:24.721865 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3b695ac20ba4205b2d8732db4740531" candidate_uuid: "df69f48656b44bb890f76587ee69cc0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196"
I20250113 02:51:24.722821 1191 tablet_bootstrap.cc:492] T d7920c0544c0466bad6bd8c09aae1a7d P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:24.723029 1191 ts_tablet_manager.cc:1397] T d7920c0544c0466bad6bd8c09aae1a7d P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.042s user 0.008s sys 0.001s
W20250113 02:51:24.723451 851 leader_election.cc:343] T d3b695ac20ba4205b2d8732db4740531 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:24.723944 1062 raft_consensus.cc:3054] T d3b695ac20ba4205b2d8732db4740531 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:24.724324 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41bcb2eafff14fa8a040982c723f99c1" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
I20250113 02:51:24.724280 1191 raft_consensus.cc:357] T d7920c0544c0466bad6bd8c09aae1a7d P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:24.724520 1191 raft_consensus.cc:383] T d7920c0544c0466bad6bd8c09aae1a7d P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:24.724628 1191 raft_consensus.cc:738] T d7920c0544c0466bad6bd8c09aae1a7d P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:24.724931 1191 consensus_queue.cc:260] T d7920c0544c0466bad6bd8c09aae1a7d P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:24.725550 1191 ts_tablet_manager.cc:1428] T d7920c0544c0466bad6bd8c09aae1a7d P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250113 02:51:24.727033 1442 leader_election.cc:290] T 41bcb2eafff14fa8a040982c723f99c1 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
I20250113 02:51:24.722554 1618 leader_election.cc:290] T 4fe54131aafb40ee9d8bc5ea299af193 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
I20250113 02:51:24.730746 1191 tablet_bootstrap.cc:492] T ba61c86743e24666b30580c36b9228c3 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:24.733610 1438 leader_election.cc:290] T d3b695ac20ba4205b2d8732db4740531 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 election: Requested vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:24.736474 1191 tablet_bootstrap.cc:654] T ba61c86743e24666b30580c36b9228c3 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:24.737779 1195 ts_tablet_manager.cc:1918] T b0361f11d0974ed19e7d4eb1c32a2f47 P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:24.737936 1195 log.cc:1198] T b0361f11d0974ed19e7d4eb1c32a2f47 P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/b0361f11d0974ed19e7d4eb1c32a2f47
I20250113 02:51:24.738420 1195 ts_tablet_manager.cc:1939] T b0361f11d0974ed19e7d4eb1c32a2f47 P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:24.739668 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet b0361f11d0974ed19e7d4eb1c32a2f47 (table default.Table_1031f897b6e24d3abd972aca00ddb48b [id=f9202f2a27134b4984d824717b8c1232]) successfully deleted
I20250113 02:51:24.739758 1195 tablet_replica.cc:331] T f1a02c37cb4f43f8a9de6484f4431958 P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:24.740043 1195 raft_consensus.cc:2238] T f1a02c37cb4f43f8a9de6484f4431958 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus shutting down.
W20250113 02:51:24.740123 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 438604e8859f422f9caccb61c5b5b125 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 438604e8859f422f9caccb61c5b5b125 already in progress: creating tablet
I20250113 02:51:24.740244 1195 raft_consensus.cc:2267] T f1a02c37cb4f43f8a9de6484f4431958 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:24.742142 1195 ts_tablet_manager.cc:1905] T f1a02c37cb4f43f8a9de6484f4431958 P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:24.744333 1196 ts_tablet_manager.cc:1918] T b0361f11d0974ed19e7d4eb1c32a2f47 P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:24.744529 1196 log.cc:1198] T b0361f11d0974ed19e7d4eb1c32a2f47 P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/b0361f11d0974ed19e7d4eb1c32a2f47
I20250113 02:51:24.745450 1062 raft_consensus.cc:2463] T d3b695ac20ba4205b2d8732db4740531 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df69f48656b44bb890f76587ee69cc0f in term 1.
I20250113 02:51:24.746251 852 leader_election.cc:304] T d3b695ac20ba4205b2d8732db4740531 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f; no voters: 1186fa869eb14812832fa6562531f196
I20250113 02:51:24.748351 1192 tablet_bootstrap.cc:492] T 32351d6de1574353a6e8073d333da477 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:24.748481 1438 raft_consensus.cc:2798] T d3b695ac20ba4205b2d8732db4740531 P df69f48656b44bb890f76587ee69cc0f [term 1 FOLLOWER]: Leader election won for term 1
I20250113 02:51:24.748513 1192 ts_tablet_manager.cc:1397] T 32351d6de1574353a6e8073d333da477 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.269s user 0.002s sys 0.005s
I20250113 02:51:24.749485 1195 ts_tablet_manager.cc:1918] T f1a02c37cb4f43f8a9de6484f4431958 P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:24.749806 1438 raft_consensus.cc:695] T d3b695ac20ba4205b2d8732db4740531 P df69f48656b44bb890f76587ee69cc0f [term 1 LEADER]: Becoming Leader. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Running, Role: LEADER
I20250113 02:51:24.750139 1438 consensus_queue.cc:237] T d3b695ac20ba4205b2d8732db4740531 P df69f48656b44bb890f76587ee69cc0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:24.755601 1196 ts_tablet_manager.cc:1939] T b0361f11d0974ed19e7d4eb1c32a2f47 P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:24.756274 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet b0361f11d0974ed19e7d4eb1c32a2f47 (table default.Table_1031f897b6e24d3abd972aca00ddb48b [id=f9202f2a27134b4984d824717b8c1232]) successfully deleted
I20250113 02:51:24.756369 1195 log.cc:1198] T f1a02c37cb4f43f8a9de6484f4431958 P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/f1a02c37cb4f43f8a9de6484f4431958
W20250113 02:51:24.756992 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 0916b41c171a419791797559fbc2807d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0916b41c171a419791797559fbc2807d already in progress: creating tablet
I20250113 02:51:24.756878 1192 raft_consensus.cc:357] T 32351d6de1574353a6e8073d333da477 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:24.757166 1192 raft_consensus.cc:383] T 32351d6de1574353a6e8073d333da477 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:24.757220 1195 ts_tablet_manager.cc:1939] T f1a02c37cb4f43f8a9de6484f4431958 P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:24.757270 1192 raft_consensus.cc:738] T 32351d6de1574353a6e8073d333da477 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
W20250113 02:51:24.757733 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 0fe5421dc3ea4be684cab084f13d8df0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0fe5421dc3ea4be684cab084f13d8df0 already in progress: creating tablet
I20250113 02:51:24.757999 1192 consensus_queue.cc:260] T 32351d6de1574353a6e8073d333da477 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
W20250113 02:51:24.758406 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 153b8f90b5a847b88944f683d306f579 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 153b8f90b5a847b88944f683d306f579 already in progress: creating tablet
I20250113 02:51:24.758678 1192 ts_tablet_manager.cc:1428] T 32351d6de1574353a6e8073d333da477 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.010s user 0.000s sys 0.001s
I20250113 02:51:24.759058 1192 tablet_bootstrap.cc:492] T 1500e4f9c0664a879503afd3edf14c66 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
W20250113 02:51:24.759094 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 42dbd37e20fa47318bc6497423dbe565 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 42dbd37e20fa47318bc6497423dbe565 already in progress: creating tablet
W20250113 02:51:24.759644 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 655b63ff61ff4f4aa08e0449df39352a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 655b63ff61ff4f4aa08e0449df39352a already in progress: creating tablet
W20250113 02:51:24.760195 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet c7842d6f2c074e3b9d1a3378db9856be because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c7842d6f2c074e3b9d1a3378db9856be already in progress: creating tablet
W20250113 02:51:24.760892 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet b4806b18a4194f9c95350d89ba62e13a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b4806b18a4194f9c95350d89ba62e13a already in progress: creating tablet
I20250113 02:51:24.761456 1192 tablet_bootstrap.cc:654] T 1500e4f9c0664a879503afd3edf14c66 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:24.762646 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet f1a02c37cb4f43f8a9de6484f4431958 (table default.Table_f8af5d3e0dc64d3fae054f82a7b33ccf [id=4b739228564344b8b394a4718750dbb0]) successfully deleted
I20250113 02:51:24.766829 1192 tablet_bootstrap.cc:492] T 1500e4f9c0664a879503afd3edf14c66 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:24.767031 1192 ts_tablet_manager.cc:1397] T 1500e4f9c0664a879503afd3edf14c66 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.002s
I20250113 02:51:24.767457 1198 ts_tablet_manager.cc:1918] T 6c8ca30720664f3a819c4bff2493934d P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:24.767601 1198 log.cc:1198] T 6c8ca30720664f3a819c4bff2493934d P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/6c8ca30720664f3a819c4bff2493934d
I20250113 02:51:24.770578 1198 ts_tablet_manager.cc:1939] T 6c8ca30720664f3a819c4bff2493934d P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
W20250113 02:51:24.770711 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet c7243d27a1c642269a3d0e67632e458c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c7243d27a1c642269a3d0e67632e458c already in progress: creating tablet
I20250113 02:51:24.771320 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41bcb2eafff14fa8a040982c723f99c1" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca"
W20250113 02:51:24.771904 709 leader_election.cc:343] T be80b631ccd747f1b9355a00787a7b55 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Not found: Tablet not found: be80b631ccd747f1b9355a00787a7b55
I20250113 02:51:24.772082 709 leader_election.cc:304] T be80b631ccd747f1b9355a00787a7b55 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
W20250113 02:51:24.772287 709 raft_consensus.cc:2681] T be80b631ccd747f1b9355a00787a7b55 P 1186fa869eb14812832fa6562531f196: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250113 02:51:24.772372 1061 raft_consensus.cc:3054] T 41bcb2eafff14fa8a040982c723f99c1 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:24.773667 1198 tablet_replica.cc:331] T f1a02c37cb4f43f8a9de6484f4431958 P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:24.774060 1618 raft_consensus.cc:491] T fc92f19390fd48a4a45ba8fb839020fd P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:24.774098 1627 raft_consensus.cc:491] T 34ab7ee573284550aaac9f814e8bd7ab P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:24.774446 1618 raft_consensus.cc:513] T fc92f19390fd48a4a45ba8fb839020fd P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:24.775128 1191 tablet_bootstrap.cc:492] T ba61c86743e24666b30580c36b9228c3 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:24.775339 1191 ts_tablet_manager.cc:1397] T ba61c86743e24666b30580c36b9228c3 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.045s user 0.007s sys 0.016s
I20250113 02:51:24.775646 1198 raft_consensus.cc:2238] T f1a02c37cb4f43f8a9de6484f4431958 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:24.775954 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe54131aafb40ee9d8bc5ea299af193" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:24.776201 917 raft_consensus.cc:2463] T 4fe54131aafb40ee9d8bc5ea299af193 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 0.
I20250113 02:51:24.773993 1192 raft_consensus.cc:357] T 1500e4f9c0664a879503afd3edf14c66 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:24.776389 1191 raft_consensus.cc:357] T ba61c86743e24666b30580c36b9228c3 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:24.776680 1191 raft_consensus.cc:383] T ba61c86743e24666b30580c36b9228c3 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:24.776772 1191 raft_consensus.cc:738] T ba61c86743e24666b30580c36b9228c3 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:24.777118 1191 consensus_queue.cc:260] T ba61c86743e24666b30580c36b9228c3 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:24.777673 1062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41bcb2eafff14fa8a040982c723f99c1" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
W20250113 02:51:24.778694 998 leader_election.cc:343] T 93a2729c5a604a27a9bbc451ccb4e128 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Not found: Tablet not found: 93a2729c5a604a27a9bbc451ccb4e128
I20250113 02:51:24.778854 1191 ts_tablet_manager.cc:1428] T ba61c86743e24666b30580c36b9228c3 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250113 02:51:24.779163 1191 tablet_bootstrap.cc:492] T b834f0acf5314bcb9d81c73de1b31842 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:24.779491 454 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_03babf194cc448f296512a09b2e3fe3e" } modify_external_catalogs: true
I20250113 02:51:24.779695 454 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_03babf194cc448f296512a09b2e3fe3e" } modify_external_catalogs: true
I20250113 02:51:24.780892 1191 tablet_bootstrap.cc:654] T b834f0acf5314bcb9d81c73de1b31842 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:24.781042 1618 leader_election.cc:290] T fc92f19390fd48a4a45ba8fb839020fd P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
I20250113 02:51:24.774528 1627 raft_consensus.cc:513] T 34ab7ee573284550aaac9f814e8bd7ab P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:24.782125 1198 raft_consensus.cc:2267] T f1a02c37cb4f43f8a9de6484f4431958 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:24.782099 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet 6c8ca30720664f3a819c4bff2493934d (table default.Table_8aa63512c7bb4bdb86f37c9ee24df209 [id=aadd55280b654ceb96c290786a0bfce0]) successfully deleted
I20250113 02:51:24.785073 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7920c0544c0466bad6bd8c09aae1a7d" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:24.787400 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41bcb2eafff14fa8a040982c723f99c1" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:24.787667 918 raft_consensus.cc:2388] T 41bcb2eafff14fa8a040982c723f99c1 P df69f48656b44bb890f76587ee69cc0f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4216a7f853d642b7bc3e5cff458f3aca in current term 1: Already voted for candidate 1186fa869eb14812832fa6562531f196 in this term.
I20250113 02:51:24.788511 1196 tablet_replica.cc:331] T 157905316fc545b6a7864ae1c4a2a584 P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
W20250113 02:51:24.791226 995 leader_election.cc:343] T fc92f19390fd48a4a45ba8fb839020fd P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Not found: Tablet not found: fc92f19390fd48a4a45ba8fb839020fd
I20250113 02:51:24.791404 998 leader_election.cc:304] T 93a2729c5a604a27a9bbc451ccb4e128 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4216a7f853d642b7bc3e5cff458f3aca; no voters: 1186fa869eb14812832fa6562531f196, df69f48656b44bb890f76587ee69cc0f
W20250113 02:51:24.792860 998 leader_election.cc:343] T d304b65eba8c40c6922333506a60540c P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:24.793006 998 leader_election.cc:304] T d304b65eba8c40c6922333506a60540c P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4216a7f853d642b7bc3e5cff458f3aca; no voters: 1186fa869eb14812832fa6562531f196, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:24.793231 1442 raft_consensus.cc:2743] T 93a2729c5a604a27a9bbc451ccb4e128 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:24.793424 1442 raft_consensus.cc:2743] T d304b65eba8c40c6922333506a60540c P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:24.796156 455 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_08def01561d44ec5a88f5b5a82d47e3b" } modify_external_catalogs: true
I20250113 02:51:24.795921 446 catalog_manager.cc:5526] T 41bcb2eafff14fa8a040982c723f99c1 P 1186fa869eb14812832fa6562531f196 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1186fa869eb14812832fa6562531f196 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "1186fa869eb14812832fa6562531f196" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } health_report { overall_health: HEALTHY } } }
I20250113 02:51:24.796334 455 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_08def01561d44ec5a88f5b5a82d47e3b" } modify_external_catalogs: true
I20250113 02:51:24.796907 998 leader_election.cc:304] T d3b695ac20ba4205b2d8732db4740531 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4216a7f853d642b7bc3e5cff458f3aca; no voters: 1186fa869eb14812832fa6562531f196, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:24.797168 1442 raft_consensus.cc:2743] T d3b695ac20ba4205b2d8732db4740531 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:24.797544 998 leader_election.cc:304] T 4fe54131aafb40ee9d8bc5ea299af193 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f; no voters: 1186fa869eb14812832fa6562531f196
I20250113 02:51:24.797696 453 master_service.cc:946] ReplaceTablet: received request to replace tablet 056bf1e67be04d179eace3d93282b145 from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:24.797775 1442 raft_consensus.cc:2798] T 4fe54131aafb40ee9d8bc5ea299af193 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 02:51:24.797852 1442 raft_consensus.cc:491] T 4fe54131aafb40ee9d8bc5ea299af193 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 02:51:24.797935 1442 raft_consensus.cc:3054] T 4fe54131aafb40ee9d8bc5ea299af193 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Advancing to term 1
W20250113 02:51:24.798523 998 leader_election.cc:343] T d7920c0544c0466bad6bd8c09aae1a7d P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:24.798645 998 leader_election.cc:304] T d7920c0544c0466bad6bd8c09aae1a7d P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4216a7f853d642b7bc3e5cff458f3aca; no voters: 1186fa869eb14812832fa6562531f196, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:24.799026 998 leader_election.cc:304] T 41bcb2eafff14fa8a040982c723f99c1 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4216a7f853d642b7bc3e5cff458f3aca; no voters: 1186fa869eb14812832fa6562531f196, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:24.800025 744 tablet_service.cc:1467] Processing CreateTablet for tablet c04be975c8c049b892b16129ed47a142 (DEFAULT_TABLE table=default.Table_eba6422342a742499b1c54520d991881 [id=36184fc3b73745c487d1d7a9a69e85ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:24.800508 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c04be975c8c049b892b16129ed47a142. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:24.800913 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6d6e462c8bfa47eaa43d3695c69d9c0a has no consensus state
I20250113 02:51:24.791687 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet 32351d6de1574353a6e8073d333da477 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:18 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:24.801828 1618 raft_consensus.cc:2743] T d7920c0544c0466bad6bd8c09aae1a7d P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:24.792431 1198 ts_tablet_manager.cc:1905] T f1a02c37cb4f43f8a9de6484f4431958 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:24.809563 741 tablet_service.cc:1467] Processing CreateTablet for tablet 8f2fb58fc24d4692a3fe9a652e076aed (DEFAULT_TABLE table=default.Table_72783655bdee4fcf8c2fff2186fe0a60 [id=8edee52f05e24c03b088ce58f4b3f4d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:24.809966 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f2fb58fc24d4692a3fe9a652e076aed. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:24.810197 898 tablet_service.cc:1467] Processing CreateTablet for tablet 8f2fb58fc24d4692a3fe9a652e076aed (DEFAULT_TABLE table=default.Table_72783655bdee4fcf8c2fff2186fe0a60 [id=8edee52f05e24c03b088ce58f4b3f4d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:24.810845 882 tablet_service.cc:1467] Processing CreateTablet for tablet d47f544f723542bea1d396f586d8d658 (DEFAULT_TABLE table=default.Table_d3474542a47c407ea91c69d7601c4af9 [id=5253ce78460b47c69168f0959713e123]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:24.811687 897 tablet_service.cc:1467] Processing CreateTablet for tablet fe3d35d9aaf64188aaa3c1ac38d279d8 (DEFAULT_TABLE table=default.Table_7e49ba949e8949fa9924acdef98c3947 [id=c8a8725a3ac44d4b8960e2ceccb0ccc6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:24.812223 754 tablet_service.cc:1467] Processing CreateTablet for tablet c73c649a4ca44471af2cf6a77500b065 (DEFAULT_TABLE table=default.Table_77b0b19284064a118a50b9bcfe72765f [id=a0bbd2c6939d449392a1371c3b5ec0a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:24.812578 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c73c649a4ca44471af2cf6a77500b065. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:24.812525 881 tablet_service.cc:1467] Processing CreateTablet for tablet 936b8ceaedd34a019a4fb89a80dd2708 (DEFAULT_TABLE table=default.Table_bb7ff740ee724b3999eeec918a03b1ae [id=1291674e6af743d08c5fff97bbb34c8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:24.812624 892 tablet_service.cc:1467] Processing CreateTablet for tablet c04be975c8c049b892b16129ed47a142 (DEFAULT_TABLE table=default.Table_eba6422342a742499b1c54520d991881 [id=36184fc3b73745c487d1d7a9a69e85ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:24.813117 899 tablet_service.cc:1467] Processing CreateTablet for tablet d59134b029a143038837366f51f00d28 (DEFAULT_TABLE table=default.Table_bb3ef05cbb904c14a9c2347246fa2fab [id=6ae8663361aa4ba989265452dfa21f89]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:24.814623 1627 leader_election.cc:290] T 34ab7ee573284550aaac9f814e8bd7ab P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
I20250113 02:51:24.815168 889 tablet_service.cc:1467] Processing CreateTablet for tablet 98bfe9620e284ab1954433ba9a036392 (DEFAULT_TABLE table=default.Table_064659c197854029a5d7af1798def6d9 [id=e9934f5e1a17427cabd573190c4f16a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:24.816226 746 tablet_service.cc:1467] Processing CreateTablet for tablet 7fb282707196444690352dea3eb1387f (DEFAULT_TABLE table=default.Table_6754736c569e4f55848afc146fe41b74 [id=521c6ec1968841128a4c4179e9d6980f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:24.816220 890 tablet_service.cc:1467] Processing CreateTablet for tablet 39a014ac69a2435a9132a75125063093 (DEFAULT_TABLE table=default.Table_02655512f28b47429ebb70e1676844ea [id=dcb8e4b96b864e75b9b47a658c2592f2]), partition=RANGE (key) PARTITION UNBOUNDED
W20250113 02:51:24.816339 995 leader_election.cc:343] T 34ab7ee573284550aaac9f814e8bd7ab P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Not found: Tablet not found: 34ab7ee573284550aaac9f814e8bd7ab
I20250113 02:51:24.816522 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7fb282707196444690352dea3eb1387f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:24.816677 1034 tablet_service.cc:1467] Processing CreateTablet for tablet c04be975c8c049b892b16129ed47a142 (DEFAULT_TABLE table=default.Table_eba6422342a742499b1c54520d991881 [id=36184fc3b73745c487d1d7a9a69e85ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:24.817173 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c04be975c8c049b892b16129ed47a142. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:24.820217 752 tablet_service.cc:1467] Processing CreateTablet for tablet d565b77f90784d87b64996772c938085 (DEFAULT_TABLE table=default.Table_915411462fb84dd7b917caa2f3a31498 [id=822489edc766460abfd5ef4bf6993ef5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:24.820554 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d565b77f90784d87b64996772c938085. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:24.820529 1029 tablet_service.cc:1467] Processing CreateTablet for tablet d59134b029a143038837366f51f00d28 (DEFAULT_TABLE table=default.Table_bb3ef05cbb904c14a9c2347246fa2fab [id=6ae8663361aa4ba989265452dfa21f89]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:24.820955 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d59134b029a143038837366f51f00d28. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:24.821900 1196 raft_consensus.cc:2238] T 157905316fc545b6a7864ae1c4a2a584 P df69f48656b44bb890f76587ee69cc0f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:24.822180 1196 raft_consensus.cc:2267] T 157905316fc545b6a7864ae1c4a2a584 P df69f48656b44bb890f76587ee69cc0f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:24.823022 1196 ts_tablet_manager.cc:1905] T 157905316fc545b6a7864ae1c4a2a584 P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:24.824498 737 tablet_service.cc:1467] Processing CreateTablet for tablet fe3d35d9aaf64188aaa3c1ac38d279d8 (DEFAULT_TABLE table=default.Table_7e49ba949e8949fa9924acdef98c3947 [id=c8a8725a3ac44d4b8960e2ceccb0ccc6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:24.824514 893 tablet_service.cc:1467] Processing CreateTablet for tablet a2781ba5061a4225b5636366857a229e (DEFAULT_TABLE table=default.Table_41d844ed7a144e59bd9e14d543ad0573 [id=16e500140b9b4bb6a7603e6ea0849cf9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:24.824882 737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe3d35d9aaf64188aaa3c1ac38d279d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:24.825927 1192 raft_consensus.cc:383] T 1500e4f9c0664a879503afd3edf14c66 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:24.828266 750 tablet_service.cc:1467] Processing CreateTablet for tablet 936b8ceaedd34a019a4fb89a80dd2708 (DEFAULT_TABLE table=default.Table_bb7ff740ee724b3999eeec918a03b1ae [id=1291674e6af743d08c5fff97bbb34c8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:24.828624 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 936b8ceaedd34a019a4fb89a80dd2708. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:24.831208 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_e5ce75ec7b044da3ae915dc3ce5421da"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:24.831935 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e5ce75ec7b044da3ae915dc3ce5421da in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:24.832250 743 tablet_service.cc:1467] Processing CreateTablet for tablet 6001369444894b48b51a48fb60ddc52a (DEFAULT_TABLE table=default.Table_ed91fb5c64de42bea271f05b8c284edc [id=b8c683d257794905b44a447d158917ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:24.832584 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6001369444894b48b51a48fb60ddc52a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:24.833704 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_2bf3c3c3eb364670a2d0633238a32fe6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:24.834288 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2bf3c3c3eb364670a2d0633238a32fe6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:24.835623 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_0b8a63bbfac5492e83f0f9bd8e0eb441"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:24.836231 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0b8a63bbfac5492e83f0f9bd8e0eb441 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:24.837778 748 tablet_service.cc:1467] Processing CreateTablet for tablet d47f544f723542bea1d396f586d8d658 (DEFAULT_TABLE table=default.Table_d3474542a47c407ea91c69d7601c4af9 [id=5253ce78460b47c69168f0959713e123]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:24.838124 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d47f544f723542bea1d396f586d8d658. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:24.840947 742 tablet_service.cc:1467] Processing CreateTablet for tablet 98bfe9620e284ab1954433ba9a036392 (DEFAULT_TABLE table=default.Table_064659c197854029a5d7af1798def6d9 [id=e9934f5e1a17427cabd573190c4f16a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:24.841276 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 98bfe9620e284ab1954433ba9a036392. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:24.843204 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_719ed613a9354add97b036faad401762" } new_table_name: "default.Table_23c1655ab4cf4a30b67185ccf4fbb76d" modify_external_catalogs: true
I20250113 02:51:24.844233 736 tablet_service.cc:1467] Processing CreateTablet for tablet 39a014ac69a2435a9132a75125063093 (DEFAULT_TABLE table=default.Table_02655512f28b47429ebb70e1676844ea [id=dcb8e4b96b864e75b9b47a658c2592f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:24.844561 736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 39a014ac69a2435a9132a75125063093. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:24.847468 1061 raft_consensus.cc:2463] T 41bcb2eafff14fa8a040982c723f99c1 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1186fa869eb14812832fa6562531f196 in term 1.
I20250113 02:51:24.848232 753 tablet_service.cc:1467] Processing CreateTablet for tablet 415a1fa4b2f14cacb43937c8a372849a (DEFAULT_TABLE table=default.Table_5181e98c2c214ebcab54c33615482528 [id=c13a3a3a70dd4dd7a493937ae16b6931]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:24.848574 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 415a1fa4b2f14cacb43937c8a372849a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:24.849256 1442 raft_consensus.cc:513] T 4fe54131aafb40ee9d8bc5ea299af193 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:24.849710 880 tablet_service.cc:1467] Processing CreateTablet for tablet 01a455eda0ee488fb3d45a6a09c43833 (DEFAULT_TABLE table=default.Table_4dae6fdf9a1e4400b85b31bf8fcc172c [id=694cd0f0a6624241947e7be0c71dac12]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:24.850646 1192 raft_consensus.cc:738] T 1500e4f9c0664a879503afd3edf14c66 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:24.853212 894 tablet_service.cc:1467] Processing CreateTablet for tablet d565b77f90784d87b64996772c938085 (DEFAULT_TABLE table=default.Table_915411462fb84dd7b917caa2f3a31498 [id=822489edc766460abfd5ef4bf6993ef5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:24.853708 891 tablet_service.cc:1467] Processing CreateTablet for tablet 7fb282707196444690352dea3eb1387f (DEFAULT_TABLE table=default.Table_6754736c569e4f55848afc146fe41b74 [id=521c6ec1968841128a4c4179e9d6980f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:24.854068 740 tablet_service.cc:1467] Processing CreateTablet for tablet a2781ba5061a4225b5636366857a229e (DEFAULT_TABLE table=default.Table_41d844ed7a144e59bd9e14d543ad0573 [id=16e500140b9b4bb6a7603e6ea0849cf9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:24.854151 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f2fb58fc24d4692a3fe9a652e076aed. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:24.854398 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2781ba5061a4225b5636366857a229e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:24.854532 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7fb282707196444690352dea3eb1387f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:24.854760 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 01a455eda0ee488fb3d45a6a09c43833. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:24.856138 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 39a014ac69a2435a9132a75125063093. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:24.857048 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d565b77f90784d87b64996772c938085. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:24.857153 747 tablet_service.cc:1467] Processing CreateTablet for tablet 01a455eda0ee488fb3d45a6a09c43833 (DEFAULT_TABLE table=default.Table_4dae6fdf9a1e4400b85b31bf8fcc172c [id=694cd0f0a6624241947e7be0c71dac12]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:24.857509 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 01a455eda0ee488fb3d45a6a09c43833. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:24.865108 1040 tablet_service.cc:1467] Processing CreateTablet for tablet 8f2fb58fc24d4692a3fe9a652e076aed (DEFAULT_TABLE table=default.Table_72783655bdee4fcf8c2fff2186fe0a60 [id=8edee52f05e24c03b088ce58f4b3f4d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:24.865553 1040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f2fb58fc24d4692a3fe9a652e076aed. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:24.866637 1618 raft_consensus.cc:2743] T 41bcb2eafff14fa8a040982c723f99c1 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:24.867635 1192 consensus_queue.cc:260] T 1500e4f9c0664a879503afd3edf14c66 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:24.868058 1028 tablet_service.cc:1467] Processing CreateTablet for tablet c73c649a4ca44471af2cf6a77500b065 (DEFAULT_TABLE table=default.Table_77b0b19284064a118a50b9bcfe72765f [id=a0bbd2c6939d449392a1371c3b5ec0a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:24.869135 895 tablet_service.cc:1467] Processing CreateTablet for tablet 6001369444894b48b51a48fb60ddc52a (DEFAULT_TABLE table=default.Table_ed91fb5c64de42bea271f05b8c284edc [id=b8c683d257794905b44a447d158917ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:24.869251 1035 tablet_service.cc:1467] Processing CreateTablet for tablet d565b77f90784d87b64996772c938085 (DEFAULT_TABLE table=default.Table_915411462fb84dd7b917caa2f3a31498 [id=822489edc766460abfd5ef4bf6993ef5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:24.869434 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 98bfe9620e284ab1954433ba9a036392. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:24.869817 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c73c649a4ca44471af2cf6a77500b065. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:24.870425 885 tablet_service.cc:1467] Processing CreateTablet for tablet 415a1fa4b2f14cacb43937c8a372849a (DEFAULT_TABLE table=default.Table_5181e98c2c214ebcab54c33615482528 [id=c13a3a3a70dd4dd7a493937ae16b6931]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:24.871225 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe54131aafb40ee9d8bc5ea299af193" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196"
I20250113 02:51:24.872485 1042 tablet_service.cc:1467] Processing CreateTablet for tablet d47f544f723542bea1d396f586d8d658 (DEFAULT_TABLE table=default.Table_d3474542a47c407ea91c69d7601c4af9 [id=5253ce78460b47c69168f0959713e123]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:24.873456 1194 tablet_bootstrap.cc:492] T d7920c0544c0466bad6bd8c09aae1a7d P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:24.874958 888 tablet_service.cc:1467] Processing CreateTablet for tablet c73c649a4ca44471af2cf6a77500b065 (DEFAULT_TABLE table=default.Table_77b0b19284064a118a50b9bcfe72765f [id=a0bbd2c6939d449392a1371c3b5ec0a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:24.875873 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2781ba5061a4225b5636366857a229e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:24.877878 1194 ts_tablet_manager.cc:1397] T d7920c0544c0466bad6bd8c09aae1a7d P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.345s user 0.009s sys 0.000s
I20250113 02:51:24.884378 1039 tablet_service.cc:1467] Processing CreateTablet for tablet 7fb282707196444690352dea3eb1387f (DEFAULT_TABLE table=default.Table_6754736c569e4f55848afc146fe41b74 [id=521c6ec1968841128a4c4179e9d6980f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:24.885697 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d565b77f90784d87b64996772c938085. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:24.892087 1033 tablet_service.cc:1467] Processing CreateTablet for tablet fe3d35d9aaf64188aaa3c1ac38d279d8 (DEFAULT_TABLE table=default.Table_7e49ba949e8949fa9924acdef98c3947 [id=c8a8725a3ac44d4b8960e2ceccb0ccc6]), partition=RANGE (key) PARTITION UNBOUNDED
W20250113 02:51:24.893792 995 leader_election.cc:343] T 4fe54131aafb40ee9d8bc5ea299af193 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:24.894929 1024 tablet_service.cc:1467] Processing CreateTablet for tablet 6001369444894b48b51a48fb60ddc52a (DEFAULT_TABLE table=default.Table_ed91fb5c64de42bea271f05b8c284edc [id=b8c683d257794905b44a447d158917ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:24.896276 1442 leader_election.cc:290] T 4fe54131aafb40ee9d8bc5ea299af193 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Requested vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
I20250113 02:51:24.897511 1192 ts_tablet_manager.cc:1428] T 1500e4f9c0664a879503afd3edf14c66 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.118s user 0.002s sys 0.002s
I20250113 02:51:24.899170 1023 tablet_service.cc:1467] Processing CreateTablet for tablet 98bfe9620e284ab1954433ba9a036392 (DEFAULT_TABLE table=default.Table_064659c197854029a5d7af1798def6d9 [id=e9934f5e1a17427cabd573190c4f16a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:24.899855 1038 tablet_service.cc:1467] Processing CreateTablet for tablet 39a014ac69a2435a9132a75125063093 (DEFAULT_TABLE table=default.Table_02655512f28b47429ebb70e1676844ea [id=dcb8e4b96b864e75b9b47a658c2592f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:24.900291 1031 tablet_service.cc:1467] Processing CreateTablet for tablet 415a1fa4b2f14cacb43937c8a372849a (DEFAULT_TABLE table=default.Table_5181e98c2c214ebcab54c33615482528 [id=c13a3a3a70dd4dd7a493937ae16b6931]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:24.900862 1025 tablet_service.cc:1467] Processing CreateTablet for tablet a2781ba5061a4225b5636366857a229e (DEFAULT_TABLE table=default.Table_41d844ed7a144e59bd9e14d543ad0573 [id=16e500140b9b4bb6a7603e6ea0849cf9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:24.906296 740 tablet_service.cc:1467] Processing CreateTablet for tablet 36726b40ac5145bfb371871b3fff079a (DEFAULT_TABLE table=default.Table_50f21224ee254e3bb13748a44e257783 [id=8ca79d8a8fec48259f1547121f37fefb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:24.907380 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7fb282707196444690352dea3eb1387f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:24.908905 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36726b40ac5145bfb371871b3fff079a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:24.908406 744 tablet_service.cc:1467] Processing CreateTablet for tablet d59134b029a143038837366f51f00d28 (DEFAULT_TABLE table=default.Table_bb3ef05cbb904c14a9c2347246fa2fab [id=6ae8663361aa4ba989265452dfa21f89]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:24.910223 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d59134b029a143038837366f51f00d28. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:24.912248 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet a2781ba5061a4225b5636366857a229e
I20250113 02:51:24.913115 752 tablet_service.cc:1514] Processing DeleteTablet for tablet a2781ba5061a4225b5636366857a229e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:24 UTC) from {username='slave'} at 127.0.0.1:55398
W20250113 02:51:24.913815 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet a2781ba5061a4225b5636366857a229e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a2781ba5061a4225b5636366857a229e already in progress: creating tablet
I20250113 02:51:24.915241 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_67ba9f96c20f458ebe9d6529f3d97852" } new_table_name: "default.Table_6de3bf6640dc47f9b977f608c5ec8ef9" modify_external_catalogs: true
I20250113 02:51:24.916739 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_c15dbcc7e55d4e3c80fb41bc4a950c81" } new_table_name: "default.Table_3b2ea4aaa2ee46acb69f2040e1ebdc41" modify_external_catalogs: true
I20250113 02:51:24.917938 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 0c408921a09a463eb464e08edcf042b3
I20250113 02:51:24.918481 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_c7fb9512ad3a4a698261ea6b58a542db" } new_table_name: "default.Table_1d910c3d2b244474a576fc4088301143" modify_external_catalogs: true
I20250113 02:51:24.919145 752 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c408921a09a463eb464e08edcf042b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:24 UTC) from {username='slave'} at 127.0.0.1:55398
W20250113 02:51:24.919781 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 0c408921a09a463eb464e08edcf042b3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0c408921a09a463eb464e08edcf042b3 already in progress: creating tablet
I20250113 02:51:24.911885 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 283fc7f1c74e480c8a63c74d3ae9eb7d
I20250113 02:51:24.921054 752 tablet_service.cc:1514] Processing DeleteTablet for tablet 283fc7f1c74e480c8a63c74d3ae9eb7d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:24 UTC) from {username='slave'} at 127.0.0.1:55398
W20250113 02:51:24.921700 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 283fc7f1c74e480c8a63c74d3ae9eb7d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 283fc7f1c74e480c8a63c74d3ae9eb7d already in progress: creating tablet
I20250113 02:51:24.921818 451 catalog_manager.cc:5618] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095) does not have the latest schema for tablet d7920c0544c0466bad6bd8c09aae1a7d (table default.Table_c9a2c2b1c62849babefee677197da844 [id=05755562cfc24370a3e6fb245a573dd4]). Expected version 1 got 0
I20250113 02:51:24.923106 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_2155bf71d1a04a93a28b13c97f636fab" } modify_external_catalogs: true
I20250113 02:51:24.923276 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_2155bf71d1a04a93a28b13c97f636fab" } modify_external_catalogs: true
I20250113 02:51:24.923390 752 tablet_service.cc:1514] Processing DeleteTablet for tablet ba61c86743e24666b30580c36b9228c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:19 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:24.923655 1195 tablet_replica.cc:331] T ba61c86743e24666b30580c36b9228c3 P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:24.923846 1195 raft_consensus.cc:2238] T ba61c86743e24666b30580c36b9228c3 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:24.924081 1195 raft_consensus.cc:2267] T ba61c86743e24666b30580c36b9228c3 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:24.926740 1195 ts_tablet_manager.cc:1905] T ba61c86743e24666b30580c36b9228c3 P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:24.928655 1030 tablet_service.cc:1467] Processing CreateTablet for tablet 936b8ceaedd34a019a4fb89a80dd2708 (DEFAULT_TABLE table=default.Table_bb7ff740ee724b3999eeec918a03b1ae [id=1291674e6af743d08c5fff97bbb34c8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:24.928995 1191 tablet_bootstrap.cc:492] T b834f0acf5314bcb9d81c73de1b31842 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:24.929133 1191 ts_tablet_manager.cc:1397] T b834f0acf5314bcb9d81c73de1b31842 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.150s user 0.011s sys 0.009s
I20250113 02:51:24.930032 1191 raft_consensus.cc:357] T b834f0acf5314bcb9d81c73de1b31842 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:24.930245 1191 raft_consensus.cc:383] T b834f0acf5314bcb9d81c73de1b31842 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:24.930341 1191 raft_consensus.cc:738] T b834f0acf5314bcb9d81c73de1b31842 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:24.931713 1191 consensus_queue.cc:260] T b834f0acf5314bcb9d81c73de1b31842 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:24.932420 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2781ba5061a4225b5636366857a229e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:24.934729 1041 tablet_service.cc:1467] Processing CreateTablet for tablet 01a455eda0ee488fb3d45a6a09c43833 (DEFAULT_TABLE table=default.Table_4dae6fdf9a1e4400b85b31bf8fcc172c [id=694cd0f0a6624241947e7be0c71dac12]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:24.934811 1195 ts_tablet_manager.cc:1918] T ba61c86743e24666b30580c36b9228c3 P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:24.935014 1195 log.cc:1198] T ba61c86743e24666b30580c36b9228c3 P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/ba61c86743e24666b30580c36b9228c3
I20250113 02:51:24.935657 1195 ts_tablet_manager.cc:1939] T ba61c86743e24666b30580c36b9228c3 P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:24.936218 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 39a014ac69a2435a9132a75125063093. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:24.936976 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet ba61c86743e24666b30580c36b9228c3 (table default.Table_3f75f23afe2f449fa7a1f9df1c0734fb [id=68a4c317da7143cf841ee2d869d2f754]) successfully deleted
I20250113 02:51:24.937520 752 tablet_service.cc:1467] Processing CreateTablet for tablet b9ec434644c14acd95c72efbef8ded59 (DEFAULT_TABLE table=default.Table_2d2546458fac45b79308ff4e7f043292 [id=3eecc618c0e249aa9ecd78c484a2cc55]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:24.937945 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b9ec434644c14acd95c72efbef8ded59. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:24.940029 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 01a455eda0ee488fb3d45a6a09c43833. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:24.943352 1191 ts_tablet_manager.cc:1428] T b834f0acf5314bcb9d81c73de1b31842 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.014s user 0.002s sys 0.002s
I20250113 02:51:24.949432 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet a2781ba5061a4225b5636366857a229e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:24 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:24.949323 1191 tablet_bootstrap.cc:492] T ebc5f1ea1e1648d49904b17603d3cd2a P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:24.950913 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 936b8ceaedd34a019a4fb89a80dd2708. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:24.953874 1442 raft_consensus.cc:491] T 23e9321f08ee4ce7b2252ad5c76c0a08 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:24.960340 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 056bf1e67be04d179eace3d93282b145
I20250113 02:51:24.960897 453 catalog_manager.cc:6596] ReplaceTablet: tablet 056bf1e67be04d179eace3d93282b145 deleted and replaced by tablet dfea918c04c1408a8b86a44bb0911a7c
I20250113 02:51:24.961292 736 tablet_service.cc:1514] Processing DeleteTablet for tablet 056bf1e67be04d179eace3d93282b145 with delete_type TABLET_DATA_DELETED (replaced by tablet dfea918c04c1408a8b86a44bb0911a7c) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:24.962708 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet bc533518faa24dffa3306ff092e8d507
I20250113 02:51:24.963378 741 tablet_service.cc:1514] Processing DeleteTablet for tablet bc533518faa24dffa3306ff092e8d507 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:24 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:24.963656 736 tablet_service.cc:1514] Processing DeleteTablet for tablet b834f0acf5314bcb9d81c73de1b31842 with delete_type TABLET_DATA_DELETED (replaced by tablet be4b2a8a3927410eaf12e7592646e4af) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:24.965049 1195 tablet_replica.cc:331] T b834f0acf5314bcb9d81c73de1b31842 P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:24.965293 1195 raft_consensus.cc:2238] T b834f0acf5314bcb9d81c73de1b31842 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:24.965478 1195 raft_consensus.cc:2267] T b834f0acf5314bcb9d81c73de1b31842 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250113 02:51:24.966181 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f9269967dea94fc3a337546ad0842e11 consensus state has no leader
I20250113 02:51:24.966984 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c408921a09a463eb464e08edcf042b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:24 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:24.967166 1037 tablet_service.cc:1514] Processing DeleteTablet for tablet 283fc7f1c74e480c8a63c74d3ae9eb7d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:24 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:24.967306 1195 ts_tablet_manager.cc:1905] T b834f0acf5314bcb9d81c73de1b31842 P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:24.967443 1442 raft_consensus.cc:513] T 23e9321f08ee4ce7b2252ad5c76c0a08 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
W20250113 02:51:24.968253 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 056bf1e67be04d179eace3d93282b145 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 056bf1e67be04d179eace3d93282b145 already in progress: creating tablet
W20250113 02:51:24.968588 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet bc533518faa24dffa3306ff092e8d507 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bc533518faa24dffa3306ff092e8d507 already in progress: creating tablet
I20250113 02:51:24.968945 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23e9321f08ee4ce7b2252ad5c76c0a08" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
W20250113 02:51:24.969635 995 leader_election.cc:343] T 23e9321f08ee4ce7b2252ad5c76c0a08 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:24.969828 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6001369444894b48b51a48fb60ddc52a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:24.970988 1192 tablet_bootstrap.cc:492] T a6055d77d168495d9b756a1a56c1a920 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:24.971652 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 415a1fa4b2f14cacb43937c8a372849a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:24.977267 1191 tablet_bootstrap.cc:654] T ebc5f1ea1e1648d49904b17603d3cd2a P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:24.978657 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 98bfe9620e284ab1954433ba9a036392. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:24.979060 736 tablet_service.cc:1467] Processing CreateTablet for tablet 9174df876fe64adc805fba0a36fb4d38 (DEFAULT_TABLE table=default.Table_37b52d3cfe0f4b0fb1ef1468117512dd [id=45aa63632c914a4e99eb69759b56c842]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:24.979553 736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9174df876fe64adc805fba0a36fb4d38. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:24.982918 1037 tablet_service.cc:1467] Processing CreateTablet for tablet 36726b40ac5145bfb371871b3fff079a (DEFAULT_TABLE table=default.Table_50f21224ee254e3bb13748a44e257783 [id=8ca79d8a8fec48259f1547121f37fefb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:24.983920 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_5be7dd3cb3674e39a393c9276c01ae4c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:24.984886 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5be7dd3cb3674e39a393c9276c01ae4c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:24.986653 1195 ts_tablet_manager.cc:1918] T b834f0acf5314bcb9d81c73de1b31842 P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:24.988327 1195 log.cc:1198] T b834f0acf5314bcb9d81c73de1b31842 P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/b834f0acf5314bcb9d81c73de1b31842
I20250113 02:51:24.990384 1195 ts_tablet_manager.cc:1939] T b834f0acf5314bcb9d81c73de1b31842 P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:24.990710 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_4a0b14872edf4030a6c3968e559d6cd3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:24.992441 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4a0b14872edf4030a6c3968e559d6cd3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:24.993206 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet b834f0acf5314bcb9d81c73de1b31842 (table default.Table_24d9af3d0b0740d69c791fc96916ef0a [id=124669ccadaf49e19d2660b1495f9647]) successfully deleted
I20250113 02:51:24.996022 1442 leader_election.cc:290] T 23e9321f08ee4ce7b2252ad5c76c0a08 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
I20250113 02:51:25.008772 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe3d35d9aaf64188aaa3c1ac38d279d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:25.014405 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet 056bf1e67be04d179eace3d93282b145 with delete_type TABLET_DATA_DELETED (replaced by tablet dfea918c04c1408a8b86a44bb0911a7c) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:25.019657 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d47f544f723542bea1d396f586d8d658. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:25.021040 880 tablet_service.cc:1467] Processing CreateTablet for tablet 36726b40ac5145bfb371871b3fff079a (DEFAULT_TABLE table=default.Table_50f21224ee254e3bb13748a44e257783 [id=8ca79d8a8fec48259f1547121f37fefb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:25.024121 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d59134b029a143038837366f51f00d28. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:25.029255 886 tablet_service.cc:1514] Processing DeleteTablet for tablet a2781ba5061a4225b5636366857a229e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:24 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:25.036037 896 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c408921a09a463eb464e08edcf042b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:24 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:25.047850 886 tablet_service.cc:1514] Processing DeleteTablet for tablet 283fc7f1c74e480c8a63c74d3ae9eb7d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:24 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:25.056403 886 tablet_service.cc:1467] Processing CreateTablet for tablet b9ec434644c14acd95c72efbef8ded59 (DEFAULT_TABLE table=default.Table_2d2546458fac45b79308ff4e7f043292 [id=3eecc618c0e249aa9ecd78c484a2cc55]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:25.058208 896 tablet_service.cc:1467] Processing CreateTablet for tablet 9174df876fe64adc805fba0a36fb4d38 (DEFAULT_TABLE table=default.Table_37b52d3cfe0f4b0fb1ef1468117512dd [id=45aa63632c914a4e99eb69759b56c842]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:25.059350 890 tablet_service.cc:1514] Processing DeleteTablet for tablet 056bf1e67be04d179eace3d93282b145 with delete_type TABLET_DATA_DELETED (replaced by tablet dfea918c04c1408a8b86a44bb0911a7c) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:25.059988 1036 tablet_service.cc:1514] Processing DeleteTablet for tablet bc533518faa24dffa3306ff092e8d507 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:24 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:25.062254 1042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d47f544f723542bea1d396f586d8d658. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:25.062883 1196 ts_tablet_manager.cc:1918] T 157905316fc545b6a7864ae1c4a2a584 P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250113 02:51:25.067426 998 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:38467, user_credentials={real_user=slave}} blocked reactor thread for 263394us
I20250113 02:51:25.069594 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_0b2076804d4c410fa414146b8bb0b91b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:25.070278 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0b2076804d4c410fa414146b8bb0b91b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:25.070222 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_cc1ce461db3e461387263aacbd6af4b3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:25.070758 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cc1ce461db3e461387263aacbd6af4b3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:25.072181 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_f683cb701e2d4ba0a3fb83f1bc3722ad"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:25.072901 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f683cb701e2d4ba0a3fb83f1bc3722ad in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:25.074769 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_acfb94f30f9c4f888bd0a4f28ff1a79e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:25.075356 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_acfb94f30f9c4f888bd0a4f28ff1a79e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:25.077839 1027 tablet_service.cc:1467] Processing CreateTablet for tablet b9ec434644c14acd95c72efbef8ded59 (DEFAULT_TABLE table=default.Table_2d2546458fac45b79308ff4e7f043292 [id=3eecc618c0e249aa9ecd78c484a2cc55]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:25.082423 1037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36726b40ac5145bfb371871b3fff079a. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:25.085173 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f4379f7fc6b54e6785f27975d9030cce consensus state has no leader
I20250113 02:51:25.086542 446 catalog_manager.cc:5526] T d3b695ac20ba4205b2d8732db4740531 P df69f48656b44bb890f76587ee69cc0f reported cstate change: term changed from 0 to 1, leader changed from <none> to df69f48656b44bb890f76587ee69cc0f (127.0.104.2). New cstate: current_term: 1 leader_uuid: "df69f48656b44bb890f76587ee69cc0f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } health_report { overall_health: UNKNOWN } } }
I20250113 02:51:25.087059 446 catalog_manager.cc:5618] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467) does not have the latest schema for tablet d3b695ac20ba4205b2d8732db4740531 (table default.Table_8b87c8469fbb465e80375aa0446de6c2 [id=f85f0c435ffa4fdca81d86de4aa4fd5f]). Expected version 3 got 0
I20250113 02:51:25.087353 446 catalog_manager.cc:5618] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467) does not have the latest schema for tablet 317c6e8013b745c290bf3190dc237ab8 (table default.Table_13fd4df9c207466cb85cd8ed9062722c [id=6bfb67a96b2b47849f7fa12c57a2f4a1]). Expected version 1 got 0
I20250113 02:51:25.067487 1194 raft_consensus.cc:357] T d7920c0544c0466bad6bd8c09aae1a7d P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:25.088184 1194 raft_consensus.cc:383] T d7920c0544c0466bad6bd8c09aae1a7d P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:25.088292 1194 raft_consensus.cc:738] T d7920c0544c0466bad6bd8c09aae1a7d P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:25.088560 1194 consensus_queue.cc:260] T d7920c0544c0466bad6bd8c09aae1a7d P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:25.089170 1194 ts_tablet_manager.cc:1428] T d7920c0544c0466bad6bd8c09aae1a7d P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.211s user 0.002s sys 0.000s
I20250113 02:51:25.089469 1194 tablet_bootstrap.cc:492] T d304b65eba8c40c6922333506a60540c P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:25.066543 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe3d35d9aaf64188aaa3c1ac38d279d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:25.091070 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36726b40ac5145bfb371871b3fff079a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:25.091298 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c04be975c8c049b892b16129ed47a142. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:25.094609 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b9ec434644c14acd95c72efbef8ded59. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:25.095999 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c73c649a4ca44471af2cf6a77500b065. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:25.101235 1192 tablet_bootstrap.cc:654] T a6055d77d168495d9b756a1a56c1a920 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
W20250113 02:51:25.104650 998 leader_election.cc:343] T fc92f19390fd48a4a45ba8fb839020fd P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Not found: Tablet not found: fc92f19390fd48a4a45ba8fb839020fd
I20250113 02:51:25.108244 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9174df876fe64adc805fba0a36fb4d38. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:25.108467 1036 tablet_service.cc:1467] Processing CreateTablet for tablet 9174df876fe64adc805fba0a36fb4d38 (DEFAULT_TABLE table=default.Table_37b52d3cfe0f4b0fb1ef1468117512dd [id=45aa63632c914a4e99eb69759b56c842]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:25.109582 1036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9174df876fe64adc805fba0a36fb4d38. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:25.110209 1191 tablet_bootstrap.cc:492] T ebc5f1ea1e1648d49904b17603d3cd2a P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:25.110383 1191 ts_tablet_manager.cc:1397] T ebc5f1ea1e1648d49904b17603d3cd2a P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.162s user 0.027s sys 0.030s
I20250113 02:51:25.111215 1191 raft_consensus.cc:357] T ebc5f1ea1e1648d49904b17603d3cd2a P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:25.073078 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23e9321f08ee4ce7b2252ad5c76c0a08" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:25.111449 1191 raft_consensus.cc:383] T ebc5f1ea1e1648d49904b17603d3cd2a P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:25.111364 998 leader_election.cc:304] T fc92f19390fd48a4a45ba8fb839020fd P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4216a7f853d642b7bc3e5cff458f3aca; no voters: 1186fa869eb14812832fa6562531f196, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:25.073230 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe54131aafb40ee9d8bc5ea299af193" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f"
I20250113 02:51:25.111521 1191 raft_consensus.cc:738] T ebc5f1ea1e1648d49904b17603d3cd2a P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:25.111783 1191 consensus_queue.cc:260] T ebc5f1ea1e1648d49904b17603d3cd2a P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:25.068377 893 tablet_service.cc:1514] Processing DeleteTablet for tablet bc533518faa24dffa3306ff092e8d507 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:24 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:25.065791 1196 log.cc:1198] T 157905316fc545b6a7864ae1c4a2a584 P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/157905316fc545b6a7864ae1c4a2a584
I20250113 02:51:25.112663 1191 ts_tablet_manager.cc:1428] T ebc5f1ea1e1648d49904b17603d3cd2a P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250113 02:51:25.112996 1191 tablet_bootstrap.cc:492] T 06f36d09bbb1443bafce906907795404 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
W20250113 02:51:25.112730 998 leader_election.cc:343] T 34ab7ee573284550aaac9f814e8bd7ab P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Not found: Tablet not found: 34ab7ee573284550aaac9f814e8bd7ab
I20250113 02:51:25.114739 1191 tablet_bootstrap.cc:654] T 06f36d09bbb1443bafce906907795404 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:25.119042 752 tablet_service.cc:1467] Processing CreateTablet for tablet e13267498917432ba92b6badb76f5d73 (DEFAULT_TABLE table=default.Table_0b855cf933544dce95621708e777674f [id=a63ccead990445fdac16fda1312bb66c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:25.119541 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e13267498917432ba92b6badb76f5d73. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:25.125033 740 tablet_service.cc:1467] Processing CreateTablet for tablet 80c647a3b5a1411e9130461115c53550 (DEFAULT_TABLE table=default.Table_82d6d0a936524f35beab62f451e5eb99 [id=c20532b7392a4ec98453d6653de242aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:25.125471 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 80c647a3b5a1411e9130461115c53550. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:25.128404 744 tablet_service.cc:1467] Processing CreateTablet for tablet f9269967dea94fc3a337546ad0842e11 (DEFAULT_TABLE table=default.Table_23c1655ab4cf4a30b67185ccf4fbb76d [id=a8e428598ca94e15b04f1df8112a4abc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:25.128808 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f9269967dea94fc3a337546ad0842e11. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:25.130774 1191 tablet_bootstrap.cc:492] T 06f36d09bbb1443bafce906907795404 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:25.130904 1191 ts_tablet_manager.cc:1397] T 06f36d09bbb1443bafce906907795404 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.002s
I20250113 02:51:25.131628 1191 raft_consensus.cc:357] T 06f36d09bbb1443bafce906907795404 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:25.131826 1191 raft_consensus.cc:383] T 06f36d09bbb1443bafce906907795404 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:25.131903 1191 raft_consensus.cc:738] T 06f36d09bbb1443bafce906907795404 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:25.132218 1191 consensus_queue.cc:260] T 06f36d09bbb1443bafce906907795404 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:25.132730 736 tablet_service.cc:1467] Processing CreateTablet for tablet 29c17fd90d774c029a35e77888358067 (DEFAULT_TABLE table=default.Table_96651718e3f643fcacec8ef593a9b30e [id=2d2795dd06ce4605a285ca2a93e20c4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:25.132915 1191 ts_tablet_manager.cc:1428] T 06f36d09bbb1443bafce906907795404 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250113 02:51:25.133144 741 tablet_service.cc:1467] Processing CreateTablet for tablet ed68a2b7ebcf4841bef54798dd3ed80d (DEFAULT_TABLE table=default.Table_7157800e639b45d8bb947e53d4319cfc [id=39997cddb4684b6c984eeeb48d715002]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:25.133554 746 tablet_service.cc:1467] Processing CreateTablet for tablet 146dad5c977f464496f7d21611030dad (DEFAULT_TABLE table=default.Table_4369a143c72643b084df58d25314c6dd [id=36ff239c08a54be99ca307354810fff3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:25.133955 750 tablet_service.cc:1467] Processing CreateTablet for tablet d8419fec408c40f6bd46c39de7b3dce9 (DEFAULT_TABLE table=default.Table_503d2ce0d8d74387bf59e25670b61bcb [id=35d1232ab27b4e37a706ab6092f9115b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:25.134370 736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 29c17fd90d774c029a35e77888358067. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:25.137504 1191 tablet_bootstrap.cc:492] T d304b65eba8c40c6922333506a60540c P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:25.141469 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 415a1fa4b2f14cacb43937c8a372849a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:25.141554 918 raft_consensus.cc:3054] T 4fe54131aafb40ee9d8bc5ea299af193 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:25.143294 1191 tablet_bootstrap.cc:654] T d304b65eba8c40c6922333506a60540c P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:25.144060 998 leader_election.cc:304] T 34ab7ee573284550aaac9f814e8bd7ab P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4216a7f853d642b7bc3e5cff458f3aca; no voters: 1186fa869eb14812832fa6562531f196, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:25.144465 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 146dad5c977f464496f7d21611030dad. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:25.147150 1061 raft_consensus.cc:3054] T 317c6e8013b745c290bf3190dc237ab8 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:25.147444 1634 raft_consensus.cc:2743] T fc92f19390fd48a4a45ba8fb839020fd P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250113 02:51:25.148923 442 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet f9269967dea94fc3a337546ad0842e11 (table default.Table_23c1655ab4cf4a30b67185ccf4fbb76d [id=a8e428598ca94e15b04f1df8112a4abc], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet f9269967dea94fc3a337546ad0842e11 consensus state has no leader
I20250113 02:51:25.149405 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6001369444894b48b51a48fb60ddc52a. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:25.149713 416 external_mini_cluster.cc:1677] Unable to check leaks on /tmp/dist-test-taskXJrxug/build/asan/bin/kudu pid 563: Timed out: CheckLeaks RPC to 127.0.104.60:42067 timed out after 1.000s (SENT)
I20250113 02:51:25.149881 416 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskXJrxug/build/asan/bin/kudu with pid 563
I20250113 02:51:25.153033 890 tablet_service.cc:1467] Processing CreateTablet for tablet f9269967dea94fc3a337546ad0842e11 (DEFAULT_TABLE table=default.Table_23c1655ab4cf4a30b67185ccf4fbb76d [id=a8e428598ca94e15b04f1df8112a4abc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:25.153970 1196 ts_tablet_manager.cc:1939] T 157905316fc545b6a7864ae1c4a2a584 P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:25.155068 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b9ec434644c14acd95c72efbef8ded59. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:25.156064 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ed68a2b7ebcf4841bef54798dd3ed80d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:25.157737 1198 ts_tablet_manager.cc:1918] T f1a02c37cb4f43f8a9de6484f4431958 P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:25.158190 1198 log.cc:1198] T f1a02c37cb4f43f8a9de6484f4431958 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/f1a02c37cb4f43f8a9de6484f4431958
I20250113 02:51:25.158504 893 tablet_service.cc:1467] Processing CreateTablet for tablet e13267498917432ba92b6badb76f5d73 (DEFAULT_TABLE table=default.Table_0b855cf933544dce95621708e777674f [id=a63ccead990445fdac16fda1312bb66c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:25.159947 737 tablet_service.cc:1467] Processing CreateTablet for tablet 303d911dfa2f4abbb053ead33db6d71c (DEFAULT_TABLE table=default.Table_96fc1675456247f2a1c6d67ce524f203 [id=034ffa2a4ba6468a94bcf891bd2e6bb3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:25.160951 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet 1500e4f9c0664a879503afd3edf14c66 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:21 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:25.161881 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f9269967dea94fc3a337546ad0842e11. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:25.162568 887 tablet_service.cc:1467] Processing CreateTablet for tablet 29c17fd90d774c029a35e77888358067 (DEFAULT_TABLE table=default.Table_96651718e3f643fcacec8ef593a9b30e [id=2d2795dd06ce4605a285ca2a93e20c4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:25.163812 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8419fec408c40f6bd46c39de7b3dce9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:25.166195 748 tablet_service.cc:1467] Processing CreateTablet for tablet 95e47db8a4d74615800ba6c049ec7131 (DEFAULT_TABLE table=default.Table_1b71e9ea2fca48daa60e8d692eab8e49 [id=383148ed836144c1bc5403dae46aad50]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:25.166893 737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 303d911dfa2f4abbb053ead33db6d71c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:25.167327 1196 tablet_replica.cc:331] T f1a02c37cb4f43f8a9de6484f4431958 P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:25.167531 1196 raft_consensus.cc:2238] T f1a02c37cb4f43f8a9de6484f4431958 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:25.167718 1196 raft_consensus.cc:2267] T f1a02c37cb4f43f8a9de6484f4431958 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:25.167966 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95e47db8a4d74615800ba6c049ec7131. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:25.168294 1196 ts_tablet_manager.cc:1905] T f1a02c37cb4f43f8a9de6484f4431958 P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:25.169718 883 tablet_service.cc:1467] Processing CreateTablet for tablet 146dad5c977f464496f7d21611030dad (DEFAULT_TABLE table=default.Table_4369a143c72643b084df58d25314c6dd [id=36ff239c08a54be99ca307354810fff3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:25.170470 884 tablet_service.cc:1467] Processing CreateTablet for tablet ed68a2b7ebcf4841bef54798dd3ed80d (DEFAULT_TABLE table=default.Table_7157800e639b45d8bb947e53d4319cfc [id=39997cddb4684b6c984eeeb48d715002]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:25.170746 891 tablet_service.cc:1467] Processing CreateTablet for tablet 80c647a3b5a1411e9130461115c53550 (DEFAULT_TABLE table=default.Table_82d6d0a936524f35beab62f451e5eb99 [id=c20532b7392a4ec98453d6653de242aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:25.173924 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e13267498917432ba92b6badb76f5d73. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:25.175884 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 29c17fd90d774c029a35e77888358067. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:25.176484 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ed68a2b7ebcf4841bef54798dd3ed80d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:25.177112 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 80c647a3b5a1411e9130461115c53550. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:25.177444 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 146dad5c977f464496f7d21611030dad. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:25.180233 1611 consensus_queue.cc:1035] T 317c6e8013b745c290bf3190dc237ab8 P df69f48656b44bb890f76587ee69cc0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20250113 02:51:25.181533 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet 157905316fc545b6a7864ae1c4a2a584 (table default.Table_5b0e60572cb54fcca8b3b5b94326ae3a [id=9f5add219a7c48da9a8afa966e22d1bd]) successfully deleted
W20250113 02:51:25.181816 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 2470b010b571444eba3f132323038f33 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2470b010b571444eba3f132323038f33 already in progress: creating tablet
W20250113 02:51:25.182013 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet b33025a462e74f2aa26cc3779f36615b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b33025a462e74f2aa26cc3779f36615b already in progress: creating tablet
W20250113 02:51:25.182204 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 65d9bbd8c15f40a4a50c1695fb5e648e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 65d9bbd8c15f40a4a50c1695fb5e648e already in progress: creating tablet
I20250113 02:51:25.182638 1198 ts_tablet_manager.cc:1939] T f1a02c37cb4f43f8a9de6484f4431958 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:25.191886 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 936b8ceaedd34a019a4fb89a80dd2708. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:25.201253 1198 tablet_replica.cc:331] T ebc5f1ea1e1648d49904b17603d3cd2a P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
W20250113 02:51:25.203030 995 consensus_peers.cc:487] T 157905316fc545b6a7864ae1c4a2a584 P 4216a7f853d642b7bc3e5cff458f3aca -> Peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Couldn't send request to peer 1186fa869eb14812832fa6562531f196. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 157905316fc545b6a7864ae1c4a2a584. This is attempt 1: this message will repeat every 5th retry.
I20250113 02:51:25.203390 1198 raft_consensus.cc:2238] T ebc5f1ea1e1648d49904b17603d3cd2a P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:25.203675 1198 raft_consensus.cc:2267] T ebc5f1ea1e1648d49904b17603d3cd2a P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:25.204444 1198 ts_tablet_manager.cc:1905] T ebc5f1ea1e1648d49904b17603d3cd2a P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:25.208142 1194 tablet_bootstrap.cc:654] T d304b65eba8c40c6922333506a60540c P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:25.223043 1191 tablet_bootstrap.cc:492] T d304b65eba8c40c6922333506a60540c P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:25.223349 1191 ts_tablet_manager.cc:1397] T d304b65eba8c40c6922333506a60540c P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.086s user 0.019s sys 0.022s
I20250113 02:51:25.224159 1634 raft_consensus.cc:2743] T 34ab7ee573284550aaac9f814e8bd7ab P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:25.224365 1191 raft_consensus.cc:357] T d304b65eba8c40c6922333506a60540c P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:25.225046 1191 raft_consensus.cc:383] T d304b65eba8c40c6922333506a60540c P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:25.225271 1191 raft_consensus.cc:738] T d304b65eba8c40c6922333506a60540c P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
W20250113 02:51:25.225356 998 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:38467, user_credentials={real_user=slave}} blocked reactor thread for 111798us
I20250113 02:51:25.225623 1191 consensus_queue.cc:260] T d304b65eba8c40c6922333506a60540c P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:25.228899 1191 ts_tablet_manager.cc:1428] T d304b65eba8c40c6922333506a60540c P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
W20250113 02:51:25.230247 998 leader_election.cc:343] T 23e9321f08ee4ce7b2252ad5c76c0a08 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:25.231627 998 leader_election.cc:304] T 23e9321f08ee4ce7b2252ad5c76c0a08 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4216a7f853d642b7bc3e5cff458f3aca; no voters: 1186fa869eb14812832fa6562531f196, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:25.231930 1192 tablet_bootstrap.cc:492] T a6055d77d168495d9b756a1a56c1a920 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:25.232494 1192 ts_tablet_manager.cc:1397] T a6055d77d168495d9b756a1a56c1a920 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.335s user 0.006s sys 0.006s
I20250113 02:51:25.232779 1634 raft_consensus.cc:2743] T 23e9321f08ee4ce7b2252ad5c76c0a08 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:25.234182 1192 raft_consensus.cc:357] T a6055d77d168495d9b756a1a56c1a920 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:25.236507 1191 tablet_bootstrap.cc:492] T 93a2729c5a604a27a9bbc451ccb4e128 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:25.238252 1191 tablet_bootstrap.cc:654] T 93a2729c5a604a27a9bbc451ccb4e128 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:25.240717 892 tablet_service.cc:1467] Processing CreateTablet for tablet d8419fec408c40f6bd46c39de7b3dce9 (DEFAULT_TABLE table=default.Table_503d2ce0d8d74387bf59e25670b61bcb [id=35d1232ab27b4e37a706ab6092f9115b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:25.244788 1192 raft_consensus.cc:383] T a6055d77d168495d9b756a1a56c1a920 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:25.245178 1192 raft_consensus.cc:738] T a6055d77d168495d9b756a1a56c1a920 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:25.245849 1192 consensus_queue.cc:260] T a6055d77d168495d9b756a1a56c1a920 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:25.246131 1191 tablet_bootstrap.cc:492] T 93a2729c5a604a27a9bbc451ccb4e128 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:25.246325 1191 ts_tablet_manager.cc:1397] T 93a2729c5a604a27a9bbc451ccb4e128 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.002s
I20250113 02:51:25.247697 1191 raft_consensus.cc:357] T 93a2729c5a604a27a9bbc451ccb4e128 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:25.247993 1191 raft_consensus.cc:383] T 93a2729c5a604a27a9bbc451ccb4e128 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:25.248157 1191 raft_consensus.cc:738] T 93a2729c5a604a27a9bbc451ccb4e128 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:25.248435 1191 consensus_queue.cc:260] T 93a2729c5a604a27a9bbc451ccb4e128 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:25.252235 1191 ts_tablet_manager.cc:1428] T 93a2729c5a604a27a9bbc451ccb4e128 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250113 02:51:25.253780 1191 tablet_bootstrap.cc:492] T 1500e4f9c0664a879503afd3edf14c66 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:25.257994 1191 tablet_bootstrap.cc:654] T 1500e4f9c0664a879503afd3edf14c66 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:25.270217 1191 tablet_bootstrap.cc:492] T 1500e4f9c0664a879503afd3edf14c66 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:25.270521 1191 ts_tablet_manager.cc:1397] T 1500e4f9c0664a879503afd3edf14c66 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.017s user 0.010s sys 0.000s
I20250113 02:51:25.271556 1191 raft_consensus.cc:357] T 1500e4f9c0664a879503afd3edf14c66 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:25.271946 1191 raft_consensus.cc:383] T 1500e4f9c0664a879503afd3edf14c66 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:25.272099 1191 raft_consensus.cc:738] T 1500e4f9c0664a879503afd3edf14c66 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:25.272450 1191 consensus_queue.cc:260] T 1500e4f9c0664a879503afd3edf14c66 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:25.276451 1191 ts_tablet_manager.cc:1428] T 1500e4f9c0664a879503afd3edf14c66 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250113 02:51:25.279147 1438 consensus_queue.cc:1035] T 317c6e8013b745c290bf3190dc237ab8 P df69f48656b44bb890f76587ee69cc0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250113 02:51:25.280498 1191 tablet_bootstrap.cc:492] T 8aac142230e244c0b59c9c2b7a7300ea P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:25.282763 1192 ts_tablet_manager.cc:1428] T a6055d77d168495d9b756a1a56c1a920 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.049s user 0.002s sys 0.000s
I20250113 02:51:25.283067 1191 tablet_bootstrap.cc:654] T 8aac142230e244c0b59c9c2b7a7300ea P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:25.284543 1192 tablet_bootstrap.cc:492] T 8aac142230e244c0b59c9c2b7a7300ea P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:25.296994 918 raft_consensus.cc:2463] T 4fe54131aafb40ee9d8bc5ea299af193 P df69f48656b44bb890f76587ee69cc0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 1.
I20250113 02:51:25.298282 1634 raft_consensus.cc:491] T a6055d77d168495d9b756a1a56c1a920 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:25.298470 895 tablet_service.cc:1467] Processing CreateTablet for tablet 95e47db8a4d74615800ba6c049ec7131 (DEFAULT_TABLE table=default.Table_1b71e9ea2fca48daa60e8d692eab8e49 [id=383148ed836144c1bc5403dae46aad50]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:25.298473 1634 raft_consensus.cc:513] T a6055d77d168495d9b756a1a56c1a920 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:25.299247 1634 leader_election.cc:290] T a6055d77d168495d9b756a1a56c1a920 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:25.300225 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6055d77d168495d9b756a1a56c1a920" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
I20250113 02:51:25.300511 1192 tablet_bootstrap.cc:654] T 8aac142230e244c0b59c9c2b7a7300ea P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
W20250113 02:51:25.301115 995 leader_election.cc:343] T a6055d77d168495d9b756a1a56c1a920 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:25.301589 897 tablet_service.cc:1467] Processing CreateTablet for tablet 303d911dfa2f4abbb053ead33db6d71c (DEFAULT_TABLE table=default.Table_96fc1675456247f2a1c6d67ce524f203 [id=034ffa2a4ba6468a94bcf891bd2e6bb3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:25.303416 1191 tablet_bootstrap.cc:492] T 8aac142230e244c0b59c9c2b7a7300ea P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:25.303629 1191 ts_tablet_manager.cc:1397] T 8aac142230e244c0b59c9c2b7a7300ea P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.024s user 0.006s sys 0.004s
I20250113 02:51:25.304629 1191 raft_consensus.cc:357] T 8aac142230e244c0b59c9c2b7a7300ea P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:25.304914 1191 raft_consensus.cc:383] T 8aac142230e244c0b59c9c2b7a7300ea P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:25.305049 1191 raft_consensus.cc:738] T 8aac142230e244c0b59c9c2b7a7300ea P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:25.305382 1191 consensus_queue.cc:260] T 8aac142230e244c0b59c9c2b7a7300ea P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:25.305555 1198 ts_tablet_manager.cc:1918] T ebc5f1ea1e1648d49904b17603d3cd2a P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:25.305684 1198 log.cc:1198] T ebc5f1ea1e1648d49904b17603d3cd2a P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/ebc5f1ea1e1648d49904b17603d3cd2a
I20250113 02:51:25.306077 1198 ts_tablet_manager.cc:1939] T ebc5f1ea1e1648d49904b17603d3cd2a P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:25.306293 1191 ts_tablet_manager.cc:1428] T 8aac142230e244c0b59c9c2b7a7300ea P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250113 02:51:25.306602 1191 tablet_bootstrap.cc:492] T 23e9321f08ee4ce7b2252ad5c76c0a08 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:25.306896 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 435) took 1045 ms. Trace:
I20250113 02:51:25.307018 1198 rpcz_store.cc:276] 0113 02:51:24.261022 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:24.261199 (+ 177us) service_pool.cc:225] Handling call
0113 02:51:25.201155 (+939956us) ts_tablet_manager.cc:1145] Deleting tablet ebc5f1ea1e1648d49904b17603d3cd2a
0113 02:51:25.201167 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:25.237586 (+ 36419us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:25.305417 (+ 67831us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:25.306862 (+ 1445us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":939773}
I20250113 02:51:25.307406 1198 tablet_replica.cc:331] T 23e9321f08ee4ce7b2252ad5c76c0a08 P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:25.307638 1198 raft_consensus.cc:2238] T 23e9321f08ee4ce7b2252ad5c76c0a08 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:25.307811 1198 raft_consensus.cc:2267] T 23e9321f08ee4ce7b2252ad5c76c0a08 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:25.310559 1191 tablet_bootstrap.cc:654] T 23e9321f08ee4ce7b2252ad5c76c0a08 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:25.313629 1192 tablet_bootstrap.cc:492] T 8aac142230e244c0b59c9c2b7a7300ea P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:25.314460 1192 ts_tablet_manager.cc:1397] T 8aac142230e244c0b59c9c2b7a7300ea P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.030s user 0.004s sys 0.000s
I20250113 02:51:25.318440 1191 tablet_bootstrap.cc:492] T 23e9321f08ee4ce7b2252ad5c76c0a08 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:25.318660 1191 ts_tablet_manager.cc:1397] T 23e9321f08ee4ce7b2252ad5c76c0a08 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.006s
I20250113 02:51:25.319445 1191 raft_consensus.cc:357] T 23e9321f08ee4ce7b2252ad5c76c0a08 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:25.319746 1191 raft_consensus.cc:383] T 23e9321f08ee4ce7b2252ad5c76c0a08 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:25.319860 1191 raft_consensus.cc:738] T 23e9321f08ee4ce7b2252ad5c76c0a08 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:25.320227 1191 consensus_queue.cc:260] T 23e9321f08ee4ce7b2252ad5c76c0a08 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:25.320770 1192 raft_consensus.cc:357] T 8aac142230e244c0b59c9c2b7a7300ea P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:25.320926 1191 ts_tablet_manager.cc:1428] T 23e9321f08ee4ce7b2252ad5c76c0a08 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250113 02:51:25.321224 1191 tablet_bootstrap.cc:492] T d3b695ac20ba4205b2d8732db4740531 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:25.322877 1191 tablet_bootstrap.cc:654] T d3b695ac20ba4205b2d8732db4740531 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:25.327358 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6055d77d168495d9b756a1a56c1a920" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:25.328497 1192 raft_consensus.cc:383] T 8aac142230e244c0b59c9c2b7a7300ea P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:25.328646 1192 raft_consensus.cc:738] T 8aac142230e244c0b59c9c2b7a7300ea P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:25.328994 1192 consensus_queue.cc:260] T 8aac142230e244c0b59c9c2b7a7300ea P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:25.331189 1191 tablet_bootstrap.cc:492] T d3b695ac20ba4205b2d8732db4740531 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:25.331380 1191 ts_tablet_manager.cc:1397] T d3b695ac20ba4205b2d8732db4740531 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.004s
I20250113 02:51:25.331852 1192 ts_tablet_manager.cc:1428] T 8aac142230e244c0b59c9c2b7a7300ea P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.012s user 0.001s sys 0.001s
I20250113 02:51:25.332716 1191 raft_consensus.cc:357] T d3b695ac20ba4205b2d8732db4740531 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:25.333386 1191 raft_consensus.cc:383] T d3b695ac20ba4205b2d8732db4740531 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:25.333604 1191 raft_consensus.cc:738] T d3b695ac20ba4205b2d8732db4740531 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:25.333967 1191 consensus_queue.cc:260] T d3b695ac20ba4205b2d8732db4740531 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:25.334237 1194 tablet_bootstrap.cc:492] T d304b65eba8c40c6922333506a60540c P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:25.334805 1191 ts_tablet_manager.cc:1428] T d3b695ac20ba4205b2d8732db4740531 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250113 02:51:25.335062 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 303d911dfa2f4abbb053ead33db6d71c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:25.335566 1194 ts_tablet_manager.cc:1397] T d304b65eba8c40c6922333506a60540c P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.245s user 0.003s sys 0.004s
I20250113 02:51:25.336836 1191 tablet_bootstrap.cc:492] T 438d1d2300e044f39a15586f335935ba P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:25.334088 998 leader_election.cc:304] T 4fe54131aafb40ee9d8bc5ea299af193 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f; no voters: 1186fa869eb14812832fa6562531f196
I20250113 02:51:25.337424 1634 raft_consensus.cc:2798] T 4fe54131aafb40ee9d8bc5ea299af193 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Leader election won for term 1
I20250113 02:51:25.337567 1634 raft_consensus.cc:695] T 4fe54131aafb40ee9d8bc5ea299af193 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: Becoming Leader. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Running, Role: LEADER
W20250113 02:51:25.337630 998 leader_election.cc:343] T a6055d77d168495d9b756a1a56c1a920 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:25.337836 998 leader_election.cc:304] T a6055d77d168495d9b756a1a56c1a920 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4216a7f853d642b7bc3e5cff458f3aca; no voters: 1186fa869eb14812832fa6562531f196, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:25.337877 1634 consensus_queue.cc:237] T 4fe54131aafb40ee9d8bc5ea299af193 P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:25.339282 1191 tablet_bootstrap.cc:654] T 438d1d2300e044f39a15586f335935ba P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:25.344121 1194 raft_consensus.cc:357] T d304b65eba8c40c6922333506a60540c P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:25.347075 1037 tablet_service.cc:1467] Processing CreateTablet for tablet e13267498917432ba92b6badb76f5d73 (DEFAULT_TABLE table=default.Table_0b855cf933544dce95621708e777674f [id=a63ccead990445fdac16fda1312bb66c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:25.347599 1037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e13267498917432ba92b6badb76f5d73. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:25.348295 1627 raft_consensus.cc:2743] T a6055d77d168495d9b756a1a56c1a920 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:25.335191 1192 tablet_bootstrap.cc:492] T ba61c86743e24666b30580c36b9228c3 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:25.350889 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet ebc5f1ea1e1648d49904b17603d3cd2a (table default.Table_3d6c2586a98845c7989a5b715c4d678e [id=f4e09f1ea2184aa78abeb307bfc00d37]) successfully deleted
I20250113 02:51:25.352334 1031 tablet_service.cc:1467] Processing CreateTablet for tablet f9269967dea94fc3a337546ad0842e11 (DEFAULT_TABLE table=default.Table_23c1655ab4cf4a30b67185ccf4fbb76d [id=a8e428598ca94e15b04f1df8112a4abc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:25.352821 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f9269967dea94fc3a337546ad0842e11. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:25.354631 1192 tablet_bootstrap.cc:654] T ba61c86743e24666b30580c36b9228c3 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:25.356315 1042 tablet_service.cc:1467] Processing CreateTablet for tablet 80c647a3b5a1411e9130461115c53550 (DEFAULT_TABLE table=default.Table_82d6d0a936524f35beab62f451e5eb99 [id=c20532b7392a4ec98453d6653de242aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:25.356737 1042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 80c647a3b5a1411e9130461115c53550. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:25.357419 1036 tablet_service.cc:1467] Processing CreateTablet for tablet 29c17fd90d774c029a35e77888358067 (DEFAULT_TABLE table=default.Table_96651718e3f643fcacec8ef593a9b30e [id=2d2795dd06ce4605a285ca2a93e20c4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:25.357832 1036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 29c17fd90d774c029a35e77888358067. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:25.360270 1029 tablet_service.cc:1467] Processing CreateTablet for tablet ed68a2b7ebcf4841bef54798dd3ed80d (DEFAULT_TABLE table=default.Table_7157800e639b45d8bb947e53d4319cfc [id=39997cddb4684b6c984eeeb48d715002]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:25.360651 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ed68a2b7ebcf4841bef54798dd3ed80d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:25.364315 1023 tablet_service.cc:1467] Processing CreateTablet for tablet 146dad5c977f464496f7d21611030dad (DEFAULT_TABLE table=default.Table_4369a143c72643b084df58d25314c6dd [id=36ff239c08a54be99ca307354810fff3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:25.364760 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 146dad5c977f464496f7d21611030dad. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:25.368278 1038 tablet_service.cc:1467] Processing CreateTablet for tablet d8419fec408c40f6bd46c39de7b3dce9 (DEFAULT_TABLE table=default.Table_503d2ce0d8d74387bf59e25670b61bcb [id=35d1232ab27b4e37a706ab6092f9115b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:25.368742 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8419fec408c40f6bd46c39de7b3dce9. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:25.371886 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 572e4eed1e924381bada27f8389821a7 consensus state has no leader
I20250113 02:51:25.376420 1194 raft_consensus.cc:383] T d304b65eba8c40c6922333506a60540c P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:25.376566 1194 raft_consensus.cc:738] T d304b65eba8c40c6922333506a60540c P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:25.377723 746 tablet_service.cc:1514] Processing DeleteTablet for tablet ebc5f1ea1e1648d49904b17603d3cd2a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:19 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:25.378065 1195 tablet_replica.cc:331] T ebc5f1ea1e1648d49904b17603d3cd2a P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:25.378334 1195 raft_consensus.cc:2238] T ebc5f1ea1e1648d49904b17603d3cd2a P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:25.378546 1195 raft_consensus.cc:2267] T ebc5f1ea1e1648d49904b17603d3cd2a P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:25.379068 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8419fec408c40f6bd46c39de7b3dce9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:25.379961 1030 tablet_service.cc:1467] Processing CreateTablet for tablet 303d911dfa2f4abbb053ead33db6d71c (DEFAULT_TABLE table=default.Table_96fc1675456247f2a1c6d67ce524f203 [id=034ffa2a4ba6468a94bcf891bd2e6bb3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:25.380380 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 303d911dfa2f4abbb053ead33db6d71c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:25.380380 1195 ts_tablet_manager.cc:1905] T ebc5f1ea1e1648d49904b17603d3cd2a P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:25.378939 1194 consensus_queue.cc:260] T d304b65eba8c40c6922333506a60540c P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
W20250113 02:51:25.386363 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7f08b72301dc431396cf8cc7790d56d4 consensus state has no leader
I20250113 02:51:25.389567 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet d9df314ffaf042ec80180868d7e7bdf7
I20250113 02:51:25.389441 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_d7db322428ae47649bf436e235e20342"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:25.390024 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d7db322428ae47649bf436e235e20342 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:25.390327 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_2b71ec43b2ee485e92d81d55f56a44fc" } new_table_name: "default.Table_a55eab4604df4ca5b767708b6533e6d3" modify_external_catalogs: true
I20250113 02:51:25.391660 447 master_service.cc:946] ReplaceTablet: received request to replace tablet 258fad818b7b4aa4b337f9949e20aa46 from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:25.392297 447 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_503d2ce0d8d74387bf59e25670b61bcb" } modify_external_catalogs: true
I20250113 02:51:25.392449 447 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_503d2ce0d8d74387bf59e25670b61bcb" } modify_external_catalogs: true
I20250113 02:51:25.393994 454 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_12a092a4543e4b04a466945f4b694c53" } modify_external_catalogs: true
I20250113 02:51:25.394151 454 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_12a092a4543e4b04a466945f4b694c53" } modify_external_catalogs: true
I20250113 02:51:25.395303 1195 ts_tablet_manager.cc:1918] T ebc5f1ea1e1648d49904b17603d3cd2a P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:25.395430 1195 log.cc:1198] T ebc5f1ea1e1648d49904b17603d3cd2a P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/ebc5f1ea1e1648d49904b17603d3cd2a
I20250113 02:51:25.395524 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_0b8a63bbfac5492e83f0f9bd8e0eb441" } modify_external_catalogs: true
I20250113 02:51:25.395699 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_0b8a63bbfac5492e83f0f9bd8e0eb441" } modify_external_catalogs: true
I20250113 02:51:25.395864 1195 ts_tablet_manager.cc:1939] T ebc5f1ea1e1648d49904b17603d3cd2a P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:25.396481 453 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_8b87c8469fbb465e80375aa0446de6c2" } modify_external_catalogs: true
I20250113 02:51:25.396617 453 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_8b87c8469fbb465e80375aa0446de6c2" } modify_external_catalogs: true
I20250113 02:51:25.397540 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_131966d78f11450584893c018d784b84"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:25.398233 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_131966d78f11450584893c018d784b84 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:25.400084 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet ebc5f1ea1e1648d49904b17603d3cd2a (table default.Table_3d6c2586a98845c7989a5b715c4d678e [id=f4e09f1ea2184aa78abeb307bfc00d37]) successfully deleted
I20250113 02:51:25.400636 746 tablet_service.cc:1514] Processing DeleteTablet for tablet d9df314ffaf042ec80180868d7e7bdf7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:24 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:25.404393 455 catalog_manager.cc:5618] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587) does not have the latest schema for tablet 317c6e8013b745c290bf3190dc237ab8 (table default.Table_13fd4df9c207466cb85cd8ed9062722c [id=6bfb67a96b2b47849f7fa12c57a2f4a1]). Expected version 1 got 0
I20250113 02:51:25.405694 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_26a65a837e114f288e826d343b8f78a1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:25.406195 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_26a65a837e114f288e826d343b8f78a1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:25.410225 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet d8419fec408c40f6bd46c39de7b3dce9
I20250113 02:51:25.411572 746 tablet_service.cc:1514] Processing DeleteTablet for tablet d8419fec408c40f6bd46c39de7b3dce9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:25 UTC) from {username='slave'} at 127.0.0.1:55398
W20250113 02:51:25.411953 451 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8cc78bf7d6324b7ba10a1b1045923f2b consensus state has no leader
I20250113 02:51:25.412283 451 master_service.cc:946] ReplaceTablet: received request to replace tablet 840773ee80c7447097ea000b106fc89d from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:25.414711 1041 tablet_service.cc:1467] Processing CreateTablet for tablet 95e47db8a4d74615800ba6c049ec7131 (DEFAULT_TABLE table=default.Table_1b71e9ea2fca48daa60e8d692eab8e49 [id=383148ed836144c1bc5403dae46aad50]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:25.415325 1042 tablet_service.cc:1514] Processing DeleteTablet for tablet d9df314ffaf042ec80180868d7e7bdf7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:24 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:25.415480 1442 raft_consensus.cc:491] T 1500e4f9c0664a879503afd3edf14c66 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:25.415627 1442 raft_consensus.cc:513] T 1500e4f9c0664a879503afd3edf14c66 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
W20250113 02:51:25.416291 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet d9df314ffaf042ec80180868d7e7bdf7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d9df314ffaf042ec80180868d7e7bdf7 already in progress: creating tablet
W20250113 02:51:25.416651 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet d8419fec408c40f6bd46c39de7b3dce9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d8419fec408c40f6bd46c39de7b3dce9 already in progress: creating tablet
I20250113 02:51:25.417120 1442 leader_election.cc:290] T 1500e4f9c0664a879503afd3edf14c66 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
I20250113 02:51:25.417451 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95e47db8a4d74615800ba6c049ec7131. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:25.419759 1042 tablet_service.cc:1514] Processing DeleteTablet for tablet d8419fec408c40f6bd46c39de7b3dce9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:25 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:25.423098 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1500e4f9c0664a879503afd3edf14c66" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
I20250113 02:51:25.423208 1194 ts_tablet_manager.cc:1428] T d304b65eba8c40c6922333506a60540c P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.049s user 0.002s sys 0.000s
I20250113 02:51:25.423303 775 raft_consensus.cc:2463] T 1500e4f9c0664a879503afd3edf14c66 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 0.
I20250113 02:51:25.423880 891 tablet_service.cc:1514] Processing DeleteTablet for tablet d9df314ffaf042ec80180868d7e7bdf7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:24 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:25.424093 884 tablet_service.cc:1514] Processing DeleteTablet for tablet d8419fec408c40f6bd46c39de7b3dce9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:25 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:25.424541 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95e47db8a4d74615800ba6c049ec7131. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:25.426023 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_f6135d2bf1204fb5b661ea8a71c526a4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:25.426774 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f6135d2bf1204fb5b661ea8a71c526a4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:25.428367 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1500e4f9c0664a879503afd3edf14c66" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:25.435444 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet f1e32e09a8244957a99bdfea5812ff31
I20250113 02:51:25.436318 454 master_service.cc:946] ReplaceTablet: received request to replace tablet 317c6e8013b745c290bf3190dc237ab8 from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:25.437963 746 tablet_service.cc:1514] Processing DeleteTablet for tablet f1e32e09a8244957a99bdfea5812ff31 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:25 UTC) from {username='slave'} at 127.0.0.1:55398
W20250113 02:51:25.438872 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet f1e32e09a8244957a99bdfea5812ff31 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f1e32e09a8244957a99bdfea5812ff31 already in progress: creating tablet
I20250113 02:51:25.442231 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_ac80d162769b4fc183613eb52f1bc4b7" } new_table_name: "default.Table_f069a5be395340ee9e86f3f25eaa5197" modify_external_catalogs: true
I20250113 02:51:25.444249 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_e1f19e9072094459a8aba69a14179bda" } new_table_name: "default.Table_e98730a6e1c74c4fbd2b39ac115701f6" modify_external_catalogs: true
I20250113 02:51:25.448522 995 leader_election.cc:304] T 1500e4f9c0664a879503afd3edf14c66 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters:
I20250113 02:51:25.449471 888 tablet_service.cc:1514] Processing DeleteTablet for tablet f1e32e09a8244957a99bdfea5812ff31 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:25 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:25.449642 1194 tablet_bootstrap.cc:492] T ebc5f1ea1e1648d49904b17603d3cd2a P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:25.452644 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 317c6e8013b745c290bf3190dc237ab8
I20250113 02:51:25.453092 454 catalog_manager.cc:6596] ReplaceTablet: tablet 317c6e8013b745c290bf3190dc237ab8 deleted and replaced by tablet c5d77927b1b24ffebb21c9f10e5cd440
I20250113 02:51:25.453766 746 tablet_service.cc:1514] Processing DeleteTablet for tablet 317c6e8013b745c290bf3190dc237ab8 with delete_type TABLET_DATA_DELETED (replaced by tablet c5d77927b1b24ffebb21c9f10e5cd440) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:25.454121 1195 tablet_replica.cc:331] T 317c6e8013b745c290bf3190dc237ab8 P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:25.454329 1195 raft_consensus.cc:2238] T 317c6e8013b745c290bf3190dc237ab8 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:25.454501 1195 raft_consensus.cc:2267] T 317c6e8013b745c290bf3190dc237ab8 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:25.455308 1195 ts_tablet_manager.cc:1905] T 317c6e8013b745c290bf3190dc237ab8 P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250113 02:51:25.455848 451 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9438472464304da9a0c7658d4f0736b6 consensus state has no leader
I20250113 02:51:25.459748 888 tablet_service.cc:1514] Processing DeleteTablet for tablet 317c6e8013b745c290bf3190dc237ab8 with delete_type TABLET_DATA_DELETED (replaced by tablet c5d77927b1b24ffebb21c9f10e5cd440) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:25.462574 1195 ts_tablet_manager.cc:1918] T 317c6e8013b745c290bf3190dc237ab8 P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250113 02:51:25.462759 1195 log.cc:1198] T 317c6e8013b745c290bf3190dc237ab8 P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/317c6e8013b745c290bf3190dc237ab8
I20250113 02:51:25.463948 1195 ts_tablet_manager.cc:1939] T 317c6e8013b745c290bf3190dc237ab8 P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
W20250113 02:51:25.464846 998 leader_election.cc:343] T 1500e4f9c0664a879503afd3edf14c66 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:25.465123 1442 raft_consensus.cc:2798] T 1500e4f9c0664a879503afd3edf14c66 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 02:51:25.465387 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet 317c6e8013b745c290bf3190dc237ab8 (table default.Table_13fd4df9c207466cb85cd8ed9062722c [id=6bfb67a96b2b47849f7fa12c57a2f4a1]) successfully deleted
W20250113 02:51:25.466442 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fe8f075d4c9a4f9abe372bfdb81a829c consensus state has no leader
I20250113 02:51:25.468555 452 catalog_manager.cc:5618] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095) does not have the latest schema for tablet d3b695ac20ba4205b2d8732db4740531 (table default.Table_8b87c8469fbb465e80375aa0446de6c2 [id=f85f0c435ffa4fdca81d86de4aa4fd5f]). Expected version 3 got 0
I20250113 02:51:25.468849 452 catalog_manager.cc:5618] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095) does not have the latest schema for tablet 06f36d09bbb1443bafce906907795404 (table default.Table_f8ea53b156c944ebaff83a83c3b75118 [id=423021f304964246bd29e4daaf470154]). Expected version 1 got 0
I20250113 02:51:25.469118 452 catalog_manager.cc:5618] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095) does not have the latest schema for tablet d304b65eba8c40c6922333506a60540c (table default.Table_e03d954cc20048e4b0d598282689643d [id=e62f8e4c51b04729903b1943275c5f7f]). Expected version 1 got 0
I20250113 02:51:25.469660 746 tablet_service.cc:1467] Processing CreateTablet for tablet 943d3a125ac5406a83f3c8d80e073b6e (DEFAULT_TABLE table=default.Table_0b8a63bbfac5492e83f0f9bd8e0eb441 [id=45c91056f5fe49ecae8e1e04e153d2ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:25.470109 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 943d3a125ac5406a83f3c8d80e073b6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:25.470788 1196 ts_tablet_manager.cc:1918] T f1a02c37cb4f43f8a9de6484f4431958 P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:25.470921 1196 log.cc:1198] T f1a02c37cb4f43f8a9de6484f4431958 P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/f1a02c37cb4f43f8a9de6484f4431958
I20250113 02:51:25.471302 1196 ts_tablet_manager.cc:1939] T f1a02c37cb4f43f8a9de6484f4431958 P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:25.472117 884 tablet_service.cc:1467] Processing CreateTablet for tablet 60fac96c00d64ec193b0c706ec64bfaf (DEFAULT_TABLE table=default.Table_5030a9f5589a4f85865ee2ba6562ee1d [id=008c98aa68334484ae38e4f10b393cb9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:25.472491 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 60fac96c00d64ec193b0c706ec64bfaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:25.473203 1196 tablet_replica.cc:331] T 317c6e8013b745c290bf3190dc237ab8 P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:25.473426 1196 raft_consensus.cc:2238] T 317c6e8013b745c290bf3190dc237ab8 P df69f48656b44bb890f76587ee69cc0f [term 1 LEADER]: Raft consensus shutting down.
I20250113 02:51:25.473840 1196 raft_consensus.cc:2267] T 317c6e8013b745c290bf3190dc237ab8 P df69f48656b44bb890f76587ee69cc0f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:25.474463 1042 tablet_service.cc:1514] Processing DeleteTablet for tablet f1e32e09a8244957a99bdfea5812ff31 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:25 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:25.475095 1196 ts_tablet_manager.cc:1905] T 317c6e8013b745c290bf3190dc237ab8 P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:25.475798 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet f1a02c37cb4f43f8a9de6484f4431958 (table default.Table_f8af5d3e0dc64d3fae054f82a7b33ccf [id=4b739228564344b8b394a4718750dbb0]) successfully deleted
I20250113 02:51:25.476562 893 tablet_service.cc:1467] Processing CreateTablet for tablet 89b6fbf30ca44488ad7969930f1356e2 (DEFAULT_TABLE table=default.Table_aebf098beedb49a4acfdb767bf5ac454 [id=291fd92b15b9498b8e6bd3695f0529f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:25.476965 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 89b6fbf30ca44488ad7969930f1356e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:25.477384 899 tablet_service.cc:1467] Processing CreateTablet for tablet 6d6e462c8bfa47eaa43d3695c69d9c0a (DEFAULT_TABLE table=default.Table_59b60c21ea364f89967c58758807a6a4 [id=f8d606a8b6c340bfa22967fe9711b17d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:25.477778 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d6e462c8bfa47eaa43d3695c69d9c0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:25.478029 741 tablet_service.cc:1467] Processing CreateTablet for tablet 6d6e462c8bfa47eaa43d3695c69d9c0a (DEFAULT_TABLE table=default.Table_59b60c21ea364f89967c58758807a6a4 [id=f8d606a8b6c340bfa22967fe9711b17d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:25.478454 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d6e462c8bfa47eaa43d3695c69d9c0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:25.478653 750 tablet_service.cc:1467] Processing CreateTablet for tablet 60fac96c00d64ec193b0c706ec64bfaf (DEFAULT_TABLE table=default.Table_5030a9f5589a4f85865ee2ba6562ee1d [id=008c98aa68334484ae38e4f10b393cb9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:25.478982 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 60fac96c00d64ec193b0c706ec64bfaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:25.479240 882 tablet_service.cc:1467] Processing CreateTablet for tablet b16dd8451f32478daf2ae118dbb9b304 (DEFAULT_TABLE table=default.Table_677784013a07442b9963695510a0a055 [id=06a7fc0682504214990e9a6c830d15ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:25.479569 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b16dd8451f32478daf2ae118dbb9b304. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:25.480227 896 tablet_service.cc:1467] Processing CreateTablet for tablet d7542beb84c94b258d2f35a4dc74eec5 (DEFAULT_TABLE table=default.Table_d1a29205746d47a6bcf4719cdaeaf285 [id=9dd99ec7e3b3474ea373bb6fd72b545d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:25.480545 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7542beb84c94b258d2f35a4dc74eec5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:25.481211 889 tablet_service.cc:1467] Processing CreateTablet for tablet 8412a588d0454ec78025e7ec30c38104 (DEFAULT_TABLE table=default.Table_a5f105b8a4ad4933b499ebaa8ecca921 [id=f1aa4a9a09434178af0610275e931a52]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:25.481546 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8412a588d0454ec78025e7ec30c38104. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:25.482988 737 tablet_service.cc:1467] Processing CreateTablet for tablet a78081b7efe34af8a053f5272cfaf346 (DEFAULT_TABLE table=default.Table_7843edf8e2e14e51b6ab66031605949e [id=b5c8764b9d1c4ddbaf888e9989822752]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:25.483361 737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a78081b7efe34af8a053f5272cfaf346. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:25.483402 887 tablet_service.cc:1467] Processing CreateTablet for tablet 61c79029dfb74f01aaac0f7f828880d0 (DEFAULT_TABLE table=default.Table_2bf3c3c3eb364670a2d0633238a32fe6 [id=ea4fc227dfb145a2b4a660062410e193]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:25.483743 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 61c79029dfb74f01aaac0f7f828880d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:25.484238 880 tablet_service.cc:1467] Processing CreateTablet for tablet dfea918c04c1408a8b86a44bb0911a7c (DEFAULT_TABLE table=default.Table_f5f150e8da7a4848bfab67568975d66f [id=5a295e63ad43499f90b2993ac92a16cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:25.484375 744 tablet_service.cc:1467] Processing CreateTablet for tablet d29899b4cccb4e37bd25978d385916f9 (DEFAULT_TABLE table=default.Table_44413b1d27e14bff98f1a50e0d1a4dd4 [id=a49d010a40304ec0b1019964e3e0e150]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:25.484544 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dfea918c04c1408a8b86a44bb0911a7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:25.484766 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d29899b4cccb4e37bd25978d385916f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:25.485137 883 tablet_service.cc:1467] Processing CreateTablet for tablet 6d5148cf8ff24c92aedf3faee5ae8723 (DEFAULT_TABLE table=default.Table_5e92e31ef2494f34a1fab99e7c6bd9f7 [id=12cf3b5d63db4f8d931d11751bf8e47a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:25.485451 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d5148cf8ff24c92aedf3faee5ae8723. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:25.485580 1191 tablet_bootstrap.cc:492] T 438d1d2300e044f39a15586f335935ba P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:25.485751 1191 ts_tablet_manager.cc:1397] T 438d1d2300e044f39a15586f335935ba P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.149s user 0.012s sys 0.027s
I20250113 02:51:25.486604 1191 raft_consensus.cc:357] T 438d1d2300e044f39a15586f335935ba P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:25.486881 1191 raft_consensus.cc:383] T 438d1d2300e044f39a15586f335935ba P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:25.486991 1191 raft_consensus.cc:738] T 438d1d2300e044f39a15586f335935ba P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:25.487285 1191 consensus_queue.cc:260] T 438d1d2300e044f39a15586f335935ba P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:25.487874 1191 ts_tablet_manager.cc:1428] T 438d1d2300e044f39a15586f335935ba P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250113 02:51:25.488257 1191 tablet_bootstrap.cc:492] T 32351d6de1574353a6e8073d333da477 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:25.488247 752 tablet_service.cc:1467] Processing CreateTablet for tablet 89b6fbf30ca44488ad7969930f1356e2 (DEFAULT_TABLE table=default.Table_aebf098beedb49a4acfdb767bf5ac454 [id=291fd92b15b9498b8e6bd3695f0529f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:25.488656 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 89b6fbf30ca44488ad7969930f1356e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:25.490226 1191 tablet_bootstrap.cc:654] T 32351d6de1574353a6e8073d333da477 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
W20250113 02:51:25.491400 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 9e736211eb3c4536b9fd1d90d2d83838 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9e736211eb3c4536b9fd1d90d2d83838 already in progress: creating tablet
W20250113 02:51:25.491741 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 438604e8859f422f9caccb61c5b5b125 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 438604e8859f422f9caccb61c5b5b125 already in progress: creating tablet
W20250113 02:51:25.492020 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet a2781ba5061a4225b5636366857a229e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a2781ba5061a4225b5636366857a229e already in progress: creating tablet
W20250113 02:51:25.492271 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 0c408921a09a463eb464e08edcf042b3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0c408921a09a463eb464e08edcf042b3 already in progress: creating tablet
I20250113 02:51:25.492301 736 tablet_service.cc:1467] Processing CreateTablet for tablet b16dd8451f32478daf2ae118dbb9b304 (DEFAULT_TABLE table=default.Table_677784013a07442b9963695510a0a055 [id=06a7fc0682504214990e9a6c830d15ea]), partition=RANGE (key) PARTITION UNBOUNDED
W20250113 02:51:25.492501 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 283fc7f1c74e480c8a63c74d3ae9eb7d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 283fc7f1c74e480c8a63c74d3ae9eb7d already in progress: creating tablet
I20250113 02:51:25.492691 736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b16dd8451f32478daf2ae118dbb9b304. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:25.492714 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 056bf1e67be04d179eace3d93282b145 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 056bf1e67be04d179eace3d93282b145 already in progress: creating tablet
W20250113 02:51:25.492962 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet bc533518faa24dffa3306ff092e8d507 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bc533518faa24dffa3306ff092e8d507 already in progress: creating tablet
W20250113 02:51:25.493186 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet d9df314ffaf042ec80180868d7e7bdf7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d9df314ffaf042ec80180868d7e7bdf7 already in progress: creating tablet
W20250113 02:51:25.493431 442 catalog_manager.cc:4839] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet d8419fec408c40f6bd46c39de7b3dce9 because the tablet was not found. No further retry: Not found: Tablet not found: d8419fec408c40f6bd46c39de7b3dce9
W20250113 02:51:25.493638 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet f1e32e09a8244957a99bdfea5812ff31 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f1e32e09a8244957a99bdfea5812ff31 already in progress: creating tablet
I20250113 02:51:25.494433 1525 raft_consensus.cc:491] T d7920c0544c0466bad6bd8c09aae1a7d P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:25.494616 1525 raft_consensus.cc:513] T d7920c0544c0466bad6bd8c09aae1a7d P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:25.495586 1525 leader_election.cc:290] T d7920c0544c0466bad6bd8c09aae1a7d P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:25.496215 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7920c0544c0466bad6bd8c09aae1a7d" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:25.496327 740 tablet_service.cc:1467] Processing CreateTablet for tablet 6d5148cf8ff24c92aedf3faee5ae8723 (DEFAULT_TABLE table=default.Table_5e92e31ef2494f34a1fab99e7c6bd9f7 [id=12cf3b5d63db4f8d931d11751bf8e47a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:25.496529 918 raft_consensus.cc:2463] T d7920c0544c0466bad6bd8c09aae1a7d P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1186fa869eb14812832fa6562531f196 in term 0.
I20250113 02:51:25.496714 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d5148cf8ff24c92aedf3faee5ae8723. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:25.496975 711 leader_election.cc:304] T d7920c0544c0466bad6bd8c09aae1a7d P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1186fa869eb14812832fa6562531f196, df69f48656b44bb890f76587ee69cc0f; no voters:
I20250113 02:51:25.499967 748 tablet_service.cc:1467] Processing CreateTablet for tablet d7542beb84c94b258d2f35a4dc74eec5 (DEFAULT_TABLE table=default.Table_d1a29205746d47a6bcf4719cdaeaf285 [id=9dd99ec7e3b3474ea373bb6fd72b545d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:25.500308 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7542beb84c94b258d2f35a4dc74eec5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:25.500928 1525 raft_consensus.cc:2798] T d7920c0544c0466bad6bd8c09aae1a7d P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 02:51:25.501091 1525 raft_consensus.cc:491] T d7920c0544c0466bad6bd8c09aae1a7d P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 02:51:25.501212 1525 raft_consensus.cc:3054] T d7920c0544c0466bad6bd8c09aae1a7d P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:25.504223 747 tablet_service.cc:1467] Processing CreateTablet for tablet 8412a588d0454ec78025e7ec30c38104 (DEFAULT_TABLE table=default.Table_a5f105b8a4ad4933b499ebaa8ecca921 [id=f1aa4a9a09434178af0610275e931a52]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:25.504596 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8412a588d0454ec78025e7ec30c38104. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:25.504832 885 tablet_service.cc:1467] Processing CreateTablet for tablet 089726074e7a40e88bb44549d04874fb (DEFAULT_TABLE table=default.Table_9ecf36f09ace4d59b57e75e0967b2956 [id=a61ea224ef9540cbab35007cc88628af]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:25.505329 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 089726074e7a40e88bb44549d04874fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:25.507321 886 tablet_service.cc:1467] Processing CreateTablet for tablet 14cea6298852460ca8cfb74e321cc124 (DEFAULT_TABLE table=default.Table_e5ce75ec7b044da3ae915dc3ce5421da [id=eb72e04ab4fc4f83b97c61c10c527032]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:25.507766 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 14cea6298852460ca8cfb74e321cc124. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:25.508219 754 tablet_service.cc:1467] Processing CreateTablet for tablet 61c79029dfb74f01aaac0f7f828880d0 (DEFAULT_TABLE table=default.Table_2bf3c3c3eb364670a2d0633238a32fe6 [id=ea4fc227dfb145a2b4a660062410e193]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:25.508518 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 61c79029dfb74f01aaac0f7f828880d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:25.509913 1194 tablet_bootstrap.cc:654] T ebc5f1ea1e1648d49904b17603d3cd2a P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:25.511864 743 tablet_service.cc:1467] Processing CreateTablet for tablet dfea918c04c1408a8b86a44bb0911a7c (DEFAULT_TABLE table=default.Table_f5f150e8da7a4848bfab67568975d66f [id=5a295e63ad43499f90b2993ac92a16cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:25.512229 753 tablet_service.cc:1467] Processing CreateTablet for tablet 089726074e7a40e88bb44549d04874fb (DEFAULT_TABLE table=default.Table_9ecf36f09ace4d59b57e75e0967b2956 [id=a61ea224ef9540cbab35007cc88628af]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:25.512497 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 089726074e7a40e88bb44549d04874fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:25.517190 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dfea918c04c1408a8b86a44bb0911a7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:25.519336 888 tablet_service.cc:1467] Processing CreateTablet for tablet 943d3a125ac5406a83f3c8d80e073b6e (DEFAULT_TABLE table=default.Table_0b8a63bbfac5492e83f0f9bd8e0eb441 [id=45c91056f5fe49ecae8e1e04e153d2ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:25.519753 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 943d3a125ac5406a83f3c8d80e073b6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:25.524464 741 tablet_service.cc:1467] Processing CreateTablet for tablet 14cea6298852460ca8cfb74e321cc124 (DEFAULT_TABLE table=default.Table_e5ce75ec7b044da3ae915dc3ce5421da [id=eb72e04ab4fc4f83b97c61c10c527032]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:25.524860 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 14cea6298852460ca8cfb74e321cc124. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:25.525765 891 tablet_service.cc:1467] Processing CreateTablet for tablet d29899b4cccb4e37bd25978d385916f9 (DEFAULT_TABLE table=default.Table_44413b1d27e14bff98f1a50e0d1a4dd4 [id=a49d010a40304ec0b1019964e3e0e150]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:25.527158 1191 tablet_bootstrap.cc:492] T 32351d6de1574353a6e8073d333da477 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:25.527307 1191 ts_tablet_manager.cc:1397] T 32351d6de1574353a6e8073d333da477 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.039s user 0.008s sys 0.016s
I20250113 02:51:25.527776 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d29899b4cccb4e37bd25978d385916f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:25.528296 1191 raft_consensus.cc:357] T 32351d6de1574353a6e8073d333da477 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:25.528533 1191 raft_consensus.cc:383] T 32351d6de1574353a6e8073d333da477 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:25.528661 1191 raft_consensus.cc:738] T 32351d6de1574353a6e8073d333da477 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:25.528942 881 tablet_service.cc:1467] Processing CreateTablet for tablet a78081b7efe34af8a053f5272cfaf346 (DEFAULT_TABLE table=default.Table_7843edf8e2e14e51b6ab66031605949e [id=b5c8764b9d1c4ddbaf888e9989822752]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:25.528928 1191 consensus_queue.cc:260] T 32351d6de1574353a6e8073d333da477 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:25.537294 1525 raft_consensus.cc:513] T d7920c0544c0466bad6bd8c09aae1a7d P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:25.538146 1525 leader_election.cc:290] T d7920c0544c0466bad6bd8c09aae1a7d P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 election: Requested vote from peers 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
W20250113 02:51:25.539111 708 connection.cc:537] client connection to 127.0.104.60:42067 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250113 02:51:25.542250 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a78081b7efe34af8a053f5272cfaf346. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:25.542239 851 connection.cc:537] client connection to 127.0.104.60:42067 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250113 02:51:25.548095 439 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.104.60:42067 (error 108) [suppressed 4 similar messages]
I20250113 02:51:25.552352 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7920c0544c0466bad6bd8c09aae1a7d" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f"
I20250113 02:51:25.564095 918 raft_consensus.cc:3054] T d7920c0544c0466bad6bd8c09aae1a7d P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:25.573786 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_bb3ef05cbb904c14a9c2347246fa2fab" } new_table_name: "default.Table_8641ddb6d21949528d69f2be6ed9011f" modify_external_catalogs: true
I20250113 02:51:25.583501 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet 317c6e8013b745c290bf3190dc237ab8 with delete_type TABLET_DATA_DELETED (replaced by tablet c5d77927b1b24ffebb21c9f10e5cd440) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:25.584156 451 master_service.cc:946] ReplaceTablet: received request to replace tablet 840773ee80c7447097ea000b106fc89d from {username='slave'} at 127.0.0.1:49258
W20250113 02:51:25.584830 439 consensus_peers.cc:487] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9 -> Peer 45e9d43ecac0401fb2c18fe5eaf089b5 (127.0.104.60:42067): Couldn't send request to peer 45e9d43ecac0401fb2c18fe5eaf089b5. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.60:42067: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250113 02:51:25.585266 998 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:38467, user_credentials={real_user=slave}} blocked reactor thread for 153455us
I20250113 02:51:25.585534 1442 raft_consensus.cc:491] T 1500e4f9c0664a879503afd3edf14c66 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 02:51:25.585680 1442 raft_consensus.cc:3054] T 1500e4f9c0664a879503afd3edf14c66 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:25.585846 1062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7920c0544c0466bad6bd8c09aae1a7d" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:25.586854 1198 ts_tablet_manager.cc:1905] T 23e9321f08ee4ce7b2252ad5c76c0a08 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:25.587467 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet d3b695ac20ba4205b2d8732db4740531
I20250113 02:51:25.588156 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 943d3a125ac5406a83f3c8d80e073b6e
W20250113 02:51:25.590763 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d59134b029a143038837366f51f00d28 consensus state has no leader
I20250113 02:51:25.590917 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_f8ea53b156c944ebaff83a83c3b75118" } new_table_name: "default.Table_2a627a757ab0406686564083ecc36c07" modify_external_catalogs: true
I20250113 02:51:25.594273 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_eec56bd1a5344ab985b794cb42a0eeba"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:25.594851 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_eec56bd1a5344ab985b794cb42a0eeba in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:25.596573 740 tablet_service.cc:1514] Processing DeleteTablet for tablet 317c6e8013b745c290bf3190dc237ab8 with delete_type TABLET_DATA_DELETED (replaced by tablet c5d77927b1b24ffebb21c9f10e5cd440) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:25.596892 753 tablet_service.cc:1514] Processing DeleteTablet for tablet 93a2729c5a604a27a9bbc451ccb4e128 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:20 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:25.597103 754 tablet_service.cc:1514] Processing DeleteTablet for tablet 23e9321f08ee4ce7b2252ad5c76c0a08 with delete_type TABLET_DATA_DELETED (replaced by tablet 6666922d677a4c678f80d7c6c47fd2a5) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:25.597090 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_e195b201eb0f468386996107b50d3bd2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:25.597296 747 tablet_service.cc:1514] Processing DeleteTablet for tablet 8aac142230e244c0b59c9c2b7a7300ea with delete_type TABLET_DATA_DELETED (replaced by tablet f31dd916336a4d8fb3ab015af7de8ccb) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:25.597544 752 tablet_service.cc:1514] Processing DeleteTablet for tablet 1500e4f9c0664a879503afd3edf14c66 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:21 UTC) from {username='slave'} at 127.0.0.1:55398
W20250113 02:51:25.597587 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e195b201eb0f468386996107b50d3bd2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:25.597543 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_ab087b92dedd4bc5a48fa4e8c9970ce1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:25.597775 736 tablet_service.cc:1514] Processing DeleteTablet for tablet 943d3a125ac5406a83f3c8d80e073b6e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:25 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:25.597975 744 tablet_service.cc:1514] Processing DeleteTablet for tablet d3b695ac20ba4205b2d8732db4740531 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:25 UTC) from {username='slave'} at 127.0.0.1:55398
W20250113 02:51:25.598083 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ab087b92dedd4bc5a48fa4e8c9970ce1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:25.598467 1195 tablet_replica.cc:331] T 93a2729c5a604a27a9bbc451ccb4e128 P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:25.598659 1195 raft_consensus.cc:2238] T 93a2729c5a604a27a9bbc451ccb4e128 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:25.598871 1195 raft_consensus.cc:2267] T 93a2729c5a604a27a9bbc451ccb4e128 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:25.598805 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_81ec0fd5c30248a5bdef2311e0164b36"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:25.599239 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_81ec0fd5c30248a5bdef2311e0164b36 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:25.600836 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_96fc1675456247f2a1c6d67ce524f203" } modify_external_catalogs: true
I20250113 02:51:25.601052 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_96fc1675456247f2a1c6d67ce524f203" } modify_external_catalogs: true
I20250113 02:51:25.602438 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_d1dafd1ed4a64d17b499f312bc298bd7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:25.602923 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d1dafd1ed4a64d17b499f312bc298bd7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:25.602815 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_d4b3b319759f44d4bbe072b1bc97cc94"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:25.603335 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d4b3b319759f44d4bbe072b1bc97cc94 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250113 02:51:25.604661 439 catalog_manager.cc:4839] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 317c6e8013b745c290bf3190dc237ab8 because the tablet was not found. No further retry: Not found: Tablet not found: 317c6e8013b745c290bf3190dc237ab8
W20250113 02:51:25.585526 995 connection.cc:537] client connection to 127.0.104.60:42067 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250113 02:51:25.611027 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7920c0544c0466bad6bd8c09aae1a7d" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca"
I20250113 02:51:25.612986 449 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_eaa71690c1224509a1b527415d09f340" } modify_external_catalogs: true
I20250113 02:51:25.613200 449 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_eaa71690c1224509a1b527415d09f340" } modify_external_catalogs: true
W20250113 02:51:25.613888 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 06f36d09bbb1443bafce906907795404 consensus state has no leader
I20250113 02:51:25.614359 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_d1ff69f5cd31453aaf9d8f6382eedcda"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:25.615015 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d1ff69f5cd31453aaf9d8f6382eedcda in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:25.615442 1027 tablet_service.cc:1467] Processing CreateTablet for tablet 943d3a125ac5406a83f3c8d80e073b6e (DEFAULT_TABLE table=default.Table_0b8a63bbfac5492e83f0f9bd8e0eb441 [id=45c91056f5fe49ecae8e1e04e153d2ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:25.616083 1195 ts_tablet_manager.cc:1905] T 93a2729c5a604a27a9bbc451ccb4e128 P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:25.616981 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_7e49ba949e8949fa9924acdef98c3947" } new_table_name: "default.Table_d7fc9abb38eb4f5a9276aa825095c20b" modify_external_catalogs: true
I20250113 02:51:25.616700 1042 tablet_service.cc:1467] Processing CreateTablet for tablet 60fac96c00d64ec193b0c706ec64bfaf (DEFAULT_TABLE table=default.Table_5030a9f5589a4f85865ee2ba6562ee1d [id=008c98aa68334484ae38e4f10b393cb9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:25.617987 1062 raft_consensus.cc:2463] T d7920c0544c0466bad6bd8c09aae1a7d P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1186fa869eb14812832fa6562531f196 in term 0.
I20250113 02:51:25.618716 1192 tablet_bootstrap.cc:492] T ba61c86743e24666b30580c36b9228c3 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:25.618868 1192 ts_tablet_manager.cc:1397] T ba61c86743e24666b30580c36b9228c3 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.284s user 0.041s sys 0.030s
I20250113 02:51:25.620760 1191 ts_tablet_manager.cc:1428] T 32351d6de1574353a6e8073d333da477 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.093s user 0.004s sys 0.045s
I20250113 02:51:25.622483 1028 tablet_service.cc:1467] Processing CreateTablet for tablet a78081b7efe34af8a053f5272cfaf346 (DEFAULT_TABLE table=default.Table_7843edf8e2e14e51b6ab66031605949e [id=b5c8764b9d1c4ddbaf888e9989822752]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:25.624155 1191 tablet_bootstrap.cc:492] T 4efa63b8be794e1db4b340f4c0c4cd5a P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:25.625013 1195 ts_tablet_manager.cc:1918] T 93a2729c5a604a27a9bbc451ccb4e128 P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:25.625149 1195 log.cc:1198] T 93a2729c5a604a27a9bbc451ccb4e128 P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/93a2729c5a604a27a9bbc451ccb4e128
I20250113 02:51:25.625607 1195 ts_tablet_manager.cc:1939] T 93a2729c5a604a27a9bbc451ccb4e128 P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:25.625634 1042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 60fac96c00d64ec193b0c706ec64bfaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:25.626164 1191 tablet_bootstrap.cc:654] T 4efa63b8be794e1db4b340f4c0c4cd5a P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:25.627151 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet 93a2729c5a604a27a9bbc451ccb4e128 (table default.Table_4f1f306815b145218e24c93821376721 [id=5d1feb17ef344c9287e571496b2b405f]) successfully deleted
I20250113 02:51:25.627274 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a78081b7efe34af8a053f5272cfaf346. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:25.627532 1660 negotiation.cc:337] Failed RPC negotiation. Trace:
0113 02:51:25.539959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.60:42067 (local address 127.0.104.1:37005)
0113 02:51:25.626837 (+ 86878us) negotiation.cc:107] Waiting for socket to connect
0113 02:51:25.627097 (+ 260us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.60:42067: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":86738,"thread_start_us":45934,"threads_started":1}
W20250113 02:51:25.627945 998 consensus_peers.cc:487] T 157905316fc545b6a7864ae1c4a2a584 P 4216a7f853d642b7bc3e5cff458f3aca -> Peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Couldn't send request to peer df69f48656b44bb890f76587ee69cc0f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 157905316fc545b6a7864ae1c4a2a584. This is attempt 1: this message will repeat every 5th retry.
I20250113 02:51:25.628103 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 943d3a125ac5406a83f3c8d80e073b6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:25.628115 897 tablet_service.cc:1514] Processing DeleteTablet for tablet d3b695ac20ba4205b2d8732db4740531 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:25 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:25.630220 1659 raft_consensus.cc:491] T d304b65eba8c40c6922333506a60540c P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:25.630381 1659 raft_consensus.cc:513] T d304b65eba8c40c6922333506a60540c P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:25.631543 1062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d304b65eba8c40c6922333506a60540c" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:25.631809 1062 raft_consensus.cc:2463] T d304b65eba8c40c6922333506a60540c P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1186fa869eb14812832fa6562531f196 in term 0.
I20250113 02:51:25.633515 709 leader_election.cc:304] T d304b65eba8c40c6922333506a60540c P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters:
I20250113 02:51:25.633908 1525 raft_consensus.cc:2798] T d304b65eba8c40c6922333506a60540c P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 02:51:25.634014 1525 raft_consensus.cc:491] T d304b65eba8c40c6922333506a60540c P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 02:51:25.634106 1525 raft_consensus.cc:3054] T d304b65eba8c40c6922333506a60540c P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:25.634972 1659 leader_election.cc:290] T d304b65eba8c40c6922333506a60540c P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:25.636936 1525 raft_consensus.cc:513] T d304b65eba8c40c6922333506a60540c P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:25.639129 1525 leader_election.cc:290] T d304b65eba8c40c6922333506a60540c P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 election: Requested vote from peers 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:25.640275 1062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d304b65eba8c40c6922333506a60540c" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca"
I20250113 02:51:25.640480 1062 raft_consensus.cc:3054] T d304b65eba8c40c6922333506a60540c P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:25.642369 1195 tablet_replica.cc:331] T 23e9321f08ee4ce7b2252ad5c76c0a08 P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:25.642513 896 tablet_service.cc:1514] Processing DeleteTablet for tablet 943d3a125ac5406a83f3c8d80e073b6e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:25 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:25.642611 1195 raft_consensus.cc:2238] T 23e9321f08ee4ce7b2252ad5c76c0a08 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:25.642778 1195 raft_consensus.cc:2267] T 23e9321f08ee4ce7b2252ad5c76c0a08 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:25.646947 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d304b65eba8c40c6922333506a60540c" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f"
I20250113 02:51:25.648182 1192 raft_consensus.cc:357] T ba61c86743e24666b30580c36b9228c3 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:25.653710 1029 tablet_service.cc:1467] Processing CreateTablet for tablet 6d6e462c8bfa47eaa43d3695c69d9c0a (DEFAULT_TABLE table=default.Table_59b60c21ea364f89967c58758807a6a4 [id=f8d606a8b6c340bfa22967fe9711b17d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:25.654193 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d6e462c8bfa47eaa43d3695c69d9c0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:25.656474 1192 raft_consensus.cc:383] T ba61c86743e24666b30580c36b9228c3 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:25.656589 1023 tablet_service.cc:1467] Processing CreateTablet for tablet d29899b4cccb4e37bd25978d385916f9 (DEFAULT_TABLE table=default.Table_44413b1d27e14bff98f1a50e0d1a4dd4 [id=a49d010a40304ec0b1019964e3e0e150]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:25.660081 1442 raft_consensus.cc:513] T 1500e4f9c0664a879503afd3edf14c66 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:25.660827 1442 leader_election.cc:290] T 1500e4f9c0664a879503afd3edf14c66 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Requested vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
I20250113 02:51:25.661151 1024 tablet_service.cc:1467] Processing CreateTablet for tablet b16dd8451f32478daf2ae118dbb9b304 (DEFAULT_TABLE table=default.Table_677784013a07442b9963695510a0a055 [id=06a7fc0682504214990e9a6c830d15ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:25.661689 1031 tablet_service.cc:1467] Processing CreateTablet for tablet 89b6fbf30ca44488ad7969930f1356e2 (DEFAULT_TABLE table=default.Table_aebf098beedb49a4acfdb767bf5ac454 [id=291fd92b15b9498b8e6bd3695f0529f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:25.663302 1627 raft_consensus.cc:491] T 32351d6de1574353a6e8073d333da477 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:25.663457 1627 raft_consensus.cc:513] T 32351d6de1574353a6e8073d333da477 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:25.664152 1627 leader_election.cc:290] T 32351d6de1574353a6e8073d333da477 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:25.666034 1039 tablet_service.cc:1467] Processing CreateTablet for tablet 8412a588d0454ec78025e7ec30c38104 (DEFAULT_TABLE table=default.Table_a5f105b8a4ad4933b499ebaa8ecca921 [id=f1aa4a9a09434178af0610275e931a52]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:25.666548 1035 tablet_service.cc:1467] Processing CreateTablet for tablet 61c79029dfb74f01aaac0f7f828880d0 (DEFAULT_TABLE table=default.Table_2bf3c3c3eb364670a2d0633238a32fe6 [id=ea4fc227dfb145a2b4a660062410e193]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:25.669457 1040 tablet_service.cc:1467] Processing CreateTablet for tablet 089726074e7a40e88bb44549d04874fb (DEFAULT_TABLE table=default.Table_9ecf36f09ace4d59b57e75e0967b2956 [id=a61ea224ef9540cbab35007cc88628af]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:25.669972 1025 tablet_service.cc:1467] Processing CreateTablet for tablet 14cea6298852460ca8cfb74e321cc124 (DEFAULT_TABLE table=default.Table_e5ce75ec7b044da3ae915dc3ce5421da [id=eb72e04ab4fc4f83b97c61c10c527032]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:25.670426 1033 tablet_service.cc:1467] Processing CreateTablet for tablet dfea918c04c1408a8b86a44bb0911a7c (DEFAULT_TABLE table=default.Table_f5f150e8da7a4848bfab67568975d66f [id=5a295e63ad43499f90b2993ac92a16cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:25.671267 1032 tablet_service.cc:1514] Processing DeleteTablet for tablet 943d3a125ac5406a83f3c8d80e073b6e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:25 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:25.670347 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d304b65eba8c40c6922333506a60540c" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:25.674702 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 89b6fbf30ca44488ad7969930f1356e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:25.675737 919 raft_consensus.cc:3054] T d304b65eba8c40c6922333506a60540c P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:25.678169 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 61c79029dfb74f01aaac0f7f828880d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:25.678272 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet d3b695ac20ba4205b2d8732db4740531 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:25 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:25.678555 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8412a588d0454ec78025e7ec30c38104. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:25.678172 1192 raft_consensus.cc:738] T ba61c86743e24666b30580c36b9228c3 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:25.680325 1192 consensus_queue.cc:260] T ba61c86743e24666b30580c36b9228c3 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:25.680907 1192 ts_tablet_manager.cc:1428] T ba61c86743e24666b30580c36b9228c3 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.062s user 0.000s sys 0.002s
I20250113 02:51:25.681205 1192 tablet_bootstrap.cc:492] T 438d1d2300e044f39a15586f335935ba P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:25.683007 1192 tablet_bootstrap.cc:654] T 438d1d2300e044f39a15586f335935ba P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:25.684113 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dfea918c04c1408a8b86a44bb0911a7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:25.688160 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 14cea6298852460ca8cfb74e321cc124. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:25.689603 1041 tablet_service.cc:1467] Processing CreateTablet for tablet 6d5148cf8ff24c92aedf3faee5ae8723 (DEFAULT_TABLE table=default.Table_5e92e31ef2494f34a1fab99e7c6bd9f7 [id=12cf3b5d63db4f8d931d11751bf8e47a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:25.690029 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d5148cf8ff24c92aedf3faee5ae8723. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:25.692059 1040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 089726074e7a40e88bb44549d04874fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:25.694617 1030 tablet_service.cc:1467] Processing CreateTablet for tablet d7542beb84c94b258d2f35a4dc74eec5 (DEFAULT_TABLE table=default.Table_d1a29205746d47a6bcf4719cdaeaf285 [id=9dd99ec7e3b3474ea373bb6fd72b545d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:25.694983 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7542beb84c94b258d2f35a4dc74eec5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:25.696236 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b16dd8451f32478daf2ae118dbb9b304. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:25.702757 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d29899b4cccb4e37bd25978d385916f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:25.704952 1525 raft_consensus.cc:491] T d3b695ac20ba4205b2d8732db4740531 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:25.705147 1525 raft_consensus.cc:513] T d3b695ac20ba4205b2d8732db4740531 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:25.705942 1525 leader_election.cc:290] T d3b695ac20ba4205b2d8732db4740531 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:25.706645 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3b695ac20ba4205b2d8732db4740531" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:25.706959 1059 raft_consensus.cc:2388] T d3b695ac20ba4205b2d8732db4740531 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1186fa869eb14812832fa6562531f196 in current term 1: Already voted for candidate df69f48656b44bb890f76587ee69cc0f in this term.
I20250113 02:51:25.719172 1062 raft_consensus.cc:2463] T d304b65eba8c40c6922333506a60540c P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1186fa869eb14812832fa6562531f196 in term 1.
I20250113 02:51:25.719851 709 leader_election.cc:304] T d304b65eba8c40c6922333506a60540c P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters:
I20250113 02:51:25.720227 1525 raft_consensus.cc:2798] T d304b65eba8c40c6922333506a60540c P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Leader election won for term 1
I20250113 02:51:25.720364 1525 raft_consensus.cc:695] T d304b65eba8c40c6922333506a60540c P 1186fa869eb14812832fa6562531f196 [term 1 LEADER]: Becoming Leader. State: Replica: 1186fa869eb14812832fa6562531f196, State: Running, Role: LEADER
I20250113 02:51:25.720659 1525 consensus_queue.cc:237] T d304b65eba8c40c6922333506a60540c P 1186fa869eb14812832fa6562531f196 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:25.720896 1194 tablet_bootstrap.cc:492] T ebc5f1ea1e1648d49904b17603d3cd2a P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:25.723163 1196 ts_tablet_manager.cc:1918] T 317c6e8013b745c290bf3190dc237ab8 P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250113 02:51:25.724109 1194 ts_tablet_manager.cc:1397] T ebc5f1ea1e1648d49904b17603d3cd2a P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.297s user 0.008s sys 0.006s
I20250113 02:51:25.708016 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1500e4f9c0664a879503afd3edf14c66" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f"
I20250113 02:51:25.725118 1196 log.cc:1198] T 317c6e8013b745c290bf3190dc237ab8 P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/317c6e8013b745c290bf3190dc237ab8
W20250113 02:51:25.737339 998 leader_election.cc:343] T 1500e4f9c0664a879503afd3edf14c66 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:25.738188 1194 raft_consensus.cc:357] T ebc5f1ea1e1648d49904b17603d3cd2a P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:25.738461 1194 raft_consensus.cc:383] T ebc5f1ea1e1648d49904b17603d3cd2a P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:25.739353 1194 raft_consensus.cc:738] T ebc5f1ea1e1648d49904b17603d3cd2a P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:25.739682 1194 consensus_queue.cc:260] T ebc5f1ea1e1648d49904b17603d3cd2a P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:25.741451 1194 ts_tablet_manager.cc:1428] T ebc5f1ea1e1648d49904b17603d3cd2a P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.016s user 0.002s sys 0.000s
I20250113 02:51:25.741889 1194 tablet_bootstrap.cc:492] T 23e9321f08ee4ce7b2252ad5c76c0a08 P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:25.716341 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32351d6de1574353a6e8073d333da477" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:25.745568 998 leader_election.cc:343] T 32351d6de1574353a6e8073d333da477 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:25.750849 1198 ts_tablet_manager.cc:1918] T 23e9321f08ee4ce7b2252ad5c76c0a08 P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:25.761847 1198 log.cc:1198] T 23e9321f08ee4ce7b2252ad5c76c0a08 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/23e9321f08ee4ce7b2252ad5c76c0a08
I20250113 02:51:25.763319 1198 ts_tablet_manager.cc:1939] T 23e9321f08ee4ce7b2252ad5c76c0a08 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:25.766760 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_ea18ad8d07b34ee4b3ef07b8f0431cfd" } modify_external_catalogs: true
I20250113 02:51:25.767035 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_ea18ad8d07b34ee4b3ef07b8f0431cfd" } modify_external_catalogs: true
I20250113 02:51:25.768469 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_ec65488dc1334a33a56ba81d2bda391c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:25.769050 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ec65488dc1334a33a56ba81d2bda391c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:25.770370 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_3318768dfa8c4d4bbc513e660dabfcc5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:25.770794 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 436) took 1508 ms. Trace:
W20250113 02:51:25.770926 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3318768dfa8c4d4bbc513e660dabfcc5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:25.771245 1198 rpcz_store.cc:276] 0113 02:51:24.261142 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:24.261379 (+ 237us) service_pool.cc:225] Handling call
0113 02:51:25.307333 (+1045954us) ts_tablet_manager.cc:1145] Deleting tablet 23e9321f08ee4ce7b2252ad5c76c0a08
0113 02:51:25.307344 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:25.665844 (+358500us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:25.750710 (+ 84866us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:25.769532 (+ 18822us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1045788}
I20250113 02:51:25.771734 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet 23e9321f08ee4ce7b2252ad5c76c0a08 (table default.Table_03b7491a47674f7187f4952feb9b44e2 [id=6e014400813c4402891b32bcbe3555e7]) successfully deleted
I20250113 02:51:25.772408 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_51b81f5fae314cd2afbbf6519d6c8cb3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:25.772871 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_51b81f5fae314cd2afbbf6519d6c8cb3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:25.773348 1198 tablet_replica.cc:331] T 157905316fc545b6a7864ae1c4a2a584 P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:25.773550 1198 raft_consensus.cc:2238] T 157905316fc545b6a7864ae1c4a2a584 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: Raft consensus shutting down.
I20250113 02:51:25.776301 1198 pending_rounds.cc:62] T 157905316fc545b6a7864ae1c4a2a584 P 4216a7f853d642b7bc3e5cff458f3aca: Trying to abort 1 pending ops.
I20250113 02:51:25.779078 1198 pending_rounds.cc:69] T 157905316fc545b6a7864ae1c4a2a584 P 4216a7f853d642b7bc3e5cff458f3aca: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113673459254206464 op_type: NO_OP noop_request { }
I20250113 02:51:25.779417 1198 raft_consensus.cc:2883] T 157905316fc545b6a7864ae1c4a2a584 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250113 02:51:25.779413 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3b695ac20ba4205b2d8732db4740531" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:25.780241 1198 raft_consensus.cc:2267] T 157905316fc545b6a7864ae1c4a2a584 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:25.781133 1198 ts_tablet_manager.cc:1905] T 157905316fc545b6a7864ae1c4a2a584 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:25.787091 1326 sys_catalog.cc:455] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 4 leader_uuid: "e13f5bcdcee34225a46c6d2651e5ede9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e13f5bcdcee34225a46c6d2651e5ede9" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44217 } } peers { permanent_uuid: "6bbc3541604a455584bb32dfc2cc4efd" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 35271 } } peers { permanent_uuid: "45e9d43ecac0401fb2c18fe5eaf089b5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42067 } } }
I20250113 02:51:25.787328 1326 sys_catalog.cc:458] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9 [sys.catalog]: This master's current role is: LEADER
I20250113 02:51:25.789366 711 leader_election.cc:304] T d3b695ac20ba4205b2d8732db4740531 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:25.789798 1525 raft_consensus.cc:3054] T d3b695ac20ba4205b2d8732db4740531 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:25.792142 918 raft_consensus.cc:2463] T d7920c0544c0466bad6bd8c09aae1a7d P df69f48656b44bb890f76587ee69cc0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1186fa869eb14812832fa6562531f196 in term 1.
I20250113 02:51:25.792518 1525 raft_consensus.cc:2743] T d3b695ac20ba4205b2d8732db4740531 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:25.792518 1198 ts_tablet_manager.cc:1918] T 157905316fc545b6a7864ae1c4a2a584 P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250113 02:51:25.792654 1198 log.cc:1198] T 157905316fc545b6a7864ae1c4a2a584 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/157905316fc545b6a7864ae1c4a2a584
I20250113 02:51:25.793085 1198 ts_tablet_manager.cc:1939] T 157905316fc545b6a7864ae1c4a2a584 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:25.796013 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1500e4f9c0664a879503afd3edf14c66" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196"
I20250113 02:51:25.796212 711 leader_election.cc:304] T d7920c0544c0466bad6bd8c09aae1a7d P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1186fa869eb14812832fa6562531f196, df69f48656b44bb890f76587ee69cc0f; no voters: 4216a7f853d642b7bc3e5cff458f3aca
I20250113 02:51:25.796576 1525 raft_consensus.cc:2798] T d7920c0544c0466bad6bd8c09aae1a7d P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Leader election won for term 1
I20250113 02:51:25.796793 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32351d6de1574353a6e8073d333da477" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
I20250113 02:51:25.797003 773 raft_consensus.cc:2463] T 32351d6de1574353a6e8073d333da477 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 0.
I20250113 02:51:25.796952 1659 consensus_queue.cc:1035] T 41bcb2eafff14fa8a040982c723f99c1 P 1186fa869eb14812832fa6562531f196 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 02:51:25.796211 775 raft_consensus.cc:3054] T 1500e4f9c0664a879503afd3edf14c66 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:25.796792 1525 raft_consensus.cc:695] T d7920c0544c0466bad6bd8c09aae1a7d P 1186fa869eb14812832fa6562531f196 [term 1 LEADER]: Becoming Leader. State: Replica: 1186fa869eb14812832fa6562531f196, State: Running, Role: LEADER
I20250113 02:51:25.798283 1525 consensus_queue.cc:237] T d7920c0544c0466bad6bd8c09aae1a7d P 1186fa869eb14812832fa6562531f196 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:25.801587 1195 ts_tablet_manager.cc:1905] T 23e9321f08ee4ce7b2252ad5c76c0a08 P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:25.806120 1438 consensus_queue.cc:1035] T d3b695ac20ba4205b2d8732db4740531 P df69f48656b44bb890f76587ee69cc0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s
I20250113 02:51:25.807116 1196 ts_tablet_manager.cc:1939] T 317c6e8013b745c290bf3190dc237ab8 P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:25.795323 1627 raft_consensus.cc:491] T 8aac142230e244c0b59c9c2b7a7300ea P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:25.795630 1192 tablet_bootstrap.cc:492] T 438d1d2300e044f39a15586f335935ba P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:25.808315 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet 317c6e8013b745c290bf3190dc237ab8 (table default.Table_13fd4df9c207466cb85cd8ed9062722c [id=6bfb67a96b2b47849f7fa12c57a2f4a1]) successfully deleted
I20250113 02:51:25.808439 1192 ts_tablet_manager.cc:1397] T 438d1d2300e044f39a15586f335935ba P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.127s user 0.009s sys 0.019s
I20250113 02:51:25.808383 1627 raft_consensus.cc:513] T 8aac142230e244c0b59c9c2b7a7300ea P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:25.809084 1627 leader_election.cc:290] T 8aac142230e244c0b59c9c2b7a7300ea P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
I20250113 02:51:25.809288 1192 raft_consensus.cc:357] T 438d1d2300e044f39a15586f335935ba P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:25.809485 1192 raft_consensus.cc:383] T 438d1d2300e044f39a15586f335935ba P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:25.799186 995 leader_election.cc:304] T 32351d6de1574353a6e8073d333da477 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters: df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:25.809571 1192 raft_consensus.cc:738] T 438d1d2300e044f39a15586f335935ba P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:25.809629 1196 tablet_replica.cc:331] T d3b695ac20ba4205b2d8732db4740531 P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:25.809819 1196 raft_consensus.cc:2238] T d3b695ac20ba4205b2d8732db4740531 P df69f48656b44bb890f76587ee69cc0f [term 1 LEADER]: Raft consensus shutting down.
I20250113 02:51:25.809906 1192 consensus_queue.cc:260] T 438d1d2300e044f39a15586f335935ba P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:25.810065 1196 pending_rounds.cc:62] T d3b695ac20ba4205b2d8732db4740531 P df69f48656b44bb890f76587ee69cc0f: Trying to abort 1 pending ops.
I20250113 02:51:25.810137 1196 pending_rounds.cc:69] T d3b695ac20ba4205b2d8732db4740531 P df69f48656b44bb890f76587ee69cc0f: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113673460739751936 op_type: NO_OP noop_request { }
I20250113 02:51:25.810482 1192 ts_tablet_manager.cc:1428] T 438d1d2300e044f39a15586f335935ba P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250113 02:51:25.810779 1192 tablet_bootstrap.cc:492] T 35e0aed5125c4e1fa5dce226eade3b78 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:25.811044 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8aac142230e244c0b59c9c2b7a7300ea" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
I20250113 02:51:25.811245 773 raft_consensus.cc:2463] T 8aac142230e244c0b59c9c2b7a7300ea P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 0.
I20250113 02:51:25.812139 775 raft_consensus.cc:2463] T 1500e4f9c0664a879503afd3edf14c66 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 1.
I20250113 02:51:25.812966 455 catalog_manager.cc:5618] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095) does not have the latest schema for tablet 438d1d2300e044f39a15586f335935ba (table default.Table_d7d1f5a34e5f4c05b474b0b61f912cd9 [id=2c7925ec0d8d4cd7b8761268d3962f15]). Expected version 3 got 0
W20250113 02:51:25.813982 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fe3d35d9aaf64188aaa3c1ac38d279d8 consensus state has no leader
I20250113 02:51:25.814365 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_0b855cf933544dce95621708e777674f" } new_table_name: "default.Table_85ede9aae0dd4e3abefcc25b836e9d0b" modify_external_catalogs: true
I20250113 02:51:25.816048 744 tablet_service.cc:1514] Processing DeleteTablet for tablet 32351d6de1574353a6e8073d333da477 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:18 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:25.816444 736 tablet_service.cc:1467] Processing CreateTablet for tablet 42a8e01bb71f49b09c83a8f5c905f38d (DEFAULT_TABLE table=default.Table_acfb94f30f9c4f888bd0a4f28ff1a79e [id=7f0887d6e06d400094e5da086d5b5aff]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:25.816540 995 leader_election.cc:304] T 8aac142230e244c0b59c9c2b7a7300ea P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters:
I20250113 02:51:25.816902 736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 42a8e01bb71f49b09c83a8f5c905f38d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:25.820108 1442 raft_consensus.cc:2798] T 8aac142230e244c0b59c9c2b7a7300ea P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 02:51:25.820226 1442 raft_consensus.cc:491] T 8aac142230e244c0b59c9c2b7a7300ea P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 02:51:25.820350 1442 raft_consensus.cc:3054] T 8aac142230e244c0b59c9c2b7a7300ea P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:25.821803 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 437) took 1560 ms. Trace:
I20250113 02:51:25.821877 1196 raft_consensus.cc:2883] T d3b695ac20ba4205b2d8732db4740531 P df69f48656b44bb890f76587ee69cc0f [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250113 02:51:25.822029 1196 raft_consensus.cc:2267] T d3b695ac20ba4205b2d8732db4740531 P df69f48656b44bb890f76587ee69cc0f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:25.821930 1198 rpcz_store.cc:276] 0113 02:51:24.261259 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:24.261404 (+ 145us) service_pool.cc:225] Handling call
0113 02:51:25.771532 (+1510128us) ts_tablet_manager.cc:1145] Deleting tablet 157905316fc545b6a7864ae1c4a2a584
0113 02:51:25.771542 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:25.786660 (+ 15118us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:25.792392 (+ 5732us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:25.821749 (+ 29357us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1509926}
I20250113 02:51:25.822280 1198 tablet_replica.cc:331] T fc92f19390fd48a4a45ba8fb839020fd P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:25.822459 1198 raft_consensus.cc:2238] T fc92f19390fd48a4a45ba8fb839020fd P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:25.822611 1198 raft_consensus.cc:2267] T fc92f19390fd48a4a45ba8fb839020fd P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:25.823160 1442 raft_consensus.cc:513] T 8aac142230e244c0b59c9c2b7a7300ea P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:25.823822 1442 leader_election.cc:290] T 8aac142230e244c0b59c9c2b7a7300ea P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Requested vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
I20250113 02:51:25.824275 1627 raft_consensus.cc:2798] T 32351d6de1574353a6e8073d333da477 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 02:51:25.824283 919 raft_consensus.cc:2463] T d304b65eba8c40c6922333506a60540c P df69f48656b44bb890f76587ee69cc0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1186fa869eb14812832fa6562531f196 in term 1.
I20250113 02:51:25.824373 1627 raft_consensus.cc:491] T 32351d6de1574353a6e8073d333da477 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 02:51:25.824476 1627 raft_consensus.cc:3054] T 32351d6de1574353a6e8073d333da477 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:25.825834 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8aac142230e244c0b59c9c2b7a7300ea" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:25.826947 1627 raft_consensus.cc:513] T 32351d6de1574353a6e8073d333da477 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:25.827543 1627 leader_election.cc:290] T 32351d6de1574353a6e8073d333da477 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Requested vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:25.831262 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32351d6de1574353a6e8073d333da477" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196"
I20250113 02:51:25.831509 773 raft_consensus.cc:3054] T 32351d6de1574353a6e8073d333da477 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:25.832425 1192 tablet_bootstrap.cc:654] T 35e0aed5125c4e1fa5dce226eade3b78 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
W20250113 02:51:25.832827 998 leader_election.cc:343] T 8aac142230e244c0b59c9c2b7a7300ea P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:25.834663 773 raft_consensus.cc:2463] T 32351d6de1574353a6e8073d333da477 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 1.
I20250113 02:51:25.834921 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32351d6de1574353a6e8073d333da477" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f"
I20250113 02:51:25.835227 995 leader_election.cc:304] T 1500e4f9c0664a879503afd3edf14c66 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters: df69f48656b44bb890f76587ee69cc0f
W20250113 02:51:25.835642 998 leader_election.cc:343] T 32351d6de1574353a6e8073d333da477 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:25.835841 995 leader_election.cc:304] T 32351d6de1574353a6e8073d333da477 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters: df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:25.836169 1442 raft_consensus.cc:2798] T 1500e4f9c0664a879503afd3edf14c66 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Leader election won for term 1
I20250113 02:51:25.836359 1442 raft_consensus.cc:695] T 1500e4f9c0664a879503afd3edf14c66 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: Becoming Leader. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Running, Role: LEADER
I20250113 02:51:25.836589 752 tablet_service.cc:1467] Processing CreateTablet for tablet 0f428c555442401285fb4eab8280b10b (DEFAULT_TABLE table=default.Table_131966d78f11450584893c018d784b84 [id=f512fbac07fb44cd8b231019d979c9fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:25.836642 1442 consensus_queue.cc:237] T 1500e4f9c0664a879503afd3edf14c66 P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:25.836997 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f428c555442401285fb4eab8280b10b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:25.837059 875 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:37356 (request call id 186) took 1113 ms. Trace:
I20250113 02:51:25.837635 744 tablet_service.cc:1467] Processing CreateTablet for tablet f6a76b17f87040dab4a4fcca07e83035 (DEFAULT_TABLE table=default.Table_f683cb701e2d4ba0a3fb83f1bc3722ad [id=71d5778c007a4f829da295569bc6fb24]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:25.838168 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f6a76b17f87040dab4a4fcca07e83035. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:25.839407 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_7c37f90c233c4b79b3e2f866b8198d0c" } new_table_name: "default.Table_d1f76dd334eb4d5890cceb429ed27e9f" modify_external_catalogs: true
I20250113 02:51:25.840029 455 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_a5f105b8a4ad4933b499ebaa8ecca921" } modify_external_catalogs: true
I20250113 02:51:25.840235 1634 raft_consensus.cc:2798] T 32351d6de1574353a6e8073d333da477 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Leader election won for term 1
I20250113 02:51:25.841295 455 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_a5f105b8a4ad4933b499ebaa8ecca921" } modify_external_catalogs: true
I20250113 02:51:25.841439 1029 tablet_service.cc:1467] Processing CreateTablet for tablet 42a8e01bb71f49b09c83a8f5c905f38d (DEFAULT_TABLE table=default.Table_acfb94f30f9c4f888bd0a4f28ff1a79e [id=7f0887d6e06d400094e5da086d5b5aff]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:25.841909 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 42a8e01bb71f49b09c83a8f5c905f38d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:25.845048 448 catalog_manager.cc:5526] T 4fe54131aafb40ee9d8bc5ea299af193 P 4216a7f853d642b7bc3e5cff458f3aca reported cstate change: term changed from 0 to 1, leader changed from <none> to 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3). New cstate: current_term: 1 leader_uuid: "4216a7f853d642b7bc3e5cff458f3aca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } health_report { overall_health: UNKNOWN } } }
I20250113 02:51:25.845865 1027 tablet_service.cc:1467] Processing CreateTablet for tablet 8b9d6b99ff8e4d4d8131c0618a9ddc1a (DEFAULT_TABLE table=default.Table_26a65a837e114f288e826d343b8f78a1 [id=96118e7e15404f62944ee8a1ef5b505a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:25.846351 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b9d6b99ff8e4d4d8131c0618a9ddc1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:25.848325 1023 tablet_service.cc:1467] Processing CreateTablet for tablet 0f428c555442401285fb4eab8280b10b (DEFAULT_TABLE table=default.Table_131966d78f11450584893c018d784b84 [id=f512fbac07fb44cd8b231019d979c9fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:25.848508 892 tablet_service.cc:1467] Processing CreateTablet for tablet 42a8e01bb71f49b09c83a8f5c905f38d (DEFAULT_TABLE table=default.Table_acfb94f30f9c4f888bd0a4f28ff1a79e [id=7f0887d6e06d400094e5da086d5b5aff]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:25.848863 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f428c555442401285fb4eab8280b10b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:25.849102 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet 157905316fc545b6a7864ae1c4a2a584 (table default.Table_5b0e60572cb54fcca8b3b5b94326ae3a [id=9f5add219a7c48da9a8afa966e22d1bd]) successfully deleted
I20250113 02:51:25.850294 1196 ts_tablet_manager.cc:1905] T d3b695ac20ba4205b2d8732db4740531 P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:25.850621 448 catalog_manager.cc:5618] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587) does not have the latest schema for tablet 4fe54131aafb40ee9d8bc5ea299af193 (table default.Table_7c37f90c233c4b79b3e2f866b8198d0c [id=70089f757a1c45f6b983f216fb167478]). Expected version 2 got 0
I20250113 02:51:25.852247 1031 tablet_service.cc:1467] Processing CreateTablet for tablet f6a76b17f87040dab4a4fcca07e83035 (DEFAULT_TABLE table=default.Table_f683cb701e2d4ba0a3fb83f1bc3722ad [id=71d5778c007a4f829da295569bc6fb24]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:25.852669 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f6a76b17f87040dab4a4fcca07e83035. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:25.854005 1024 tablet_service.cc:1467] Processing CreateTablet for tablet 9083a6e1940e467cb73644c38c0bcc78 (DEFAULT_TABLE table=default.Table_cc1ce461db3e461387263aacbd6af4b3 [id=7d33ae86b61841629e84d6a3dcb7ce63]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:25.854410 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9083a6e1940e467cb73644c38c0bcc78. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:25.847476 875 rpcz_store.cc:276] 0113 02:51:24.723686 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:24.725401 (+ 1715us) service_pool.cc:225] Handling call
0113 02:51:25.084105 (+358704us) spinlock_profiling.cc:243] Waited 119 ms on lock 0x616000113854. stack: 00007f0b1ea7cfd6 00007f0b1ea7cd90 00007f0b1da5c48d 000000000096d544 000000000096d50d 000000000096ceda 00007f0b2a2effd0 00007f0b2e02cb3f 00007f0b289878e6 00007f0b28987615 00007f0b257e430e 00007f0b257e31ee 00007f0b257e9425 00007f0b257e9d4f 00007f0b257e9b78 0000000000a71ecc
0113 02:51:25.377035 (+292930us) spinlock_profiling.cc:243] Waited 73.1 ms on lock 0x6160000b2c54. stack: 00007f0b1ea7cfd6 00007f0b1ea7cd90 00007f0b1da5c48d 000000000096d544 000000000096d50d 000000000096ceda 00007f0b2a2effd0 00007f0b2e02cb3f 00007f0b289878e6 00007f0b28987615 00007f0b257e430e 00007f0b257e31ee 00007f0b257e9425 00007f0b257e9d4f 00007f0b257e9b78 0000000000a71ecc
0113 02:51:25.828816 (+451781us) spinlock_profiling.cc:243] Waited 116 ms on lock 0x6160000c2e54. stack: 00007f0b1ea7cfd6 00007f0b1ea7cd90 00007f0b1da5c48d 000000000096d544 000000000096d50d 000000000096ceda 00007f0b2a2effd0 00007f0b2e02cb3f 00007f0b289878e6 00007f0b28987615 00007f0b257e430e 00007f0b257e31ee 00007f0b257e9425 00007f0b257e9d4f 00007f0b257e9b78 0000000000a71ecc
0113 02:51:25.837005 (+ 8189us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":708971520}
I20250113 02:51:25.857020 1035 tablet_service.cc:1467] Processing CreateTablet for tablet f2fb1d39c9ca45ccb2da0be75ccf4acf (DEFAULT_TABLE table=default.Table_d7db322428ae47649bf436e235e20342 [id=f5acea5e1480460ebd9319a762a7b908]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:25.857358 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2fb1d39c9ca45ccb2da0be75ccf4acf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:25.864128 1028 tablet_service.cc:1467] Processing CreateTablet for tablet fd663de3fbb441dc8cc8f1608cd6fd79 (DEFAULT_TABLE table=default.Table_0b2076804d4c410fa414146b8bb0b91b [id=8b8ba624e22c4cebbb9b3591cf832b5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:25.864538 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd663de3fbb441dc8cc8f1608cd6fd79. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:25.865854 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 42a8e01bb71f49b09c83a8f5c905f38d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:25.866680 889 tablet_service.cc:1467] Processing CreateTablet for tablet f2fb1d39c9ca45ccb2da0be75ccf4acf (DEFAULT_TABLE table=default.Table_d7db322428ae47649bf436e235e20342 [id=f5acea5e1480460ebd9319a762a7b908]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:25.867139 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8aac142230e244c0b59c9c2b7a7300ea" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f"
I20250113 02:51:25.868664 1033 tablet_service.cc:1467] Processing CreateTablet for tablet c5d77927b1b24ffebb21c9f10e5cd440 (DEFAULT_TABLE table=default.Table_13fd4df9c207466cb85cd8ed9062722c [id=6bfb67a96b2b47849f7fa12c57a2f4a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:25.869148 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c5d77927b1b24ffebb21c9f10e5cd440. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:25.870566 886 tablet_service.cc:1467] Processing CreateTablet for tablet 8b9d6b99ff8e4d4d8131c0618a9ddc1a (DEFAULT_TABLE table=default.Table_26a65a837e114f288e826d343b8f78a1 [id=96118e7e15404f62944ee8a1ef5b505a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:25.871212 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2fb1d39c9ca45ccb2da0be75ccf4acf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:25.872131 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b9d6b99ff8e4d4d8131c0618a9ddc1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:25.872401 1040 tablet_service.cc:1467] Processing CreateTablet for tablet 88a53da280b0470abc7b522134d05c8f (DEFAULT_TABLE table=default.Table_4a0b14872edf4030a6c3968e559d6cd3 [id=9c66628c6bc3432f87bd4e6692fbe8ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:25.872797 1040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 88a53da280b0470abc7b522134d05c8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:25.876195 895 tablet_service.cc:1467] Processing CreateTablet for tablet f6a76b17f87040dab4a4fcca07e83035 (DEFAULT_TABLE table=default.Table_f683cb701e2d4ba0a3fb83f1bc3722ad [id=71d5778c007a4f829da295569bc6fb24]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:25.876454 1042 tablet_service.cc:1467] Processing CreateTablet for tablet d52b9d0f60714a31918c1645178f3525 (DEFAULT_TABLE table=default.Table_f6135d2bf1204fb5b661ea8a71c526a4 [id=2df4d30aa22b4a0d916a4afc9deae195]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:25.876610 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f6a76b17f87040dab4a4fcca07e83035. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:25.877007 1042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d52b9d0f60714a31918c1645178f3525. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:25.879300 1634 raft_consensus.cc:695] T 32351d6de1574353a6e8073d333da477 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: Becoming Leader. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Running, Role: LEADER
I20250113 02:51:25.879694 1634 consensus_queue.cc:237] T 32351d6de1574353a6e8073d333da477 P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:25.884065 883 tablet_service.cc:1467] Processing CreateTablet for tablet 9083a6e1940e467cb73644c38c0bcc78 (DEFAULT_TABLE table=default.Table_cc1ce461db3e461387263aacbd6af4b3 [id=7d33ae86b61841629e84d6a3dcb7ce63]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:25.884505 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9083a6e1940e467cb73644c38c0bcc78. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:25.891321 899 tablet_service.cc:1467] Processing CreateTablet for tablet c5d77927b1b24ffebb21c9f10e5cd440 (DEFAULT_TABLE table=default.Table_13fd4df9c207466cb85cd8ed9062722c [id=6bfb67a96b2b47849f7fa12c57a2f4a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:25.892536 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c5d77927b1b24ffebb21c9f10e5cd440. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:25.896435 1035 tablet_service.cc:1467] Processing CreateTablet for tablet 7fa8c252d6824373a9b9e7b615d14725 (DEFAULT_TABLE table=default.Table_5be7dd3cb3674e39a393c9276c01ae4c [id=514eb247311f4e0992db7e974a49064c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:25.897078 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7fa8c252d6824373a9b9e7b615d14725. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:25.911598 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_4a0b14872edf4030a6c3968e559d6cd3" } modify_external_catalogs: true
I20250113 02:51:25.911828 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_4a0b14872edf4030a6c3968e559d6cd3" } modify_external_catalogs: true
I20250113 02:51:25.913586 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_6f2972dcacb44ccb9e4f3d828ee536ec" } new_table_name: "default.Table_71699c9b0b1a46cda2db76dee82b90b4" modify_external_catalogs: true
I20250113 02:51:25.913767 754 tablet_service.cc:1467] Processing CreateTablet for tablet 9083a6e1940e467cb73644c38c0bcc78 (DEFAULT_TABLE table=default.Table_cc1ce461db3e461387263aacbd6af4b3 [id=7d33ae86b61841629e84d6a3dcb7ce63]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:25.914196 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9083a6e1940e467cb73644c38c0bcc78. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:25.840831 711 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.2:38467, user_credentials={real_user=slave}} blocked reactor thread for 43697.1us
I20250113 02:51:25.916163 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8aac142230e244c0b59c9c2b7a7300ea" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196"
I20250113 02:51:25.917239 747 tablet_service.cc:1467] Processing CreateTablet for tablet 8b9d6b99ff8e4d4d8131c0618a9ddc1a (DEFAULT_TABLE table=default.Table_26a65a837e114f288e826d343b8f78a1 [id=96118e7e15404f62944ee8a1ef5b505a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:25.917956 753 tablet_service.cc:1467] Processing CreateTablet for tablet f2fb1d39c9ca45ccb2da0be75ccf4acf (DEFAULT_TABLE table=default.Table_d7db322428ae47649bf436e235e20342 [id=f5acea5e1480460ebd9319a762a7b908]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:25.918473 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 8412a588d0454ec78025e7ec30c38104
I20250113 02:51:25.919274 740 tablet_service.cc:1467] Processing CreateTablet for tablet d52b9d0f60714a31918c1645178f3525 (DEFAULT_TABLE table=default.Table_f6135d2bf1204fb5b661ea8a71c526a4 [id=2df4d30aa22b4a0d916a4afc9deae195]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:25.919325 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_96651718e3f643fcacec8ef593a9b30e" } new_table_name: "default.Table_c05a2a3b8f804fb5bde195c57b87e094" modify_external_catalogs: true
I20250113 02:51:25.920181 743 tablet_service.cc:1467] Processing CreateTablet for tablet 7fa8c252d6824373a9b9e7b615d14725 (DEFAULT_TABLE table=default.Table_5be7dd3cb3674e39a393c9276c01ae4c [id=514eb247311f4e0992db7e974a49064c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:25.920526 741 tablet_service.cc:1514] Processing DeleteTablet for tablet 8412a588d0454ec78025e7ec30c38104 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:25 UTC) from {username='slave'} at 127.0.0.1:55398
W20250113 02:51:25.920683 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e13267498917432ba92b6badb76f5d73 consensus state has no leader
I20250113 02:51:25.920850 737 tablet_service.cc:1467] Processing CreateTablet for tablet fd663de3fbb441dc8cc8f1608cd6fd79 (DEFAULT_TABLE table=default.Table_0b2076804d4c410fa414146b8bb0b91b [id=8b8ba624e22c4cebbb9b3591cf832b5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:25.921053 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_9c812746e36d4df38dd75d3a561d7d57"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:25.921598 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9c812746e36d4df38dd75d3a561d7d57 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:25.922854 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet b0b267b0d9c54ba496c89f6cb287dffd
I20250113 02:51:25.923472 449 master_service.cc:946] ReplaceTablet: received request to replace tablet 67baaab1e36e415381579d7385d4ab5a from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:25.924227 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 77cf8b0c824f401aba1880eb7fd1807b
W20250113 02:51:25.925019 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4fe54131aafb40ee9d8bc5ea299af193 consensus state has no leader
I20250113 02:51:25.925081 750 tablet_service.cc:1467] Processing CreateTablet for tablet 88a53da280b0470abc7b522134d05c8f (DEFAULT_TABLE table=default.Table_4a0b14872edf4030a6c3968e559d6cd3 [id=9c66628c6bc3432f87bd4e6692fbe8ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:25.925403 1042 tablet_service.cc:1514] Processing DeleteTablet for tablet b0b267b0d9c54ba496c89f6cb287dffd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:25 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:25.925494 748 tablet_service.cc:1467] Processing CreateTablet for tablet c5d77927b1b24ffebb21c9f10e5cd440 (DEFAULT_TABLE table=default.Table_13fd4df9c207466cb85cd8ed9062722c [id=6bfb67a96b2b47849f7fa12c57a2f4a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:25.925788 1033 tablet_service.cc:1514] Processing DeleteTablet for tablet 77cf8b0c824f401aba1880eb7fd1807b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:25 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:25.926017 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet 8412a588d0454ec78025e7ec30c38104 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:25 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:25.926805 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b9d6b99ff8e4d4d8131c0618a9ddc1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:25.927887 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2fb1d39c9ca45ccb2da0be75ccf4acf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:25.928248 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 303d911dfa2f4abbb053ead33db6d71c
I20250113 02:51:25.928925 1033 tablet_service.cc:1514] Processing DeleteTablet for tablet a6055d77d168495d9b756a1a56c1a920 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:19 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:25.929266 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet 8aac142230e244c0b59c9c2b7a7300ea with delete_type TABLET_DATA_DELETED (replaced by tablet f31dd916336a4d8fb3ab015af7de8ccb) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:25.929803 454 catalog_manager.cc:5618] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467) does not have the latest schema for tablet d7920c0544c0466bad6bd8c09aae1a7d (table default.Table_c9a2c2b1c62849babefee677197da844 [id=05755562cfc24370a3e6fb245a573dd4]). Expected version 1 got 0
I20250113 02:51:25.930039 454 catalog_manager.cc:5618] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467) does not have the latest schema for tablet d304b65eba8c40c6922333506a60540c (table default.Table_e03d954cc20048e4b0d598282689643d [id=e62f8e4c51b04729903b1943275c5f7f]). Expected version 1 got 0
I20250113 02:51:25.931514 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet 303d911dfa2f4abbb053ead33db6d71c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:25 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:25.823137 1198 ts_tablet_manager.cc:1905] T fc92f19390fd48a4a45ba8fb839020fd P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:25.933699 775 raft_consensus.cc:3054] T 8aac142230e244c0b59c9c2b7a7300ea P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:25.935237 1659 raft_consensus.cc:491] T 06f36d09bbb1443bafce906907795404 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:25.935371 1659 raft_consensus.cc:513] T 06f36d09bbb1443bafce906907795404 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:25.937340 746 tablet_service.cc:1514] Processing DeleteTablet for tablet 77cf8b0c824f401aba1880eb7fd1807b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:25 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:25.937449 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7fa8c252d6824373a9b9e7b615d14725. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:25.938315 741 tablet_service.cc:1514] Processing DeleteTablet for tablet b0b267b0d9c54ba496c89f6cb287dffd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:25 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:25.940279 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c5d77927b1b24ffebb21c9f10e5cd440. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:25.941380 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d52b9d0f60714a31918c1645178f3525. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:25.941743 741 tablet_service.cc:1514] Processing DeleteTablet for tablet 303d911dfa2f4abbb053ead33db6d71c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:25 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:25.942006 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 88a53da280b0470abc7b522134d05c8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:25.942097 891 tablet_service.cc:1467] Processing CreateTablet for tablet 0f428c555442401285fb4eab8280b10b (DEFAULT_TABLE table=default.Table_131966d78f11450584893c018d784b84 [id=f512fbac07fb44cd8b231019d979c9fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:25.942884 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06f36d09bbb1443bafce906907795404" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:25.944923 737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd663de3fbb441dc8cc8f1608cd6fd79. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:25.945544 709 leader_election.cc:343] T 06f36d09bbb1443bafce906907795404 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:25.946338 1659 leader_election.cc:290] T 06f36d09bbb1443bafce906907795404 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:25.947280 885 tablet_service.cc:1467] Processing CreateTablet for tablet 7fa8c252d6824373a9b9e7b615d14725 (DEFAULT_TABLE table=default.Table_5be7dd3cb3674e39a393c9276c01ae4c [id=514eb247311f4e0992db7e974a49064c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:25.948450 881 tablet_service.cc:1467] Processing CreateTablet for tablet 88a53da280b0470abc7b522134d05c8f (DEFAULT_TABLE table=default.Table_4a0b14872edf4030a6c3968e559d6cd3 [id=9c66628c6bc3432f87bd4e6692fbe8ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:25.951272 880 tablet_service.cc:1467] Processing CreateTablet for tablet d52b9d0f60714a31918c1645178f3525 (DEFAULT_TABLE table=default.Table_f6135d2bf1204fb5b661ea8a71c526a4 [id=2df4d30aa22b4a0d916a4afc9deae195]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:25.953286 1198 ts_tablet_manager.cc:1918] T fc92f19390fd48a4a45ba8fb839020fd P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:25.953419 1198 log.cc:1198] T fc92f19390fd48a4a45ba8fb839020fd P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/fc92f19390fd48a4a45ba8fb839020fd
I20250113 02:51:25.953475 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06f36d09bbb1443bafce906907795404" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:25.953799 1198 ts_tablet_manager.cc:1939] T fc92f19390fd48a4a45ba8fb839020fd P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:25.953847 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f428c555442401285fb4eab8280b10b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:25.954543 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 438) took 1693 ms. Trace:
I20250113 02:51:25.954648 1198 rpcz_store.cc:276] 0113 02:51:24.261370 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:24.261566 (+ 196us) service_pool.cc:225] Handling call
0113 02:51:25.822191 (+1560625us) ts_tablet_manager.cc:1145] Deleting tablet fc92f19390fd48a4a45ba8fb839020fd
0113 02:51:25.822201 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:25.942502 (+120301us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:25.953173 (+ 10671us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:25.954509 (+ 1336us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1560469}
I20250113 02:51:25.954890 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7fa8c252d6824373a9b9e7b615d14725. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:25.956378 711 leader_election.cc:343] T 06f36d09bbb1443bafce906907795404 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:25.956533 711 leader_election.cc:304] T 06f36d09bbb1443bafce906907795404 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:25.957175 1659 raft_consensus.cc:2743] T 06f36d09bbb1443bafce906907795404 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:25.957769 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet fc92f19390fd48a4a45ba8fb839020fd (table default.Table_c94783b201764702b6de4f3219090efe [id=285c6d11c414470f8c40eefdf6a8a9e8]) successfully deleted
I20250113 02:51:25.958003 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 88a53da280b0470abc7b522134d05c8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:25.961589 1194 tablet_bootstrap.cc:654] T 23e9321f08ee4ce7b2252ad5c76c0a08 P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:25.961951 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d52b9d0f60714a31918c1645178f3525. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:25.962340 888 tablet_service.cc:1514] Processing DeleteTablet for tablet 8412a588d0454ec78025e7ec30c38104 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:25 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:25.964217 1198 tablet_replica.cc:331] T b0361f11d0974ed19e7d4eb1c32a2f47 P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:25.964474 1198 raft_consensus.cc:2238] T b0361f11d0974ed19e7d4eb1c32a2f47 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:25.964684 1198 raft_consensus.cc:2267] T b0361f11d0974ed19e7d4eb1c32a2f47 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:25.965294 1198 ts_tablet_manager.cc:1905] T b0361f11d0974ed19e7d4eb1c32a2f47 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:25.966051 898 tablet_service.cc:1514] Processing DeleteTablet for tablet b0b267b0d9c54ba496c89f6cb287dffd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:25 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:25.966840 898 tablet_service.cc:1514] Processing DeleteTablet for tablet 77cf8b0c824f401aba1880eb7fd1807b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:25 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:25.971491 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_f410165a9c1c45229555f7b89641c3db"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:25.972297 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_bbda194bd2cd4df2b7e7d429700e5f3c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:25.972512 1669 consensus_queue.cc:1035] T 41bcb2eafff14fa8a040982c723f99c1 P 1186fa869eb14812832fa6562531f196 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250113 02:51:25.972893 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bbda194bd2cd4df2b7e7d429700e5f3c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:25.974346 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_8a3a3b6c452a437dae09bdd5ee43c8a0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:25.974776 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8a3a3b6c452a437dae09bdd5ee43c8a0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:25.975901 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_3203c45c9ef34fcdb7c9ebcafb066412"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:25.976344 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3203c45c9ef34fcdb7c9ebcafb066412 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250113 02:51:25.977805 995 consensus_peers.cc:487] T 4fe54131aafb40ee9d8bc5ea299af193 P 4216a7f853d642b7bc3e5cff458f3aca -> Peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Couldn't send request to peer 1186fa869eb14812832fa6562531f196. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250113 02:51:25.978612 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f410165a9c1c45229555f7b89641c3db in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:25.980261 1198 ts_tablet_manager.cc:1918] T b0361f11d0974ed19e7d4eb1c32a2f47 P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:25.983826 775 raft_consensus.cc:2463] T 8aac142230e244c0b59c9c2b7a7300ea P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 1.
I20250113 02:51:25.953378 887 tablet_service.cc:1467] Processing CreateTablet for tablet fd663de3fbb441dc8cc8f1608cd6fd79 (DEFAULT_TABLE table=default.Table_0b2076804d4c410fa414146b8bb0b91b [id=8b8ba624e22c4cebbb9b3591cf832b5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:26.000118 1198 log.cc:1198] T b0361f11d0974ed19e7d4eb1c32a2f47 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/b0361f11d0974ed19e7d4eb1c32a2f47
I20250113 02:51:26.000885 1198 ts_tablet_manager.cc:1939] T b0361f11d0974ed19e7d4eb1c32a2f47 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:26.003301 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 433) took 1742 ms. Trace:
I20250113 02:51:26.005551 995 leader_election.cc:304] T 8aac142230e244c0b59c9c2b7a7300ea P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters:
I20250113 02:51:26.008894 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd663de3fbb441dc8cc8f1608cd6fd79. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:26.008944 1191 tablet_bootstrap.cc:492] T 4efa63b8be794e1db4b340f4c0c4cd5a P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:26.009102 1191 ts_tablet_manager.cc:1397] T 4efa63b8be794e1db4b340f4c0c4cd5a P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.385s user 0.093s sys 0.094s
I20250113 02:51:26.009088 1442 raft_consensus.cc:2798] T 8aac142230e244c0b59c9c2b7a7300ea P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Leader election won for term 1
I20250113 02:51:26.009214 1442 raft_consensus.cc:695] T 8aac142230e244c0b59c9c2b7a7300ea P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: Becoming Leader. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Running, Role: LEADER
I20250113 02:51:26.009495 1442 consensus_queue.cc:237] T 8aac142230e244c0b59c9c2b7a7300ea P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:26.009891 1191 raft_consensus.cc:357] T 4efa63b8be794e1db4b340f4c0c4cd5a P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:26.010135 1191 raft_consensus.cc:383] T 4efa63b8be794e1db4b340f4c0c4cd5a P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:26.010214 1191 raft_consensus.cc:738] T 4efa63b8be794e1db4b340f4c0c4cd5a P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:26.010454 1191 consensus_queue.cc:260] T 4efa63b8be794e1db4b340f4c0c4cd5a P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:26.011024 1191 ts_tablet_manager.cc:1428] T 4efa63b8be794e1db4b340f4c0c4cd5a P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250113 02:51:26.014878 1192 tablet_bootstrap.cc:492] T 35e0aed5125c4e1fa5dce226eade3b78 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:26.015033 1192 ts_tablet_manager.cc:1397] T 35e0aed5125c4e1fa5dce226eade3b78 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.204s user 0.014s sys 0.010s
I20250113 02:51:26.016547 898 tablet_service.cc:1514] Processing DeleteTablet for tablet ebc5f1ea1e1648d49904b17603d3cd2a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:19 UTC) from {username='slave'} at 127.0.0.1:37366
W20250113 02:51:26.017472 998 leader_election.cc:343] T 8aac142230e244c0b59c9c2b7a7300ea P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:26.018041 1191 tablet_bootstrap.cc:492] T 4fe54131aafb40ee9d8bc5ea299af193 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:26.018237 917 raft_consensus.cc:1270] T 4fe54131aafb40ee9d8bc5ea299af193 P df69f48656b44bb890f76587ee69cc0f [term 1 FOLLOWER]: Refusing update from remote peer 4216a7f853d642b7bc3e5cff458f3aca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250113 02:51:26.018788 1192 raft_consensus.cc:357] T 35e0aed5125c4e1fa5dce226eade3b78 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:26.019017 1192 raft_consensus.cc:383] T 35e0aed5125c4e1fa5dce226eade3b78 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:26.019098 1192 raft_consensus.cc:738] T 35e0aed5125c4e1fa5dce226eade3b78 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:26.019964 1191 tablet_bootstrap.cc:654] T 4fe54131aafb40ee9d8bc5ea299af193 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:26.019966 1192 consensus_queue.cc:260] T 35e0aed5125c4e1fa5dce226eade3b78 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:26.020737 1674 consensus_queue.cc:1035] T 4fe54131aafb40ee9d8bc5ea299af193 P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 02:51:26.021312 886 tablet_service.cc:1514] Processing DeleteTablet for tablet 317c6e8013b745c290bf3190dc237ab8 with delete_type TABLET_DATA_DELETED (replaced by tablet c5d77927b1b24ffebb21c9f10e5cd440) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:26.022862 1195 ts_tablet_manager.cc:1918] T 23e9321f08ee4ce7b2252ad5c76c0a08 P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:26.023012 1195 log.cc:1198] T 23e9321f08ee4ce7b2252ad5c76c0a08 P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/23e9321f08ee4ce7b2252ad5c76c0a08
I20250113 02:51:26.023370 1195 ts_tablet_manager.cc:1939] T 23e9321f08ee4ce7b2252ad5c76c0a08 P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:26.023767 1192 ts_tablet_manager.cc:1428] T 35e0aed5125c4e1fa5dce226eade3b78 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20250113 02:51:26.026948 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 88a53da280b0470abc7b522134d05c8f
I20250113 02:51:26.029063 743 tablet_service.cc:1514] Processing DeleteTablet for tablet 88a53da280b0470abc7b522134d05c8f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:25 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:26.030715 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet 88a53da280b0470abc7b522134d05c8f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:25 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:26.032792 1192 tablet_bootstrap.cc:492] T 06f36d09bbb1443bafce906907795404 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:26.033398 886 tablet_service.cc:1514] Processing DeleteTablet for tablet 88a53da280b0470abc7b522134d05c8f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:25 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:26.036981 1192 tablet_bootstrap.cc:654] T 06f36d09bbb1443bafce906907795404 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:26.042161 1196 ts_tablet_manager.cc:1918] T d3b695ac20ba4205b2d8732db4740531 P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250113 02:51:26.042364 1196 log.cc:1198] T d3b695ac20ba4205b2d8732db4740531 P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/d3b695ac20ba4205b2d8732db4740531
I20250113 02:51:26.043018 1196 ts_tablet_manager.cc:1939] T d3b695ac20ba4205b2d8732db4740531 P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:26.049903 1198 rpcz_store.cc:276] 0113 02:51:24.260803 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:24.261750 (+ 947us) service_pool.cc:225] Handling call
0113 02:51:25.964102 (+1702352us) ts_tablet_manager.cc:1145] Deleting tablet b0361f11d0974ed19e7d4eb1c32a2f47
0113 02:51:25.964159 (+ 57us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:25.970986 (+ 6827us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:25.980158 (+ 9172us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:26.003247 (+ 23089us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1702194}
I20250113 02:51:26.050416 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet b0361f11d0974ed19e7d4eb1c32a2f47 (table default.Table_1031f897b6e24d3abd972aca00ddb48b [id=f9202f2a27134b4984d824717b8c1232]) successfully deleted
I20250113 02:51:26.051054 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 439) took 1777 ms. Trace:
I20250113 02:51:26.051245 1198 rpcz_store.cc:276] 0113 02:51:24.273040 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:24.273106 (+ 66us) service_pool.cc:225] Handling call
0113 02:51:26.050654 (+1777548us) ts_tablet_manager.cc:1145] Deleting tablet c7243d27a1c642269a3d0e67632e458c
0113 02:51:26.050670 (+ 16us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:26.050971 (+ 301us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1776900}
W20250113 02:51:26.051833 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet c7243d27a1c642269a3d0e67632e458c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c7243d27a1c642269a3d0e67632e458c already in progress: creating tablet
I20250113 02:51:26.052909 1195 tablet_replica.cc:331] T 8aac142230e244c0b59c9c2b7a7300ea P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:26.052966 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet 23e9321f08ee4ce7b2252ad5c76c0a08 (table default.Table_03b7491a47674f7187f4952feb9b44e2 [id=6e014400813c4402891b32bcbe3555e7]) successfully deleted
I20250113 02:51:26.053170 1195 raft_consensus.cc:2238] T 8aac142230e244c0b59c9c2b7a7300ea P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:26.053375 1195 raft_consensus.cc:2267] T 8aac142230e244c0b59c9c2b7a7300ea P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:26.054010 1195 ts_tablet_manager.cc:1905] T 8aac142230e244c0b59c9c2b7a7300ea P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:26.054008 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet d3b695ac20ba4205b2d8732db4740531 (table default.Table_8b87c8469fbb465e80375aa0446de6c2 [id=f85f0c435ffa4fdca81d86de4aa4fd5f]) successfully deleted
W20250113 02:51:26.054400 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 943d3a125ac5406a83f3c8d80e073b6e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 943d3a125ac5406a83f3c8d80e073b6e already in progress: creating tablet
W20250113 02:51:26.054731 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 8412a588d0454ec78025e7ec30c38104 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8412a588d0454ec78025e7ec30c38104 already in progress: creating tablet
I20250113 02:51:26.045882 881 tablet_service.cc:1514] Processing DeleteTablet for tablet 303d911dfa2f4abbb053ead33db6d71c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:25 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:26.057397 1192 tablet_bootstrap.cc:492] T 06f36d09bbb1443bafce906907795404 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:26.057595 1192 ts_tablet_manager.cc:1397] T 06f36d09bbb1443bafce906907795404 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.025s user 0.003s sys 0.000s
I20250113 02:51:26.058620 1192 raft_consensus.cc:357] T 06f36d09bbb1443bafce906907795404 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:26.058920 1192 raft_consensus.cc:383] T 06f36d09bbb1443bafce906907795404 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:26.059022 1192 raft_consensus.cc:738] T 06f36d09bbb1443bafce906907795404 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:26.059335 1192 consensus_queue.cc:260] T 06f36d09bbb1443bafce906907795404 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:26.059901 1192 ts_tablet_manager.cc:1428] T 06f36d09bbb1443bafce906907795404 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250113 02:51:26.061803 1191 tablet_bootstrap.cc:492] T 4fe54131aafb40ee9d8bc5ea299af193 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:26.062064 1191 ts_tablet_manager.cc:1397] T 4fe54131aafb40ee9d8bc5ea299af193 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.051s user 0.006s sys 0.002s
I20250113 02:51:26.063074 1191 raft_consensus.cc:357] T 4fe54131aafb40ee9d8bc5ea299af193 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:26.063395 1191 raft_consensus.cc:383] T 4fe54131aafb40ee9d8bc5ea299af193 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:26.063633 1191 raft_consensus.cc:738] T 4fe54131aafb40ee9d8bc5ea299af193 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:26.064141 1191 consensus_queue.cc:260] T 4fe54131aafb40ee9d8bc5ea299af193 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:26.066375 1191 ts_tablet_manager.cc:1428] T 4fe54131aafb40ee9d8bc5ea299af193 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20250113 02:51:26.067734 1191 tablet_bootstrap.cc:492] T 35e0aed5125c4e1fa5dce226eade3b78 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:26.060081 1198 tablet_replica.cc:331] T 93a2729c5a604a27a9bbc451ccb4e128 P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:26.069193 1198 raft_consensus.cc:2238] T 93a2729c5a604a27a9bbc451ccb4e128 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:26.069474 1198 raft_consensus.cc:2267] T 93a2729c5a604a27a9bbc451ccb4e128 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:26.070492 1198 ts_tablet_manager.cc:1905] T 93a2729c5a604a27a9bbc451ccb4e128 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:26.072887 1196 tablet_replica.cc:331] T ebc5f1ea1e1648d49904b17603d3cd2a P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
W20250113 02:51:26.073397 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet b0b267b0d9c54ba496c89f6cb287dffd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b0b267b0d9c54ba496c89f6cb287dffd already in progress: creating tablet
I20250113 02:51:26.073431 1192 tablet_bootstrap.cc:492] T d5fbf97536a2472fa091168363bf9549 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
W20250113 02:51:26.073895 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 77cf8b0c824f401aba1880eb7fd1807b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 77cf8b0c824f401aba1880eb7fd1807b already in progress: creating tablet
I20250113 02:51:26.077368 1442 raft_consensus.cc:491] T 35e0aed5125c4e1fa5dce226eade3b78 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:26.077603 1442 raft_consensus.cc:513] T 35e0aed5125c4e1fa5dce226eade3b78 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:26.078239 1442 leader_election.cc:290] T 35e0aed5125c4e1fa5dce226eade3b78 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
I20250113 02:51:26.078850 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35e0aed5125c4e1fa5dce226eade3b78" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:26.079178 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35e0aed5125c4e1fa5dce226eade3b78" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
W20250113 02:51:26.079425 998 leader_election.cc:343] T 35e0aed5125c4e1fa5dce226eade3b78 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
W20250113 02:51:26.079895 995 leader_election.cc:343] T 35e0aed5125c4e1fa5dce226eade3b78 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:26.083621 995 leader_election.cc:304] T 35e0aed5125c4e1fa5dce226eade3b78 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4216a7f853d642b7bc3e5cff458f3aca; no voters: 1186fa869eb14812832fa6562531f196, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:26.083920 1442 raft_consensus.cc:2743] T 35e0aed5125c4e1fa5dce226eade3b78 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:26.084193 1196 raft_consensus.cc:2238] T ebc5f1ea1e1648d49904b17603d3cd2a P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:26.084416 1196 raft_consensus.cc:2267] T ebc5f1ea1e1648d49904b17603d3cd2a P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:26.084340 1191 tablet_bootstrap.cc:654] T 35e0aed5125c4e1fa5dce226eade3b78 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:26.085394 1196 ts_tablet_manager.cc:1905] T ebc5f1ea1e1648d49904b17603d3cd2a P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:26.094115 1192 tablet_bootstrap.cc:654] T d5fbf97536a2472fa091168363bf9549 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:26.102152 1191 tablet_bootstrap.cc:492] T 35e0aed5125c4e1fa5dce226eade3b78 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:26.102373 1191 ts_tablet_manager.cc:1397] T 35e0aed5125c4e1fa5dce226eade3b78 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.035s user 0.000s sys 0.005s
I20250113 02:51:26.103181 1191 raft_consensus.cc:357] T 35e0aed5125c4e1fa5dce226eade3b78 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:26.103495 1191 raft_consensus.cc:383] T 35e0aed5125c4e1fa5dce226eade3b78 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:26.103590 1191 raft_consensus.cc:738] T 35e0aed5125c4e1fa5dce226eade3b78 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:26.103893 1191 consensus_queue.cc:260] T 35e0aed5125c4e1fa5dce226eade3b78 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:26.107569 1191 ts_tablet_manager.cc:1428] T 35e0aed5125c4e1fa5dce226eade3b78 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250113 02:51:26.108284 1194 tablet_bootstrap.cc:492] T 23e9321f08ee4ce7b2252ad5c76c0a08 P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:26.108507 1194 ts_tablet_manager.cc:1397] T 23e9321f08ee4ce7b2252ad5c76c0a08 P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.367s user 0.003s sys 0.002s
I20250113 02:51:26.109309 1194 raft_consensus.cc:357] T 23e9321f08ee4ce7b2252ad5c76c0a08 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:26.109593 1194 raft_consensus.cc:383] T 23e9321f08ee4ce7b2252ad5c76c0a08 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:26.109714 1194 raft_consensus.cc:738] T 23e9321f08ee4ce7b2252ad5c76c0a08 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:26.110033 1194 consensus_queue.cc:260] T 23e9321f08ee4ce7b2252ad5c76c0a08 P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:26.110630 1191 tablet_bootstrap.cc:492] T 7f08b72301dc431396cf8cc7790d56d4 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:26.112404 1191 tablet_bootstrap.cc:654] T 7f08b72301dc431396cf8cc7790d56d4 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:26.113374 1194 ts_tablet_manager.cc:1428] T 23e9321f08ee4ce7b2252ad5c76c0a08 P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250113 02:51:26.114771 1194 tablet_bootstrap.cc:492] T b834f0acf5314bcb9d81c73de1b31842 P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:26.115895 1192 tablet_bootstrap.cc:492] T d5fbf97536a2472fa091168363bf9549 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:26.116089 1192 ts_tablet_manager.cc:1397] T d5fbf97536a2472fa091168363bf9549 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.044s user 0.006s sys 0.000s
I20250113 02:51:26.117112 1192 raft_consensus.cc:357] T d5fbf97536a2472fa091168363bf9549 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:26.117377 1192 raft_consensus.cc:383] T d5fbf97536a2472fa091168363bf9549 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:26.117486 1192 raft_consensus.cc:738] T d5fbf97536a2472fa091168363bf9549 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:26.117887 1192 consensus_queue.cc:260] T d5fbf97536a2472fa091168363bf9549 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:26.120610 1192 ts_tablet_manager.cc:1428] T d5fbf97536a2472fa091168363bf9549 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20250113 02:51:26.123652 1195 ts_tablet_manager.cc:1918] T 8aac142230e244c0b59c9c2b7a7300ea P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:26.125380 1194 tablet_bootstrap.cc:654] T b834f0acf5314bcb9d81c73de1b31842 P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:26.125169 1195 log.cc:1198] T 8aac142230e244c0b59c9c2b7a7300ea P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/8aac142230e244c0b59c9c2b7a7300ea
I20250113 02:51:26.126228 1192 tablet_bootstrap.cc:492] T 64cda8796dff46ce95c37439395c77ec P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:26.126804 1442 raft_consensus.cc:491] T ba61c86743e24666b30580c36b9228c3 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:26.127022 1442 raft_consensus.cc:513] T ba61c86743e24666b30580c36b9228c3 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:26.127775 1442 leader_election.cc:290] T ba61c86743e24666b30580c36b9228c3 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
I20250113 02:51:26.128618 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba61c86743e24666b30580c36b9228c3" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:26.128660 995 leader_election.cc:343] T ba61c86743e24666b30580c36b9228c3 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Not found: Tablet not found: ba61c86743e24666b30580c36b9228c3
W20250113 02:51:26.129235 998 leader_election.cc:343] T ba61c86743e24666b30580c36b9228c3 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:26.129359 998 leader_election.cc:304] T ba61c86743e24666b30580c36b9228c3 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4216a7f853d642b7bc3e5cff458f3aca; no voters: 1186fa869eb14812832fa6562531f196, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:26.129812 1442 raft_consensus.cc:2743] T ba61c86743e24666b30580c36b9228c3 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:26.131213 1198 ts_tablet_manager.cc:1918] T 93a2729c5a604a27a9bbc451ccb4e128 P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:26.132093 1198 log.cc:1198] T 93a2729c5a604a27a9bbc451ccb4e128 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/93a2729c5a604a27a9bbc451ccb4e128
I20250113 02:51:26.132458 1195 ts_tablet_manager.cc:1939] T 8aac142230e244c0b59c9c2b7a7300ea P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:26.135118 1198 ts_tablet_manager.cc:1939] T 93a2729c5a604a27a9bbc451ccb4e128 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:26.138697 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet 8aac142230e244c0b59c9c2b7a7300ea (table default.Table_37b1d9d091924f8d9a8a2cbf1d06af1a [id=4d85a53a49d642b798af44eef60b3f78]) successfully deleted
I20250113 02:51:26.139582 1195 tablet_replica.cc:331] T 1500e4f9c0664a879503afd3edf14c66 P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:26.140482 1195 raft_consensus.cc:2238] T 1500e4f9c0664a879503afd3edf14c66 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:26.141508 1195 raft_consensus.cc:2267] T 1500e4f9c0664a879503afd3edf14c66 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:26.144476 1191 tablet_bootstrap.cc:492] T 7f08b72301dc431396cf8cc7790d56d4 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:26.144665 1191 ts_tablet_manager.cc:1397] T 7f08b72301dc431396cf8cc7790d56d4 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.034s user 0.007s sys 0.004s
I20250113 02:51:26.147553 1195 ts_tablet_manager.cc:1905] T 1500e4f9c0664a879503afd3edf14c66 P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:26.148470 1191 raft_consensus.cc:357] T 7f08b72301dc431396cf8cc7790d56d4 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:26.149784 1191 raft_consensus.cc:383] T 7f08b72301dc431396cf8cc7790d56d4 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:26.157042 1191 raft_consensus.cc:738] T 7f08b72301dc431396cf8cc7790d56d4 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:26.157680 1191 consensus_queue.cc:260] T 7f08b72301dc431396cf8cc7790d56d4 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
W20250113 02:51:26.158151 442 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet f9269967dea94fc3a337546ad0842e11 (table default.Table_23c1655ab4cf4a30b67185ccf4fbb76d [id=a8e428598ca94e15b04f1df8112a4abc], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet f9269967dea94fc3a337546ad0842e11 consensus state has no leader
I20250113 02:51:26.158810 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 429) took 1886 ms. Trace:
I20250113 02:51:26.160552 1690 tablet.cc:1721] T 4fe54131aafb40ee9d8bc5ea299af193 P 4216a7f853d642b7bc3e5cff458f3aca: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250113 02:51:26.161146 1198 rpcz_store.cc:276] 0113 02:51:24.260201 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:24.277498 (+ 17297us) service_pool.cc:225] Handling call
0113 02:51:26.052124 (+1774626us) ts_tablet_manager.cc:1145] Deleting tablet 93a2729c5a604a27a9bbc451ccb4e128
0113 02:51:26.052163 (+ 39us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:26.098802 (+ 46639us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:26.130191 (+ 31389us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:26.146392 (+ 16201us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":19713,"tablet-delete.queue_time_us":1774358}
I20250113 02:51:26.161633 1191 ts_tablet_manager.cc:1428] T 7f08b72301dc431396cf8cc7790d56d4 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.016s user 0.001s sys 0.002s
I20250113 02:51:26.162124 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 462) took 1595 ms. Trace:
I20250113 02:51:26.162106 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet 93a2729c5a604a27a9bbc451ccb4e128 (table default.Table_4f1f306815b145218e24c93821376721 [id=5d1feb17ef344c9287e571496b2b405f]) successfully deleted
I20250113 02:51:26.162405 1198 rpcz_store.cc:276] 0113 02:51:24.566982 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:24.567247 (+ 265us) service_pool.cc:225] Handling call
0113 02:51:26.161974 (+1594727us) ts_tablet_manager.cc:1145] Deleting tablet b33025a462e74f2aa26cc3779f36615b
0113 02:51:26.161985 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:26.162090 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1561646}
I20250113 02:51:26.162329 1192 tablet_bootstrap.cc:654] T 64cda8796dff46ce95c37439395c77ec P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
W20250113 02:51:26.163038 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet b33025a462e74f2aa26cc3779f36615b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b33025a462e74f2aa26cc3779f36615b already in progress: creating tablet
I20250113 02:51:26.163349 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 463) took 1592 ms. Trace:
I20250113 02:51:26.163753 1191 tablet_bootstrap.cc:492] T d5fbf97536a2472fa091168363bf9549 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:26.163888 1198 rpcz_store.cc:276] 0113 02:51:24.571285 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:24.571370 (+ 85us) service_pool.cc:225] Handling call
0113 02:51:26.163227 (+1591857us) ts_tablet_manager.cc:1145] Deleting tablet 2470b010b571444eba3f132323038f33
0113 02:51:26.163236 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:26.163318 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1562013}
W20250113 02:51:26.164490 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 2470b010b571444eba3f132323038f33 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2470b010b571444eba3f132323038f33 already in progress: creating tablet
I20250113 02:51:26.164566 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 464) took 1593 ms. Trace:
I20250113 02:51:26.165817 1198 rpcz_store.cc:276] 0113 02:51:24.571409 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:24.571451 (+ 42us) service_pool.cc:225] Handling call
0113 02:51:26.164452 (+1593001us) ts_tablet_manager.cc:1145] Deleting tablet 65d9bbd8c15f40a4a50c1695fb5e648e
0113 02:51:26.164460 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:26.164538 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1563091}
W20250113 02:51:26.166481 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 65d9bbd8c15f40a4a50c1695fb5e648e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 65d9bbd8c15f40a4a50c1695fb5e648e already in progress: creating tablet
I20250113 02:51:26.166824 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 466) took 1554 ms. Trace:
I20250113 02:51:26.167421 1191 tablet_bootstrap.cc:654] T d5fbf97536a2472fa091168363bf9549 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:26.169466 1198 rpcz_store.cc:276] 0113 02:51:24.612317 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:24.612372 (+ 55us) service_pool.cc:225] Handling call
0113 02:51:26.166674 (+1554302us) ts_tablet_manager.cc:1145] Deleting tablet 9e736211eb3c4536b9fd1d90d2d83838
0113 02:51:26.166699 (+ 25us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:26.166794 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1550144}
W20250113 02:51:26.169889 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 9e736211eb3c4536b9fd1d90d2d83838 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9e736211eb3c4536b9fd1d90d2d83838 already in progress: creating tablet
I20250113 02:51:26.170014 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 467) took 1528 ms. Trace:
I20250113 02:51:26.170140 1198 rpcz_store.cc:276] 0113 02:51:24.641700 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:24.643818 (+ 2118us) service_pool.cc:225] Handling call
0113 02:51:26.169862 (+1526044us) ts_tablet_manager.cc:1145] Deleting tablet 438604e8859f422f9caccb61c5b5b125
0113 02:51:26.169871 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:26.169952 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1525773}
I20250113 02:51:26.170540 1198 tablet_replica.cc:331] T 34ab7ee573284550aaac9f814e8bd7ab P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
W20250113 02:51:26.170522 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 438604e8859f422f9caccb61c5b5b125 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 438604e8859f422f9caccb61c5b5b125 already in progress: creating tablet
I20250113 02:51:26.170781 1198 raft_consensus.cc:2238] T 34ab7ee573284550aaac9f814e8bd7ab P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:26.170954 1198 raft_consensus.cc:2267] T 34ab7ee573284550aaac9f814e8bd7ab P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:26.182250 1196 ts_tablet_manager.cc:1918] T ebc5f1ea1e1648d49904b17603d3cd2a P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:26.182279 1195 ts_tablet_manager.cc:1918] T 1500e4f9c0664a879503afd3edf14c66 P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:26.182410 1196 log.cc:1198] T ebc5f1ea1e1648d49904b17603d3cd2a P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/ebc5f1ea1e1648d49904b17603d3cd2a
I20250113 02:51:26.182448 1195 log.cc:1198] T 1500e4f9c0664a879503afd3edf14c66 P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/1500e4f9c0664a879503afd3edf14c66
I20250113 02:51:26.182803 1196 ts_tablet_manager.cc:1939] T ebc5f1ea1e1648d49904b17603d3cd2a P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:26.182864 1195 ts_tablet_manager.cc:1939] T 1500e4f9c0664a879503afd3edf14c66 P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:26.183400 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet ebc5f1ea1e1648d49904b17603d3cd2a (table default.Table_3d6c2586a98845c7989a5b715c4d678e [id=f4e09f1ea2184aa78abeb307bfc00d37]) successfully deleted
W20250113 02:51:26.184602 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fd0b8592997d4a61b8ca854ecbb67c55 consensus state has no leader
I20250113 02:51:26.184649 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet 1500e4f9c0664a879503afd3edf14c66 (table default.Table_d05b7517b14c4168bea99385f1a7d8b6 [id=05cf81a8622149d29f0f4c567a44371c]) successfully deleted
W20250113 02:51:26.185459 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 943d3a125ac5406a83f3c8d80e073b6e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 943d3a125ac5406a83f3c8d80e073b6e already in progress: creating tablet
I20250113 02:51:26.185818 1195 tablet_replica.cc:331] T d3b695ac20ba4205b2d8732db4740531 P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:26.185747 1689 tablet.cc:1721] T 4fe54131aafb40ee9d8bc5ea299af193 P df69f48656b44bb890f76587ee69cc0f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250113 02:51:26.186071 1195 raft_consensus.cc:2238] T d3b695ac20ba4205b2d8732db4740531 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:26.186275 1195 raft_consensus.cc:2267] T d3b695ac20ba4205b2d8732db4740531 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:26.186540 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_9181be1f89094ceea4944414dd995df0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:26.186929 1195 ts_tablet_manager.cc:1905] T d3b695ac20ba4205b2d8732db4740531 P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250113 02:51:26.187134 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9181be1f89094ceea4944414dd995df0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250113 02:51:26.188217 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 29c17fd90d774c029a35e77888358067 consensus state has no leader
I20250113 02:51:26.188634 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_15d236fec5f24590b15d5d21f67e3941" } new_table_name: "default.Table_3463038bce4545df8ec6c3834403a95d" modify_external_catalogs: true
I20250113 02:51:26.190578 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 67baaab1e36e415381579d7385d4ab5a
I20250113 02:51:26.191112 449 catalog_manager.cc:6596] ReplaceTablet: tablet 67baaab1e36e415381579d7385d4ab5a deleted and replaced by tablet c368b5b0c3a940c4b4efb64c946375c3
I20250113 02:51:26.191185 1690 tablet.cc:1716] T 4fe54131aafb40ee9d8bc5ea299af193 P 4216a7f853d642b7bc3e5cff458f3aca: Skipping requested alter to schema version 3, tablet already version 3
I20250113 02:51:26.191344 449 master_service.cc:946] ReplaceTablet: received request to replace tablet 6d4d8b5cc8ad4c9e935d8afaf5766e66 from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:26.193003 743 tablet_service.cc:1514] Processing DeleteTablet for tablet 67baaab1e36e415381579d7385d4ab5a with delete_type TABLET_DATA_DELETED (replaced by tablet c368b5b0c3a940c4b4efb64c946375c3) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:26.193374 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_6d300fedf96048a5a3bdbc9f36ac0b6c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:26.193946 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6d300fedf96048a5a3bdbc9f36ac0b6c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:26.194540 883 tablet_service.cc:1514] Processing DeleteTablet for tablet 67baaab1e36e415381579d7385d4ab5a with delete_type TABLET_DATA_DELETED (replaced by tablet c368b5b0c3a940c4b4efb64c946375c3) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:26.195474 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet 67baaab1e36e415381579d7385d4ab5a with delete_type TABLET_DATA_DELETED (replaced by tablet c368b5b0c3a940c4b4efb64c946375c3) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:26.196964 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_6431cb4cd5374cbcaceda152cbc19e5b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:26.197464 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6431cb4cd5374cbcaceda152cbc19e5b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:26.197584 1195 ts_tablet_manager.cc:1918] T d3b695ac20ba4205b2d8732db4740531 P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:26.197706 1195 log.cc:1198] T d3b695ac20ba4205b2d8732db4740531 P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/d3b695ac20ba4205b2d8732db4740531
I20250113 02:51:26.198235 1195 ts_tablet_manager.cc:1939] T d3b695ac20ba4205b2d8732db4740531 P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:26.199260 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet d3b695ac20ba4205b2d8732db4740531 (table default.Table_8b87c8469fbb465e80375aa0446de6c2 [id=f85f0c435ffa4fdca81d86de4aa4fd5f]) successfully deleted
I20250113 02:51:26.199293 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_9e1b92761ab84f80905c2b4f57738caf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:26.199599 1195 tablet_replica.cc:331] T 32351d6de1574353a6e8073d333da477 P 1186fa869eb14812832fa6562531f196: stopping tablet replica
W20250113 02:51:26.199769 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9e1b92761ab84f80905c2b4f57738caf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:26.200249 1195 raft_consensus.cc:2238] T 32351d6de1574353a6e8073d333da477 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:26.200414 1195 raft_consensus.cc:2267] T 32351d6de1574353a6e8073d333da477 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:26.200961 1195 ts_tablet_manager.cc:1905] T 32351d6de1574353a6e8073d333da477 P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:26.202843 1659 raft_consensus.cc:491] T 4efa63b8be794e1db4b340f4c0c4cd5a P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:26.202972 1659 raft_consensus.cc:513] T 4efa63b8be794e1db4b340f4c0c4cd5a P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:26.203655 1659 leader_election.cc:290] T 4efa63b8be794e1db4b340f4c0c4cd5a P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:26.204221 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_eaece5fe552b47fb9c0f3d73031f40f4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:26.204825 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_eaece5fe552b47fb9c0f3d73031f40f4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250113 02:51:26.205693 709 leader_election.cc:343] T 4efa63b8be794e1db4b340f4c0c4cd5a P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Not found: Tablet not found: 4efa63b8be794e1db4b340f4c0c4cd5a
W20250113 02:51:26.206566 711 leader_election.cc:343] T 4efa63b8be794e1db4b340f4c0c4cd5a P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Not found: Tablet not found: 4efa63b8be794e1db4b340f4c0c4cd5a
I20250113 02:51:26.206702 711 leader_election.cc:304] T 4efa63b8be794e1db4b340f4c0c4cd5a P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:26.207072 1659 raft_consensus.cc:2743] T 4efa63b8be794e1db4b340f4c0c4cd5a P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250113 02:51:26.210966 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f105f8b32a654e4ca154e8fc0cb96091 consensus state has no leader
I20250113 02:51:26.211333 455 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_51b81f5fae314cd2afbbf6519d6c8cb3" } modify_external_catalogs: true
I20250113 02:51:26.211500 455 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_51b81f5fae314cd2afbbf6519d6c8cb3" } modify_external_catalogs: true
I20250113 02:51:26.213053 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_b09c6cee53c64995a167959c15f5e288" } new_table_name: "default.Table_3fa4f9f57be84226a01935b139545f6e" modify_external_catalogs: true
I20250113 02:51:26.214717 446 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_d7db322428ae47649bf436e235e20342" } modify_external_catalogs: true
I20250113 02:51:26.215691 446 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_d7db322428ae47649bf436e235e20342" } modify_external_catalogs: true
I20250113 02:51:26.216457 448 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_3ed64b1d864b4b79bdaef850c30c486b" } modify_external_catalogs: true
I20250113 02:51:26.216742 448 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_3ed64b1d864b4b79bdaef850c30c486b" } modify_external_catalogs: true
I20250113 02:51:26.216908 1195 ts_tablet_manager.cc:1918] T 32351d6de1574353a6e8073d333da477 P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:26.217052 1195 log.cc:1198] T 32351d6de1574353a6e8073d333da477 P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/32351d6de1574353a6e8073d333da477
I20250113 02:51:26.217514 1195 ts_tablet_manager.cc:1939] T 32351d6de1574353a6e8073d333da477 P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:26.217510 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_90fe60d176ce484680307442a262e8d5" } new_table_name: "default.Table_3f1e7146fed34d53896f138e43a4d26e" modify_external_catalogs: true
I20250113 02:51:26.219108 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet 32351d6de1574353a6e8073d333da477 (table default.Table_78c0bcf1e2d64abc81253301920cf566 [id=e5831758ca184c1998f551fe0285d010]) successfully deleted
W20250113 02:51:26.220518 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 8412a588d0454ec78025e7ec30c38104 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8412a588d0454ec78025e7ec30c38104 already in progress: creating tablet
W20250113 02:51:26.223001 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 77cf8b0c824f401aba1880eb7fd1807b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 77cf8b0c824f401aba1880eb7fd1807b already in progress: creating tablet
W20250113 02:51:26.223652 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet b0b267b0d9c54ba496c89f6cb287dffd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b0b267b0d9c54ba496c89f6cb287dffd already in progress: creating tablet
W20250113 02:51:26.224352 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 303d911dfa2f4abbb053ead33db6d71c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 303d911dfa2f4abbb053ead33db6d71c already in progress: creating tablet
W20250113 02:51:26.225087 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 88a53da280b0470abc7b522134d05c8f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 88a53da280b0470abc7b522134d05c8f already in progress: creating tablet
W20250113 02:51:26.225760 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 67baaab1e36e415381579d7385d4ab5a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 67baaab1e36e415381579d7385d4ab5a already in progress: creating tablet
W20250113 02:51:26.229152 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1d2fdc75d00f4e749d322ac6cd5fc410 consensus state has no leader
I20250113 02:51:26.228345 1191 tablet_bootstrap.cc:492] T d5fbf97536a2472fa091168363bf9549 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:26.229599 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_ba98ed25e93a45a288784d4e5bdee029" } modify_external_catalogs: true
I20250113 02:51:26.230574 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_ba98ed25e93a45a288784d4e5bdee029" } modify_external_catalogs: true
I20250113 02:51:26.232388 1191 ts_tablet_manager.cc:1397] T d5fbf97536a2472fa091168363bf9549 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.068s user 0.002s sys 0.009s
I20250113 02:51:26.233052 454 catalog_manager.cc:5618] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587) does not have the latest schema for tablet 438d1d2300e044f39a15586f335935ba (table default.Table_d7d1f5a34e5f4c05b474b0b61f912cd9 [id=2c7925ec0d8d4cd7b8761268d3962f15]). Expected version 3 got 0
I20250113 02:51:26.234155 450 catalog_manager.cc:5526] T d7920c0544c0466bad6bd8c09aae1a7d P 1186fa869eb14812832fa6562531f196 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1186fa869eb14812832fa6562531f196 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "1186fa869eb14812832fa6562531f196" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } health_report { overall_health: HEALTHY } } }
I20250113 02:51:26.234675 450 catalog_manager.cc:5618] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095) does not have the latest schema for tablet d7920c0544c0466bad6bd8c09aae1a7d (table default.Table_c9a2c2b1c62849babefee677197da844 [id=05755562cfc24370a3e6fb245a573dd4]). Expected version 1 got 0
I20250113 02:51:26.234980 450 catalog_manager.cc:5526] T d304b65eba8c40c6922333506a60540c P 1186fa869eb14812832fa6562531f196 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1186fa869eb14812832fa6562531f196 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "1186fa869eb14812832fa6562531f196" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } health_report { overall_health: HEALTHY } } }
I20250113 02:51:26.235405 450 catalog_manager.cc:5618] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095) does not have the latest schema for tablet d304b65eba8c40c6922333506a60540c (table default.Table_e03d954cc20048e4b0d598282689643d [id=e62f8e4c51b04729903b1943275c5f7f]). Expected version 1 got 0
I20250113 02:51:26.235603 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet ba61c86743e24666b30580c36b9228c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:19 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:26.237143 880 tablet_service.cc:1467] Processing CreateTablet for tablet 656a0dda5c0b46309ed472d34a42ff94 (DEFAULT_TABLE table=default.Table_d4b3b319759f44d4bbe072b1bc97cc94 [id=2d80f898223a4982aa73dfca00277e28]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:26.237632 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 656a0dda5c0b46309ed472d34a42ff94. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:26.237951 1023 tablet_service.cc:1467] Processing CreateTablet for tablet 656a0dda5c0b46309ed472d34a42ff94 (DEFAULT_TABLE table=default.Table_d4b3b319759f44d4bbe072b1bc97cc94 [id=2d80f898223a4982aa73dfca00277e28]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:26.238387 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 656a0dda5c0b46309ed472d34a42ff94. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:26.240289 883 tablet_service.cc:1467] Processing CreateTablet for tablet 795a96fc8cb243e2841739b3a492d0f8 (DEFAULT_TABLE table=default.Table_d1ff69f5cd31453aaf9d8f6382eedcda [id=b2ee3441903a403488dad5cdaf2f9986]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:26.240697 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 795a96fc8cb243e2841739b3a492d0f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:26.242897 1035 tablet_service.cc:1467] Processing CreateTablet for tablet 795a96fc8cb243e2841739b3a492d0f8 (DEFAULT_TABLE table=default.Table_d1ff69f5cd31453aaf9d8f6382eedcda [id=b2ee3441903a403488dad5cdaf2f9986]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:26.243228 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 795a96fc8cb243e2841739b3a492d0f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:26.245157 892 tablet_service.cc:1467] Processing CreateTablet for tablet 695f3f27513d41ad8a4d6eedef6db2b3 (DEFAULT_TABLE table=default.Table_d1dafd1ed4a64d17b499f312bc298bd7 [id=48bed7c08d9849a19479b744b4c65a68]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:26.245543 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 695f3f27513d41ad8a4d6eedef6db2b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:26.246451 455 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet 6c3aea00fe1d4bb68d111787c4258a22
W20250113 02:51:26.248183 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet aa5c172a69bf4ca79999f4b58df08de8 consensus state has no leader
I20250113 02:51:26.250284 1024 tablet_service.cc:1467] Processing CreateTablet for tablet 130f8540b9a6477e84bb9595e7c39994 (DEFAULT_TABLE table=default.Table_eec56bd1a5344ab985b794cb42a0eeba [id=5c83feb8a24b422fbde3c5fdbc7c63e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:26.250669 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 130f8540b9a6477e84bb9595e7c39994. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:26.250901 895 tablet_service.cc:1467] Processing CreateTablet for tablet 130f8540b9a6477e84bb9595e7c39994 (DEFAULT_TABLE table=default.Table_eec56bd1a5344ab985b794cb42a0eeba [id=5c83feb8a24b422fbde3c5fdbc7c63e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:26.251284 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 130f8540b9a6477e84bb9595e7c39994. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:26.253494 453 catalog_manager.cc:5618] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587) does not have the latest schema for tablet 06f36d09bbb1443bafce906907795404 (table default.Table_2a627a757ab0406686564083ecc36c07 [id=423021f304964246bd29e4daaf470154]). Expected version 2 got 0
I20250113 02:51:26.256469 1033 tablet_service.cc:1467] Processing CreateTablet for tablet 695f3f27513d41ad8a4d6eedef6db2b3 (DEFAULT_TABLE table=default.Table_d1dafd1ed4a64d17b499f312bc298bd7 [id=48bed7c08d9849a19479b744b4c65a68]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:26.256678 899 tablet_service.cc:1467] Processing CreateTablet for tablet c6820d6331e44656a27e23afe973e2e2 (DEFAULT_TABLE table=default.Table_ab087b92dedd4bc5a48fa4e8c9970ce1 [id=1bd2830a315340249fb100fa422faf73]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:26.256875 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 695f3f27513d41ad8a4d6eedef6db2b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:26.257066 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c6820d6331e44656a27e23afe973e2e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:26.260391 1027 tablet_service.cc:1467] Processing CreateTablet for tablet c6820d6331e44656a27e23afe973e2e2 (DEFAULT_TABLE table=default.Table_ab087b92dedd4bc5a48fa4e8c9970ce1 [id=1bd2830a315340249fb100fa422faf73]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:26.260686 889 tablet_service.cc:1467] Processing CreateTablet for tablet 93c3a547f2674149a89fd0a68abd1d61 (DEFAULT_TABLE table=default.Table_81ec0fd5c30248a5bdef2311e0164b36 [id=d5161fc08be4432eb41b684f3b5e6da1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:26.260862 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c6820d6331e44656a27e23afe973e2e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:26.261147 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 93c3a547f2674149a89fd0a68abd1d61. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:26.264679 1042 tablet_service.cc:1467] Processing CreateTablet for tablet 93c3a547f2674149a89fd0a68abd1d61 (DEFAULT_TABLE table=default.Table_81ec0fd5c30248a5bdef2311e0164b36 [id=d5161fc08be4432eb41b684f3b5e6da1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:26.265110 1042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 93c3a547f2674149a89fd0a68abd1d61. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:26.268396 1198 ts_tablet_manager.cc:1905] T 34ab7ee573284550aaac9f814e8bd7ab P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:26.268332 1028 tablet_service.cc:1467] Processing CreateTablet for tablet 9cf14ed03b3c418c941cd20ed371d7e2 (DEFAULT_TABLE table=default.Table_e195b201eb0f468386996107b50d3bd2 [id=525848fa5c2540c7b187bf64389aaade]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:26.268941 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9cf14ed03b3c418c941cd20ed371d7e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:26.272837 455 master_service.cc:946] ReplaceTablet: received request to replace tablet 7d53de6379b84103ba905effb0835676 from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:26.273953 454 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_349bd0d1105d42e4889232adbacc4d3a" } modify_external_catalogs: true
I20250113 02:51:26.274224 454 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_349bd0d1105d42e4889232adbacc4d3a" } modify_external_catalogs: true
I20250113 02:51:26.276808 1192 tablet_bootstrap.cc:492] T 64cda8796dff46ce95c37439395c77ec P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:26.277051 1192 ts_tablet_manager.cc:1397] T 64cda8796dff46ce95c37439395c77ec P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.152s user 0.005s sys 0.000s
I20250113 02:51:26.277985 1192 raft_consensus.cc:357] T 64cda8796dff46ce95c37439395c77ec P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:26.278242 1192 raft_consensus.cc:383] T 64cda8796dff46ce95c37439395c77ec P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:26.278329 1192 raft_consensus.cc:738] T 64cda8796dff46ce95c37439395c77ec P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:26.278628 1192 consensus_queue.cc:260] T 64cda8796dff46ce95c37439395c77ec P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:26.279255 1192 ts_tablet_manager.cc:1428] T 64cda8796dff46ce95c37439395c77ec P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250113 02:51:26.280411 1192 tablet_bootstrap.cc:492] T 5b3f012b88c04cb68ed7a1848147b70b P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:26.282392 1192 tablet_bootstrap.cc:654] T 5b3f012b88c04cb68ed7a1848147b70b P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:26.289275 1194 tablet_bootstrap.cc:492] T b834f0acf5314bcb9d81c73de1b31842 P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:26.289503 1194 ts_tablet_manager.cc:1397] T b834f0acf5314bcb9d81c73de1b31842 P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.175s user 0.004s sys 0.004s
I20250113 02:51:26.289697 1634 raft_consensus.cc:491] T 438d1d2300e044f39a15586f335935ba P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:26.289897 1634 raft_consensus.cc:513] T 438d1d2300e044f39a15586f335935ba P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:26.290472 1194 raft_consensus.cc:357] T b834f0acf5314bcb9d81c73de1b31842 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:26.290668 1194 raft_consensus.cc:383] T b834f0acf5314bcb9d81c73de1b31842 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:26.290807 1194 raft_consensus.cc:738] T b834f0acf5314bcb9d81c73de1b31842 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:26.291049 1194 consensus_queue.cc:260] T b834f0acf5314bcb9d81c73de1b31842 P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:26.291622 1194 ts_tablet_manager.cc:1428] T b834f0acf5314bcb9d81c73de1b31842 P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250113 02:51:26.292263 1634 leader_election.cc:290] T 438d1d2300e044f39a15586f335935ba P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:26.297291 899 tablet_service.cc:1467] Processing CreateTablet for tablet 9cf14ed03b3c418c941cd20ed371d7e2 (DEFAULT_TABLE table=default.Table_e195b201eb0f468386996107b50d3bd2 [id=525848fa5c2540c7b187bf64389aaade]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:26.304281 1194 tablet_bootstrap.cc:492] T 32351d6de1574353a6e8073d333da477 P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:26.306330 1194 tablet_bootstrap.cc:654] T 32351d6de1574353a6e8073d333da477 P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:26.308523 1198 ts_tablet_manager.cc:1918] T 34ab7ee573284550aaac9f814e8bd7ab P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:26.308780 1198 log.cc:1198] T 34ab7ee573284550aaac9f814e8bd7ab P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/34ab7ee573284550aaac9f814e8bd7ab
I20250113 02:51:26.309209 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "438d1d2300e044f39a15586f335935ba" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:26.309520 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9cf14ed03b3c418c941cd20ed371d7e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:26.310333 917 raft_consensus.cc:2463] T 438d1d2300e044f39a15586f335935ba P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 0.
I20250113 02:51:26.310753 998 leader_election.cc:304] T 438d1d2300e044f39a15586f335935ba P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f; no voters:
I20250113 02:51:26.311048 1634 raft_consensus.cc:2798] T 438d1d2300e044f39a15586f335935ba P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 02:51:26.311156 1634 raft_consensus.cc:491] T 438d1d2300e044f39a15586f335935ba P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 02:51:26.311236 1634 raft_consensus.cc:3054] T 438d1d2300e044f39a15586f335935ba P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:26.311529 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 6d4d8b5cc8ad4c9e935d8afaf5766e66
I20250113 02:51:26.312052 449 catalog_manager.cc:6596] ReplaceTablet: tablet 6d4d8b5cc8ad4c9e935d8afaf5766e66 deleted and replaced by tablet e57ac440265e4646bfc5d5d9ca4c35d2
I20250113 02:51:26.313928 1634 raft_consensus.cc:513] T 438d1d2300e044f39a15586f335935ba P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:26.314144 1033 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d4d8b5cc8ad4c9e935d8afaf5766e66 with delete_type TABLET_DATA_DELETED (replaced by tablet e57ac440265e4646bfc5d5d9ca4c35d2) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:26.314895 1634 leader_election.cc:290] T 438d1d2300e044f39a15586f335935ba P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Requested vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:26.315193 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "438d1d2300e044f39a15586f335935ba" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f"
I20250113 02:51:26.315392 919 raft_consensus.cc:3054] T 438d1d2300e044f39a15586f335935ba P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:26.318343 919 raft_consensus.cc:2463] T 438d1d2300e044f39a15586f335935ba P df69f48656b44bb890f76587ee69cc0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 1.
I20250113 02:51:26.318737 998 leader_election.cc:304] T 438d1d2300e044f39a15586f335935ba P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f; no voters:
I20250113 02:51:26.319041 1634 raft_consensus.cc:2798] T 438d1d2300e044f39a15586f335935ba P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Leader election won for term 1
I20250113 02:51:26.319303 1634 raft_consensus.cc:695] T 438d1d2300e044f39a15586f335935ba P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: Becoming Leader. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Running, Role: LEADER
I20250113 02:51:26.319577 1198 ts_tablet_manager.cc:1939] T 34ab7ee573284550aaac9f814e8bd7ab P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:26.319634 1634 consensus_queue.cc:237] T 438d1d2300e044f39a15586f335935ba P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:26.320442 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 468) took 1677 ms. Trace:
I20250113 02:51:26.320578 1198 rpcz_store.cc:276] 0113 02:51:24.642482 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:24.644950 (+ 2468us) service_pool.cc:225] Handling call
0113 02:51:26.170481 (+1525531us) ts_tablet_manager.cc:1145] Deleting tablet 34ab7ee573284550aaac9f814e8bd7ab
0113 02:51:26.170490 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:26.276342 (+105852us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:26.308401 (+ 32059us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:26.320379 (+ 11978us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1525347}
I20250113 02:51:26.321245 1198 tablet_replica.cc:331] T 32351d6de1574353a6e8073d333da477 P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:26.321202 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet 34ab7ee573284550aaac9f814e8bd7ab (table default.Table_c59cac1b4e0249129c7671a4919283c8 [id=1cd50765f4964c15a341d307ff2d449f]) successfully deleted
I20250113 02:51:26.321611 1198 raft_consensus.cc:2238] T 32351d6de1574353a6e8073d333da477 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: Raft consensus shutting down.
I20250113 02:51:26.321921 1198 pending_rounds.cc:62] T 32351d6de1574353a6e8073d333da477 P 4216a7f853d642b7bc3e5cff458f3aca: Trying to abort 1 pending ops.
I20250113 02:51:26.322043 1198 pending_rounds.cc:69] T 32351d6de1574353a6e8073d333da477 P 4216a7f853d642b7bc3e5cff458f3aca: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113673465366806528 op_type: NO_OP noop_request { }
I20250113 02:51:26.322273 1198 raft_consensus.cc:2883] T 32351d6de1574353a6e8073d333da477 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250113 02:51:26.322455 1198 raft_consensus.cc:2267] T 32351d6de1574353a6e8073d333da477 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:26.322739 889 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d4d8b5cc8ad4c9e935d8afaf5766e66 with delete_type TABLET_DATA_DELETED (replaced by tablet e57ac440265e4646bfc5d5d9ca4c35d2) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:26.236541 743 tablet_service.cc:1467] Processing CreateTablet for tablet 795a96fc8cb243e2841739b3a492d0f8 (DEFAULT_TABLE table=default.Table_d1ff69f5cd31453aaf9d8f6382eedcda [id=b2ee3441903a403488dad5cdaf2f9986]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:26.323592 1198 ts_tablet_manager.cc:1905] T 32351d6de1574353a6e8073d333da477 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:26.323798 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 795a96fc8cb243e2841739b3a492d0f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:26.237258 750 tablet_service.cc:1467] Processing CreateTablet for tablet 656a0dda5c0b46309ed472d34a42ff94 (DEFAULT_TABLE table=default.Table_d4b3b319759f44d4bbe072b1bc97cc94 [id=2d80f898223a4982aa73dfca00277e28]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:26.324234 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 656a0dda5c0b46309ed472d34a42ff94. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:26.324324 1191 raft_consensus.cc:357] T d5fbf97536a2472fa091168363bf9549 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:26.324548 1191 raft_consensus.cc:383] T d5fbf97536a2472fa091168363bf9549 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:26.324620 1191 raft_consensus.cc:738] T d5fbf97536a2472fa091168363bf9549 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:26.324853 1191 consensus_queue.cc:260] T d5fbf97536a2472fa091168363bf9549 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:26.325419 1191 ts_tablet_manager.cc:1428] T d5fbf97536a2472fa091168363bf9549 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250113 02:51:26.325748 1191 tablet_bootstrap.cc:492] T 0fb75c551b4c424186ef2021db742e8e P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:26.327589 1191 tablet_bootstrap.cc:654] T 0fb75c551b4c424186ef2021db742e8e P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:26.248987 744 tablet_service.cc:1467] Processing CreateTablet for tablet 130f8540b9a6477e84bb9595e7c39994 (DEFAULT_TABLE table=default.Table_eec56bd1a5344ab985b794cb42a0eeba [id=5c83feb8a24b422fbde3c5fdbc7c63e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:26.328281 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 130f8540b9a6477e84bb9595e7c39994. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:26.249353 753 tablet_service.cc:1467] Processing CreateTablet for tablet 695f3f27513d41ad8a4d6eedef6db2b3 (DEFAULT_TABLE table=default.Table_d1dafd1ed4a64d17b499f312bc298bd7 [id=48bed7c08d9849a19479b744b4c65a68]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:26.249696 747 tablet_service.cc:1467] Processing CreateTablet for tablet c6820d6331e44656a27e23afe973e2e2 (DEFAULT_TABLE table=default.Table_ab087b92dedd4bc5a48fa4e8c9970ce1 [id=1bd2830a315340249fb100fa422faf73]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:26.250036 754 tablet_service.cc:1467] Processing CreateTablet for tablet 93c3a547f2674149a89fd0a68abd1d61 (DEFAULT_TABLE table=default.Table_81ec0fd5c30248a5bdef2311e0164b36 [id=d5161fc08be4432eb41b684f3b5e6da1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:26.250327 737 tablet_service.cc:1467] Processing CreateTablet for tablet 9cf14ed03b3c418c941cd20ed371d7e2 (DEFAULT_TABLE table=default.Table_e195b201eb0f468386996107b50d3bd2 [id=525848fa5c2540c7b187bf64389aaade]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:26.329589 1192 tablet_bootstrap.cc:492] T 5b3f012b88c04cb68ed7a1848147b70b P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:26.264673 1696 raft_consensus.cc:491] T 438d1d2300e044f39a15586f335935ba P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:26.284567 1659 raft_consensus.cc:491] T 35e0aed5125c4e1fa5dce226eade3b78 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:26.329757 1192 ts_tablet_manager.cc:1397] T 5b3f012b88c04cb68ed7a1848147b70b P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.049s user 0.003s sys 0.000s
I20250113 02:51:26.291281 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "438d1d2300e044f39a15586f335935ba" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
I20250113 02:51:26.312667 752 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d4d8b5cc8ad4c9e935d8afaf5766e66 with delete_type TABLET_DATA_DELETED (replaced by tablet e57ac440265e4646bfc5d5d9ca4c35d2) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:26.315068 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "438d1d2300e044f39a15586f335935ba" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196"
I20250113 02:51:26.330472 1192 raft_consensus.cc:357] T 5b3f012b88c04cb68ed7a1848147b70b P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:26.330757 1192 raft_consensus.cc:383] T 5b3f012b88c04cb68ed7a1848147b70b P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:26.330893 1192 raft_consensus.cc:738] T 5b3f012b88c04cb68ed7a1848147b70b P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:26.330852 1194 tablet_bootstrap.cc:492] T 32351d6de1574353a6e8073d333da477 P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:26.331224 1192 consensus_queue.cc:260] T 5b3f012b88c04cb68ed7a1848147b70b P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
W20250113 02:51:26.331589 442 catalog_manager.cc:4839] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 317c6e8013b745c290bf3190dc237ab8 because the tablet was not found. No further retry: Not found: Tablet not found: 317c6e8013b745c290bf3190dc237ab8
I20250113 02:51:26.332017 1194 ts_tablet_manager.cc:1397] T 32351d6de1574353a6e8073d333da477 P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.028s user 0.002s sys 0.002s
W20250113 02:51:26.336397 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 88a53da280b0470abc7b522134d05c8f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 88a53da280b0470abc7b522134d05c8f already in progress: creating tablet
I20250113 02:51:26.336517 451 catalog_manager.cc:6071] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: default.Table_d1f76dd334eb4d5890cceb429ed27e9f [id=70089f757a1c45f6b983f216fb167478] alter complete (version 3)
I20250113 02:51:26.337211 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet f2fb1d39c9ca45ccb2da0be75ccf4acf
I20250113 02:51:26.337348 880 tablet_service.cc:1514] Processing DeleteTablet for tablet 23e9321f08ee4ce7b2252ad5c76c0a08 with delete_type TABLET_DATA_DELETED (replaced by tablet 6666922d677a4c678f80d7c6c47fd2a5) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:26.339337 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet d8bb60fdd06946739b75b255303d9d24
I20250113 02:51:26.339900 1194 raft_consensus.cc:357] T 32351d6de1574353a6e8073d333da477 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:26.341343 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet b217cb8c09fb41cab4f53e605c37efe8
I20250113 02:51:26.341197 1194 raft_consensus.cc:383] T 32351d6de1574353a6e8073d333da477 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:26.342451 1194 raft_consensus.cc:738] T 32351d6de1574353a6e8073d333da477 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:26.343017 892 tablet_service.cc:1514] Processing DeleteTablet for tablet b217cb8c09fb41cab4f53e605c37efe8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:26 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:26.343328 889 tablet_service.cc:1514] Processing DeleteTablet for tablet d8bb60fdd06946739b75b255303d9d24 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:26 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:26.344309 1689 tablet.cc:1716] T 4fe54131aafb40ee9d8bc5ea299af193 P df69f48656b44bb890f76587ee69cc0f: Skipping requested alter to schema version 3, tablet already version 3
I20250113 02:51:26.343793 1194 consensus_queue.cc:260] T 32351d6de1574353a6e8073d333da477 P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:26.345933 899 tablet_service.cc:1514] Processing DeleteTablet for tablet f2fb1d39c9ca45ccb2da0be75ccf4acf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:26 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:26.346805 1194 ts_tablet_manager.cc:1428] T 32351d6de1574353a6e8073d333da477 P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.014s user 0.004s sys 0.000s
I20250113 02:51:26.348315 1194 tablet_bootstrap.cc:492] T 1500e4f9c0664a879503afd3edf14c66 P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:26.349028 899 tablet_service.cc:1514] Processing DeleteTablet for tablet b834f0acf5314bcb9d81c73de1b31842 with delete_type TABLET_DATA_DELETED (replaced by tablet be4b2a8a3927410eaf12e7592646e4af) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:26.350524 1033 tablet_service.cc:1514] Processing DeleteTablet for tablet f2fb1d39c9ca45ccb2da0be75ccf4acf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:26 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:26.351292 899 tablet_service.cc:1514] Processing DeleteTablet for tablet 32351d6de1574353a6e8073d333da477 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:18 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:26.353283 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet d5fbf97536a2472fa091168363bf9549 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:18 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:26.353713 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet 35e0aed5125c4e1fa5dce226eade3b78 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:17 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:26.354066 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet b217cb8c09fb41cab4f53e605c37efe8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:26 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:26.354378 1042 tablet_service.cc:1514] Processing DeleteTablet for tablet d8bb60fdd06946739b75b255303d9d24 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:26 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:26.354609 1192 ts_tablet_manager.cc:1428] T 5b3f012b88c04cb68ed7a1848147b70b P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250113 02:51:26.355743 1192 tablet_bootstrap.cc:492] T b2a35fd891c645d68d79c61a2c832ba7 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
W20250113 02:51:26.357486 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 303d911dfa2f4abbb053ead33db6d71c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 303d911dfa2f4abbb053ead33db6d71c already in progress: creating tablet
W20250113 02:51:26.357821 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 67baaab1e36e415381579d7385d4ab5a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 67baaab1e36e415381579d7385d4ab5a already in progress: creating tablet
I20250113 02:51:26.357861 1198 ts_tablet_manager.cc:1918] T 32351d6de1574353a6e8073d333da477 P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250113 02:51:26.357983 1198 log.cc:1198] T 32351d6de1574353a6e8073d333da477 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/32351d6de1574353a6e8073d333da477
W20250113 02:51:26.358139 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 6d4d8b5cc8ad4c9e935d8afaf5766e66 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6d4d8b5cc8ad4c9e935d8afaf5766e66 already in progress: creating tablet
W20250113 02:51:26.358357 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet b217cb8c09fb41cab4f53e605c37efe8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b217cb8c09fb41cab4f53e605c37efe8 already in progress: creating tablet
W20250113 02:51:26.358597 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet d8bb60fdd06946739b75b255303d9d24 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d8bb60fdd06946739b75b255303d9d24 already in progress: creating tablet
I20250113 02:51:26.358752 1198 ts_tablet_manager.cc:1939] T 32351d6de1574353a6e8073d333da477 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:26.368029 1192 tablet_bootstrap.cc:654] T b2a35fd891c645d68d79c61a2c832ba7 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:26.374866 880 tablet_service.cc:1467] Processing CreateTablet for tablet acae3ff634104b5d931ece75d1eba5d6 (DEFAULT_TABLE table=default.Table_eaece5fe552b47fb9c0f3d73031f40f4 [id=27d19697fb0f42d09faee637ba85cefd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:26.375376 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet acae3ff634104b5d931ece75d1eba5d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:26.377281 899 tablet_service.cc:1467] Processing CreateTablet for tablet 17fc320eb01f4ac2b9361aeab5b452fc (DEFAULT_TABLE table=default.Table_6431cb4cd5374cbcaceda152cbc19e5b [id=3a485245494241b5a902dc44980c7c8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:26.377506 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 469) took 1586 ms. Trace:
I20250113 02:51:26.377621 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 17fc320eb01f4ac2b9361aeab5b452fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:26.377614 1198 rpcz_store.cc:276] 0113 02:51:24.791305 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:24.791582 (+ 277us) service_pool.cc:225] Handling call
0113 02:51:26.321163 (+1529581us) ts_tablet_manager.cc:1145] Deleting tablet 32351d6de1574353a6e8073d333da477
0113 02:51:26.321173 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:26.326908 (+ 5735us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:26.357730 (+ 30822us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:26.377323 (+ 19593us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1519547}
I20250113 02:51:26.378655 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet 32351d6de1574353a6e8073d333da477 (table default.Table_78c0bcf1e2d64abc81253301920cf566 [id=e5831758ca184c1998f551fe0285d010]) successfully deleted
I20250113 02:51:26.378958 1042 tablet_service.cc:1467] Processing CreateTablet for tablet acae3ff634104b5d931ece75d1eba5d6 (DEFAULT_TABLE table=default.Table_eaece5fe552b47fb9c0f3d73031f40f4 [id=27d19697fb0f42d09faee637ba85cefd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:26.379349 1042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet acae3ff634104b5d931ece75d1eba5d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:26.379637 1024 tablet_service.cc:1467] Processing CreateTablet for tablet 5d4912055c5d4c8d8be8b65121f53200 (DEFAULT_TABLE table=default.Table_bbda194bd2cd4df2b7e7d429700e5f3c [id=a578f9aa01d44ba7923d8d93a1facf68]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:26.380039 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d4912055c5d4c8d8be8b65121f53200. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:26.380250 892 tablet_service.cc:1467] Processing CreateTablet for tablet 5d4912055c5d4c8d8be8b65121f53200 (DEFAULT_TABLE table=default.Table_bbda194bd2cd4df2b7e7d429700e5f3c [id=a578f9aa01d44ba7923d8d93a1facf68]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:26.380551 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d4912055c5d4c8d8be8b65121f53200. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:26.384251 1027 tablet_service.cc:1467] Processing CreateTablet for tablet 17fc320eb01f4ac2b9361aeab5b452fc (DEFAULT_TABLE table=default.Table_6431cb4cd5374cbcaceda152cbc19e5b [id=3a485245494241b5a902dc44980c7c8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:26.384469 889 tablet_service.cc:1467] Processing CreateTablet for tablet ef6dfe0431ef4a73a957559f60f24756 (DEFAULT_TABLE table=default.Table_9e1b92761ab84f80905c2b4f57738caf [id=bc667764370048d0a6211fa11ac9c55b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:26.384652 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 17fc320eb01f4ac2b9361aeab5b452fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:26.384804 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef6dfe0431ef4a73a957559f60f24756. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:26.386067 1028 tablet_service.cc:1467] Processing CreateTablet for tablet fa9052eea538439a9cb4c527e97897a4 (DEFAULT_TABLE table=default.Table_ec65488dc1334a33a56ba81d2bda391c [id=2061c64fe8b545069da97dad04c40637]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:26.386411 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fa9052eea538439a9cb4c527e97897a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:26.388981 1025 tablet_service.cc:1467] Processing CreateTablet for tablet 4cbc27d8ac2a46c58bffc33d8af17be2 (DEFAULT_TABLE table=default.Table_8a3a3b6c452a437dae09bdd5ee43c8a0 [id=cd5695fc47644c2cb6c33e932a74550b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:26.389281 1194 tablet_bootstrap.cc:654] T 1500e4f9c0664a879503afd3edf14c66 P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:26.389330 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4cbc27d8ac2a46c58bffc33d8af17be2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:26.389955 891 tablet_service.cc:1467] Processing CreateTablet for tablet fa9052eea538439a9cb4c527e97897a4 (DEFAULT_TABLE table=default.Table_ec65488dc1334a33a56ba81d2bda391c [id=2061c64fe8b545069da97dad04c40637]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:26.390292 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fa9052eea538439a9cb4c527e97897a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:26.392242 881 tablet_service.cc:1467] Processing CreateTablet for tablet c368b5b0c3a940c4b4efb64c946375c3 (DEFAULT_TABLE table=default.Table_5182a1c5c7a2462ba930940ae04b505f [id=6666a614063942c99a78b95d35add037]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:26.392221 1023 tablet_service.cc:1467] Processing CreateTablet for tablet c368b5b0c3a940c4b4efb64c946375c3 (DEFAULT_TABLE table=default.Table_5182a1c5c7a2462ba930940ae04b505f [id=6666a614063942c99a78b95d35add037]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:26.392297 1037 tablet_service.cc:1467] Processing CreateTablet for tablet e52429bc9c944c75922a1306acd38fea (DEFAULT_TABLE table=default.Table_3203c45c9ef34fcdb7c9ebcafb066412 [id=46f200ae89ac4e00829a5be6414589f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:26.392555 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c368b5b0c3a940c4b4efb64c946375c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:26.392581 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c368b5b0c3a940c4b4efb64c946375c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:26.384253 1033 tablet_service.cc:1467] Processing CreateTablet for tablet bd48dbbfe20342569e6f153353f1ef9c (DEFAULT_TABLE table=default.Table_f410165a9c1c45229555f7b89641c3db [id=4496eba45fa043bc91553e1d77c53234]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:26.393579 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd48dbbfe20342569e6f153353f1ef9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:26.395823 1040 tablet_service.cc:1467] Processing CreateTablet for tablet 37b7e05f7bbd4a4180cc40e76868469b (DEFAULT_TABLE table=default.Table_9c812746e36d4df38dd75d3a561d7d57 [id=da9f9fb691454444b316e0f782345ee1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:26.396169 1040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37b7e05f7bbd4a4180cc40e76868469b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:26.396239 885 tablet_service.cc:1467] Processing CreateTablet for tablet 9fa6a1c4f6ef4f98b4de114210af08c5 (DEFAULT_TABLE table=default.Table_6d300fedf96048a5a3bdbc9f36ac0b6c [id=84cbc9ab3d23459988b3681c8c6004da]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:26.396533 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9fa6a1c4f6ef4f98b4de114210af08c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:26.397228 1037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e52429bc9c944c75922a1306acd38fea. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:26.399639 1031 tablet_service.cc:1467] Processing CreateTablet for tablet 9fa6a1c4f6ef4f98b4de114210af08c5 (DEFAULT_TABLE table=default.Table_6d300fedf96048a5a3bdbc9f36ac0b6c [id=84cbc9ab3d23459988b3681c8c6004da]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:26.400034 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9fa6a1c4f6ef4f98b4de114210af08c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:26.400425 886 tablet_service.cc:1467] Processing CreateTablet for tablet 0953b2f89e35402c9e9e84f3589c23ac (DEFAULT_TABLE table=default.Table_3318768dfa8c4d4bbc513e660dabfcc5 [id=dffb2926dc0a4a7db6878bd4c92f2bcd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:26.400746 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0953b2f89e35402c9e9e84f3589c23ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:26.403429 1029 tablet_service.cc:1467] Processing CreateTablet for tablet 0953b2f89e35402c9e9e84f3589c23ac (DEFAULT_TABLE table=default.Table_3318768dfa8c4d4bbc513e660dabfcc5 [id=dffb2926dc0a4a7db6878bd4c92f2bcd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:26.403767 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0953b2f89e35402c9e9e84f3589c23ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:26.404244 898 tablet_service.cc:1467] Processing CreateTablet for tablet 4cbc27d8ac2a46c58bffc33d8af17be2 (DEFAULT_TABLE table=default.Table_8a3a3b6c452a437dae09bdd5ee43c8a0 [id=cd5695fc47644c2cb6c33e932a74550b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:26.404561 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4cbc27d8ac2a46c58bffc33d8af17be2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:26.407696 448 master_service.cc:946] ReplaceTablet: received request to replace tablet 6d63f0e5835540a78c82020b20a07dac from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:26.379292 1035 tablet_service.cc:1467] Processing CreateTablet for tablet ef6dfe0431ef4a73a957559f60f24756 (DEFAULT_TABLE table=default.Table_9e1b92761ab84f80905c2b4f57738caf [id=bc667764370048d0a6211fa11ac9c55b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:26.409509 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef6dfe0431ef4a73a957559f60f24756. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:26.411370 888 tablet_service.cc:1467] Processing CreateTablet for tablet e52429bc9c944c75922a1306acd38fea (DEFAULT_TABLE table=default.Table_3203c45c9ef34fcdb7c9ebcafb066412 [id=46f200ae89ac4e00829a5be6414589f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:26.411723 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e52429bc9c944c75922a1306acd38fea. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:26.413547 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 569d666c128a45b7a526c13be5090ae3
I20250113 02:51:26.414958 1036 tablet_service.cc:1514] Processing DeleteTablet for tablet 569d666c128a45b7a526c13be5090ae3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:26 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:26.395907 887 tablet_service.cc:1467] Processing CreateTablet for tablet 37b7e05f7bbd4a4180cc40e76868469b (DEFAULT_TABLE table=default.Table_9c812746e36d4df38dd75d3a561d7d57 [id=da9f9fb691454444b316e0f782345ee1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:26.416354 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37b7e05f7bbd4a4180cc40e76868469b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:26.422169 1196 tablet_replica.cc:331] T 23e9321f08ee4ce7b2252ad5c76c0a08 P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:26.422415 895 tablet_service.cc:1467] Processing CreateTablet for tablet bd48dbbfe20342569e6f153353f1ef9c (DEFAULT_TABLE table=default.Table_f410165a9c1c45229555f7b89641c3db [id=4496eba45fa043bc91553e1d77c53234]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:26.422745 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd48dbbfe20342569e6f153353f1ef9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:26.422736 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 485) took 1480 ms. Trace:
I20250113 02:51:26.422833 1198 rpcz_store.cc:276] 0113 02:51:24.942701 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:24.942770 (+ 69us) service_pool.cc:225] Handling call
0113 02:51:26.422526 (+1479756us) ts_tablet_manager.cc:1145] Deleting tablet a2781ba5061a4225b5636366857a229e
0113 02:51:26.422536 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:26.422659 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1472881}
W20250113 02:51:26.424445 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet a2781ba5061a4225b5636366857a229e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a2781ba5061a4225b5636366857a229e already in progress: creating tablet
I20250113 02:51:26.425004 454 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_2d289859f3a2422b9f1c5bd6b3083088" } new_table_name: "default.Table_9494f6e5f96141dbaec6769ed51d39b2" modify_external_catalogs: true
I20250113 02:51:26.426419 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_322cee4c328e4e2ca3dcfed1ac77749d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:26.422421 1196 raft_consensus.cc:2238] T 23e9321f08ee4ce7b2252ad5c76c0a08 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
W20250113 02:51:26.426939 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_322cee4c328e4e2ca3dcfed1ac77749d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:26.428881 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_4d5b54216098465fb88c7b9d8b605344"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:26.429491 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4d5b54216098465fb88c7b9d8b605344 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:26.430725 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_67eab203697544008a476488a93df8a9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:26.431293 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_67eab203697544008a476488a93df8a9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:26.432705 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_3c89afa5af12403b9a48046ee6c81da3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:26.433260 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3c89afa5af12403b9a48046ee6c81da3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:26.434634 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_d86534ff16f04a45977372ccee26a336"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:26.435220 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d86534ff16f04a45977372ccee26a336 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:26.436291 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_f11af4ff436842b8800e70008fb2c7b5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:26.436846 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f11af4ff436842b8800e70008fb2c7b5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:26.443802 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_7f15d522310c4e8a8afc5d1b77bcaeec" } new_table_name: "default.Table_f7097c6cf1fd4839872c42a30fa5f64f" modify_external_catalogs: true
I20250113 02:51:26.444432 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_632d3a63935c4ff2ba48037a311708c4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:26.445046 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_632d3a63935c4ff2ba48037a311708c4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:26.428251 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_769b925f82bb49ab81662917c3011720" } new_table_name: "default.Table_f5459a21934542d9b62792bcd109f836" modify_external_catalogs: true
I20250113 02:51:26.455255 1192 tablet_bootstrap.cc:492] T b2a35fd891c645d68d79c61a2c832ba7 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:26.455601 1192 ts_tablet_manager.cc:1397] T b2a35fd891c645d68d79c61a2c832ba7 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.100s user 0.004s sys 0.003s
I20250113 02:51:26.457774 1634 raft_consensus.cc:491] T 06f36d09bbb1443bafce906907795404 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:26.457950 1634 raft_consensus.cc:513] T 06f36d09bbb1443bafce906907795404 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
W20250113 02:51:26.458556 454 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 788ab82626424b6e833e31761a15b9b2 consensus state has no leader
I20250113 02:51:26.458658 1634 leader_election.cc:290] T 06f36d09bbb1443bafce906907795404 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
I20250113 02:51:26.459630 1037 tablet_service.cc:1467] Processing CreateTablet for tablet 25abb5d0f24e4407907f12caa55c5c8b (DEFAULT_TABLE table=default.Table_9181be1f89094ceea4944414dd995df0 [id=92ac4d0187294a428a7ab739cce23cdd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:26.460045 1037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 25abb5d0f24e4407907f12caa55c5c8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:26.464983 1025 tablet_service.cc:1467] Processing CreateTablet for tablet e57ac440265e4646bfc5d5d9ca4c35d2 (DEFAULT_TABLE table=default.Table_54119752ebc147b8b34429f1c38b71d9 [id=2ab34074f7624b419c1cd36e9a09b4c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:26.474611 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e57ac440265e4646bfc5d5d9ca4c35d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:26.475627 1192 raft_consensus.cc:357] T b2a35fd891c645d68d79c61a2c832ba7 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:26.477172 1192 raft_consensus.cc:383] T b2a35fd891c645d68d79c61a2c832ba7 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:26.479547 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 486) took 1526 ms. Trace:
I20250113 02:51:26.483845 1198 rpcz_store.cc:276] 0113 02:51:24.952983 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:24.953047 (+ 64us) service_pool.cc:225] Handling call
0113 02:51:26.479383 (+1526336us) ts_tablet_manager.cc:1145] Deleting tablet 0c408921a09a463eb464e08edcf042b3
0113 02:51:26.479393 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:26.479491 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1512256}
I20250113 02:51:26.484110 1438 raft_consensus.cc:491] T b834f0acf5314bcb9d81c73de1b31842 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:26.484295 1438 raft_consensus.cc:513] T b834f0acf5314bcb9d81c73de1b31842 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:26.485046 1438 leader_election.cc:290] T b834f0acf5314bcb9d81c73de1b31842 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587), 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
I20250113 02:51:26.485174 883 tablet_service.cc:1514] Processing DeleteTablet for tablet 569d666c128a45b7a526c13be5090ae3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:26 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:26.485548 884 tablet_service.cc:1467] Processing CreateTablet for tablet 25abb5d0f24e4407907f12caa55c5c8b (DEFAULT_TABLE table=default.Table_9181be1f89094ceea4944414dd995df0 [id=92ac4d0187294a428a7ab739cce23cdd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:26.485850 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 487) took 1532 ms. Trace:
I20250113 02:51:26.485944 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 25abb5d0f24e4407907f12caa55c5c8b. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:26.485889 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 0c408921a09a463eb464e08edcf042b3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0c408921a09a463eb464e08edcf042b3 already in progress: creating tablet
I20250113 02:51:26.485937 1198 rpcz_store.cc:276] 0113 02:51:24.953607 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:24.953665 (+ 58us) service_pool.cc:225] Handling call
0113 02:51:26.485665 (+1532000us) ts_tablet_manager.cc:1145] Deleting tablet 283fc7f1c74e480c8a63c74d3ae9eb7d
0113 02:51:26.485674 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:26.485798 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1518414}
I20250113 02:51:26.486339 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 489) took 1494 ms. Trace:
W20250113 02:51:26.486335 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 283fc7f1c74e480c8a63c74d3ae9eb7d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 283fc7f1c74e480c8a63c74d3ae9eb7d already in progress: creating tablet
I20250113 02:51:26.486418 1198 rpcz_store.cc:276] 0113 02:51:24.992015 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:24.992131 (+ 116us) service_pool.cc:225] Handling call
0113 02:51:26.486174 (+1494043us) ts_tablet_manager.cc:1145] Deleting tablet 056bf1e67be04d179eace3d93282b145
0113 02:51:26.486182 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:26.486293 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1471537}
W20250113 02:51:26.486852 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 056bf1e67be04d179eace3d93282b145 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 056bf1e67be04d179eace3d93282b145 already in progress: creating tablet
I20250113 02:51:26.487123 896 tablet_service.cc:1467] Processing CreateTablet for tablet e57ac440265e4646bfc5d5d9ca4c35d2 (DEFAULT_TABLE table=default.Table_54119752ebc147b8b34429f1c38b71d9 [id=2ab34074f7624b419c1cd36e9a09b4c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:26.487461 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e57ac440265e4646bfc5d5d9ca4c35d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:26.488611 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06f36d09bbb1443bafce906907795404" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:26.477490 1192 raft_consensus.cc:738] T b2a35fd891c645d68d79c61a2c832ba7 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
W20250113 02:51:26.489241 998 leader_election.cc:343] T 06f36d09bbb1443bafce906907795404 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:26.489809 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 490) took 1479 ms. Trace:
I20250113 02:51:26.489909 1198 rpcz_store.cc:276] 0113 02:51:25.010674 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:25.010758 (+ 84us) service_pool.cc:225] Handling call
0113 02:51:26.489632 (+1478874us) ts_tablet_manager.cc:1145] Deleting tablet bc533518faa24dffa3306ff092e8d507
0113 02:51:26.489642 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:26.489757 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1429461}
W20250113 02:51:26.490295 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet bc533518faa24dffa3306ff092e8d507 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bc533518faa24dffa3306ff092e8d507 already in progress: creating tablet
I20250113 02:51:26.492324 1196 raft_consensus.cc:2267] T 23e9321f08ee4ce7b2252ad5c76c0a08 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:26.491655 1192 consensus_queue.cc:260] T b2a35fd891c645d68d79c61a2c832ba7 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:26.495837 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_5f73c3e74d984bb69330b294fbc610de"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:26.496074 1196 ts_tablet_manager.cc:1905] T 23e9321f08ee4ce7b2252ad5c76c0a08 P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250113 02:51:26.496455 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5f73c3e74d984bb69330b294fbc610de in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:26.498001 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_fc22714e47dd45e6890bf2935057344e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:26.498519 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fc22714e47dd45e6890bf2935057344e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:26.500150 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_c1b46fe79c2e4aabbb2a97139f4078c3" } new_table_name: "default.Table_874095afd8f14f9b97268aafb4a38bee" modify_external_catalogs: true
W20250113 02:51:26.500962 442 catalog_manager.cc:4945] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): alter failed for tablet 317c6e8013b745c290bf3190dc237ab8 (table default.Table_13fd4df9c207466cb85cd8ed9062722c [id=6bfb67a96b2b47849f7fa12c57a2f4a1]),no further retry: Not found: Tablet not found: 317c6e8013b745c290bf3190dc237ab8
W20250113 02:51:26.499346 852 leader_election.cc:343] T b834f0acf5314bcb9d81c73de1b31842 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Not found: Tablet not found: b834f0acf5314bcb9d81c73de1b31842
I20250113 02:51:26.501734 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_e5ce75ec7b044da3ae915dc3ce5421da" } modify_external_catalogs: true
I20250113 02:51:26.501924 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_e5ce75ec7b044da3ae915dc3ce5421da" } modify_external_catalogs: true
I20250113 02:51:26.512079 1192 ts_tablet_manager.cc:1428] T b2a35fd891c645d68d79c61a2c832ba7 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.056s user 0.001s sys 0.002s
I20250113 02:51:26.515177 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_e195b201eb0f468386996107b50d3bd2" } new_table_name: "default.Table_5a88d7e91d464a8aa804fec7b174d051" modify_external_catalogs: true
I20250113 02:51:26.515612 1192 tablet_bootstrap.cc:492] T 572e4eed1e924381bada27f8389821a7 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
W20250113 02:51:26.518502 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 79aaab73797c474bae06831fa308b066 consensus state has no leader
W20250113 02:51:26.518971 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5b1076a5a17b4fa495975285616401cf consensus state has no leader
I20250113 02:51:26.522054 454 catalog_manager.cc:5526] T 438d1d2300e044f39a15586f335935ba P 4216a7f853d642b7bc3e5cff458f3aca reported cstate change: term changed from 0 to 1, leader changed from <none> to 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3). New cstate: current_term: 1 leader_uuid: "4216a7f853d642b7bc3e5cff458f3aca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } health_report { overall_health: HEALTHY } } }
I20250113 02:51:26.522500 454 catalog_manager.cc:5618] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587) does not have the latest schema for tablet 438d1d2300e044f39a15586f335935ba (table default.Table_d7d1f5a34e5f4c05b474b0b61f912cd9 [id=2c7925ec0d8d4cd7b8761268d3962f15]). Expected version 3 got 0
I20250113 02:51:26.525209 1192 tablet_bootstrap.cc:654] T 572e4eed1e924381bada27f8389821a7 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:26.528678 1659 raft_consensus.cc:513] T 35e0aed5125c4e1fa5dce226eade3b78 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:26.528896 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c6820d6331e44656a27e23afe973e2e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:26.529966 1659 leader_election.cc:290] T 35e0aed5125c4e1fa5dce226eade3b78 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:26.531570 1062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35e0aed5125c4e1fa5dce226eade3b78" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:26.531793 1062 raft_consensus.cc:2463] T 35e0aed5125c4e1fa5dce226eade3b78 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1186fa869eb14812832fa6562531f196 in term 0.
I20250113 02:51:26.532210 1669 raft_consensus.cc:491] T 7f08b72301dc431396cf8cc7790d56d4 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:26.532670 1198 tablet_replica.cc:331] T 1500e4f9c0664a879503afd3edf14c66 P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:26.532887 1198 raft_consensus.cc:2238] T 1500e4f9c0664a879503afd3edf14c66 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: Raft consensus shutting down.
I20250113 02:51:26.533162 1198 pending_rounds.cc:62] T 1500e4f9c0664a879503afd3edf14c66 P 4216a7f853d642b7bc3e5cff458f3aca: Trying to abort 1 pending ops.
I20250113 02:51:26.533226 1198 pending_rounds.cc:69] T 1500e4f9c0664a879503afd3edf14c66 P 4216a7f853d642b7bc3e5cff458f3aca: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113673465189941248 op_type: NO_OP noop_request { }
I20250113 02:51:26.533777 1669 raft_consensus.cc:513] T 7f08b72301dc431396cf8cc7790d56d4 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:26.535665 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 93c3a547f2674149a89fd0a68abd1d61. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:26.538030 740 tablet_service.cc:1514] Processing DeleteTablet for tablet 32351d6de1574353a6e8073d333da477 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:18 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:26.538419 752 tablet_service.cc:1514] Processing DeleteTablet for tablet f2fb1d39c9ca45ccb2da0be75ccf4acf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:26 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:26.535406 1696 raft_consensus.cc:513] T 438d1d2300e044f39a15586f335935ba P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:26.542598 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "438d1d2300e044f39a15586f335935ba" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:26.543179 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 695f3f27513d41ad8a4d6eedef6db2b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:26.543409 709 leader_election.cc:304] T 35e0aed5125c4e1fa5dce226eade3b78 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters:
I20250113 02:51:26.533349 1198 raft_consensus.cc:2883] T 1500e4f9c0664a879503afd3edf14c66 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250113 02:51:26.543740 1198 raft_consensus.cc:2267] T 1500e4f9c0664a879503afd3edf14c66 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Raft consensus is shut down!
W20250113 02:51:26.544363 421 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.104.2:38467, user_credentials={real_user=slave}} blocked reactor thread for 180432us
I20250113 02:51:26.545573 448 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_9494f6e5f96141dbaec6769ed51d39b2" } modify_external_catalogs: true
I20250113 02:51:26.545608 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_0f4e49c7b953418aba49a768de9b199b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:26.545799 448 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_9494f6e5f96141dbaec6769ed51d39b2" } modify_external_catalogs: true
W20250113 02:51:26.546193 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0f4e49c7b953418aba49a768de9b199b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:26.547196 453 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_f2da084a4aee4d19b45e784e56f926a6" } modify_external_catalogs: true
I20250113 02:51:26.547266 1062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "438d1d2300e044f39a15586f335935ba" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:26.547358 453 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_f2da084a4aee4d19b45e784e56f926a6" } modify_external_catalogs: true
I20250113 02:51:26.548702 449 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_2f3720d8d897466388c3f9dde157d701" } modify_external_catalogs: true
I20250113 02:51:26.548879 449 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_2f3720d8d897466388c3f9dde157d701" } modify_external_catalogs: true
I20250113 02:51:26.549432 752 tablet_service.cc:1514] Processing DeleteTablet for tablet d8bb60fdd06946739b75b255303d9d24 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:26 UTC) from {username='slave'} at 127.0.0.1:55398
W20250113 02:51:26.552783 851 leader_election.cc:343] T b834f0acf5314bcb9d81c73de1b31842 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Not found: Tablet not found: b834f0acf5314bcb9d81c73de1b31842
I20250113 02:51:26.552938 851 leader_election.cc:304] T b834f0acf5314bcb9d81c73de1b31842 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df69f48656b44bb890f76587ee69cc0f; no voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca
I20250113 02:51:26.552879 1062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f08b72301dc431396cf8cc7790d56d4" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:26.553658 1194 tablet_bootstrap.cc:492] T 1500e4f9c0664a879503afd3edf14c66 P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:26.553943 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskXJrxug/build/asan/bin/kudu
/tmp/dist-test-taskXJrxug/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.60:42067
--webserver_interface=127.0.104.60
--webserver_port=38411
--builtin_ntp_servers=127.0.104.20:45573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:44217,127.0.104.61:35271,127.0.104.60:42067
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250113 02:51:26.554877 1659 raft_consensus.cc:2798] T 35e0aed5125c4e1fa5dce226eade3b78 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 02:51:26.555544 917 raft_consensus.cc:2388] T 438d1d2300e044f39a15586f335935ba P df69f48656b44bb890f76587ee69cc0f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1186fa869eb14812832fa6562531f196 in current term 1: Already voted for candidate 4216a7f853d642b7bc3e5cff458f3aca in this term.
I20250113 02:51:26.556046 775 raft_consensus.cc:2463] T 438d1d2300e044f39a15586f335935ba P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 0.
I20250113 02:51:26.556619 1198 ts_tablet_manager.cc:1905] T 1500e4f9c0664a879503afd3edf14c66 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:26.557756 1669 leader_election.cc:290] T 7f08b72301dc431396cf8cc7790d56d4 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:26.558229 736 tablet_service.cc:1467] Processing CreateTablet for tablet ef6dfe0431ef4a73a957559f60f24756 (DEFAULT_TABLE table=default.Table_9e1b92761ab84f80905c2b4f57738caf [id=bc667764370048d0a6211fa11ac9c55b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:26.558346 752 tablet_service.cc:1467] Processing CreateTablet for tablet acae3ff634104b5d931ece75d1eba5d6 (DEFAULT_TABLE table=default.Table_eaece5fe552b47fb9c0f3d73031f40f4 [id=27d19697fb0f42d09faee637ba85cefd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:26.559767 1192 tablet_bootstrap.cc:492] T 572e4eed1e924381bada27f8389821a7 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:26.559922 1192 ts_tablet_manager.cc:1397] T 572e4eed1e924381bada27f8389821a7 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.044s user 0.003s sys 0.000s
I20250113 02:51:26.560761 1192 raft_consensus.cc:357] T 572e4eed1e924381bada27f8389821a7 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:26.560969 1192 raft_consensus.cc:383] T 572e4eed1e924381bada27f8389821a7 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:26.561101 1192 raft_consensus.cc:738] T 572e4eed1e924381bada27f8389821a7 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:26.561079 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35e0aed5125c4e1fa5dce226eade3b78" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:26.561319 1192 consensus_queue.cc:260] T 572e4eed1e924381bada27f8389821a7 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:26.561429 1438 raft_consensus.cc:2743] T b834f0acf5314bcb9d81c73de1b31842 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:26.561439 1194 ts_tablet_manager.cc:1397] T 1500e4f9c0664a879503afd3edf14c66 P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.206s user 0.026s sys 0.032s
I20250113 02:51:26.561830 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f08b72301dc431396cf8cc7790d56d4" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:26.561908 1192 ts_tablet_manager.cc:1428] T 572e4eed1e924381bada27f8389821a7 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250113 02:51:26.562206 1192 tablet_bootstrap.cc:492] T bcb59573e82743858bd39bde64627d6d P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:26.564123 746 tablet_service.cc:1467] Processing CreateTablet for tablet bd48dbbfe20342569e6f153353f1ef9c (DEFAULT_TABLE table=default.Table_f410165a9c1c45229555f7b89641c3db [id=4496eba45fa043bc91553e1d77c53234]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:26.566941 740 tablet_service.cc:1514] Processing DeleteTablet for tablet b217cb8c09fb41cab4f53e605c37efe8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:26 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:26.567142 742 tablet_service.cc:1467] Processing CreateTablet for tablet fa9052eea538439a9cb4c527e97897a4 (DEFAULT_TABLE table=default.Table_ec65488dc1334a33a56ba81d2bda391c [id=2061c64fe8b545069da97dad04c40637]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:26.567214 711 leader_election.cc:304] T 438d1d2300e044f39a15586f335935ba P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:26.569384 1192 tablet_bootstrap.cc:654] T bcb59573e82743858bd39bde64627d6d P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:26.570885 1198 ts_tablet_manager.cc:1918] T 1500e4f9c0664a879503afd3edf14c66 P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250113 02:51:26.570683 1696 leader_election.cc:290] T 438d1d2300e044f39a15586f335935ba P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:26.574216 740 tablet_service.cc:1467] Processing CreateTablet for tablet c368b5b0c3a940c4b4efb64c946375c3 (DEFAULT_TABLE table=default.Table_5182a1c5c7a2462ba930940ae04b505f [id=6666a614063942c99a78b95d35add037]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:26.574824 749 tablet_service.cc:1467] Processing CreateTablet for tablet 9fa6a1c4f6ef4f98b4de114210af08c5 (DEFAULT_TABLE table=default.Table_6d300fedf96048a5a3bdbc9f36ac0b6c [id=84cbc9ab3d23459988b3681c8c6004da]), partition=RANGE (key) PARTITION UNBOUNDED
W20250113 02:51:26.575335 711 leader_election.cc:343] T 35e0aed5125c4e1fa5dce226eade3b78 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:26.575827 748 tablet_service.cc:1467] Processing CreateTablet for tablet 17fc320eb01f4ac2b9361aeab5b452fc (DEFAULT_TABLE table=default.Table_6431cb4cd5374cbcaceda152cbc19e5b [id=3a485245494241b5a902dc44980c7c8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:26.576355 741 tablet_service.cc:1467] Processing CreateTablet for tablet 5d4912055c5d4c8d8be8b65121f53200 (DEFAULT_TABLE table=default.Table_bbda194bd2cd4df2b7e7d429700e5f3c [id=a578f9aa01d44ba7923d8d93a1facf68]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:26.576761 737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9cf14ed03b3c418c941cd20ed371d7e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:26.576699 739 tablet_service.cc:1467] Processing CreateTablet for tablet 37b7e05f7bbd4a4180cc40e76868469b (DEFAULT_TABLE table=default.Table_9c812746e36d4df38dd75d3a561d7d57 [id=da9f9fb691454444b316e0f782345ee1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:26.577080 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 17fc320eb01f4ac2b9361aeab5b452fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:26.577280 738 tablet_service.cc:1467] Processing CreateTablet for tablet 0953b2f89e35402c9e9e84f3589c23ac (DEFAULT_TABLE table=default.Table_3318768dfa8c4d4bbc513e660dabfcc5 [id=dffb2926dc0a4a7db6878bd4c92f2bcd]), partition=RANGE (key) PARTITION UNBOUNDED
W20250113 02:51:26.577553 709 leader_election.cc:343] T 7f08b72301dc431396cf8cc7790d56d4 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:26.578298 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37b7e05f7bbd4a4180cc40e76868469b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:26.578974 1669 raft_consensus.cc:3054] T 438d1d2300e044f39a15586f335935ba P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:26.580657 755 tablet_service.cc:1467] Processing CreateTablet for tablet 4cbc27d8ac2a46c58bffc33d8af17be2 (DEFAULT_TABLE table=default.Table_8a3a3b6c452a437dae09bdd5ee43c8a0 [id=cd5695fc47644c2cb6c33e932a74550b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:26.581131 751 tablet_service.cc:1467] Processing CreateTablet for tablet e52429bc9c944c75922a1306acd38fea (DEFAULT_TABLE table=default.Table_3203c45c9ef34fcdb7c9ebcafb066412 [id=46f200ae89ac4e00829a5be6414589f0]), partition=RANGE (key) PARTITION UNBOUNDED
W20250113 02:51:26.582311 711 leader_election.cc:343] T 7f08b72301dc431396cf8cc7790d56d4 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:26.584080 736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef6dfe0431ef4a73a957559f60f24756. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:26.584493 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 32cba8f3efc54e78b4df41c15df82eea consensus state has no leader
I20250113 02:51:26.586596 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e52429bc9c944c75922a1306acd38fea. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:26.586813 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4cbc27d8ac2a46c58bffc33d8af17be2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:26.588323 1196 ts_tablet_manager.cc:1918] T 23e9321f08ee4ce7b2252ad5c76c0a08 P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:26.589416 745 tablet_service.cc:1514] Processing DeleteTablet for tablet 569d666c128a45b7a526c13be5090ae3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:26 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:26.590266 1196 log.cc:1198] T 23e9321f08ee4ce7b2252ad5c76c0a08 P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/23e9321f08ee4ce7b2252ad5c76c0a08
I20250113 02:51:26.590649 1196 ts_tablet_manager.cc:1939] T 23e9321f08ee4ce7b2252ad5c76c0a08 P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:26.591259 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet 23e9321f08ee4ce7b2252ad5c76c0a08 (table default.Table_03b7491a47674f7187f4952feb9b44e2 [id=6e014400813c4402891b32bcbe3555e7]) successfully deleted
I20250113 02:51:26.591719 1659 raft_consensus.cc:491] T 35e0aed5125c4e1fa5dce226eade3b78 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 02:51:26.591862 1659 raft_consensus.cc:3054] T 35e0aed5125c4e1fa5dce226eade3b78 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:26.591858 711 leader_election.cc:304] T 7f08b72301dc431396cf8cc7790d56d4 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:26.592622 1696 raft_consensus.cc:2743] T 7f08b72301dc431396cf8cc7790d56d4 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250113 02:51:26.592762 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9cf14ed03b3c418c941cd20ed371d7e2 consensus state has no leader
I20250113 02:51:26.593181 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 14cea6298852460ca8cfb74e321cc124
W20250113 02:51:26.595424 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 6d4d8b5cc8ad4c9e935d8afaf5766e66 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6d4d8b5cc8ad4c9e935d8afaf5766e66 already in progress: creating tablet
I20250113 02:51:26.595688 446 catalog_manager.cc:5618] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095) does not have the latest schema for tablet 4fe54131aafb40ee9d8bc5ea299af193 (table default.Table_d1f76dd334eb4d5890cceb429ed27e9f [id=70089f757a1c45f6b983f216fb167478]). Expected version 3 got 0
W20250113 02:51:26.595965 439 catalog_manager.cc:4839] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 32351d6de1574353a6e8073d333da477 because the tablet was not found. No further retry: Not found: Tablet not found: 32351d6de1574353a6e8073d333da477
I20250113 02:51:26.596097 446 catalog_manager.cc:5618] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095) does not have the latest schema for tablet 7f08b72301dc431396cf8cc7790d56d4 (table default.Table_6de3bf6640dc47f9b977f608c5ec8ef9 [id=bd14ce0e2b0a4a60a63022be2bc80fef]). Expected version 3 got 0
W20250113 02:51:26.596258 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet f2fb1d39c9ca45ccb2da0be75ccf4acf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f2fb1d39c9ca45ccb2da0be75ccf4acf already in progress: creating tablet
W20250113 02:51:26.596474 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet d8bb60fdd06946739b75b255303d9d24 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d8bb60fdd06946739b75b255303d9d24 already in progress: creating tablet
W20250113 02:51:26.596668 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet b217cb8c09fb41cab4f53e605c37efe8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b217cb8c09fb41cab4f53e605c37efe8 already in progress: creating tablet
I20250113 02:51:26.597005 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d4912055c5d4c8d8be8b65121f53200. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:26.598309 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet f2fb1d39c9ca45ccb2da0be75ccf4acf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f2fb1d39c9ca45ccb2da0be75ccf4acf already in progress: creating tablet
I20250113 02:51:26.598699 745 tablet_service.cc:1467] Processing CreateTablet for tablet e57ac440265e4646bfc5d5d9ca4c35d2 (DEFAULT_TABLE table=default.Table_54119752ebc147b8b34429f1c38b71d9 [id=2ab34074f7624b419c1cd36e9a09b4c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:26.599087 884 tablet_service.cc:1514] Processing DeleteTablet for tablet 14cea6298852460ca8cfb74e321cc124 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:26 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:26.599086 750 tablet_service.cc:1467] Processing CreateTablet for tablet 25abb5d0f24e4407907f12caa55c5c8b (DEFAULT_TABLE table=default.Table_9181be1f89094ceea4944414dd995df0 [id=92ac4d0187294a428a7ab739cce23cdd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:26.599884 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fa9052eea538439a9cb4c527e97897a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:26.600044 1194 raft_consensus.cc:357] T 1500e4f9c0664a879503afd3edf14c66 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:26.600257 1194 raft_consensus.cc:383] T 1500e4f9c0664a879503afd3edf14c66 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:26.600397 1194 raft_consensus.cc:738] T 1500e4f9c0664a879503afd3edf14c66 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:26.600615 1194 consensus_queue.cc:260] T 1500e4f9c0664a879503afd3edf14c66 P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:26.601352 1194 ts_tablet_manager.cc:1428] T 1500e4f9c0664a879503afd3edf14c66 P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.040s user 0.002s sys 0.000s
I20250113 02:51:26.608675 918 raft_consensus.cc:1270] T d7920c0544c0466bad6bd8c09aae1a7d P df69f48656b44bb890f76587ee69cc0f [term 1 FOLLOWER]: Refusing update from remote peer 1186fa869eb14812832fa6562531f196: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250113 02:51:26.609310 1194 tablet_bootstrap.cc:492] T 8aac142230e244c0b59c9c2b7a7300ea P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:26.609974 1696 consensus_queue.cc:1035] T d7920c0544c0466bad6bd8c09aae1a7d P 1186fa869eb14812832fa6562531f196 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250113 02:51:26.611838 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0953b2f89e35402c9e9e84f3589c23ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:26.613319 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet 14cea6298852460ca8cfb74e321cc124 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:26 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:26.615211 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e57ac440265e4646bfc5d5d9ca4c35d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:26.615939 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06f36d09bbb1443bafce906907795404" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
I20250113 02:51:26.616485 775 raft_consensus.cc:2463] T 06f36d09bbb1443bafce906907795404 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 0.
I20250113 02:51:26.617601 446 catalog_manager.cc:5618] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587) does not have the latest schema for tablet 572e4eed1e924381bada27f8389821a7 (table default.Table_1d910c3d2b244474a576fc4088301143 [id=75cb0745bbb84e1c883cb13b546ecb11]). Expected version 4 got 0
I20250113 02:51:26.617707 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9fa6a1c4f6ef4f98b4de114210af08c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:26.617674 773 raft_consensus.cc:3054] T 4fe54131aafb40ee9d8bc5ea299af193 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:26.618331 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd48dbbfe20342569e6f153353f1ef9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:26.619450 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b3f012b88c04cb68ed7a1848147b70b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:18 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:26.619803 1062 raft_consensus.cc:3054] T d7920c0544c0466bad6bd8c09aae1a7d P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:26.619899 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet b2a35fd891c645d68d79c61a2c832ba7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:20 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:26.619822 1037 tablet_service.cc:1514] Processing DeleteTablet for tablet 64cda8796dff46ce95c37439395c77ec with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:20 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:26.621369 1196 tablet_replica.cc:331] T b834f0acf5314bcb9d81c73de1b31842 P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:26.622841 1062 raft_consensus.cc:1270] T d7920c0544c0466bad6bd8c09aae1a7d P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Refusing update from remote peer 1186fa869eb14812832fa6562531f196: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250113 02:51:26.619808 1198 log.cc:1198] T 1500e4f9c0664a879503afd3edf14c66 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/1500e4f9c0664a879503afd3edf14c66
I20250113 02:51:26.628930 1196 raft_consensus.cc:2238] T b834f0acf5314bcb9d81c73de1b31842 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:26.629283 1198 ts_tablet_manager.cc:1939] T 1500e4f9c0664a879503afd3edf14c66 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:26.629371 1196 raft_consensus.cc:2267] T b834f0acf5314bcb9d81c73de1b31842 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:26.629495 446 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_5181e98c2c214ebcab54c33615482528" } modify_external_catalogs: true
I20250113 02:51:26.629714 446 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_5181e98c2c214ebcab54c33615482528" } modify_external_catalogs: true
I20250113 02:51:26.630513 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 492) took 1531 ms. Trace:
I20250113 02:51:26.630602 1198 rpcz_store.cc:276] 0113 02:51:25.099192 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:25.114751 (+ 15559us) service_pool.cc:225] Handling call
0113 02:51:26.532576 (+1417825us) ts_tablet_manager.cc:1145] Deleting tablet 1500e4f9c0664a879503afd3edf14c66
0113 02:51:26.532586 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:26.566412 (+ 33826us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:26.570766 (+ 4354us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:26.630459 (+ 59693us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1371353}
I20250113 02:51:26.631072 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 503) took 1217 ms. Trace:
I20250113 02:51:26.631167 1198 rpcz_store.cc:276] 0113 02:51:25.413756 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:25.413824 (+ 68us) service_pool.cc:225] Handling call
0113 02:51:26.630882 (+1217058us) ts_tablet_manager.cc:1145] Deleting tablet d9df314ffaf042ec80180868d7e7bdf7
0113 02:51:26.630891 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:26.631020 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1215420}
I20250113 02:51:26.631292 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 788ab82626424b6e833e31761a15b9b2
I20250113 02:51:26.632442 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet 1500e4f9c0664a879503afd3edf14c66 (table default.Table_d05b7517b14c4168bea99385f1a7d8b6 [id=05cf81a8622149d29f0f4c567a44371c]) successfully deleted
W20250113 02:51:26.632730 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet d9df314ffaf042ec80180868d7e7bdf7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d9df314ffaf042ec80180868d7e7bdf7 already in progress: creating tablet
I20250113 02:51:26.632849 1037 tablet_service.cc:1514] Processing DeleteTablet for tablet 788ab82626424b6e833e31761a15b9b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:26 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:26.634027 1194 tablet_bootstrap.cc:654] T 8aac142230e244c0b59c9c2b7a7300ea P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:26.634896 1196 ts_tablet_manager.cc:1905] T b834f0acf5314bcb9d81c73de1b31842 P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:26.635818 895 tablet_service.cc:1514] Processing DeleteTablet for tablet 788ab82626424b6e833e31761a15b9b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:26 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:26.636592 1696 consensus_queue.cc:1035] T d7920c0544c0466bad6bd8c09aae1a7d P 1186fa869eb14812832fa6562531f196 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.008s
I20250113 02:51:26.636852 1192 tablet_bootstrap.cc:492] T bcb59573e82743858bd39bde64627d6d P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:26.637013 1192 ts_tablet_manager.cc:1397] T bcb59573e82743858bd39bde64627d6d P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.075s user 0.001s sys 0.003s
I20250113 02:51:26.637005 895 tablet_service.cc:1514] Processing DeleteTablet for tablet 1500e4f9c0664a879503afd3edf14c66 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:21 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:26.637073 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 504) took 1217 ms. Trace:
I20250113 02:51:26.637169 1198 rpcz_store.cc:276] 0113 02:51:25.419638 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:25.419687 (+ 49us) service_pool.cc:225] Handling call
0113 02:51:26.636887 (+1217200us) ts_tablet_manager.cc:1145] Deleting tablet d8419fec408c40f6bd46c39de7b3dce9
0113 02:51:26.636899 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:26.637013 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1211519}
I20250113 02:51:26.637682 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 505) took 1190 ms. Trace:
W20250113 02:51:26.637696 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet d8419fec408c40f6bd46c39de7b3dce9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d8419fec408c40f6bd46c39de7b3dce9 already in progress: creating tablet
I20250113 02:51:26.639480 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c368b5b0c3a940c4b4efb64c946375c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:26.645716 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet acae3ff634104b5d931ece75d1eba5d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:26.651768 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 1968c7e9357c4c6e901d334ace0467fb
I20250113 02:51:26.652568 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 3b1910496e0849b193e390a5ef1a6b2b
I20250113 02:51:26.654590 449 catalog_manager.cc:5618] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467) does not have the latest schema for tablet d7920c0544c0466bad6bd8c09aae1a7d (table default.Table_c9a2c2b1c62849babefee677197da844 [id=05755562cfc24370a3e6fb245a573dd4]). Expected version 1 got 0
I20250113 02:51:26.655365 895 tablet_service.cc:1514] Processing DeleteTablet for tablet 1968c7e9357c4c6e901d334ace0467fb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:26 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:26.656536 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 25abb5d0f24e4407907f12caa55c5c8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:26.659122 1701 tablet.cc:1721] T d7920c0544c0466bad6bd8c09aae1a7d P df69f48656b44bb890f76587ee69cc0f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250113 02:51:26.662648 449 catalog_manager.cc:5618] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467) does not have the latest schema for tablet d7920c0544c0466bad6bd8c09aae1a7d (table default.Table_c9a2c2b1c62849babefee677197da844 [id=05755562cfc24370a3e6fb245a573dd4]). Expected version 1 got 0
I20250113 02:51:26.662612 1198 rpcz_store.cc:276] 0113 02:51:25.447550 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:25.447648 (+ 98us) service_pool.cc:225] Handling call
0113 02:51:26.637530 (+1189882us) ts_tablet_manager.cc:1145] Deleting tablet f1e32e09a8244957a99bdfea5812ff31
0113 02:51:26.637538 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:26.637630 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1162915}
I20250113 02:51:26.663194 1198 tablet_replica.cc:331] T 317c6e8013b745c290bf3190dc237ab8 P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:26.637810 1192 raft_consensus.cc:357] T bcb59573e82743858bd39bde64627d6d P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
W20250113 02:51:26.663213 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet f1e32e09a8244957a99bdfea5812ff31 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f1e32e09a8244957a99bdfea5812ff31 already in progress: creating tablet
I20250113 02:51:26.663372 1192 raft_consensus.cc:383] T bcb59573e82743858bd39bde64627d6d P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:26.663424 1198 raft_consensus.cc:2238] T 317c6e8013b745c290bf3190dc237ab8 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:26.663475 1192 raft_consensus.cc:738] T bcb59573e82743858bd39bde64627d6d P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:26.663623 1198 raft_consensus.cc:2267] T 317c6e8013b745c290bf3190dc237ab8 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:26.663751 1192 consensus_queue.cc:260] T bcb59573e82743858bd39bde64627d6d P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:26.664331 1192 ts_tablet_manager.cc:1428] T bcb59573e82743858bd39bde64627d6d P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.027s user 0.000s sys 0.002s
I20250113 02:51:26.664448 1198 ts_tablet_manager.cc:1905] T 317c6e8013b745c290bf3190dc237ab8 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:26.664675 1192 tablet_bootstrap.cc:492] T 5b1076a5a17b4fa495975285616401cf P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:26.667007 1192 tablet_bootstrap.cc:654] T 5b1076a5a17b4fa495975285616401cf P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:26.667531 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_6d300fedf96048a5a3bdbc9f36ac0b6c" } new_table_name: "default.Table_dbec5dd582c846b893ff6250a548bad9" modify_external_catalogs: true
I20250113 02:51:26.668865 896 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b1910496e0849b193e390a5ef1a6b2b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:26 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:26.653946 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b1910496e0849b193e390a5ef1a6b2b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:26 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:26.653756 1037 tablet_service.cc:1514] Processing DeleteTablet for tablet 1968c7e9357c4c6e901d334ace0467fb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:26 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:26.670511 1669 raft_consensus.cc:2743] T 438d1d2300e044f39a15586f335935ba P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:26.685286 744 tablet_service.cc:1514] Processing DeleteTablet for tablet 14cea6298852460ca8cfb74e321cc124 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:26 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:26.686554 1659 raft_consensus.cc:513] T 35e0aed5125c4e1fa5dce226eade3b78 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:26.687992 744 tablet_service.cc:1514] Processing DeleteTablet for tablet 4efa63b8be794e1db4b340f4c0c4cd5a with delete_type TABLET_DATA_DELETED (replaced by tablet eceed0f52c9f46bbaa78ca773a7afde5) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:26.688329 744 tablet_service.cc:1514] Processing DeleteTablet for tablet 35e0aed5125c4e1fa5dce226eade3b78 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:17 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:26.688596 744 tablet_service.cc:1514] Processing DeleteTablet for tablet d5fbf97536a2472fa091168363bf9549 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:18 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:26.691938 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 415a1fa4b2f14cacb43937c8a372849a
I20250113 02:51:26.694972 896 tablet_service.cc:1514] Processing DeleteTablet for tablet 415a1fa4b2f14cacb43937c8a372849a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:26 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:26.695106 1037 tablet_service.cc:1514] Processing DeleteTablet for tablet 415a1fa4b2f14cacb43937c8a372849a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:26 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:26.695590 744 tablet_service.cc:1514] Processing DeleteTablet for tablet 788ab82626424b6e833e31761a15b9b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:26 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:26.695873 1194 tablet_bootstrap.cc:492] T 8aac142230e244c0b59c9c2b7a7300ea P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:26.696116 1194 ts_tablet_manager.cc:1397] T 8aac142230e244c0b59c9c2b7a7300ea P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.087s user 0.005s sys 0.001s
I20250113 02:51:26.696988 1194 raft_consensus.cc:357] T 8aac142230e244c0b59c9c2b7a7300ea P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:26.697261 1194 raft_consensus.cc:383] T 8aac142230e244c0b59c9c2b7a7300ea P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:26.697345 1194 raft_consensus.cc:738] T 8aac142230e244c0b59c9c2b7a7300ea P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:26.697695 1194 consensus_queue.cc:260] T 8aac142230e244c0b59c9c2b7a7300ea P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:26.699643 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_901b9f58fa9941bea7289bfcb72b22bb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:26.700258 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_901b9f58fa9941bea7289bfcb72b22bb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:26.700742 1194 ts_tablet_manager.cc:1428] T 8aac142230e244c0b59c9c2b7a7300ea P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250113 02:51:26.701620 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_13f74f4092644f87b45e1d86f43bba41"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:26.701972 1194 tablet_bootstrap.cc:492] T 35e0aed5125c4e1fa5dce226eade3b78 P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
W20250113 02:51:26.702149 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_13f74f4092644f87b45e1d86f43bba41 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:26.703625 1194 tablet_bootstrap.cc:654] T 35e0aed5125c4e1fa5dce226eade3b78 P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:26.703796 744 tablet_service.cc:1514] Processing DeleteTablet for tablet 1968c7e9357c4c6e901d334ace0467fb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:26 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:26.704340 1195 tablet_replica.cc:331] T 4efa63b8be794e1db4b340f4c0c4cd5a P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:26.704550 1195 raft_consensus.cc:2238] T 4efa63b8be794e1db4b340f4c0c4cd5a P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:26.704727 1195 raft_consensus.cc:2267] T 4efa63b8be794e1db4b340f4c0c4cd5a P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:26.704794 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_4e6b1d81ffef42c3a7f83bf18c1fbc48"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:26.705258 1195 ts_tablet_manager.cc:1905] T 4efa63b8be794e1db4b340f4c0c4cd5a P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250113 02:51:26.705274 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4e6b1d81ffef42c3a7f83bf18c1fbc48 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:26.707001 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35e0aed5125c4e1fa5dce226eade3b78" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca"
I20250113 02:51:26.708550 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_657257f5b9604a1994dc736b6e69330c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:26.709059 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_657257f5b9604a1994dc736b6e69330c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250113 02:51:26.710103 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 569d666c128a45b7a526c13be5090ae3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 569d666c128a45b7a526c13be5090ae3 already in progress: creating tablet
I20250113 02:51:26.711026 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35e0aed5125c4e1fa5dce226eade3b78" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f"
I20250113 02:51:26.711681 995 leader_election.cc:304] T 06f36d09bbb1443bafce906907795404 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters: df69f48656b44bb890f76587ee69cc0f
W20250113 02:51:26.714211 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 14cea6298852460ca8cfb74e321cc124 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 14cea6298852460ca8cfb74e321cc124 already in progress: creating tablet
I20250113 02:51:26.714758 1659 leader_election.cc:290] T 35e0aed5125c4e1fa5dce226eade3b78 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 election: Requested vote from peers 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:26.715018 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b1910496e0849b193e390a5ef1a6b2b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:26 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:26.716063 1198 ts_tablet_manager.cc:1918] T 317c6e8013b745c290bf3190dc237ab8 P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250113 02:51:26.716207 1634 raft_consensus.cc:2798] T 06f36d09bbb1443bafce906907795404 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 02:51:26.716317 1634 raft_consensus.cc:491] T 06f36d09bbb1443bafce906907795404 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 02:51:26.716451 1634 raft_consensus.cc:3054] T 06f36d09bbb1443bafce906907795404 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Advancing to term 1
W20250113 02:51:26.716153 711 leader_election.cc:343] T 35e0aed5125c4e1fa5dce226eade3b78 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:26.717878 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_3f69e970777d4d2f84a8275f667a875c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:26.718329 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3f69e970777d4d2f84a8275f667a875c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:26.720628 1634 raft_consensus.cc:513] T 06f36d09bbb1443bafce906907795404 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:26.716104 1059 raft_consensus.cc:3054] T 35e0aed5125c4e1fa5dce226eade3b78 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:26.721354 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_4deab2d4aa9e4d6f8e165675d04c119a" } new_table_name: "default.Table_f8a6812c7fb94ea69cd9bf775b914ea6" modify_external_catalogs: true
I20250113 02:51:26.722569 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_243aff1bbfc6454b92f66035f64a7b84" } new_table_name: "default.Table_3b76cf9fac334ebf9b1ecb9f8d07ea74" modify_external_catalogs: true
I20250113 02:51:26.724463 1696 consensus_queue.cc:1035] T d304b65eba8c40c6922333506a60540c P 1186fa869eb14812832fa6562531f196 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.011s
I20250113 02:51:26.724879 447 catalog_manager.cc:5618] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095) does not have the latest schema for tablet d7920c0544c0466bad6bd8c09aae1a7d (table default.Table_c9a2c2b1c62849babefee677197da844 [id=05755562cfc24370a3e6fb245a573dd4]). Expected version 1 got 0
I20250113 02:51:26.724967 1634 leader_election.cc:290] T 06f36d09bbb1443bafce906907795404 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Requested vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
I20250113 02:51:26.725785 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06f36d09bbb1443bafce906907795404" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196"
I20250113 02:51:26.726287 1059 raft_consensus.cc:2463] T 35e0aed5125c4e1fa5dce226eade3b78 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1186fa869eb14812832fa6562531f196 in term 1.
I20250113 02:51:26.726774 744 tablet_service.cc:1514] Processing DeleteTablet for tablet 415a1fa4b2f14cacb43937c8a372849a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:26 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:26.727063 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06f36d09bbb1443bafce906907795404" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f"
W20250113 02:51:26.727815 998 leader_election.cc:343] T 06f36d09bbb1443bafce906907795404 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:26.728612 919 raft_consensus.cc:1270] T 438d1d2300e044f39a15586f335935ba P df69f48656b44bb890f76587ee69cc0f [term 1 FOLLOWER]: Refusing update from remote peer 4216a7f853d642b7bc3e5cff458f3aca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250113 02:51:26.728966 775 raft_consensus.cc:3054] T 06f36d09bbb1443bafce906907795404 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:26.729730 1192 tablet_bootstrap.cc:492] T 5b1076a5a17b4fa495975285616401cf P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:26.729863 1192 ts_tablet_manager.cc:1397] T 5b1076a5a17b4fa495975285616401cf P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.065s user 0.007s sys 0.003s
I20250113 02:51:26.730573 772 raft_consensus.cc:1270] T 438d1d2300e044f39a15586f335935ba P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Refusing update from remote peer 4216a7f853d642b7bc3e5cff458f3aca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250113 02:51:26.730921 1192 raft_consensus.cc:357] T 5b1076a5a17b4fa495975285616401cf P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:26.731125 1192 raft_consensus.cc:383] T 5b1076a5a17b4fa495975285616401cf P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:26.731207 1192 raft_consensus.cc:738] T 5b1076a5a17b4fa495975285616401cf P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:26.731444 1192 consensus_queue.cc:260] T 5b1076a5a17b4fa495975285616401cf P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:26.732172 1192 ts_tablet_manager.cc:1428] T 5b1076a5a17b4fa495975285616401cf P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250113 02:51:26.732453 1192 tablet_bootstrap.cc:492] T 7f08b72301dc431396cf8cc7790d56d4 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:26.733382 1634 consensus_queue.cc:1035] T 438d1d2300e044f39a15586f335935ba P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 02:51:26.734100 1634 consensus_queue.cc:1035] T 438d1d2300e044f39a15586f335935ba P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 02:51:26.740530 1192 tablet_bootstrap.cc:654] T 7f08b72301dc431396cf8cc7790d56d4 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:26.746258 775 raft_consensus.cc:2463] T 06f36d09bbb1443bafce906907795404 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 1.
I20250113 02:51:26.751703 1196 ts_tablet_manager.cc:1918] T b834f0acf5314bcb9d81c73de1b31842 P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:26.751946 1196 log.cc:1198] T b834f0acf5314bcb9d81c73de1b31842 P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/b834f0acf5314bcb9d81c73de1b31842
I20250113 02:51:26.752516 1196 ts_tablet_manager.cc:1939] T b834f0acf5314bcb9d81c73de1b31842 P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
W20250113 02:51:26.752662 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9fa6a1c4f6ef4f98b4de114210af08c5 consensus state has no leader
I20250113 02:51:26.753197 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_1c82125f87e542e0932f372cd1d99dab"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:26.753749 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1c82125f87e542e0932f372cd1d99dab in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:26.754859 453 catalog_manager.cc:6071] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: default.Table_c9a2c2b1c62849babefee677197da844 [id=05755562cfc24370a3e6fb245a573dd4] alter complete (version 1)
I20250113 02:51:26.755322 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_e6bf5c2eb70745c69afb72092d93bc22"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:26.755815 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e6bf5c2eb70745c69afb72092d93bc22 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:26.757377 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet b834f0acf5314bcb9d81c73de1b31842 (table default.Table_24d9af3d0b0740d69c791fc96916ef0a [id=124669ccadaf49e19d2660b1495f9647]) successfully deleted
I20250113 02:51:26.758117 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_e39cea4b271447f1a10c258d62b5a217"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:26.758556 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e39cea4b271447f1a10c258d62b5a217 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:26.761487 709 leader_election.cc:304] T 35e0aed5125c4e1fa5dce226eade3b78 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters: df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:26.762584 995 leader_election.cc:304] T 06f36d09bbb1443bafce906907795404 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters: df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:26.762866 1634 raft_consensus.cc:2798] T 06f36d09bbb1443bafce906907795404 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Leader election won for term 1
I20250113 02:51:26.763212 1634 raft_consensus.cc:695] T 06f36d09bbb1443bafce906907795404 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: Becoming Leader. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Running, Role: LEADER
I20250113 02:51:26.763548 1634 consensus_queue.cc:237] T 06f36d09bbb1443bafce906907795404 P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:26.767180 1191 tablet_bootstrap.cc:492] T 0fb75c551b4c424186ef2021db742e8e P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:26.767376 1191 ts_tablet_manager.cc:1397] T 0fb75c551b4c424186ef2021db742e8e P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.442s user 0.002s sys 0.015s
I20250113 02:51:26.768179 1191 raft_consensus.cc:357] T 0fb75c551b4c424186ef2021db742e8e P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:26.768388 1191 raft_consensus.cc:383] T 0fb75c551b4c424186ef2021db742e8e P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:26.768491 1191 raft_consensus.cc:738] T 0fb75c551b4c424186ef2021db742e8e P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:26.768710 1191 consensus_queue.cc:260] T 0fb75c551b4c424186ef2021db742e8e P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:26.769294 1191 ts_tablet_manager.cc:1428] T 0fb75c551b4c424186ef2021db742e8e P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250113 02:51:26.769644 1191 tablet_bootstrap.cc:492] T a6055d77d168495d9b756a1a56c1a920 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:26.771925 1191 tablet_bootstrap.cc:654] T a6055d77d168495d9b756a1a56c1a920 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:26.772359 1659 consensus_queue.cc:1035] T d304b65eba8c40c6922333506a60540c P 1186fa869eb14812832fa6562531f196 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.011s
I20250113 02:51:26.772563 1669 raft_consensus.cc:2798] T 35e0aed5125c4e1fa5dce226eade3b78 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Leader election won for term 1
I20250113 02:51:26.773463 1669 raft_consensus.cc:695] T 35e0aed5125c4e1fa5dce226eade3b78 P 1186fa869eb14812832fa6562531f196 [term 1 LEADER]: Becoming Leader. State: Replica: 1186fa869eb14812832fa6562531f196, State: Running, Role: LEADER
I20250113 02:51:26.773778 1669 consensus_queue.cc:237] T 35e0aed5125c4e1fa5dce226eade3b78 P 1186fa869eb14812832fa6562531f196 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:26.781482 1192 tablet_bootstrap.cc:492] T 7f08b72301dc431396cf8cc7790d56d4 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:26.781694 1192 ts_tablet_manager.cc:1397] T 7f08b72301dc431396cf8cc7790d56d4 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.049s user 0.004s sys 0.003s
I20250113 02:51:26.785043 1192 raft_consensus.cc:357] T 7f08b72301dc431396cf8cc7790d56d4 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:26.785334 1192 raft_consensus.cc:383] T 7f08b72301dc431396cf8cc7790d56d4 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:26.785427 1192 raft_consensus.cc:738] T 7f08b72301dc431396cf8cc7790d56d4 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:26.792869 1192 consensus_queue.cc:260] T 7f08b72301dc431396cf8cc7790d56d4 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:26.716234 1198 log.cc:1198] T 317c6e8013b745c290bf3190dc237ab8 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/317c6e8013b745c290bf3190dc237ab8
I20250113 02:51:26.794916 1198 ts_tablet_manager.cc:1939] T 317c6e8013b745c290bf3190dc237ab8 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:26.798821 1194 tablet_bootstrap.cc:492] T 35e0aed5125c4e1fa5dce226eade3b78 P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:26.787227 1712 tablet.cc:1721] T d7920c0544c0466bad6bd8c09aae1a7d P 1186fa869eb14812832fa6562531f196: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250113 02:51:26.801239 1194 ts_tablet_manager.cc:1397] T 35e0aed5125c4e1fa5dce226eade3b78 P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.099s user 0.004s sys 0.000s
I20250113 02:51:26.799343 1196 tablet_replica.cc:331] T 32351d6de1574353a6e8073d333da477 P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:26.802592 1196 raft_consensus.cc:2238] T 32351d6de1574353a6e8073d333da477 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:26.805948 448 catalog_manager.cc:5618] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587) does not have the latest schema for tablet d7920c0544c0466bad6bd8c09aae1a7d (table default.Table_c9a2c2b1c62849babefee677197da844 [id=05755562cfc24370a3e6fb245a573dd4]). Expected version 1 got 0
I20250113 02:51:26.806743 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_ec0ab13bb2604f4da00a662572ae1734"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:26.807328 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ec0ab13bb2604f4da00a662572ae1734 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:26.807462 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_6716ec8ca07b476fbaa9480c4d821568" } new_table_name: "default.Table_459ff6639996470cb94e898c9ef4880e" modify_external_catalogs: true
I20250113 02:51:26.808404 454 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_8f20c6ca51ab41a58970945c7946a768" } new_table_name: "default.Table_c2be66cb47964190b4a1a67e3a9f6787" modify_external_catalogs: true
I20250113 02:51:26.808749 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_82a014a9f48c43b4b14d63522bac7124" } modify_external_catalogs: true
I20250113 02:51:26.808928 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_82a014a9f48c43b4b14d63522bac7124" } modify_external_catalogs: true
I20250113 02:51:26.813508 755 tablet_service.cc:1467] Processing CreateTablet for tablet cd21f10d74c946a2872a84cadccf7950 (DEFAULT_TABLE table=default.Table_4d5b54216098465fb88c7b9d8b605344 [id=e992f40fef2e466e9a49b8d3307e2a1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:26.813712 450 master_service.cc:946] ReplaceTablet: received request to replace tablet c826a0b8327f44f9a00e5afe3a36cd8a from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:26.813910 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cd21f10d74c946a2872a84cadccf7950. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:26.814317 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1a63f485caf54068abd6b50d7de3d14a consensus state has no leader
W20250113 02:51:26.814689 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e8ab4516789e4585a101c8766d338af3 consensus state has no leader
I20250113 02:51:26.816272 736 tablet_service.cc:1467] Processing CreateTablet for tablet fc9189a9d1e4418e9f9205386f473232 (DEFAULT_TABLE table=default.Table_3c89afa5af12403b9a48046ee6c81da3 [id=6d2aefcd0eac438f8dc6defb8cb8947e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:26.812466 887 tablet_service.cc:1467] Processing CreateTablet for tablet 13cc9b4d2d6e41b781a342f6060c6c70 (DEFAULT_TABLE table=default.Table_f11af4ff436842b8800e70008fb2c7b5 [id=9aa3eaef86b940ce928053d5866fdd95]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:26.812618 884 tablet_service.cc:1467] Processing CreateTablet for tablet cd21f10d74c946a2872a84cadccf7950 (DEFAULT_TABLE table=default.Table_4d5b54216098465fb88c7b9d8b605344 [id=e992f40fef2e466e9a49b8d3307e2a1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:26.812985 895 tablet_service.cc:1467] Processing CreateTablet for tablet c4b7196ef5d54459921977e81a2432db (DEFAULT_TABLE table=default.Table_632d3a63935c4ff2ba48037a311708c4 [id=1acc8aa6d6404f58a124530a4445521b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:26.813261 880 tablet_service.cc:1467] Processing CreateTablet for tablet df6299f38ffa4c2f9d6c8a5e5285905f (DEFAULT_TABLE table=default.Table_67eab203697544008a476488a93df8a9 [id=577c95b7ff7a474fb345c08b733b4cb5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:26.816632 896 tablet_service.cc:1467] Processing CreateTablet for tablet fc9189a9d1e4418e9f9205386f473232 (DEFAULT_TABLE table=default.Table_3c89afa5af12403b9a48046ee6c81da3 [id=6d2aefcd0eac438f8dc6defb8cb8947e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:26.819007 1196 raft_consensus.cc:2267] T 32351d6de1574353a6e8073d333da477 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:26.822609 739 tablet_service.cc:1467] Processing CreateTablet for tablet c4b7196ef5d54459921977e81a2432db (DEFAULT_TABLE table=default.Table_632d3a63935c4ff2ba48037a311708c4 [id=1acc8aa6d6404f58a124530a4445521b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:26.823400 736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc9189a9d1e4418e9f9205386f473232. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:26.824196 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c4b7196ef5d54459921977e81a2432db. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:26.824808 741 tablet_service.cc:1467] Processing CreateTablet for tablet 13cc9b4d2d6e41b781a342f6060c6c70 (DEFAULT_TABLE table=default.Table_f11af4ff436842b8800e70008fb2c7b5 [id=9aa3eaef86b940ce928053d5866fdd95]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:26.825603 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 13cc9b4d2d6e41b781a342f6060c6c70. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:26.827538 748 tablet_service.cc:1467] Processing CreateTablet for tablet df6299f38ffa4c2f9d6c8a5e5285905f (DEFAULT_TABLE table=default.Table_67eab203697544008a476488a93df8a9 [id=577c95b7ff7a474fb345c08b733b4cb5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:26.827798 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 506) took 1370 ms. Trace:
I20250113 02:51:26.827999 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df6299f38ffa4c2f9d6c8a5e5285905f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:26.827992 1198 rpcz_store.cc:276] 0113 02:51:25.457532 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:25.457590 (+ 58us) service_pool.cc:225] Handling call
0113 02:51:26.663099 (+1205509us) ts_tablet_manager.cc:1145] Deleting tablet 317c6e8013b745c290bf3190dc237ab8
0113 02:51:26.663112 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:26.680006 (+ 16894us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:26.683172 (+ 3166us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:26.827745 (+144573us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":31897,"tablet-delete.queue_time_us":1079361}
I20250113 02:51:26.828719 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 522) took 1163 ms. Trace:
I20250113 02:51:26.833262 455 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_02655512f28b47429ebb70e1676844ea" } modify_external_catalogs: true
I20250113 02:51:26.833490 455 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_02655512f28b47429ebb70e1676844ea" } modify_external_catalogs: true
I20250113 02:51:26.834861 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_d43add0d74a8498f835de5cd0acf0a25"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:26.835479 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d43add0d74a8498f835de5cd0acf0a25 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:26.835888 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 13cc9b4d2d6e41b781a342f6060c6c70. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:26.831130 1674 raft_consensus.cc:491] T 64cda8796dff46ce95c37439395c77ec P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:26.840457 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df6299f38ffa4c2f9d6c8a5e5285905f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:26.844123 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c4b7196ef5d54459921977e81a2432db. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:26.848132 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cd21f10d74c946a2872a84cadccf7950. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:26.851438 1192 ts_tablet_manager.cc:1428] T 7f08b72301dc431396cf8cc7790d56d4 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.048s user 0.001s sys 0.001s
I20250113 02:51:26.856418 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_4369a143c72643b084df58d25314c6dd" } modify_external_catalogs: true
I20250113 02:51:26.856657 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_4369a143c72643b084df58d25314c6dd" } modify_external_catalogs: true
I20250113 02:51:26.857079 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc9189a9d1e4418e9f9205386f473232. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:26.859436 1195 ts_tablet_manager.cc:1918] T 4efa63b8be794e1db4b340f4c0c4cd5a P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:26.859617 1195 log.cc:1198] T 4efa63b8be794e1db4b340f4c0c4cd5a P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/4efa63b8be794e1db4b340f4c0c4cd5a
I20250113 02:51:26.860145 1195 ts_tablet_manager.cc:1939] T 4efa63b8be794e1db4b340f4c0c4cd5a P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:26.861549 1195 tablet_replica.cc:331] T 35e0aed5125c4e1fa5dce226eade3b78 P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:26.868865 1712 tablet.cc:1721] T 4fe54131aafb40ee9d8bc5ea299af193 P 1186fa869eb14812832fa6562531f196: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250113 02:51:26.870087 1674 raft_consensus.cc:513] T 64cda8796dff46ce95c37439395c77ec P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:26.870846 1674 leader_election.cc:290] T 64cda8796dff46ce95c37439395c77ec P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
I20250113 02:51:26.871795 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64cda8796dff46ce95c37439395c77ec" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:26.873394 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64cda8796dff46ce95c37439395c77ec" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
W20250113 02:51:26.876035 998 leader_election.cc:343] T 64cda8796dff46ce95c37439395c77ec P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
W20250113 02:51:26.876924 995 leader_election.cc:343] T 64cda8796dff46ce95c37439395c77ec P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:26.877130 995 leader_election.cc:304] T 64cda8796dff46ce95c37439395c77ec P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4216a7f853d642b7bc3e5cff458f3aca; no voters: 1186fa869eb14812832fa6562531f196, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:26.877473 1674 raft_consensus.cc:2743] T 64cda8796dff46ce95c37439395c77ec P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:26.877883 1192 tablet_bootstrap.cc:492] T 0fb75c551b4c424186ef2021db742e8e P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:26.879736 1192 tablet_bootstrap.cc:654] T 0fb75c551b4c424186ef2021db742e8e P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:26.880774 1198 rpcz_store.cc:276] 0113 02:51:25.665321 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:25.667056 (+ 1735us) service_pool.cc:225] Handling call
0113 02:51:26.828525 (+1161469us) ts_tablet_manager.cc:1145] Deleting tablet 943d3a125ac5406a83f3c8d80e073b6e
0113 02:51:26.828540 (+ 15us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:26.828683 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1157122}
I20250113 02:51:26.881206 1198 tablet_replica.cc:331] T d3b695ac20ba4205b2d8732db4740531 P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:26.881459 1198 raft_consensus.cc:2238] T d3b695ac20ba4205b2d8732db4740531 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:26.881614 1196 ts_tablet_manager.cc:1905] T 32351d6de1574353a6e8073d333da477 P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:26.881675 1198 pending_rounds.cc:62] T d3b695ac20ba4205b2d8732db4740531 P 4216a7f853d642b7bc3e5cff458f3aca: Trying to abort 1 pending ops.
I20250113 02:51:26.881807 1198 pending_rounds.cc:69] T d3b695ac20ba4205b2d8732db4740531 P 4216a7f853d642b7bc3e5cff458f3aca: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113673460739751936 op_type: NO_OP noop_request { }
I20250113 02:51:26.882037 1198 raft_consensus.cc:2883] T d3b695ac20ba4205b2d8732db4740531 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20250113 02:51:26.882231 1198 raft_consensus.cc:2267] T d3b695ac20ba4205b2d8732db4740531 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:26.882998 1198 ts_tablet_manager.cc:1905] T d3b695ac20ba4205b2d8732db4740531 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:26.886413 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet 4efa63b8be794e1db4b340f4c0c4cd5a (table default.Table_6a0195b054a043269e133f6ed8dd5979 [id=4b2bd999368c4b628c70cb3ee8c2b1f3]) successfully deleted
I20250113 02:51:26.889278 1198 ts_tablet_manager.cc:1918] T d3b695ac20ba4205b2d8732db4740531 P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250113 02:51:26.897498 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet 317c6e8013b745c290bf3190dc237ab8 (table default.Table_13fd4df9c207466cb85cd8ed9062722c [id=6bfb67a96b2b47849f7fa12c57a2f4a1]) successfully deleted
W20250113 02:51:26.897826 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 943d3a125ac5406a83f3c8d80e073b6e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 943d3a125ac5406a83f3c8d80e073b6e already in progress: creating tablet
I20250113 02:51:26.914103 1729 tablet.cc:1716] T 4fe54131aafb40ee9d8bc5ea299af193 P 1186fa869eb14812832fa6562531f196: Skipping requested alter to schema version 3, tablet already version 3
I20250113 02:51:26.918556 1195 raft_consensus.cc:2238] T 35e0aed5125c4e1fa5dce226eade3b78 P 1186fa869eb14812832fa6562531f196 [term 1 LEADER]: Raft consensus shutting down.
I20250113 02:51:26.919274 1195 pending_rounds.cc:62] T 35e0aed5125c4e1fa5dce226eade3b78 P 1186fa869eb14812832fa6562531f196: Trying to abort 1 pending ops.
I20250113 02:51:26.919461 1195 pending_rounds.cc:69] T 35e0aed5125c4e1fa5dce226eade3b78 P 1186fa869eb14812832fa6562531f196: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113673469028429824 op_type: NO_OP noop_request { }
I20250113 02:51:26.919639 1195 raft_consensus.cc:2883] T 35e0aed5125c4e1fa5dce226eade3b78 P 1186fa869eb14812832fa6562531f196 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250113 02:51:26.919951 1195 raft_consensus.cc:2267] T 35e0aed5125c4e1fa5dce226eade3b78 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:26.921348 1195 ts_tablet_manager.cc:1905] T 35e0aed5125c4e1fa5dce226eade3b78 P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:26.921466 1191 tablet_bootstrap.cc:492] T a6055d77d168495d9b756a1a56c1a920 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:26.921712 1191 ts_tablet_manager.cc:1397] T a6055d77d168495d9b756a1a56c1a920 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.152s user 0.002s sys 0.035s
I20250113 02:51:26.926465 447 catalog_manager.cc:5618] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095) does not have the latest schema for tablet 438d1d2300e044f39a15586f335935ba (table default.Table_d7d1f5a34e5f4c05b474b0b61f912cd9 [id=2c7925ec0d8d4cd7b8761268d3962f15]). Expected version 3 got 0
I20250113 02:51:26.928221 1191 raft_consensus.cc:357] T a6055d77d168495d9b756a1a56c1a920 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:26.928548 1191 raft_consensus.cc:383] T a6055d77d168495d9b756a1a56c1a920 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:26.928732 1191 raft_consensus.cc:738] T a6055d77d168495d9b756a1a56c1a920 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:26.929140 1191 consensus_queue.cc:260] T a6055d77d168495d9b756a1a56c1a920 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:26.929350 449 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_3203c45c9ef34fcdb7c9ebcafb066412" } modify_external_catalogs: true
I20250113 02:51:26.929531 449 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_3203c45c9ef34fcdb7c9ebcafb066412" } modify_external_catalogs: true
I20250113 02:51:26.930639 1191 ts_tablet_manager.cc:1428] T a6055d77d168495d9b756a1a56c1a920 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.009s user 0.001s sys 0.000s
I20250113 02:51:26.927903 447 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_26a65a837e114f288e826d343b8f78a1" } modify_external_catalogs: true
I20250113 02:51:26.932921 447 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_26a65a837e114f288e826d343b8f78a1" } modify_external_catalogs: true
I20250113 02:51:26.938609 755 tablet_service.cc:1514] Processing DeleteTablet for tablet 0fb75c551b4c424186ef2021db742e8e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:20 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:26.939604 1191 tablet_bootstrap.cc:492] T bcb59573e82743858bd39bde64627d6d P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
W20250113 02:51:26.940539 454 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c508724edea147e5bf6e20bca2f50397 consensus state has no leader
I20250113 02:51:26.941668 1191 tablet_bootstrap.cc:654] T bcb59573e82743858bd39bde64627d6d P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:26.943048 1729 tablet.cc:1721] T 438d1d2300e044f39a15586f335935ba P 1186fa869eb14812832fa6562531f196: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
W20250113 02:51:26.943600 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ba8980ad3a364c888a825affbcbc9b1f consensus state has no leader
I20250113 02:51:26.944798 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 6a4c6dc08df744e6856bdf99ab65c575
I20250113 02:51:26.946447 739 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a4c6dc08df744e6856bdf99ab65c575 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:26 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:26.948128 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 8b9d6b99ff8e4d4d8131c0618a9ddc1a
I20250113 02:51:26.949458 739 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b9d6b99ff8e4d4d8131c0618a9ddc1a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:26 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:26.953296 451 master_service.cc:946] ReplaceTablet: received request to replace tablet 0fe5421dc3ea4be684cab084f13d8df0 from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:26.956801 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet e52429bc9c944c75922a1306acd38fea
I20250113 02:51:26.957293 896 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a4c6dc08df744e6856bdf99ab65c575 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:26 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:26.957770 880 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b9d6b99ff8e4d4d8131c0618a9ddc1a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:26 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:26.958117 739 tablet_service.cc:1514] Processing DeleteTablet for tablet e52429bc9c944c75922a1306acd38fea with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:26 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:26.958252 1194 raft_consensus.cc:357] T 35e0aed5125c4e1fa5dce226eade3b78 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:26.961151 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_5be7dd3cb3674e39a393c9276c01ae4c" } new_table_name: "default.Table_12a9dbdb62784359a7d952862eea0677" modify_external_catalogs: true
I20250113 02:51:26.962276 755 tablet_service.cc:1514] Processing DeleteTablet for tablet a6055d77d168495d9b756a1a56c1a920 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:19 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:26.964368 447 catalog_manager.cc:6071] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: default.Table_d7d1f5a34e5f4c05b474b0b61f912cd9 [id=2c7925ec0d8d4cd7b8761268d3962f15] alter complete (version 3)
I20250113 02:51:26.964514 1194 raft_consensus.cc:383] T 35e0aed5125c4e1fa5dce226eade3b78 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:26.968036 447 catalog_manager.cc:5618] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095) does not have the latest schema for tablet d304b65eba8c40c6922333506a60540c (table default.Table_e03d954cc20048e4b0d598282689643d [id=e62f8e4c51b04729903b1943275c5f7f]). Expected version 1 got 0
I20250113 02:51:26.968899 1195 ts_tablet_manager.cc:1918] T 35e0aed5125c4e1fa5dce226eade3b78 P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250113 02:51:26.969038 1195 log.cc:1198] T 35e0aed5125c4e1fa5dce226eade3b78 P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/35e0aed5125c4e1fa5dce226eade3b78
I20250113 02:51:26.897038 1728 tablet.cc:1721] T d7920c0544c0466bad6bd8c09aae1a7d P 4216a7f853d642b7bc3e5cff458f3aca: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250113 02:51:26.969954 1198 log.cc:1198] T d3b695ac20ba4205b2d8732db4740531 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/d3b695ac20ba4205b2d8732db4740531
I20250113 02:51:26.899233 1037 tablet_service.cc:1467] Processing CreateTablet for tablet fc9189a9d1e4418e9f9205386f473232 (DEFAULT_TABLE table=default.Table_3c89afa5af12403b9a48046ee6c81da3 [id=6d2aefcd0eac438f8dc6defb8cb8947e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:26.970474 1198 ts_tablet_manager.cc:1939] T d3b695ac20ba4205b2d8732db4740531 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:26.970747 1037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc9189a9d1e4418e9f9205386f473232. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:26.971381 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 521) took 1306 ms. Trace:
I20250113 02:51:26.971535 453 catalog_manager.cc:5618] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587) does not have the latest schema for tablet 5b1076a5a17b4fa495975285616401cf (table default.Table_f5459a21934542d9b62792bcd109f836 [id=bc2aa39c74b04c7ca71a6d0acd780853]). Expected version 2 got 0
I20250113 02:51:26.971506 1198 rpcz_store.cc:276] 0113 02:51:25.665251 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:25.667035 (+ 1784us) service_pool.cc:225] Handling call
0113 02:51:26.881139 (+1214104us) ts_tablet_manager.cc:1145] Deleting tablet d3b695ac20ba4205b2d8732db4740531
0113 02:51:26.881150 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:26.886468 (+ 5318us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:26.889171 (+ 2703us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:26.971338 (+ 82167us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1202746}
I20250113 02:51:26.972010 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 535) took 1048 ms. Trace:
I20250113 02:51:26.971843 453 catalog_manager.cc:5526] T 06f36d09bbb1443bafce906907795404 P 4216a7f853d642b7bc3e5cff458f3aca reported cstate change: term changed from 0 to 1, leader changed from <none> to 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3). New cstate: current_term: 1 leader_uuid: "4216a7f853d642b7bc3e5cff458f3aca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } health_report { overall_health: UNKNOWN } } }
I20250113 02:51:26.972128 1198 rpcz_store.cc:276] 0113 02:51:25.923385 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:25.925316 (+ 1931us) service_pool.cc:225] Handling call
0113 02:51:26.971809 (+1046493us) ts_tablet_manager.cc:1145] Deleting tablet b0b267b0d9c54ba496c89f6cb287dffd
0113 02:51:26.971819 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:26.971948 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1046270}
I20250113 02:51:26.972337 453 catalog_manager.cc:5618] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587) does not have the latest schema for tablet 06f36d09bbb1443bafce906907795404 (table default.Table_2a627a757ab0406686564083ecc36c07 [id=423021f304964246bd29e4daaf470154]). Expected version 2 got 0
I20250113 02:51:26.972551 453 catalog_manager.cc:5618] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587) does not have the latest schema for tablet 7f08b72301dc431396cf8cc7790d56d4 (table default.Table_6de3bf6640dc47f9b977f608c5ec8ef9 [id=bd14ce0e2b0a4a60a63022be2bc80fef]). Expected version 3 got 0
I20250113 02:51:26.972546 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 536) took 1047 ms. Trace:
I20250113 02:51:26.972721 453 catalog_manager.cc:5618] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587) does not have the latest schema for tablet d7920c0544c0466bad6bd8c09aae1a7d (table default.Table_c9a2c2b1c62849babefee677197da844 [id=05755562cfc24370a3e6fb245a573dd4]). Expected version 1 got 0
I20250113 02:51:26.972678 1198 rpcz_store.cc:276] 0113 02:51:25.924829 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:25.925697 (+ 868us) service_pool.cc:225] Handling call
0113 02:51:26.972351 (+1046654us) ts_tablet_manager.cc:1145] Deleting tablet 77cf8b0c824f401aba1880eb7fd1807b
0113 02:51:26.972369 (+ 18us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:26.972491 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1046454}
I20250113 02:51:26.972872 453 catalog_manager.cc:5618] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587) does not have the latest schema for tablet d304b65eba8c40c6922333506a60540c (table default.Table_e03d954cc20048e4b0d598282689643d [id=e62f8e4c51b04729903b1943275c5f7f]). Expected version 1 got 0
I20250113 02:51:26.973019 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 534) took 1052 ms. Trace:
I20250113 02:51:26.973126 1198 rpcz_store.cc:276] 0113 02:51:25.920213 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:25.925955 (+ 5742us) service_pool.cc:225] Handling call
0113 02:51:26.972886 (+1046931us) ts_tablet_manager.cc:1145] Deleting tablet 8412a588d0454ec78025e7ec30c38104
0113 02:51:26.972895 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:26.972971 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1046735}
I20250113 02:51:26.973390 1198 tablet_replica.cc:331] T a6055d77d168495d9b756a1a56c1a920 P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:26.973595 1198 raft_consensus.cc:2238] T a6055d77d168495d9b756a1a56c1a920 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:26.973763 1198 raft_consensus.cc:2267] T a6055d77d168495d9b756a1a56c1a920 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:26.974280 1198 ts_tablet_manager.cc:1905] T a6055d77d168495d9b756a1a56c1a920 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:26.902208 1025 tablet_service.cc:1467] Processing CreateTablet for tablet c4b7196ef5d54459921977e81a2432db (DEFAULT_TABLE table=default.Table_632d3a63935c4ff2ba48037a311708c4 [id=1acc8aa6d6404f58a124530a4445521b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:26.975904 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c4b7196ef5d54459921977e81a2432db. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:26.977752 1195 ts_tablet_manager.cc:1939] T 35e0aed5125c4e1fa5dce226eade3b78 P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
W20250113 02:51:26.978365 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7fa8c252d6824373a9b9e7b615d14725 consensus state has no leader
I20250113 02:51:26.978745 1195 tablet_replica.cc:331] T d5fbf97536a2472fa091168363bf9549 P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:26.978962 1195 raft_consensus.cc:2238] T d5fbf97536a2472fa091168363bf9549 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:26.979125 1195 raft_consensus.cc:2267] T d5fbf97536a2472fa091168363bf9549 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:26.982321 739 tablet_service.cc:1467] Processing CreateTablet for tablet f4af42b9bae44e3fb009310212cec587 (DEFAULT_TABLE table=default.Table_657257f5b9604a1994dc736b6e69330c [id=c93d391e7c1a43218d99a2b4f0a8de4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:26.982381 880 tablet_service.cc:1514] Processing DeleteTablet for tablet e52429bc9c944c75922a1306acd38fea with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:26 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:26.902849 1029 tablet_service.cc:1467] Processing CreateTablet for tablet 13cc9b4d2d6e41b781a342f6060c6c70 (DEFAULT_TABLE table=default.Table_f11af4ff436842b8800e70008fb2c7b5 [id=9aa3eaef86b940ce928053d5866fdd95]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:26.902938 1040 tablet_service.cc:1467] Processing CreateTablet for tablet cd21f10d74c946a2872a84cadccf7950 (DEFAULT_TABLE table=default.Table_4d5b54216098465fb88c7b9d8b605344 [id=e992f40fef2e466e9a49b8d3307e2a1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:26.903496 1023 tablet_service.cc:1467] Processing CreateTablet for tablet df6299f38ffa4c2f9d6c8a5e5285905f (DEFAULT_TABLE table=default.Table_67eab203697544008a476488a93df8a9 [id=577c95b7ff7a474fb345c08b733b4cb5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:26.905766 1727 raft_consensus.cc:491] T d5fbf97536a2472fa091168363bf9549 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:26.909797 1634 raft_consensus.cc:491] T b2a35fd891c645d68d79c61a2c832ba7 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:26.952280 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a4c6dc08df744e6856bdf99ab65c575 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:26 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:26.983443 896 tablet_service.cc:1467] Processing CreateTablet for tablet 214ee6d7403745728d42cfc7db2ad4f9 (DEFAULT_TABLE table=default.Table_13f74f4092644f87b45e1d86f43bba41 [id=d3c1486098664f2c9482782ffcf507d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:26.952648 1031 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b9d6b99ff8e4d4d8131c0618a9ddc1a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:26 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:26.984035 895 tablet_service.cc:1467] Processing CreateTablet for tablet f4af42b9bae44e3fb009310212cec587 (DEFAULT_TABLE table=default.Table_657257f5b9604a1994dc736b6e69330c [id=c93d391e7c1a43218d99a2b4f0a8de4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:26.984701 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet d3b695ac20ba4205b2d8732db4740531 (table default.Table_8b87c8469fbb465e80375aa0446de6c2 [id=f85f0c435ffa4fdca81d86de4aa4fd5f]) successfully deleted
W20250113 02:51:26.984999 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet b0b267b0d9c54ba496c89f6cb287dffd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b0b267b0d9c54ba496c89f6cb287dffd already in progress: creating tablet
W20250113 02:51:26.985220 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 77cf8b0c824f401aba1880eb7fd1807b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 77cf8b0c824f401aba1880eb7fd1807b already in progress: creating tablet
W20250113 02:51:26.985421 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 8412a588d0454ec78025e7ec30c38104 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8412a588d0454ec78025e7ec30c38104 already in progress: creating tablet
I20250113 02:51:26.962661 1192 tablet_bootstrap.cc:492] T 0fb75c551b4c424186ef2021db742e8e P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:26.965188 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet e52429bc9c944c75922a1306acd38fea with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:26 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:26.986771 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet 35e0aed5125c4e1fa5dce226eade3b78 (table default.Table_be2bc384adeb441093bc75976bb51c0b [id=74c4c856f172482d955d7f6d0b62608f]) successfully deleted
I20250113 02:51:26.987200 736 tablet_service.cc:1467] Processing CreateTablet for tablet a0385f2c6aad45748ebf86d2cfdd40d6 (DEFAULT_TABLE table=default.Table_4e6b1d81ffef42c3a7f83bf18c1fbc48 [id=b2dd127145b344afaa96b2af37244b9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:26.987622 741 tablet_service.cc:1467] Processing CreateTablet for tablet b5205b359d234babb89707a54a7f968f (DEFAULT_TABLE table=default.Table_0f4e49c7b953418aba49a768de9b199b [id=62dbe4b6a4d7433d907cf6c97eaeec43]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:26.988204 738 tablet_service.cc:1467] Processing CreateTablet for tablet 7b16286181684916ac84823f5fec563e (DEFAULT_TABLE table=default.Table_5f73c3e74d984bb69330b294fbc610de [id=3eb7267c2c8b41ef9a48984eda3d26fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:26.988366 750 tablet_service.cc:1467] Processing CreateTablet for tablet c4761d692adc405e88d45ca6e748140b (DEFAULT_TABLE table=default.Table_fc22714e47dd45e6890bf2935057344e [id=4646b623a05c4f339d2c34eae2a04c7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:26.988489 1191 tablet_bootstrap.cc:492] T bcb59573e82743858bd39bde64627d6d P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:26.988951 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f4af42b9bae44e3fb009310212cec587. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:26.989792 1195 ts_tablet_manager.cc:1905] T d5fbf97536a2472fa091168363bf9549 P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:26.989881 1191 ts_tablet_manager.cc:1397] T bcb59573e82743858bd39bde64627d6d P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.049s user 0.003s sys 0.001s
I20250113 02:51:26.992245 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 214ee6d7403745728d42cfc7db2ad4f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:26.994607 449 catalog_manager.cc:5618] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467) does not have the latest schema for tablet 438d1d2300e044f39a15586f335935ba (table default.Table_d7d1f5a34e5f4c05b474b0b61f912cd9 [id=2c7925ec0d8d4cd7b8761268d3962f15]). Expected version 3 got 0
I20250113 02:51:26.992451 1727 raft_consensus.cc:513] T d5fbf97536a2472fa091168363bf9549 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:26.996568 1192 ts_tablet_manager.cc:1397] T 0fb75c551b4c424186ef2021db742e8e P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.135s user 0.002s sys 0.003s
I20250113 02:51:26.997799 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f4af42b9bae44e3fb009310212cec587. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:26.998103 887 tablet_service.cc:1467] Processing CreateTablet for tablet a0385f2c6aad45748ebf86d2cfdd40d6 (DEFAULT_TABLE table=default.Table_4e6b1d81ffef42c3a7f83bf18c1fbc48 [id=b2dd127145b344afaa96b2af37244b9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:26.998914 891 tablet_service.cc:1467] Processing CreateTablet for tablet b5205b359d234babb89707a54a7f968f (DEFAULT_TABLE table=default.Table_0f4e49c7b953418aba49a768de9b199b [id=62dbe4b6a4d7433d907cf6c97eaeec43]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:26.999264 1194 raft_consensus.cc:738] T 35e0aed5125c4e1fa5dce226eade3b78 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:26.998812 886 tablet_service.cc:1467] Processing CreateTablet for tablet f24cda32b6e947c7983ae2f1f9a64c23 (DEFAULT_TABLE table=default.Table_322cee4c328e4e2ca3dcfed1ac77749d [id=1c537c551dee416c93b0605227438914]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:26.996400 1634 raft_consensus.cc:513] T b2a35fd891c645d68d79c61a2c832ba7 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:27.000154 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 13cc9b4d2d6e41b781a342f6060c6c70. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:27.000288 899 tablet_service.cc:1467] Processing CreateTablet for tablet c4761d692adc405e88d45ca6e748140b (DEFAULT_TABLE table=default.Table_fc22714e47dd45e6890bf2935057344e [id=4646b623a05c4f339d2c34eae2a04c7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:27.000555 1701 tablet.cc:1721] T 438d1d2300e044f39a15586f335935ba P df69f48656b44bb890f76587ee69cc0f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250113 02:51:27.000847 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7b16286181684916ac84823f5fec563e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:27.001477 881 tablet_service.cc:1467] Processing CreateTablet for tablet 7b16286181684916ac84823f5fec563e (DEFAULT_TABLE table=default.Table_5f73c3e74d984bb69330b294fbc610de [id=3eb7267c2c8b41ef9a48984eda3d26fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:27.001849 884 tablet_service.cc:1467] Processing CreateTablet for tablet 03f24cc9d3bb413485b132228d73d272 (DEFAULT_TABLE table=default.Table_901b9f58fa9941bea7289bfcb72b22bb [id=3766f848cfe549558c91a0542a24ef42]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:27.002292 1031 tablet_service.cc:1467] Processing CreateTablet for tablet f4af42b9bae44e3fb009310212cec587 (DEFAULT_TABLE table=default.Table_657257f5b9604a1994dc736b6e69330c [id=c93d391e7c1a43218d99a2b4f0a8de4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:27.002547 1035 tablet_service.cc:1467] Processing CreateTablet for tablet 214ee6d7403745728d42cfc7db2ad4f9 (DEFAULT_TABLE table=default.Table_13f74f4092644f87b45e1d86f43bba41 [id=d3c1486098664f2c9482782ffcf507d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:27.002971 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5205b359d234babb89707a54a7f968f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:27.004294 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5205b359d234babb89707a54a7f968f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:27.004881 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 146dad5c977f464496f7d21611030dad
I20250113 02:51:27.004940 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2a35fd891c645d68d79c61a2c832ba7" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
I20250113 02:51:27.005899 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f24cda32b6e947c7983ae2f1f9a64c23. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:27.005973 742 tablet_service.cc:1514] Processing DeleteTablet for tablet 146dad5c977f464496f7d21611030dad with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:26 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:27.006057 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 39a014ac69a2435a9132a75125063093
I20250113 02:51:27.006150 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c4761d692adc405e88d45ca6e748140b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:27.006736 736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a0385f2c6aad45748ebf86d2cfdd40d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:27.007293 880 tablet_service.cc:1514] Processing DeleteTablet for tablet 8aac142230e244c0b59c9c2b7a7300ea with delete_type TABLET_DATA_DELETED (replaced by tablet f31dd916336a4d8fb3ab015af7de8ccb) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:27.008090 742 tablet_service.cc:1514] Processing DeleteTablet for tablet 39a014ac69a2435a9132a75125063093 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:26 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:27.008664 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7b16286181684916ac84823f5fec563e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:27.011116 455 catalog_manager.cc:5618] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467) does not have the latest schema for tablet d304b65eba8c40c6922333506a60540c (table default.Table_e03d954cc20048e4b0d598282689643d [id=e62f8e4c51b04729903b1943275c5f7f]). Expected version 1 got 0
I20250113 02:51:27.013512 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_4078d475fb2e4e52ad184b9318ec9c7d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:27.014012 748 tablet_service.cc:1467] Processing CreateTablet for tablet 03f24cc9d3bb413485b132228d73d272 (DEFAULT_TABLE table=default.Table_901b9f58fa9941bea7289bfcb72b22bb [id=3766f848cfe549558c91a0542a24ef42]), partition=RANGE (key) PARTITION UNBOUNDED
W20250113 02:51:27.014098 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4078d475fb2e4e52ad184b9318ec9c7d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:27.014430 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03f24cc9d3bb413485b132228d73d272. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:27.014835 889 tablet_service.cc:1514] Processing DeleteTablet for tablet 39a014ac69a2435a9132a75125063093 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:26 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:27.015064 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2a35fd891c645d68d79c61a2c832ba7" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:27.015333 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_45fa4077686e44b988c4c789591fcc22"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:27.015769 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_45fa4077686e44b988c4c789591fcc22 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:27.016901 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_693a3c46ae2043a79678c8fa42ec801b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:27.017355 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_693a3c46ae2043a79678c8fa42ec801b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:27.019565 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_7bb48c9e5f084d96980d7efdd395f685"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:27.014366 1194 consensus_queue.cc:260] T 35e0aed5125c4e1fa5dce226eade3b78 P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
W20250113 02:51:27.019995 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7bb48c9e5f084d96980d7efdd395f685 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:27.020112 880 tablet_service.cc:1514] Processing DeleteTablet for tablet 146dad5c977f464496f7d21611030dad with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:26 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:27.020941 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_3bb52edb283b4667b984b8e0370430a5" } new_table_name: "default.Table_c6708c29033245ec913eacfe9c3255c2" modify_external_catalogs: true
I20250113 02:51:27.021107 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c4761d692adc405e88d45ca6e748140b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:27.022042 448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_ffb4a25d44a04760b8306ea2678f4b42" } new_table_name: "default.Table_21648ca05a3344848aeac2ad300f7400" modify_external_catalogs: true
I20250113 02:51:27.024051 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_2c0aebce064848d7ace63d6a47b5daf7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:27.024467 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2c0aebce064848d7ace63d6a47b5daf7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:27.030200 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a0385f2c6aad45748ebf86d2cfdd40d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:27.033179 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df6299f38ffa4c2f9d6c8a5e5285905f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:27.034953 1634 leader_election.cc:290] T b2a35fd891c645d68d79c61a2c832ba7 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
I20250113 02:51:27.035568 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03f24cc9d3bb413485b132228d73d272. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:27.035081 995 leader_election.cc:343] T b2a35fd891c645d68d79c61a2c832ba7 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:27.036295 880 tablet_service.cc:1467] Processing CreateTablet for tablet 70554c1314ef4778bb60dd2f09d08b5d (DEFAULT_TABLE table=default.Table_d86534ff16f04a45977372ccee26a336 [id=aacd92c99f2a4ab7bcc9167e947e94f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:27.036777 1027 tablet_service.cc:1467] Processing CreateTablet for tablet 03f24cc9d3bb413485b132228d73d272 (DEFAULT_TABLE table=default.Table_901b9f58fa9941bea7289bfcb72b22bb [id=3766f848cfe549558c91a0542a24ef42]), partition=RANGE (key) PARTITION UNBOUNDED
W20250113 02:51:27.037118 998 leader_election.cc:343] T b2a35fd891c645d68d79c61a2c832ba7 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:27.037453 742 tablet_service.cc:1467] Processing CreateTablet for tablet f24cda32b6e947c7983ae2f1f9a64c23 (DEFAULT_TABLE table=default.Table_322cee4c328e4e2ca3dcfed1ac77749d [id=1c537c551dee416c93b0605227438914]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:27.037624 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 214ee6d7403745728d42cfc7db2ad4f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:27.037676 745 tablet_service.cc:1467] Processing CreateTablet for tablet 70554c1314ef4778bb60dd2f09d08b5d (DEFAULT_TABLE table=default.Table_d86534ff16f04a45977372ccee26a336 [id=aacd92c99f2a4ab7bcc9167e947e94f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:27.044848 1024 tablet_service.cc:1467] Processing CreateTablet for tablet b5205b359d234babb89707a54a7f968f (DEFAULT_TABLE table=default.Table_0f4e49c7b953418aba49a768de9b199b [id=62dbe4b6a4d7433d907cf6c97eaeec43]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:27.045981 1033 tablet_service.cc:1467] Processing CreateTablet for tablet a0385f2c6aad45748ebf86d2cfdd40d6 (DEFAULT_TABLE table=default.Table_4e6b1d81ffef42c3a7f83bf18c1fbc48 [id=b2dd127145b344afaa96b2af37244b9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:27.046305 998 leader_election.cc:304] T b2a35fd891c645d68d79c61a2c832ba7 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4216a7f853d642b7bc3e5cff458f3aca; no voters: 1186fa869eb14812832fa6562531f196, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:27.046600 1634 raft_consensus.cc:2743] T b2a35fd891c645d68d79c61a2c832ba7 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:27.048290 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03f24cc9d3bb413485b132228d73d272. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:27.049297 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5fbf97536a2472fa091168363bf9549" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:27.050848 1194 ts_tablet_manager.cc:1428] T 35e0aed5125c4e1fa5dce226eade3b78 P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.229s user 0.000s sys 0.002s
W20250113 02:51:27.051155 448 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1d15c7a01f854f0aaabcd30ec3166e31 consensus state has no leader
W20250113 02:51:27.051558 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet dd566a49d3ff4d3889b80d8d2ccbee0e consensus state has no leader
I20250113 02:51:27.055013 1194 tablet_bootstrap.cc:492] T 06f36d09bbb1443bafce906907795404 P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:27.056185 1727 leader_election.cc:290] T d5fbf97536a2472fa091168363bf9549 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
I20250113 02:51:27.057339 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70554c1314ef4778bb60dd2f09d08b5d. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:27.061406 998 leader_election.cc:343] T d5fbf97536a2472fa091168363bf9549 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:27.061415 1030 tablet_service.cc:1514] Processing DeleteTablet for tablet 39a014ac69a2435a9132a75125063093 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:26 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:27.066905 1029 tablet_service.cc:1467] Processing CreateTablet for tablet c4761d692adc405e88d45ca6e748140b (DEFAULT_TABLE table=default.Table_fc22714e47dd45e6890bf2935057344e [id=4646b623a05c4f339d2c34eae2a04c7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:27.067225 447 master_service.cc:946] ReplaceTablet: received request to replace tablet 11950e996c814590bc9f3f22c0354902 from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:27.068954 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5205b359d234babb89707a54a7f968f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:27.069263 1040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cd21f10d74c946a2872a84cadccf7950. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:27.046324 1042 tablet_service.cc:1467] Processing CreateTablet for tablet 7b16286181684916ac84823f5fec563e (DEFAULT_TABLE table=default.Table_5f73c3e74d984bb69330b294fbc610de [id=3eb7267c2c8b41ef9a48984eda3d26fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:27.071053 1041 tablet_service.cc:1467] Processing CreateTablet for tablet f24cda32b6e947c7983ae2f1f9a64c23 (DEFAULT_TABLE table=default.Table_322cee4c328e4e2ca3dcfed1ac77749d [id=1c537c551dee416c93b0605227438914]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:27.072299 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet bcb59573e82743858bd39bde64627d6d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:20 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:27.075959 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f4af42b9bae44e3fb009310212cec587. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:27.076843 1727 raft_consensus.cc:491] T 5b3f012b88c04cb68ed7a1848147b70b P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:27.077692 1036 tablet_service.cc:1514] Processing DeleteTablet for tablet 146dad5c977f464496f7d21611030dad with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:26 UTC) from {username='slave'} at 127.0.0.1:60230
W20250113 02:51:27.078426 995 consensus_peers.cc:487] T 8aac142230e244c0b59c9c2b7a7300ea P 4216a7f853d642b7bc3e5cff458f3aca -> Peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Couldn't send request to peer 1186fa869eb14812832fa6562531f196. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8aac142230e244c0b59c9c2b7a7300ea. This is attempt 1: this message will repeat every 5th retry.
I20250113 02:51:27.079066 1634 raft_consensus.cc:491] T 572e4eed1e924381bada27f8389821a7 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:27.080804 1042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7b16286181684916ac84823f5fec563e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:27.083024 1025 tablet_service.cc:1467] Processing CreateTablet for tablet 70554c1314ef4778bb60dd2f09d08b5d (DEFAULT_TABLE table=default.Table_d86534ff16f04a45977372ccee26a336 [id=aacd92c99f2a4ab7bcc9167e947e94f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:27.083720 1192 raft_consensus.cc:357] T 0fb75c551b4c424186ef2021db742e8e P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:27.083902 1192 raft_consensus.cc:383] T 0fb75c551b4c424186ef2021db742e8e P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:27.084002 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f24cda32b6e947c7983ae2f1f9a64c23. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:27.084633 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f24cda32b6e947c7983ae2f1f9a64c23. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:27.083369 1634 raft_consensus.cc:513] T 572e4eed1e924381bada27f8389821a7 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:27.086685 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70554c1314ef4778bb60dd2f09d08b5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:27.086982 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5fbf97536a2472fa091168363bf9549" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
W20250113 02:51:27.087597 995 leader_election.cc:343] T d5fbf97536a2472fa091168363bf9549 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Illegal state: Tablet not RUNNING: STOPPED
I20250113 02:51:27.091917 755 tablet_service.cc:1467] Processing CreateTablet for tablet 214ee6d7403745728d42cfc7db2ad4f9 (DEFAULT_TABLE table=default.Table_13f74f4092644f87b45e1d86f43bba41 [id=d3c1486098664f2c9482782ffcf507d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:27.092415 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 214ee6d7403745728d42cfc7db2ad4f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:27.092674 1192 raft_consensus.cc:738] T 0fb75c551b4c424186ef2021db742e8e P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:27.093554 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a0385f2c6aad45748ebf86d2cfdd40d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:27.093737 1634 leader_election.cc:290] T 572e4eed1e924381bada27f8389821a7 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:27.095202 1727 raft_consensus.cc:513] T 5b3f012b88c04cb68ed7a1848147b70b P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:27.095814 1727 leader_election.cc:290] T 5b3f012b88c04cb68ed7a1848147b70b P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:27.096154 1192 consensus_queue.cc:260] T 0fb75c551b4c424186ef2021db742e8e P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:27.097200 1192 ts_tablet_manager.cc:1428] T 0fb75c551b4c424186ef2021db742e8e P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.100s user 0.000s sys 0.002s
I20250113 02:51:27.093583 995 leader_election.cc:304] T d5fbf97536a2472fa091168363bf9549 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4216a7f853d642b7bc3e5cff458f3aca; no voters: 1186fa869eb14812832fa6562531f196, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:27.099572 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "572e4eed1e924381bada27f8389821a7" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
I20250113 02:51:27.099568 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b3f012b88c04cb68ed7a1848147b70b" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
I20250113 02:51:27.101032 1727 raft_consensus.cc:2743] T d5fbf97536a2472fa091168363bf9549 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:27.103158 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70554c1314ef4778bb60dd2f09d08b5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:27.103423 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "572e4eed1e924381bada27f8389821a7" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:27.104331 1192 tablet_bootstrap.cc:492] T ee7582ee04494c0db1179359bcb311c8 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
W20250113 02:51:27.105592 995 leader_election.cc:343] T 572e4eed1e924381bada27f8389821a7 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:27.105887 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b3f012b88c04cb68ed7a1848147b70b" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:27.108747 1191 raft_consensus.cc:357] T bcb59573e82743858bd39bde64627d6d P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:27.109011 1191 raft_consensus.cc:383] T bcb59573e82743858bd39bde64627d6d P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:27.109149 1191 raft_consensus.cc:738] T bcb59573e82743858bd39bde64627d6d P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:27.109455 1191 consensus_queue.cc:260] T bcb59573e82743858bd39bde64627d6d P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:27.112452 1191 ts_tablet_manager.cc:1428] T bcb59573e82743858bd39bde64627d6d P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.122s user 0.000s sys 0.002s
I20250113 02:51:27.112771 1191 tablet_bootstrap.cc:492] T 572e4eed1e924381bada27f8389821a7 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:27.112890 1198 ts_tablet_manager.cc:1918] T a6055d77d168495d9b756a1a56c1a920 P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:27.112138 1735 tablet.cc:1721] T 438d1d2300e044f39a15586f335935ba P 4216a7f853d642b7bc3e5cff458f3aca: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250113 02:51:27.128996 1191 tablet_bootstrap.cc:654] T 572e4eed1e924381bada27f8389821a7 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:27.130004 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_2302c6aa93864d4c98218c4b2299c03d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:27.130645 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2302c6aa93864d4c98218c4b2299c03d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:27.131047 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_76a8b4c723fb4930bcda1dafea1b055c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:27.131587 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_76a8b4c723fb4930bcda1dafea1b055c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:27.131683 1195 ts_tablet_manager.cc:1918] T d5fbf97536a2472fa091168363bf9549 P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:27.131842 1195 log.cc:1198] T d5fbf97536a2472fa091168363bf9549 P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/d5fbf97536a2472fa091168363bf9549
I20250113 02:51:27.132822 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_db6d505f52eb4388a2004fdd604f20ea"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:27.133296 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_db6d505f52eb4388a2004fdd604f20ea in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:27.133498 1195 ts_tablet_manager.cc:1939] T d5fbf97536a2472fa091168363bf9549 P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
W20250113 02:51:27.133692 995 leader_election.cc:343] T 5b3f012b88c04cb68ed7a1848147b70b P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Illegal state: must be running to vote when last-logged opid is not known
W20250113 02:51:27.134544 998 leader_election.cc:343] T 572e4eed1e924381bada27f8389821a7 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:27.134797 998 leader_election.cc:304] T 572e4eed1e924381bada27f8389821a7 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4216a7f853d642b7bc3e5cff458f3aca; no voters: 1186fa869eb14812832fa6562531f196, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:27.135317 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet d5fbf97536a2472fa091168363bf9549 (table default.Table_abea3b2ec5144f1fadd95a623c48917c [id=cdb71df855274f13a405028421724d37]) successfully deleted
W20250113 02:51:27.136019 998 leader_election.cc:343] T 5b3f012b88c04cb68ed7a1848147b70b P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:27.136147 998 leader_election.cc:304] T 5b3f012b88c04cb68ed7a1848147b70b P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4216a7f853d642b7bc3e5cff458f3aca; no voters: 1186fa869eb14812832fa6562531f196, df69f48656b44bb890f76587ee69cc0f
W20250113 02:51:27.136703 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 788ab82626424b6e833e31761a15b9b2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 788ab82626424b6e833e31761a15b9b2 already in progress: creating tablet
I20250113 02:51:27.137069 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c4761d692adc405e88d45ca6e748140b. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:27.137338 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 1968c7e9357c4c6e901d334ace0467fb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1968c7e9357c4c6e901d334ace0467fb already in progress: creating tablet
I20250113 02:51:27.138065 1634 raft_consensus.cc:2743] T 5b3f012b88c04cb68ed7a1848147b70b P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250113 02:51:27.141935 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 3b1910496e0849b193e390a5ef1a6b2b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3b1910496e0849b193e390a5ef1a6b2b already in progress: creating tablet
I20250113 02:51:27.142905 1198 log.cc:1198] T a6055d77d168495d9b756a1a56c1a920 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/a6055d77d168495d9b756a1a56c1a920
W20250113 02:51:27.143193 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 415a1fa4b2f14cacb43937c8a372849a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 415a1fa4b2f14cacb43937c8a372849a already in progress: creating tablet
I20250113 02:51:27.143455 1195 tablet_replica.cc:331] T 0fb75c551b4c424186ef2021db742e8e P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:27.143724 1195 raft_consensus.cc:2238] T 0fb75c551b4c424186ef2021db742e8e P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:27.143942 1195 raft_consensus.cc:2267] T 0fb75c551b4c424186ef2021db742e8e P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:27.144681 1195 ts_tablet_manager.cc:1905] T 0fb75c551b4c424186ef2021db742e8e P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:27.144896 1198 ts_tablet_manager.cc:1939] T a6055d77d168495d9b756a1a56c1a920 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:27.145961 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 538) took 1219 ms. Trace:
I20250113 02:51:27.146062 1198 rpcz_store.cc:276] 0113 02:51:25.926523 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:25.928827 (+ 2304us) service_pool.cc:225] Handling call
0113 02:51:26.973326 (+1044499us) ts_tablet_manager.cc:1145] Deleting tablet a6055d77d168495d9b756a1a56c1a920
0113 02:51:26.973335 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:27.033160 (+ 59825us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:27.106699 (+ 73539us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:27.145580 (+ 38881us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1044238}
I20250113 02:51:27.146895 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet a6055d77d168495d9b756a1a56c1a920 (table default.Table_44f1c4e8cc2b4e83adce8d411ae53448 [id=a9e0aae9f5af42b796d29a75140f9c29]) successfully deleted
I20250113 02:51:27.147301 1198 tablet_replica.cc:331] T 8aac142230e244c0b59c9c2b7a7300ea P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:27.147536 1198 raft_consensus.cc:2238] T 8aac142230e244c0b59c9c2b7a7300ea P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: Raft consensus shutting down.
I20250113 02:51:27.148167 1198 pending_rounds.cc:62] T 8aac142230e244c0b59c9c2b7a7300ea P 4216a7f853d642b7bc3e5cff458f3aca: Trying to abort 1 pending ops.
I20250113 02:51:27.148919 1734 raft_consensus.cc:2743] T 572e4eed1e924381bada27f8389821a7 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:27.149312 1198 pending_rounds.cc:69] T 8aac142230e244c0b59c9c2b7a7300ea P 4216a7f853d642b7bc3e5cff458f3aca: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113673465897664512 op_type: NO_OP noop_request { }
I20250113 02:51:27.150511 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_3f69e970777d4d2f84a8275f667a875c" } new_table_name: "default.Table_09819d1f01c94668aba0478d44f251f8" modify_external_catalogs: true
I20250113 02:51:27.150920 454 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_59b60c21ea364f89967c58758807a6a4" } new_table_name: "default.Table_8fe4dfea88ce420f8b780417adb4909b" modify_external_catalogs: true
W20250113 02:51:27.164745 439 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet dd566a49d3ff4d3889b80d8d2ccbee0e (table default.Table_c6708c29033245ec913eacfe9c3255c2 [id=0e2055968e5a492492305d4513f0edba], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet dd566a49d3ff4d3889b80d8d2ccbee0e consensus state has no leader
I20250113 02:51:27.168797 1198 raft_consensus.cc:2883] T 8aac142230e244c0b59c9c2b7a7300ea P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250113 02:51:27.173350 1192 tablet_bootstrap.cc:654] T ee7582ee04494c0db1179359bcb311c8 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:27.174010 1195 ts_tablet_manager.cc:1918] T 0fb75c551b4c424186ef2021db742e8e P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:27.174079 1198 raft_consensus.cc:2267] T 8aac142230e244c0b59c9c2b7a7300ea P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:27.175734 1198 ts_tablet_manager.cc:1905] T 8aac142230e244c0b59c9c2b7a7300ea P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:27.177525 1195 log.cc:1198] T 0fb75c551b4c424186ef2021db742e8e P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/0fb75c551b4c424186ef2021db742e8e
I20250113 02:51:27.178499 1195 ts_tablet_manager.cc:1939] T 0fb75c551b4c424186ef2021db742e8e P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:27.179690 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet 0fb75c551b4c424186ef2021db742e8e (table default.Table_2e62961b127b4bb3ac9b5d30f408c299 [id=e2dbf349032c4342818c5f8a891b7f99]) successfully deleted
W20250113 02:51:27.181746 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 6a4c6dc08df744e6856bdf99ab65c575 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6a4c6dc08df744e6856bdf99ab65c575 already in progress: creating tablet
W20250113 02:51:27.187716 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 8b9d6b99ff8e4d4d8131c0618a9ddc1a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8b9d6b99ff8e4d4d8131c0618a9ddc1a already in progress: creating tablet
W20250113 02:51:27.188792 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet e52429bc9c944c75922a1306acd38fea because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e52429bc9c944c75922a1306acd38fea already in progress: creating tablet
I20250113 02:51:27.189409 1195 tablet_replica.cc:331] T a6055d77d168495d9b756a1a56c1a920 P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:27.189855 1195 raft_consensus.cc:2238] T a6055d77d168495d9b756a1a56c1a920 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:27.192266 1195 raft_consensus.cc:2267] T a6055d77d168495d9b756a1a56c1a920 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:27.197295 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_a85aae70bdd1465786bae1f4505e333c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:27.197918 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a85aae70bdd1465786bae1f4505e333c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:27.195941 755 tablet_service.cc:1514] Processing DeleteTablet for tablet bcb59573e82743858bd39bde64627d6d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:20 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:27.193425 1195 ts_tablet_manager.cc:1905] T a6055d77d168495d9b756a1a56c1a920 P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:27.200230 451 catalog_manager.cc:5618] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587) does not have the latest schema for tablet d304b65eba8c40c6922333506a60540c (table default.Table_e03d954cc20048e4b0d598282689643d [id=e62f8e4c51b04729903b1943275c5f7f]). Expected version 1 got 0
I20250113 02:51:27.204785 1196 ts_tablet_manager.cc:1918] T 32351d6de1574353a6e8073d333da477 P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:27.208292 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_f5459a21934542d9b62792bcd109f836" } modify_external_catalogs: true
I20250113 02:51:27.208544 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_f5459a21934542d9b62792bcd109f836" } modify_external_catalogs: true
I20250113 02:51:27.207441 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_0785edda3a764ff6a35bd4d6276d6e40"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:27.209722 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0785edda3a764ff6a35bd4d6276d6e40 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:27.211647 455 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_1b71e9ea2fca48daa60e8d692eab8e49" } modify_external_catalogs: true
I20250113 02:51:27.211853 455 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_1b71e9ea2fca48daa60e8d692eab8e49" } modify_external_catalogs: true
I20250113 02:51:27.217099 1196 log.cc:1198] T 32351d6de1574353a6e8073d333da477 P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/32351d6de1574353a6e8073d333da477
I20250113 02:51:27.217341 1192 tablet_bootstrap.cc:492] T ee7582ee04494c0db1179359bcb311c8 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:27.217530 1196 ts_tablet_manager.cc:1939] T 32351d6de1574353a6e8073d333da477 P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:27.218353 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet 32351d6de1574353a6e8073d333da477 (table default.Table_78c0bcf1e2d64abc81253301920cf566 [id=e5831758ca184c1998f551fe0285d010]) successfully deleted
I20250113 02:51:27.219170 881 tablet_service.cc:1514] Processing DeleteTablet for tablet 35e0aed5125c4e1fa5dce226eade3b78 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:17 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:27.223088 1192 ts_tablet_manager.cc:1397] T ee7582ee04494c0db1179359bcb311c8 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.113s user 0.002s sys 0.007s
I20250113 02:51:27.223595 1194 tablet_bootstrap.cc:654] T 06f36d09bbb1443bafce906907795404 P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
W20250113 02:51:27.228468 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 569d666c128a45b7a526c13be5090ae3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 569d666c128a45b7a526c13be5090ae3 already in progress: creating tablet
I20250113 02:51:27.231267 1198 ts_tablet_manager.cc:1918] T 8aac142230e244c0b59c9c2b7a7300ea P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250113 02:51:27.231421 1191 tablet_bootstrap.cc:492] T 572e4eed1e924381bada27f8389821a7 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:27.231518 1198 log.cc:1198] T 8aac142230e244c0b59c9c2b7a7300ea P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/8aac142230e244c0b59c9c2b7a7300ea
I20250113 02:51:27.231601 1191 ts_tablet_manager.cc:1397] T 572e4eed1e924381bada27f8389821a7 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.119s user 0.007s sys 0.012s
I20250113 02:51:27.231956 1198 ts_tablet_manager.cc:1939] T 8aac142230e244c0b59c9c2b7a7300ea P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:27.232602 1191 raft_consensus.cc:357] T 572e4eed1e924381bada27f8389821a7 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:27.233017 1191 raft_consensus.cc:383] T 572e4eed1e924381bada27f8389821a7 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:27.233193 1191 raft_consensus.cc:738] T 572e4eed1e924381bada27f8389821a7 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:27.233186 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 537) took 1306 ms. Trace:
I20250113 02:51:27.233281 1198 rpcz_store.cc:276] 0113 02:51:25.926381 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:25.929178 (+ 2797us) service_pool.cc:225] Handling call
0113 02:51:27.146335 (+1217157us) ts_tablet_manager.cc:1145] Deleting tablet 8aac142230e244c0b59c9c2b7a7300ea
0113 02:51:27.146344 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:27.178972 (+ 32628us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:27.231148 (+ 52176us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:27.232916 (+ 1768us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1216931}
I20250113 02:51:27.233579 1191 consensus_queue.cc:260] T 572e4eed1e924381bada27f8389821a7 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:27.238006 1191 ts_tablet_manager.cc:1428] T 572e4eed1e924381bada27f8389821a7 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
W20250113 02:51:27.238008 454 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6d6e462c8bfa47eaa43d3695c69d9c0a consensus state has no leader
I20250113 02:51:27.239454 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 539) took 1302 ms. Trace:
I20250113 02:51:27.239655 1198 rpcz_store.cc:276] 0113 02:51:25.931403 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:25.931452 (+ 49us) service_pool.cc:225] Handling call
0113 02:51:27.233761 (+1302309us) ts_tablet_manager.cc:1145] Deleting tablet 303d911dfa2f4abbb053ead33db6d71c
0113 02:51:27.233772 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:27.233872 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1302117}
I20250113 02:51:27.240291 1191 tablet_bootstrap.cc:492] T 1a26e2b5e1054aa6afe7f44bf8d43108 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:27.241513 1734 raft_consensus.cc:491] T 7f08b72301dc431396cf8cc7790d56d4 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:27.242024 1734 raft_consensus.cc:513] T 7f08b72301dc431396cf8cc7790d56d4 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:27.242722 1734 leader_election.cc:290] T 7f08b72301dc431396cf8cc7790d56d4 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
I20250113 02:51:27.240494 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 542) took 1212 ms. Trace:
I20250113 02:51:27.243280 1198 rpcz_store.cc:276] 0113 02:51:26.028490 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:26.030582 (+ 2092us) service_pool.cc:225] Handling call
0113 02:51:27.240321 (+1209739us) ts_tablet_manager.cc:1145] Deleting tablet 88a53da280b0470abc7b522134d05c8f
0113 02:51:27.240332 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:27.240444 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1209393}
I20250113 02:51:27.243870 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 543) took 1048 ms. Trace:
I20250113 02:51:27.244207 1198 rpcz_store.cc:276] 0113 02:51:26.195322 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:26.195376 (+ 54us) service_pool.cc:225] Handling call
0113 02:51:27.243595 (+1048219us) ts_tablet_manager.cc:1145] Deleting tablet 67baaab1e36e415381579d7385d4ab5a
0113 02:51:27.243603 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:27.243822 (+ 219us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1047992}
I20250113 02:51:27.244490 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f08b72301dc431396cf8cc7790d56d4" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
I20250113 02:51:27.244666 1198 tablet_replica.cc:331] T ba61c86743e24666b30580c36b9228c3 P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:27.244848 1198 raft_consensus.cc:2238] T ba61c86743e24666b30580c36b9228c3 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:27.245016 1198 raft_consensus.cc:2267] T ba61c86743e24666b30580c36b9228c3 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:27.245602 1198 ts_tablet_manager.cc:1905] T ba61c86743e24666b30580c36b9228c3 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:27.246065 773 raft_consensus.cc:2463] T 7f08b72301dc431396cf8cc7790d56d4 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 0.
I20250113 02:51:27.246740 1195 ts_tablet_manager.cc:1918] T a6055d77d168495d9b756a1a56c1a920 P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:27.246964 1195 log.cc:1198] T a6055d77d168495d9b756a1a56c1a920 P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/a6055d77d168495d9b756a1a56c1a920
I20250113 02:51:27.247411 1195 ts_tablet_manager.cc:1939] T a6055d77d168495d9b756a1a56c1a920 P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
W20250113 02:51:27.248327 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 14cea6298852460ca8cfb74e321cc124 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 14cea6298852460ca8cfb74e321cc124 already in progress: creating tablet
W20250113 02:51:27.249468 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 788ab82626424b6e833e31761a15b9b2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 788ab82626424b6e833e31761a15b9b2 already in progress: creating tablet
I20250113 02:51:27.250311 1196 tablet_replica.cc:331] T 1500e4f9c0664a879503afd3edf14c66 P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:27.241549 1192 raft_consensus.cc:357] T ee7582ee04494c0db1179359bcb311c8 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:27.247628 995 leader_election.cc:304] T 7f08b72301dc431396cf8cc7790d56d4 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters:
I20250113 02:51:27.250543 1192 raft_consensus.cc:383] T ee7582ee04494c0db1179359bcb311c8 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:27.250674 1192 raft_consensus.cc:738] T ee7582ee04494c0db1179359bcb311c8 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:27.250820 1734 raft_consensus.cc:2798] T 7f08b72301dc431396cf8cc7790d56d4 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 02:51:27.250921 1734 raft_consensus.cc:491] T 7f08b72301dc431396cf8cc7790d56d4 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 02:51:27.251049 1734 raft_consensus.cc:3054] T 7f08b72301dc431396cf8cc7790d56d4 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:27.250999 1192 consensus_queue.cc:260] T ee7582ee04494c0db1179359bcb311c8 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:27.253211 1192 ts_tablet_manager.cc:1428] T ee7582ee04494c0db1179359bcb311c8 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.030s user 0.001s sys 0.001s
I20250113 02:51:27.253540 1192 tablet_bootstrap.cc:492] T 7d53de6379b84103ba905effb0835676 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:27.253986 1734 raft_consensus.cc:513] T 7f08b72301dc431396cf8cc7790d56d4 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:27.254698 1734 leader_election.cc:290] T 7f08b72301dc431396cf8cc7790d56d4 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Requested vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
I20250113 02:51:27.252527 1191 tablet_bootstrap.cc:654] T 1a26e2b5e1054aa6afe7f44bf8d43108 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:27.256009 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f08b72301dc431396cf8cc7790d56d4" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196"
I20250113 02:51:27.256222 773 raft_consensus.cc:3054] T 7f08b72301dc431396cf8cc7790d56d4 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:27.256898 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f08b72301dc431396cf8cc7790d56d4" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f"
W20250113 02:51:27.257459 998 leader_election.cc:343] T 7f08b72301dc431396cf8cc7790d56d4 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:27.257586 1196 raft_consensus.cc:2238] T 1500e4f9c0664a879503afd3edf14c66 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:27.257761 1192 tablet_bootstrap.cc:654] T 7d53de6379b84103ba905effb0835676 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:27.257783 1196 raft_consensus.cc:2267] T 1500e4f9c0664a879503afd3edf14c66 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:27.258333 1196 ts_tablet_manager.cc:1905] T 1500e4f9c0664a879503afd3edf14c66 P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:27.258859 773 raft_consensus.cc:2463] T 7f08b72301dc431396cf8cc7790d56d4 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 1.
I20250113 02:51:27.259383 995 leader_election.cc:304] T 7f08b72301dc431396cf8cc7790d56d4 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters: df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:27.260447 1195 tablet_replica.cc:331] T bcb59573e82743858bd39bde64627d6d P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:27.260689 1195 raft_consensus.cc:2238] T bcb59573e82743858bd39bde64627d6d P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:27.260924 1195 raft_consensus.cc:2267] T bcb59573e82743858bd39bde64627d6d P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:27.261579 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f08b72301dc431396cf8cc7790d56d4" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:27.262177 998 leader_election.cc:343] T 7f08b72301dc431396cf8cc7790d56d4 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:27.267374 1634 raft_consensus.cc:2798] T 7f08b72301dc431396cf8cc7790d56d4 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Leader election won for term 1
I20250113 02:51:27.267524 1634 raft_consensus.cc:695] T 7f08b72301dc431396cf8cc7790d56d4 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: Becoming Leader. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Running, Role: LEADER
I20250113 02:51:27.267781 1634 consensus_queue.cc:237] T 7f08b72301dc431396cf8cc7790d56d4 P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:27.268118 1191 tablet_bootstrap.cc:492] T 1a26e2b5e1054aa6afe7f44bf8d43108 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:27.268317 1191 ts_tablet_manager.cc:1397] T 1a26e2b5e1054aa6afe7f44bf8d43108 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.028s user 0.006s sys 0.000s
I20250113 02:51:27.268797 1734 raft_consensus.cc:491] T bcb59573e82743858bd39bde64627d6d P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:27.269163 1191 raft_consensus.cc:357] T 1a26e2b5e1054aa6afe7f44bf8d43108 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:27.269127 1734 raft_consensus.cc:513] T bcb59573e82743858bd39bde64627d6d P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:27.269480 1191 raft_consensus.cc:383] T 1a26e2b5e1054aa6afe7f44bf8d43108 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:27.269613 1191 raft_consensus.cc:738] T 1a26e2b5e1054aa6afe7f44bf8d43108 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:27.269953 1191 consensus_queue.cc:260] T 1a26e2b5e1054aa6afe7f44bf8d43108 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:27.270676 1734 leader_election.cc:290] T bcb59573e82743858bd39bde64627d6d P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
I20250113 02:51:27.271152 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcb59573e82743858bd39bde64627d6d" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
I20250113 02:51:27.271323 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcb59573e82743858bd39bde64627d6d" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:27.271840 998 leader_election.cc:343] T bcb59573e82743858bd39bde64627d6d P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
W20250113 02:51:27.272009 995 leader_election.cc:343] T bcb59573e82743858bd39bde64627d6d P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:27.272138 995 leader_election.cc:304] T bcb59573e82743858bd39bde64627d6d P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4216a7f853d642b7bc3e5cff458f3aca; no voters: 1186fa869eb14812832fa6562531f196, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:27.272579 1734 raft_consensus.cc:2743] T bcb59573e82743858bd39bde64627d6d P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:27.273032 1191 ts_tablet_manager.cc:1428] T 1a26e2b5e1054aa6afe7f44bf8d43108 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250113 02:51:27.273382 1191 tablet_bootstrap.cc:492] T 5b3f012b88c04cb68ed7a1848147b70b P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:27.275136 1191 tablet_bootstrap.cc:654] T 5b3f012b88c04cb68ed7a1848147b70b P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:27.277689 1192 tablet_bootstrap.cc:492] T 7d53de6379b84103ba905effb0835676 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:27.277834 1192 ts_tablet_manager.cc:1397] T 7d53de6379b84103ba905effb0835676 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.024s user 0.001s sys 0.001s
I20250113 02:51:27.259948 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet a6055d77d168495d9b756a1a56c1a920 (table default.Table_44f1c4e8cc2b4e83adce8d411ae53448 [id=a9e0aae9f5af42b796d29a75140f9c29]) successfully deleted
I20250113 02:51:27.278615 1192 raft_consensus.cc:357] T 7d53de6379b84103ba905effb0835676 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
W20250113 02:51:27.278735 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 146dad5c977f464496f7d21611030dad because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 146dad5c977f464496f7d21611030dad already in progress: creating tablet
I20250113 02:51:27.278805 1192 raft_consensus.cc:383] T 7d53de6379b84103ba905effb0835676 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:27.278898 1192 raft_consensus.cc:738] T 7d53de6379b84103ba905effb0835676 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
W20250113 02:51:27.278970 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 39a014ac69a2435a9132a75125063093 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 39a014ac69a2435a9132a75125063093 already in progress: creating tablet
I20250113 02:51:27.279129 1192 consensus_queue.cc:260] T 7d53de6379b84103ba905effb0835676 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:27.282930 1198 ts_tablet_manager.cc:1918] T ba61c86743e24666b30580c36b9228c3 P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:27.284176 1194 tablet_bootstrap.cc:492] T 06f36d09bbb1443bafce906907795404 P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:27.284353 1194 ts_tablet_manager.cc:1397] T 06f36d09bbb1443bafce906907795404 P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.232s user 0.003s sys 0.004s
W20250113 02:51:27.285131 440 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.61:35271, user_credentials={real_user=slave}} blocked reactor thread for 49224.6us
I20250113 02:51:27.285864 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet 8aac142230e244c0b59c9c2b7a7300ea (table default.Table_37b1d9d091924f8d9a8a2cbf1d06af1a [id=4d85a53a49d642b798af44eef60b3f78]) successfully deleted
W20250113 02:51:27.286242 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 303d911dfa2f4abbb053ead33db6d71c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 303d911dfa2f4abbb053ead33db6d71c already in progress: creating tablet
W20250113 02:51:27.296116 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 88a53da280b0470abc7b522134d05c8f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 88a53da280b0470abc7b522134d05c8f already in progress: creating tablet
W20250113 02:51:27.296500 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 67baaab1e36e415381579d7385d4ab5a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 67baaab1e36e415381579d7385d4ab5a already in progress: creating tablet
I20250113 02:51:27.297364 1195 ts_tablet_manager.cc:1905] T bcb59573e82743858bd39bde64627d6d P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:27.305227 1198 log.cc:1198] T ba61c86743e24666b30580c36b9228c3 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/ba61c86743e24666b30580c36b9228c3
I20250113 02:51:27.289481 1040 tablet_service.cc:1514] Processing DeleteTablet for tablet 0fb75c551b4c424186ef2021db742e8e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:20 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:27.284019 1192 ts_tablet_manager.cc:1428] T 7d53de6379b84103ba905effb0835676 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20250113 02:51:27.305480 1194 raft_consensus.cc:357] T 06f36d09bbb1443bafce906907795404 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:27.306056 1192 tablet_bootstrap.cc:492] T 1a26e2b5e1054aa6afe7f44bf8d43108 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:27.307188 1198 ts_tablet_manager.cc:1939] T ba61c86743e24666b30580c36b9228c3 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
W20250113 02:51:27.307770 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 12a28557b0cc42fdad927c86fcfa7b96 has no consensus state
I20250113 02:51:27.312417 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_855f1069eae9472998eb347f20b003b8" } modify_external_catalogs: true
I20250113 02:51:27.312588 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_855f1069eae9472998eb347f20b003b8" } modify_external_catalogs: true
I20250113 02:51:27.312780 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 544) took 1077 ms. Trace:
I20250113 02:51:27.312887 1198 rpcz_store.cc:276] 0113 02:51:26.235077 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:26.235507 (+ 430us) service_pool.cc:225] Handling call
0113 02:51:27.244585 (+1009078us) ts_tablet_manager.cc:1145] Deleting tablet ba61c86743e24666b30580c36b9228c3
0113 02:51:27.244594 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:27.279768 (+ 35174us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:27.282806 (+ 3038us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:27.312743 (+ 29937us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1008823}
I20250113 02:51:27.313467 448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_e81c865deb634a57bf1906b18b31a60a" } new_table_name: "default.Table_685277f86f8f48fc9577c4adef84e006" modify_external_catalogs: true
I20250113 02:51:27.313805 1192 tablet_bootstrap.cc:654] T 1a26e2b5e1054aa6afe7f44bf8d43108 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:27.315414 1198 tablet_replica.cc:331] T d5fbf97536a2472fa091168363bf9549 P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:27.315683 1198 raft_consensus.cc:2238] T d5fbf97536a2472fa091168363bf9549 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:27.315896 1198 raft_consensus.cc:2267] T d5fbf97536a2472fa091168363bf9549 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:27.316465 1198 ts_tablet_manager.cc:1905] T d5fbf97536a2472fa091168363bf9549 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:27.316717 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_3fa4f9f57be84226a01935b139545f6e" } modify_external_catalogs: true
I20250113 02:51:27.316903 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_3fa4f9f57be84226a01935b139545f6e" } modify_external_catalogs: true
I20250113 02:51:27.318256 449 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_eec56bd1a5344ab985b794cb42a0eeba" } modify_external_catalogs: true
I20250113 02:51:27.318425 449 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_eec56bd1a5344ab985b794cb42a0eeba" } modify_external_catalogs: true
I20250113 02:51:27.320210 453 master_service.cc:946] ReplaceTablet: received request to replace tablet 58053845187f466599750f2a62f4c3e1 from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:27.321300 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet ba61c86743e24666b30580c36b9228c3 (table default.Table_3f75f23afe2f449fa7a1f9df1c0734fb [id=68a4c317da7143cf841ee2d869d2f754]) successfully deleted
W20250113 02:51:27.321842 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 6d4d8b5cc8ad4c9e935d8afaf5766e66 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6d4d8b5cc8ad4c9e935d8afaf5766e66 already in progress: creating tablet
W20250113 02:51:27.322141 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet f2fb1d39c9ca45ccb2da0be75ccf4acf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f2fb1d39c9ca45ccb2da0be75ccf4acf already in progress: creating tablet
I20250113 02:51:27.330216 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 95e47db8a4d74615800ba6c049ec7131
I20250113 02:51:27.331081 887 tablet_service.cc:1514] Processing DeleteTablet for tablet 95e47db8a4d74615800ba6c049ec7131 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:27 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:27.331197 755 tablet_service.cc:1514] Processing DeleteTablet for tablet 95e47db8a4d74615800ba6c049ec7131 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:27 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:27.332185 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_e32cce9feadb4573a31e4b3ddf77e835"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:27.332770 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e32cce9feadb4573a31e4b3ddf77e835 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:27.336668 1040 tablet_service.cc:1514] Processing DeleteTablet for tablet 95e47db8a4d74615800ba6c049ec7131 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:27 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:27.337349 1192 tablet_bootstrap.cc:492] T 1a26e2b5e1054aa6afe7f44bf8d43108 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:27.337602 1192 ts_tablet_manager.cc:1397] T 1a26e2b5e1054aa6afe7f44bf8d43108 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.032s user 0.000s sys 0.003s
W20250113 02:51:27.338346 448 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet acd83024592e431898e9675c17f50d4f consensus state has no leader
I20250113 02:51:27.338719 448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_ed91fb5c64de42bea271f05b8c284edc" } new_table_name: "default.Table_7e24f64b8bff42a0a429a531f7b588b6" modify_external_catalogs: true
I20250113 02:51:27.338675 1192 raft_consensus.cc:357] T 1a26e2b5e1054aa6afe7f44bf8d43108 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:27.338963 1192 raft_consensus.cc:383] T 1a26e2b5e1054aa6afe7f44bf8d43108 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:27.339044 1192 raft_consensus.cc:738] T 1a26e2b5e1054aa6afe7f44bf8d43108 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:27.339383 1192 consensus_queue.cc:260] T 1a26e2b5e1054aa6afe7f44bf8d43108 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:27.341300 1194 raft_consensus.cc:383] T 06f36d09bbb1443bafce906907795404 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:27.341470 1194 raft_consensus.cc:738] T 06f36d09bbb1443bafce906907795404 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:27.341827 1194 consensus_queue.cc:260] T 06f36d09bbb1443bafce906907795404 P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:27.342645 1194 ts_tablet_manager.cc:1428] T 06f36d09bbb1443bafce906907795404 P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.058s user 0.000s sys 0.002s
I20250113 02:51:27.343047 1194 tablet_bootstrap.cc:492] T ba61c86743e24666b30580c36b9228c3 P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:27.343199 1196 ts_tablet_manager.cc:1918] T 1500e4f9c0664a879503afd3edf14c66 P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:27.343930 1196 log.cc:1198] T 1500e4f9c0664a879503afd3edf14c66 P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/1500e4f9c0664a879503afd3edf14c66
I20250113 02:51:27.347374 1194 tablet_bootstrap.cc:654] T ba61c86743e24666b30580c36b9228c3 P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:27.349874 1438 raft_consensus.cc:491] T 8aac142230e244c0b59c9c2b7a7300ea P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:27.350044 1438 raft_consensus.cc:513] T 8aac142230e244c0b59c9c2b7a7300ea P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:27.350836 1438 leader_election.cc:290] T 8aac142230e244c0b59c9c2b7a7300ea P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587), 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
I20250113 02:51:27.351541 1192 ts_tablet_manager.cc:1428] T 1a26e2b5e1054aa6afe7f44bf8d43108 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.014s user 0.001s sys 0.001s
W20250113 02:51:27.352301 851 leader_election.cc:343] T 8aac142230e244c0b59c9c2b7a7300ea P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Not found: Tablet not found: 8aac142230e244c0b59c9c2b7a7300ea
W20250113 02:51:27.354588 852 leader_election.cc:343] T 8aac142230e244c0b59c9c2b7a7300ea P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Not found: Tablet not found: 8aac142230e244c0b59c9c2b7a7300ea
I20250113 02:51:27.354702 852 leader_election.cc:304] T 8aac142230e244c0b59c9c2b7a7300ea P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df69f48656b44bb890f76587ee69cc0f; no voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca
I20250113 02:51:27.355105 1438 raft_consensus.cc:2743] T 8aac142230e244c0b59c9c2b7a7300ea P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:27.359169 1192 tablet_bootstrap.cc:492] T 501d4e1ae84e4795913035f8a19cc32d P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:27.361893 446 catalog_manager.cc:5526] T 7f08b72301dc431396cf8cc7790d56d4 P 4216a7f853d642b7bc3e5cff458f3aca reported cstate change: term changed from 0 to 1, leader changed from <none> to 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3). New cstate: current_term: 1 leader_uuid: "4216a7f853d642b7bc3e5cff458f3aca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } health_report { overall_health: UNKNOWN } } }
I20250113 02:51:27.362310 446 catalog_manager.cc:5618] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587) does not have the latest schema for tablet 7f08b72301dc431396cf8cc7790d56d4 (table default.Table_6de3bf6640dc47f9b977f608c5ec8ef9 [id=bd14ce0e2b0a4a60a63022be2bc80fef]). Expected version 3 got 0
I20250113 02:51:27.362798 887 tablet_service.cc:1467] Processing CreateTablet for tablet 28772907fb6e43cb8832901f36537a25 (DEFAULT_TABLE table=default.Table_45fa4077686e44b988c4c789591fcc22 [id=1ccc6f4d4f8d4ba69b1844ed9d9ba9ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:27.364485 1198 ts_tablet_manager.cc:1918] T d5fbf97536a2472fa091168363bf9549 P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:27.364637 1198 log.cc:1198] T d5fbf97536a2472fa091168363bf9549 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/d5fbf97536a2472fa091168363bf9549
I20250113 02:51:27.364908 738 tablet_service.cc:1467] Processing CreateTablet for tablet 02bf7042d4ce4077851a44bbfb6c4b46 (DEFAULT_TABLE table=default.Table_4078d475fb2e4e52ad184b9318ec9c7d [id=6e6f2eaf51294f6a8a1228cc66b89575]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:27.364904 895 tablet_service.cc:1467] Processing CreateTablet for tablet 344fa45adf1749f1b5b299d9a6d893c9 (DEFAULT_TABLE table=default.Table_693a3c46ae2043a79678c8fa42ec801b [id=d2ecded77d744b94913a3efbd4f477bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:27.365041 1198 ts_tablet_manager.cc:1939] T d5fbf97536a2472fa091168363bf9549 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:27.365327 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 02bf7042d4ce4077851a44bbfb6c4b46. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:27.365483 884 tablet_service.cc:1467] Processing CreateTablet for tablet 02bf7042d4ce4077851a44bbfb6c4b46 (DEFAULT_TABLE table=default.Table_4078d475fb2e4e52ad184b9318ec9c7d [id=6e6f2eaf51294f6a8a1228cc66b89575]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:27.365935 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 557) took 1019 ms. Trace:
I20250113 02:51:27.366031 880 tablet_service.cc:1467] Processing CreateTablet for tablet a9d1e3336a5b4cc092b847e0ff229a85 (DEFAULT_TABLE table=default.Table_e39cea4b271447f1a10c258d62b5a217 [id=efb6c8d5f28e4eeb957a9d086e2db9e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:27.366061 1198 rpcz_store.cc:276] 0113 02:51:26.346864 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:26.346928 (+ 64us) service_pool.cc:225] Handling call
0113 02:51:27.315362 (+968434us) ts_tablet_manager.cc:1145] Deleting tablet d5fbf97536a2472fa091168363bf9549
0113 02:51:27.315369 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:27.329818 (+ 14449us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:27.364359 (+ 34541us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:27.365866 (+ 1507us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":961920}
I20250113 02:51:27.366614 881 tablet_service.cc:1467] Processing CreateTablet for tablet 3a12785c43294c39b4c2aa0dc66b5b4c (DEFAULT_TABLE table=default.Table_1c82125f87e542e0932f372cd1d99dab [id=e5940280d5b140ea8b9bfcbfcaec056e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:27.367094 886 tablet_service.cc:1467] Processing CreateTablet for tablet a3506ba26c9c4f0295cb1b2798b4a001 (DEFAULT_TABLE table=default.Table_d43add0d74a8498f835de5cd0acf0a25 [id=ede6fcfd16854af1bb2e6893af7afa19]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:27.367278 899 tablet_service.cc:1467] Processing CreateTablet for tablet 32d70f20f9f7440b89e861edb5aa57e1 (DEFAULT_TABLE table=default.Table_7bb48c9e5f084d96980d7efdd395f685 [id=9a06212e79a64014ac4d8056d87ed535]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:27.368209 745 tablet_service.cc:1467] Processing CreateTablet for tablet 344fa45adf1749f1b5b299d9a6d893c9 (DEFAULT_TABLE table=default.Table_693a3c46ae2043a79678c8fa42ec801b [id=d2ecded77d744b94913a3efbd4f477bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:27.368654 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 344fa45adf1749f1b5b299d9a6d893c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:27.370631 896 tablet_service.cc:1467] Processing CreateTablet for tablet 33ccb3f7d2024f03b23d1e1c1046072a (DEFAULT_TABLE table=default.Table_ec0ab13bb2604f4da00a662572ae1734 [id=e152f00c8900469aa5000b1c5b638953]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:27.373867 891 tablet_service.cc:1467] Processing CreateTablet for tablet c7da2c406e64426faf123c4bddfaf213 (DEFAULT_TABLE table=default.Table_e6bf5c2eb70745c69afb72092d93bc22 [id=7eb31eded10f498b918c00ec5e3f860a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:27.374671 755 tablet_service.cc:1467] Processing CreateTablet for tablet 28772907fb6e43cb8832901f36537a25 (DEFAULT_TABLE table=default.Table_45fa4077686e44b988c4c789591fcc22 [id=1ccc6f4d4f8d4ba69b1844ed9d9ba9ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:27.375144 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 28772907fb6e43cb8832901f36537a25. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:27.379545 889 tablet_service.cc:1467] Processing CreateTablet for tablet 12a28557b0cc42fdad927c86fcfa7b96 (DEFAULT_TABLE table=default.Table_09819d1f01c94668aba0478d44f251f8 [id=6841a1fc1ac44da8b4fc51c6250bdca5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:27.380229 1196 ts_tablet_manager.cc:1939] T 1500e4f9c0664a879503afd3edf14c66 P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:27.380843 1192 tablet_bootstrap.cc:654] T 501d4e1ae84e4795913035f8a19cc32d P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:27.380892 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet 1500e4f9c0664a879503afd3edf14c66 (table default.Table_d05b7517b14c4168bea99385f1a7d8b6 [id=05cf81a8622149d29f0f4c567a44371c]) successfully deleted
I20250113 02:51:27.381736 1196 tablet_replica.cc:331] T 8aac142230e244c0b59c9c2b7a7300ea P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
W20250113 02:51:27.381829 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 1968c7e9357c4c6e901d334ace0467fb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1968c7e9357c4c6e901d334ace0467fb already in progress: creating tablet
I20250113 02:51:27.381937 1196 raft_consensus.cc:2238] T 8aac142230e244c0b59c9c2b7a7300ea P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:27.382104 1196 raft_consensus.cc:2267] T 8aac142230e244c0b59c9c2b7a7300ea P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
W20250113 02:51:27.382170 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 3b1910496e0849b193e390a5ef1a6b2b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3b1910496e0849b193e390a5ef1a6b2b already in progress: creating tablet
W20250113 02:51:27.382411 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 415a1fa4b2f14cacb43937c8a372849a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 415a1fa4b2f14cacb43937c8a372849a already in progress: creating tablet
W20250113 02:51:27.382606 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 6a4c6dc08df744e6856bdf99ab65c575 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6a4c6dc08df744e6856bdf99ab65c575 already in progress: creating tablet
I20250113 02:51:27.382651 1196 ts_tablet_manager.cc:1905] T 8aac142230e244c0b59c9c2b7a7300ea P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250113 02:51:27.382814 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 8b9d6b99ff8e4d4d8131c0618a9ddc1a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8b9d6b99ff8e4d4d8131c0618a9ddc1a already in progress: creating tablet
I20250113 02:51:27.383311 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 28772907fb6e43cb8832901f36537a25. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:27.383473 750 tablet_service.cc:1467] Processing CreateTablet for tablet a3506ba26c9c4f0295cb1b2798b4a001 (DEFAULT_TABLE table=default.Table_d43add0d74a8498f835de5cd0acf0a25 [id=ede6fcfd16854af1bb2e6893af7afa19]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:27.383841 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3506ba26c9c4f0295cb1b2798b4a001. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:27.383968 738 tablet_service.cc:1467] Processing CreateTablet for tablet a9d1e3336a5b4cc092b847e0ff229a85 (DEFAULT_TABLE table=default.Table_e39cea4b271447f1a10c258d62b5a217 [id=efb6c8d5f28e4eeb957a9d086e2db9e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:27.384330 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a9d1e3336a5b4cc092b847e0ff229a85. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:27.386951 748 tablet_service.cc:1467] Processing CreateTablet for tablet 32d70f20f9f7440b89e861edb5aa57e1 (DEFAULT_TABLE table=default.Table_7bb48c9e5f084d96980d7efdd395f685 [id=9a06212e79a64014ac4d8056d87ed535]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:27.387286 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 32d70f20f9f7440b89e861edb5aa57e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:27.388060 736 tablet_service.cc:1467] Processing CreateTablet for tablet 3a12785c43294c39b4c2aa0dc66b5b4c (DEFAULT_TABLE table=default.Table_1c82125f87e542e0932f372cd1d99dab [id=e5940280d5b140ea8b9bfcbfcaec056e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:27.388204 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c7da2c406e64426faf123c4bddfaf213. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:27.388399 736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a12785c43294c39b4c2aa0dc66b5b4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:27.391055 1198 tablet_replica.cc:331] T 35e0aed5125c4e1fa5dce226eade3b78 P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:27.391285 1198 raft_consensus.cc:2238] T 35e0aed5125c4e1fa5dce226eade3b78 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:27.391495 1198 raft_consensus.cc:2267] T 35e0aed5125c4e1fa5dce226eade3b78 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:27.392117 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 33ccb3f7d2024f03b23d1e1c1046072a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:27.392226 742 tablet_service.cc:1467] Processing CreateTablet for tablet c7da2c406e64426faf123c4bddfaf213 (DEFAULT_TABLE table=default.Table_e6bf5c2eb70745c69afb72092d93bc22 [id=7eb31eded10f498b918c00ec5e3f860a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:27.392570 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c7da2c406e64426faf123c4bddfaf213. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:27.396275 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 12a28557b0cc42fdad927c86fcfa7b96. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:27.396682 1198 ts_tablet_manager.cc:1905] T 35e0aed5125c4e1fa5dce226eade3b78 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:27.396876 741 tablet_service.cc:1467] Processing CreateTablet for tablet 12a28557b0cc42fdad927c86fcfa7b96 (DEFAULT_TABLE table=default.Table_09819d1f01c94668aba0478d44f251f8 [id=6841a1fc1ac44da8b4fc51c6250bdca5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:27.392230 739 tablet_service.cc:1467] Processing CreateTablet for tablet 33ccb3f7d2024f03b23d1e1c1046072a (DEFAULT_TABLE table=default.Table_ec0ab13bb2604f4da00a662572ae1734 [id=e152f00c8900469aa5000b1c5b638953]), partition=RANGE (key) PARTITION UNBOUNDED
W20250113 02:51:27.398365 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet e52429bc9c944c75922a1306acd38fea because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e52429bc9c944c75922a1306acd38fea already in progress: creating tablet
I20250113 02:51:27.400274 455 catalog_manager.cc:5618] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467) does not have the latest schema for tablet 06f36d09bbb1443bafce906907795404 (table default.Table_2a627a757ab0406686564083ecc36c07 [id=423021f304964246bd29e4daaf470154]). Expected version 2 got 0
W20250113 02:51:27.401240 448 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6001369444894b48b51a48fb60ddc52a consensus state has no leader
I20250113 02:51:27.401618 448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_e6bf5c2eb70745c69afb72092d93bc22" } new_table_name: "default.Table_a5a1020e5fc44e0e878aa129c40eed5d" modify_external_catalogs: true
I20250113 02:51:27.402456 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_0ba8bb338353424c992f00c5940f3fbc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:27.403018 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0ba8bb338353424c992f00c5940f3fbc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:27.408068 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3506ba26c9c4f0295cb1b2798b4a001. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:27.408998 454 catalog_manager.cc:5618] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095) does not have the latest schema for tablet 572e4eed1e924381bada27f8389821a7 (table default.Table_1d910c3d2b244474a576fc4088301143 [id=75cb0745bbb84e1c883cb13b546ecb11]). Expected version 4 got 0
I20250113 02:51:27.409533 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a12785c43294c39b4c2aa0dc66b5b4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:27.409888 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_e392f2cbf2f14aa39320676634a32819"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:27.410862 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e392f2cbf2f14aa39320676634a32819 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:27.414165 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 02bf7042d4ce4077851a44bbfb6c4b46. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:27.414814 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a9d1e3336a5b4cc092b847e0ff229a85. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:27.416443 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 32d70f20f9f7440b89e861edb5aa57e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:27.423642 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 344fa45adf1749f1b5b299d9a6d893c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:27.423838 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 12a28557b0cc42fdad927c86fcfa7b96. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:27.430807 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 33ccb3f7d2024f03b23d1e1c1046072a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:27.438731 1198 ts_tablet_manager.cc:1918] T 35e0aed5125c4e1fa5dce226eade3b78 P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:27.442391 1198 log.cc:1198] T 35e0aed5125c4e1fa5dce226eade3b78 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/35e0aed5125c4e1fa5dce226eade3b78
I20250113 02:51:27.443500 1198 ts_tablet_manager.cc:1939] T 35e0aed5125c4e1fa5dce226eade3b78 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
W20250113 02:51:27.449120 440 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.61:35271, user_credentials={real_user=slave}} blocked reactor thread for 94587.8us
I20250113 02:51:27.450212 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet d5fbf97536a2472fa091168363bf9549 (table default.Table_abea3b2ec5144f1fadd95a623c48917c [id=cdb71df855274f13a405028421724d37]) successfully deleted
I20250113 02:51:27.451269 1040 tablet_service.cc:1467] Processing CreateTablet for tablet 28772907fb6e43cb8832901f36537a25 (DEFAULT_TABLE table=default.Table_45fa4077686e44b988c4c789591fcc22 [id=1ccc6f4d4f8d4ba69b1844ed9d9ba9ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:27.451709 1040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 28772907fb6e43cb8832901f36537a25. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:27.456369 1035 tablet_service.cc:1467] Processing CreateTablet for tablet 344fa45adf1749f1b5b299d9a6d893c9 (DEFAULT_TABLE table=default.Table_693a3c46ae2043a79678c8fa42ec801b [id=d2ecded77d744b94913a3efbd4f477bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:27.456813 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 344fa45adf1749f1b5b299d9a6d893c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:27.460358 1025 tablet_service.cc:1467] Processing CreateTablet for tablet 02bf7042d4ce4077851a44bbfb6c4b46 (DEFAULT_TABLE table=default.Table_4078d475fb2e4e52ad184b9318ec9c7d [id=6e6f2eaf51294f6a8a1228cc66b89575]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:27.460721 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 02bf7042d4ce4077851a44bbfb6c4b46. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:27.463711 1029 tablet_service.cc:1467] Processing CreateTablet for tablet a9d1e3336a5b4cc092b847e0ff229a85 (DEFAULT_TABLE table=default.Table_e39cea4b271447f1a10c258d62b5a217 [id=efb6c8d5f28e4eeb957a9d086e2db9e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:27.464136 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a9d1e3336a5b4cc092b847e0ff229a85. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:27.464210 439 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.60:42067, user_credentials={real_user=slave}} blocked reactor thread for 61198.6us
I20250113 02:51:27.465387 1191 tablet_bootstrap.cc:492] T 5b3f012b88c04cb68ed7a1848147b70b P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:27.466881 1195 ts_tablet_manager.cc:1918] T bcb59573e82743858bd39bde64627d6d P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:27.467017 1195 log.cc:1198] T bcb59573e82743858bd39bde64627d6d P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/bcb59573e82743858bd39bde64627d6d
I20250113 02:51:27.467144 1031 tablet_service.cc:1467] Processing CreateTablet for tablet 3a12785c43294c39b4c2aa0dc66b5b4c (DEFAULT_TABLE table=default.Table_1c82125f87e542e0932f372cd1d99dab [id=e5940280d5b140ea8b9bfcbfcaec056e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:27.467355 1195 ts_tablet_manager.cc:1939] T bcb59573e82743858bd39bde64627d6d P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:27.467458 1191 ts_tablet_manager.cc:1397] T 5b3f012b88c04cb68ed7a1848147b70b P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.192s user 0.017s sys 0.034s
I20250113 02:51:27.467504 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a12785c43294c39b4c2aa0dc66b5b4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:27.468329 1191 raft_consensus.cc:357] T 5b3f012b88c04cb68ed7a1848147b70b P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:27.468621 1191 raft_consensus.cc:383] T 5b3f012b88c04cb68ed7a1848147b70b P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:27.468703 1191 raft_consensus.cc:738] T 5b3f012b88c04cb68ed7a1848147b70b P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:27.468999 1191 consensus_queue.cc:260] T 5b3f012b88c04cb68ed7a1848147b70b P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:27.469619 1191 ts_tablet_manager.cc:1428] T 5b3f012b88c04cb68ed7a1848147b70b P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250113 02:51:27.472407 1042 tablet_service.cc:1467] Processing CreateTablet for tablet a3506ba26c9c4f0295cb1b2798b4a001 (DEFAULT_TABLE table=default.Table_d43add0d74a8498f835de5cd0acf0a25 [id=ede6fcfd16854af1bb2e6893af7afa19]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:27.472815 1042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3506ba26c9c4f0295cb1b2798b4a001. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:27.476758 1024 tablet_service.cc:1467] Processing CreateTablet for tablet 12a28557b0cc42fdad927c86fcfa7b96 (DEFAULT_TABLE table=default.Table_09819d1f01c94668aba0478d44f251f8 [id=6841a1fc1ac44da8b4fc51c6250bdca5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:27.477203 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 12a28557b0cc42fdad927c86fcfa7b96. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:27.480329 1033 tablet_service.cc:1467] Processing CreateTablet for tablet 32d70f20f9f7440b89e861edb5aa57e1 (DEFAULT_TABLE table=default.Table_7bb48c9e5f084d96980d7efdd395f685 [id=9a06212e79a64014ac4d8056d87ed535]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:27.480336 1027 tablet_service.cc:1467] Processing CreateTablet for tablet c7da2c406e64426faf123c4bddfaf213 (DEFAULT_TABLE table=default.Table_e6bf5c2eb70745c69afb72092d93bc22 [id=7eb31eded10f498b918c00ec5e3f860a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:27.480767 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c7da2c406e64426faf123c4bddfaf213. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:27.481245 1041 tablet_service.cc:1467] Processing CreateTablet for tablet 33ccb3f7d2024f03b23d1e1c1046072a (DEFAULT_TABLE table=default.Table_ec0ab13bb2604f4da00a662572ae1734 [id=e152f00c8900469aa5000b1c5b638953]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:27.481593 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 33ccb3f7d2024f03b23d1e1c1046072a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:27.484025 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 32d70f20f9f7440b89e861edb5aa57e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:27.496644 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 11950e996c814590bc9f3f22c0354902
I20250113 02:51:27.497154 447 catalog_manager.cc:6596] ReplaceTablet: tablet 11950e996c814590bc9f3f22c0354902 deleted and replaced by tablet 00d4381876234f16b19336523640bbc8
I20250113 02:51:27.498214 447 catalog_manager.cc:5618] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467) does not have the latest schema for tablet 06f36d09bbb1443bafce906907795404 (table default.Table_2a627a757ab0406686564083ecc36c07 [id=423021f304964246bd29e4daaf470154]). Expected version 2 got 0
I20250113 02:51:27.499003 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_ff26c7d175d24168a34825fdee4c72a0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:27.499548 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ff26c7d175d24168a34825fdee4c72a0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:27.500205 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 82505f1389184d0ca12ec0e2d6084d22
I20250113 02:51:27.500694 452 master_service.cc:946] ReplaceTablet: received request to replace tablet 30ca8a716bfc4ab789b216e17567785e from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:27.501919 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 5b1076a5a17b4fa495975285616401cf
I20250113 02:51:27.502579 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_73371ec3d0f94477b851fc6ce3028d8e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:27.503062 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_73371ec3d0f94477b851fc6ce3028d8e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:27.504688 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 1d2fdc75d00f4e749d322ac6cd5fc410
I20250113 02:51:27.505316 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_693a3c46ae2043a79678c8fa42ec801b" } new_table_name: "default.Table_0cfdfcf64e1e47a7951f9170072704c8" modify_external_catalogs: true
I20250113 02:51:27.506639 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet 82505f1389184d0ca12ec0e2d6084d22 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:27 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:27.506888 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b1076a5a17b4fa495975285616401cf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:27 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:27.507428 446 master_service.cc:946] ReplaceTablet: received request to replace tablet a2781ba5061a4225b5636366857a229e from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:27.508517 1192 tablet_bootstrap.cc:492] T 501d4e1ae84e4795913035f8a19cc32d P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:27.508718 1192 ts_tablet_manager.cc:1397] T 501d4e1ae84e4795913035f8a19cc32d P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.150s user 0.001s sys 0.013s
I20250113 02:51:27.509847 1041 tablet_service.cc:1514] Processing DeleteTablet for tablet 11950e996c814590bc9f3f22c0354902 with delete_type TABLET_DATA_DELETED (replaced by tablet 00d4381876234f16b19336523640bbc8) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:27.510464 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d53de6379b84103ba905effb0835676 with delete_type TABLET_DATA_DELETED (replaced by tablet fd0b8592997d4a61b8ca854ecbb67c55) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:27.510496 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 130f8540b9a6477e84bb9595e7c39994
I20250113 02:51:27.511757 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_9e1b92761ab84f80905c2b4f57738caf" } new_table_name: "default.Table_f4ebf22f164942e581f60ba75f899497" modify_external_catalogs: true
W20250113 02:51:27.514077 448 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c7da2c406e64426faf123c4bddfaf213 consensus state has no leader
I20250113 02:51:27.514655 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet 130f8540b9a6477e84bb9595e7c39994 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:27 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:27.515858 1192 raft_consensus.cc:357] T 501d4e1ae84e4795913035f8a19cc32d P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:27.516130 1192 raft_consensus.cc:383] T 501d4e1ae84e4795913035f8a19cc32d P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:27.516228 1192 raft_consensus.cc:738] T 501d4e1ae84e4795913035f8a19cc32d P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:27.516463 1192 consensus_queue.cc:260] T 501d4e1ae84e4795913035f8a19cc32d P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:27.517103 1192 ts_tablet_manager.cc:1428] T 501d4e1ae84e4795913035f8a19cc32d P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20250113 02:51:27.521441 1041 tablet_service.cc:1514] Processing DeleteTablet for tablet ee7582ee04494c0db1179359bcb311c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:19 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:27.521764 1033 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d2fdc75d00f4e749d322ac6cd5fc410 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:27 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:27.522578 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 556) took 1175 ms. Trace:
I20250113 02:51:27.523048 1192 tablet_bootstrap.cc:492] T 601c71d5f7624fbb87b365d2e775d33e P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:27.526125 1198 rpcz_store.cc:276] 0113 02:51:26.346745 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:26.347789 (+ 1044us) service_pool.cc:225] Handling call
0113 02:51:27.390951 (+1043162us) ts_tablet_manager.cc:1145] Deleting tablet 35e0aed5125c4e1fa5dce226eade3b78
0113 02:51:27.390964 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:27.416563 (+ 25599us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:27.438550 (+ 21987us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:27.522525 (+ 83975us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1037097}
I20250113 02:51:27.526566 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 555) took 1179 ms. Trace:
I20250113 02:51:27.526764 1198 rpcz_store.cc:276] 0113 02:51:26.346637 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:26.349590 (+ 2953us) service_pool.cc:225] Handling call
0113 02:51:27.526419 (+1176829us) ts_tablet_manager.cc:1145] Deleting tablet b217cb8c09fb41cab4f53e605c37efe8
0113 02:51:27.526429 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:27.526533 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1172236}
I20250113 02:51:27.527104 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 554) took 1180 ms. Trace:
I20250113 02:51:27.527205 1198 rpcz_store.cc:276] 0113 02:51:26.346543 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:26.350110 (+ 3567us) service_pool.cc:225] Handling call
0113 02:51:27.526988 (+1176878us) ts_tablet_manager.cc:1145] Deleting tablet d8bb60fdd06946739b75b255303d9d24
0113 02:51:27.526997 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:27.527076 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1172501}
I20250113 02:51:27.527565 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 570) took 1112 ms. Trace:
I20250113 02:51:27.527689 1198 rpcz_store.cc:276] 0113 02:51:26.414820 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:26.414875 (+ 55us) service_pool.cc:225] Handling call
0113 02:51:27.527414 (+1112539us) ts_tablet_manager.cc:1145] Deleting tablet 569d666c128a45b7a526c13be5090ae3
0113 02:51:27.527423 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:27.527534 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1112306}
I20250113 02:51:27.528399 1198 tablet_replica.cc:331] T 5b3f012b88c04cb68ed7a1848147b70b P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:27.528672 1198 raft_consensus.cc:2238] T 5b3f012b88c04cb68ed7a1848147b70b P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:27.528857 1198 raft_consensus.cc:2267] T 5b3f012b88c04cb68ed7a1848147b70b P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:27.529417 1198 ts_tablet_manager.cc:1905] T 5b3f012b88c04cb68ed7a1848147b70b P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:27.529336 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_ba61ec17b9374461a322a08a44edecf7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:27.529906 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ba61ec17b9374461a322a08a44edecf7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:27.530578 1192 tablet_bootstrap.cc:654] T 601c71d5f7624fbb87b365d2e775d33e P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:27.532011 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_daf955e3f47f4bdf8c76884ea4054463"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:27.532486 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_daf955e3f47f4bdf8c76884ea4054463 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:27.535004 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_47a3c6a8e5d44fc6b95c32c776da62df"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:27.535537 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_47a3c6a8e5d44fc6b95c32c776da62df in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:27.536940 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 58053845187f466599750f2a62f4c3e1
I20250113 02:51:27.537518 453 catalog_manager.cc:6596] ReplaceTablet: tablet 58053845187f466599750f2a62f4c3e1 deleted and replaced by tablet a04cec5ed9f34a16a80e112e5c9671e6
I20250113 02:51:27.538308 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_771ecccb82dd4b5ba2f748c98f40eeff"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:27.538899 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_771ecccb82dd4b5ba2f748c98f40eeff in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:27.543119 1198 ts_tablet_manager.cc:1918] T 5b3f012b88c04cb68ed7a1848147b70b P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:27.544021 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet 35e0aed5125c4e1fa5dce226eade3b78 (table default.Table_be2bc384adeb441093bc75976bb51c0b [id=74c4c856f172482d955d7f6d0b62608f]) successfully deleted
W20250113 02:51:27.544302 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet b217cb8c09fb41cab4f53e605c37efe8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b217cb8c09fb41cab4f53e605c37efe8 already in progress: creating tablet
W20250113 02:51:27.544519 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet d8bb60fdd06946739b75b255303d9d24 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d8bb60fdd06946739b75b255303d9d24 already in progress: creating tablet
W20250113 02:51:27.544696 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 569d666c128a45b7a526c13be5090ae3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 569d666c128a45b7a526c13be5090ae3 already in progress: creating tablet
W20250113 02:51:27.544924 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 14cea6298852460ca8cfb74e321cc124 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 14cea6298852460ca8cfb74e321cc124 already in progress: creating tablet
I20250113 02:51:27.545694 1198 log.cc:1198] T 5b3f012b88c04cb68ed7a1848147b70b P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/5b3f012b88c04cb68ed7a1848147b70b
I20250113 02:51:27.546729 1198 ts_tablet_manager.cc:1939] T 5b3f012b88c04cb68ed7a1848147b70b P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:27.547134 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet 58053845187f466599750f2a62f4c3e1 with delete_type TABLET_DATA_DELETED (replaced by tablet a04cec5ed9f34a16a80e112e5c9671e6) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:27.547765 1198 tablet_replica.cc:331] T b2a35fd891c645d68d79c61a2c832ba7 P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:27.549572 1198 raft_consensus.cc:2238] T b2a35fd891c645d68d79c61a2c832ba7 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:27.549818 1198 raft_consensus.cc:2267] T b2a35fd891c645d68d79c61a2c832ba7 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:27.551303 1198 ts_tablet_manager.cc:1905] T b2a35fd891c645d68d79c61a2c832ba7 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:27.552356 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 30ca8a716bfc4ab789b216e17567785e
I20250113 02:51:27.552498 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet 5b3f012b88c04cb68ed7a1848147b70b (table default.Table_ce0f77e091564c29bf54791dda458d78 [id=86a46eed92bc4bddad870e392232cb51]) successfully deleted
I20250113 02:51:27.552874 452 catalog_manager.cc:6596] ReplaceTablet: tablet 30ca8a716bfc4ab789b216e17567785e deleted and replaced by tablet 5f7c91d8c018455e847e5969f7b5e1cc
I20250113 02:51:27.555075 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet 30ca8a716bfc4ab789b216e17567785e with delete_type TABLET_DATA_DELETED (replaced by tablet 5f7c91d8c018455e847e5969f7b5e1cc) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:27.558786 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_e1594759ceea473eb9f01fff92b93b10"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:27.559278 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e1594759ceea473eb9f01fff92b93b10 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:27.561245 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_ab087b92dedd4bc5a48fa4e8c9970ce1" } modify_external_catalogs: true
I20250113 02:51:27.561429 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_ab087b92dedd4bc5a48fa4e8c9970ce1" } modify_external_catalogs: true
I20250113 02:51:27.564059 745 tablet_service.cc:1514] Processing DeleteTablet for tablet 11950e996c814590bc9f3f22c0354902 with delete_type TABLET_DATA_DELETED (replaced by tablet 00d4381876234f16b19336523640bbc8) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:27.564631 745 tablet_service.cc:1514] Processing DeleteTablet for tablet 82505f1389184d0ca12ec0e2d6084d22 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:27 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:27.565143 745 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b1076a5a17b4fa495975285616401cf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:27 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:27.565654 739 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d2fdc75d00f4e749d322ac6cd5fc410 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:27 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:27.567279 739 tablet_service.cc:1514] Processing DeleteTablet for tablet 58053845187f466599750f2a62f4c3e1 with delete_type TABLET_DATA_DELETED (replaced by tablet a04cec5ed9f34a16a80e112e5c9671e6) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:27.566161 745 tablet_service.cc:1514] Processing DeleteTablet for tablet 130f8540b9a6477e84bb9595e7c39994 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:27 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:27.567412 1198 ts_tablet_manager.cc:1918] T b2a35fd891c645d68d79c61a2c832ba7 P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:27.567584 1198 log.cc:1198] T b2a35fd891c645d68d79c61a2c832ba7 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/b2a35fd891c645d68d79c61a2c832ba7
I20250113 02:51:27.567893 745 tablet_service.cc:1514] Processing DeleteTablet for tablet 30ca8a716bfc4ab789b216e17567785e with delete_type TABLET_DATA_DELETED (replaced by tablet 5f7c91d8c018455e847e5969f7b5e1cc) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:27.568007 1198 ts_tablet_manager.cc:1939] T b2a35fd891c645d68d79c61a2c832ba7 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:27.569535 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet b2a35fd891c645d68d79c61a2c832ba7 (table default.Table_266561dfe36245ecb8230fdf1904cb02 [id=60925cf1a1224988b0d349b621ab5368]) successfully deleted
I20250113 02:51:27.569976 1198 tablet_replica.cc:331] T 64cda8796dff46ce95c37439395c77ec P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:27.570225 1198 raft_consensus.cc:2238] T 64cda8796dff46ce95c37439395c77ec P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:27.570387 1198 raft_consensus.cc:2267] T 64cda8796dff46ce95c37439395c77ec P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:27.570950 1198 ts_tablet_manager.cc:1905] T 64cda8796dff46ce95c37439395c77ec P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:27.573894 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a26e2b5e1054aa6afe7f44bf8d43108 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:21 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:27.575155 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet bcb59573e82743858bd39bde64627d6d (table default.Table_9cd2b585fc704fb6a2d0da56a774b8bd [id=7b5976daebfb49eeb72f919006fa149b]) successfully deleted
I20250113 02:51:27.576884 1192 tablet_bootstrap.cc:492] T 601c71d5f7624fbb87b365d2e775d33e P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:27.577082 1192 ts_tablet_manager.cc:1397] T 601c71d5f7624fbb87b365d2e775d33e P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.054s user 0.009s sys 0.008s
W20250113 02:51:27.577597 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 95e47db8a4d74615800ba6c049ec7131 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 95e47db8a4d74615800ba6c049ec7131 already in progress: creating tablet
I20250113 02:51:27.577994 1192 raft_consensus.cc:357] T 601c71d5f7624fbb87b365d2e775d33e P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:27.578243 1192 raft_consensus.cc:383] T 601c71d5f7624fbb87b365d2e775d33e P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:27.578344 1192 raft_consensus.cc:738] T 601c71d5f7624fbb87b365d2e775d33e P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:27.578613 1192 consensus_queue.cc:260] T 601c71d5f7624fbb87b365d2e775d33e P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:27.579200 1192 ts_tablet_manager.cc:1428] T 601c71d5f7624fbb87b365d2e775d33e P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250113 02:51:27.579680 1192 tablet_bootstrap.cc:492] T 8adfa738c17a4169a5d99f6ceca75100 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:27.582101 1192 tablet_bootstrap.cc:654] T 8adfa738c17a4169a5d99f6ceca75100 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
W20250113 02:51:27.582784 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 11950e996c814590bc9f3f22c0354902 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 11950e996c814590bc9f3f22c0354902 already in progress: creating tablet
W20250113 02:51:27.583190 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 82505f1389184d0ca12ec0e2d6084d22 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 82505f1389184d0ca12ec0e2d6084d22 already in progress: creating tablet
W20250113 02:51:27.584442 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 5b1076a5a17b4fa495975285616401cf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5b1076a5a17b4fa495975285616401cf already in progress: creating tablet
I20250113 02:51:27.584743 1191 tablet_bootstrap.cc:492] T 64cda8796dff46ce95c37439395c77ec P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:27.586392 1198 ts_tablet_manager.cc:1918] T 64cda8796dff46ce95c37439395c77ec P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:27.586578 1198 log.cc:1198] T 64cda8796dff46ce95c37439395c77ec P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/64cda8796dff46ce95c37439395c77ec
I20250113 02:51:27.588116 896 tablet_service.cc:1514] Processing DeleteTablet for tablet 11950e996c814590bc9f3f22c0354902 with delete_type TABLET_DATA_DELETED (replaced by tablet 00d4381876234f16b19336523640bbc8) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:27.591140 1198 ts_tablet_manager.cc:1939] T 64cda8796dff46ce95c37439395c77ec P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:27.592303 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet 64cda8796dff46ce95c37439395c77ec (table default.Table_45ad8711e23e44f0b7b83b7a6128c427 [id=983d3b296c064375a1f2747ce556f01d]) successfully deleted
W20250113 02:51:27.595645 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 788ab82626424b6e833e31761a15b9b2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 788ab82626424b6e833e31761a15b9b2 already in progress: creating tablet
W20250113 02:51:27.595909 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 3b1910496e0849b193e390a5ef1a6b2b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3b1910496e0849b193e390a5ef1a6b2b already in progress: creating tablet
I20250113 02:51:27.596134 898 tablet_service.cc:1514] Processing DeleteTablet for tablet 130f8540b9a6477e84bb9595e7c39994 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:27 UTC) from {username='slave'} at 127.0.0.1:37366
W20250113 02:51:27.596187 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 1968c7e9357c4c6e901d334ace0467fb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1968c7e9357c4c6e901d334ace0467fb already in progress: creating tablet
W20250113 02:51:27.596738 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 415a1fa4b2f14cacb43937c8a372849a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 415a1fa4b2f14cacb43937c8a372849a already in progress: creating tablet
I20250113 02:51:27.597117 885 tablet_service.cc:1514] Processing DeleteTablet for tablet 58053845187f466599750f2a62f4c3e1 with delete_type TABLET_DATA_DELETED (replaced by tablet a04cec5ed9f34a16a80e112e5c9671e6) from {username='slave'} at 127.0.0.1:37366
W20250113 02:51:27.597486 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 6a4c6dc08df744e6856bdf99ab65c575 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6a4c6dc08df744e6856bdf99ab65c575 already in progress: creating tablet
W20250113 02:51:27.598259 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 8b9d6b99ff8e4d4d8131c0618a9ddc1a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8b9d6b99ff8e4d4d8131c0618a9ddc1a already in progress: creating tablet
I20250113 02:51:27.598798 896 tablet_service.cc:1514] Processing DeleteTablet for tablet 82505f1389184d0ca12ec0e2d6084d22 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:27 UTC) from {username='slave'} at 127.0.0.1:37366
W20250113 02:51:27.599066 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet e52429bc9c944c75922a1306acd38fea because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e52429bc9c944c75922a1306acd38fea already in progress: creating tablet
I20250113 02:51:27.599532 1194 tablet_bootstrap.cc:492] T ba61c86743e24666b30580c36b9228c3 P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:27.599665 1194 ts_tablet_manager.cc:1397] T ba61c86743e24666b30580c36b9228c3 P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.257s user 0.001s sys 0.024s
W20250113 02:51:27.600033 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 39a014ac69a2435a9132a75125063093 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 39a014ac69a2435a9132a75125063093 already in progress: creating tablet
I20250113 02:51:27.599867 888 tablet_service.cc:1514] Processing DeleteTablet for tablet 30ca8a716bfc4ab789b216e17567785e with delete_type TABLET_DATA_DELETED (replaced by tablet 5f7c91d8c018455e847e5969f7b5e1cc) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:27.599280 887 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d2fdc75d00f4e749d322ac6cd5fc410 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:27 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:27.600649 1198 tablet_replica.cc:331] T bcb59573e82743858bd39bde64627d6d P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:27.600883 1198 raft_consensus.cc:2238] T bcb59573e82743858bd39bde64627d6d P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:27.601083 1198 raft_consensus.cc:2267] T bcb59573e82743858bd39bde64627d6d P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:27.601677 1198 ts_tablet_manager.cc:1905] T bcb59573e82743858bd39bde64627d6d P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:27.601737 1194 raft_consensus.cc:357] T ba61c86743e24666b30580c36b9228c3 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:27.602337 1194 raft_consensus.cc:383] T ba61c86743e24666b30580c36b9228c3 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:27.602741 1194 raft_consensus.cc:738] T ba61c86743e24666b30580c36b9228c3 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:27.602977 1194 consensus_queue.cc:260] T ba61c86743e24666b30580c36b9228c3 P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:27.604094 1194 ts_tablet_manager.cc:1428] T ba61c86743e24666b30580c36b9228c3 P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250113 02:51:27.612332 1194 tablet_bootstrap.cc:492] T a6055d77d168495d9b756a1a56c1a920 P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:27.615195 1194 tablet_bootstrap.cc:654] T a6055d77d168495d9b756a1a56c1a920 P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:27.621290 1734 raft_consensus.cc:491] T 5b1076a5a17b4fa495975285616401cf P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:27.621500 1734 raft_consensus.cc:513] T 5b1076a5a17b4fa495975285616401cf P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:27.621837 1727 raft_consensus.cc:491] T 7d53de6379b84103ba905effb0835676 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:27.621809 1669 raft_consensus.cc:491] T 572e4eed1e924381bada27f8389821a7 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:27.622002 1727 raft_consensus.cc:513] T 7d53de6379b84103ba905effb0835676 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:27.622001 1669 raft_consensus.cc:513] T 572e4eed1e924381bada27f8389821a7 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:27.622291 1734 leader_election.cc:290] T 5b1076a5a17b4fa495975285616401cf P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
I20250113 02:51:27.622575 1727 leader_election.cc:290] T 7d53de6379b84103ba905effb0835676 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
I20250113 02:51:27.622601 1669 leader_election.cc:290] T 572e4eed1e924381bada27f8389821a7 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:27.623127 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1076a5a17b4fa495975285616401cf" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:27.623277 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d53de6379b84103ba905effb0835676" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:27.623657 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "572e4eed1e924381bada27f8389821a7" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:27.624289 711 leader_election.cc:343] T 572e4eed1e924381bada27f8389821a7 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
W20250113 02:51:27.624557 998 leader_election.cc:343] T 5b1076a5a17b4fa495975285616401cf P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
W20250113 02:51:27.624790 998 leader_election.cc:343] T 7d53de6379b84103ba905effb0835676 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:27.625754 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d53de6379b84103ba905effb0835676" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
I20250113 02:51:27.625749 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1076a5a17b4fa495975285616401cf" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
W20250113 02:51:27.626376 995 leader_election.cc:343] T 5b1076a5a17b4fa495975285616401cf P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:27.628374 1196 ts_tablet_manager.cc:1918] T 8aac142230e244c0b59c9c2b7a7300ea P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:27.628643 1196 log.cc:1198] T 8aac142230e244c0b59c9c2b7a7300ea P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/8aac142230e244c0b59c9c2b7a7300ea
I20250113 02:51:27.631527 1192 tablet_bootstrap.cc:492] T 8adfa738c17a4169a5d99f6ceca75100 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:27.626406 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "572e4eed1e924381bada27f8389821a7" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:27.631742 1059 raft_consensus.cc:2463] T 572e4eed1e924381bada27f8389821a7 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1186fa869eb14812832fa6562531f196 in term 0.
I20250113 02:51:27.631747 1192 ts_tablet_manager.cc:1397] T 8adfa738c17a4169a5d99f6ceca75100 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.052s user 0.008s sys 0.008s
W20250113 02:51:27.631899 998 consensus_peers.cc:487] T 7f08b72301dc431396cf8cc7790d56d4 P 4216a7f853d642b7bc3e5cff458f3aca -> Peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Couldn't send request to peer df69f48656b44bb890f76587ee69cc0f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250113 02:51:27.632375 709 leader_election.cc:304] T 572e4eed1e924381bada27f8389821a7 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters: df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:27.632895 1669 raft_consensus.cc:2798] T 572e4eed1e924381bada27f8389821a7 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 02:51:27.633019 1669 raft_consensus.cc:491] T 572e4eed1e924381bada27f8389821a7 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 02:51:27.633147 1669 raft_consensus.cc:3054] T 572e4eed1e924381bada27f8389821a7 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:27.634672 891 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b1076a5a17b4fa495975285616401cf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:27 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:27.635874 1669 raft_consensus.cc:513] T 572e4eed1e924381bada27f8389821a7 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:27.638135 1669 leader_election.cc:290] T 572e4eed1e924381bada27f8389821a7 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 election: Requested vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:27.641160 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "572e4eed1e924381bada27f8389821a7" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f"
I20250113 02:51:27.642004 1196 ts_tablet_manager.cc:1939] T 8aac142230e244c0b59c9c2b7a7300ea P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
W20250113 02:51:27.642798 711 leader_election.cc:343] T 572e4eed1e924381bada27f8389821a7 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:27.646147 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet 8aac142230e244c0b59c9c2b7a7300ea (table default.Table_37b1d9d091924f8d9a8a2cbf1d06af1a [id=4d85a53a49d642b798af44eef60b3f78]) successfully deleted
I20250113 02:51:27.646489 1194 tablet_bootstrap.cc:492] T a6055d77d168495d9b756a1a56c1a920 P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:27.646664 1194 ts_tablet_manager.cc:1397] T a6055d77d168495d9b756a1a56c1a920 P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.041s user 0.004s sys 0.008s
W20250113 02:51:27.647152 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 39a014ac69a2435a9132a75125063093 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 39a014ac69a2435a9132a75125063093 already in progress: creating tablet
I20250113 02:51:27.647445 1194 raft_consensus.cc:357] T a6055d77d168495d9b756a1a56c1a920 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:27.647679 1194 raft_consensus.cc:383] T a6055d77d168495d9b756a1a56c1a920 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:27.647771 1194 raft_consensus.cc:738] T a6055d77d168495d9b756a1a56c1a920 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
W20250113 02:51:27.648043 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 146dad5c977f464496f7d21611030dad because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 146dad5c977f464496f7d21611030dad already in progress: creating tablet
I20250113 02:51:27.648088 1194 consensus_queue.cc:260] T a6055d77d168495d9b756a1a56c1a920 P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:27.648469 1196 tablet_replica.cc:331] T 35e0aed5125c4e1fa5dce226eade3b78 P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:27.648734 1194 ts_tablet_manager.cc:1428] T a6055d77d168495d9b756a1a56c1a920 P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
W20250113 02:51:27.649076 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 1d2fdc75d00f4e749d322ac6cd5fc410 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1d2fdc75d00f4e749d322ac6cd5fc410 already in progress: creating tablet
I20250113 02:51:27.631551 995 leader_election.cc:304] T 5b1076a5a17b4fa495975285616401cf P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4216a7f853d642b7bc3e5cff458f3aca; no voters: 1186fa869eb14812832fa6562531f196, df69f48656b44bb890f76587ee69cc0f
W20250113 02:51:27.649749 995 leader_election.cc:343] T 7d53de6379b84103ba905effb0835676 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:27.649806 1196 raft_consensus.cc:2238] T 35e0aed5125c4e1fa5dce226eade3b78 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:27.650072 1196 raft_consensus.cc:2267] T 35e0aed5125c4e1fa5dce226eade3b78 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:27.650847 1196 ts_tablet_manager.cc:1905] T 35e0aed5125c4e1fa5dce226eade3b78 P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:27.651182 995 leader_election.cc:304] T 7d53de6379b84103ba905effb0835676 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4216a7f853d642b7bc3e5cff458f3aca; no voters: 1186fa869eb14812832fa6562531f196, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:27.650640 1192 raft_consensus.cc:357] T 8adfa738c17a4169a5d99f6ceca75100 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:27.651413 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "572e4eed1e924381bada27f8389821a7" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca"
I20250113 02:51:27.651531 1192 raft_consensus.cc:383] T 8adfa738c17a4169a5d99f6ceca75100 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:27.651597 1059 raft_consensus.cc:3054] T 572e4eed1e924381bada27f8389821a7 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:27.651892 1192 raft_consensus.cc:738] T 8adfa738c17a4169a5d99f6ceca75100 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:27.652204 1192 consensus_queue.cc:260] T 8adfa738c17a4169a5d99f6ceca75100 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:27.653378 1192 ts_tablet_manager.cc:1428] T 8adfa738c17a4169a5d99f6ceca75100 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.021s user 0.002s sys 0.000s
I20250113 02:51:27.654421 772 raft_consensus.cc:1270] T 7f08b72301dc431396cf8cc7790d56d4 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Refusing update from remote peer 4216a7f853d642b7bc3e5cff458f3aca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250113 02:51:27.655076 1192 tablet_bootstrap.cc:492] T 258fad818b7b4aa4b337f9949e20aa46 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:27.655090 1734 raft_consensus.cc:2743] T 7d53de6379b84103ba905effb0835676 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:27.657008 1194 tablet_bootstrap.cc:492] T d5fbf97536a2472fa091168363bf9549 P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:27.657217 1192 tablet_bootstrap.cc:654] T 258fad818b7b4aa4b337f9949e20aa46 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:27.657289 1734 consensus_queue.cc:1035] T 7f08b72301dc431396cf8cc7790d56d4 P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 02:51:27.658200 1198 ts_tablet_manager.cc:1918] T bcb59573e82743858bd39bde64627d6d P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:27.658345 1198 log.cc:1198] T bcb59573e82743858bd39bde64627d6d P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/bcb59573e82743858bd39bde64627d6d
I20250113 02:51:27.658607 1194 tablet_bootstrap.cc:654] T d5fbf97536a2472fa091168363bf9549 P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:27.658753 1198 ts_tablet_manager.cc:1939] T bcb59573e82743858bd39bde64627d6d P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:27.659646 1059 raft_consensus.cc:2463] T 572e4eed1e924381bada27f8389821a7 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1186fa869eb14812832fa6562531f196 in term 1.
I20250113 02:51:27.659957 1198 tablet_replica.cc:331] T 0fb75c551b4c424186ef2021db742e8e P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:27.660199 1198 raft_consensus.cc:2238] T 0fb75c551b4c424186ef2021db742e8e P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:27.660228 709 leader_election.cc:304] T 572e4eed1e924381bada27f8389821a7 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters: df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:27.660363 1198 raft_consensus.cc:2267] T 0fb75c551b4c424186ef2021db742e8e P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:27.660794 1196 ts_tablet_manager.cc:1918] T 35e0aed5125c4e1fa5dce226eade3b78 P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:27.660897 1196 log.cc:1198] T 35e0aed5125c4e1fa5dce226eade3b78 P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/35e0aed5125c4e1fa5dce226eade3b78
I20250113 02:51:27.660941 1198 ts_tablet_manager.cc:1905] T 0fb75c551b4c424186ef2021db742e8e P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:27.662503 1196 ts_tablet_manager.cc:1939] T 35e0aed5125c4e1fa5dce226eade3b78 P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:27.663619 1669 raft_consensus.cc:2798] T 572e4eed1e924381bada27f8389821a7 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Leader election won for term 1
I20250113 02:51:27.664041 1727 raft_consensus.cc:2743] T 5b1076a5a17b4fa495975285616401cf P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:27.673054 1194 tablet_bootstrap.cc:492] T d5fbf97536a2472fa091168363bf9549 P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:27.673228 1194 ts_tablet_manager.cc:1397] T d5fbf97536a2472fa091168363bf9549 P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.024s user 0.008s sys 0.003s
I20250113 02:51:27.677143 1198 ts_tablet_manager.cc:1918] T 0fb75c551b4c424186ef2021db742e8e P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:27.677846 1198 log.cc:1198] T 0fb75c551b4c424186ef2021db742e8e P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/0fb75c551b4c424186ef2021db742e8e
I20250113 02:51:27.678469 1198 ts_tablet_manager.cc:1939] T 0fb75c551b4c424186ef2021db742e8e P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:27.680076 1191 tablet_bootstrap.cc:654] T 64cda8796dff46ce95c37439395c77ec P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:27.680128 1194 raft_consensus.cc:357] T d5fbf97536a2472fa091168363bf9549 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:27.680374 1194 raft_consensus.cc:383] T d5fbf97536a2472fa091168363bf9549 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:27.680456 1194 raft_consensus.cc:738] T d5fbf97536a2472fa091168363bf9549 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:27.680676 1194 consensus_queue.cc:260] T d5fbf97536a2472fa091168363bf9549 P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:27.663475 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet 35e0aed5125c4e1fa5dce226eade3b78 (table default.Table_be2bc384adeb441093bc75976bb51c0b [id=74c4c856f172482d955d7f6d0b62608f]) successfully deleted
I20250113 02:51:27.664613 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet bcb59573e82743858bd39bde64627d6d (table default.Table_9cd2b585fc704fb6a2d0da56a774b8bd [id=7b5976daebfb49eeb72f919006fa149b]) successfully deleted
W20250113 02:51:27.683305 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 146dad5c977f464496f7d21611030dad because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 146dad5c977f464496f7d21611030dad already in progress: creating tablet
W20250113 02:51:27.683458 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 95e47db8a4d74615800ba6c049ec7131 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 95e47db8a4d74615800ba6c049ec7131 already in progress: creating tablet
I20250113 02:51:27.683563 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet 0fb75c551b4c424186ef2021db742e8e (table default.Table_2e62961b127b4bb3ac9b5d30f408c299 [id=e2dbf349032c4342818c5f8a891b7f99]) successfully deleted
I20250113 02:51:27.683641 1194 ts_tablet_manager.cc:1428] T d5fbf97536a2472fa091168363bf9549 P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.008s user 0.000s sys 0.002s
W20250113 02:51:27.683671 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 11950e996c814590bc9f3f22c0354902 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 11950e996c814590bc9f3f22c0354902 already in progress: creating tablet
W20250113 02:51:27.683779 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 95e47db8a4d74615800ba6c049ec7131 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 95e47db8a4d74615800ba6c049ec7131 already in progress: creating tablet
I20250113 02:51:27.683941 1194 tablet_bootstrap.cc:492] T bcb59573e82743858bd39bde64627d6d P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
W20250113 02:51:27.684018 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 82505f1389184d0ca12ec0e2d6084d22 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 82505f1389184d0ca12ec0e2d6084d22 already in progress: creating tablet
W20250113 02:51:27.683909 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 130f8540b9a6477e84bb9595e7c39994 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 130f8540b9a6477e84bb9595e7c39994 already in progress: creating tablet
W20250113 02:51:27.684360 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 58053845187f466599750f2a62f4c3e1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 58053845187f466599750f2a62f4c3e1 already in progress: creating tablet
I20250113 02:51:27.684459 1198 tablet_replica.cc:331] T 5b1076a5a17b4fa495975285616401cf P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
W20250113 02:51:27.684604 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 82505f1389184d0ca12ec0e2d6084d22 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 82505f1389184d0ca12ec0e2d6084d22 already in progress: creating tablet
I20250113 02:51:27.684676 1198 raft_consensus.cc:2238] T 5b1076a5a17b4fa495975285616401cf P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus shutting down.
W20250113 02:51:27.684813 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 30ca8a716bfc4ab789b216e17567785e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 30ca8a716bfc4ab789b216e17567785e already in progress: creating tablet
W20250113 02:51:27.685081 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 1d2fdc75d00f4e749d322ac6cd5fc410 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1d2fdc75d00f4e749d322ac6cd5fc410 already in progress: creating tablet
W20250113 02:51:27.685356 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 5b1076a5a17b4fa495975285616401cf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5b1076a5a17b4fa495975285616401cf already in progress: creating tablet
I20250113 02:51:27.685544 1198 raft_consensus.cc:2267] T 5b1076a5a17b4fa495975285616401cf P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:27.686144 1198 ts_tablet_manager.cc:1905] T 5b1076a5a17b4fa495975285616401cf P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250113 02:51:27.686647 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ef6dfe0431ef4a73a957559f60f24756 consensus state has no leader
W20250113 02:51:27.691900 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 344fa45adf1749f1b5b299d9a6d893c9 consensus state has no leader
I20250113 02:51:27.691877 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet 501d4e1ae84e4795913035f8a19cc32d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:18 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:27.692291 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_a627be2353214265addee118a16ade57" } new_table_name: "default.Table_7bb55657ad1b451c86260845c932b23e" modify_external_catalogs: true
I20250113 02:51:27.692704 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_517c2399a90d4b36925a3b7fb36d5963"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:27.693317 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_517c2399a90d4b36925a3b7fb36d5963 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:27.694427 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_f621d2678f0e47c0997a05a2f1964785" } new_table_name: "default.Table_2abef959c5c34459a8dfa3034e207995" modify_external_catalogs: true
I20250113 02:51:27.695652 452 master_service.cc:946] ReplaceTablet: received request to replace tablet a97efc70946c4af487e33a85abf6b90c from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:27.696676 454 catalog_manager.cc:5618] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095) does not have the latest schema for tablet 572e4eed1e924381bada27f8389821a7 (table default.Table_1d910c3d2b244474a576fc4088301143 [id=75cb0745bbb84e1c883cb13b546ecb11]). Expected version 4 got 0
I20250113 02:51:27.699851 1192 tablet_bootstrap.cc:492] T 258fad818b7b4aa4b337f9949e20aa46 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:27.700306 453 catalog_manager.cc:5618] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587) does not have the latest schema for tablet 8adfa738c17a4169a5d99f6ceca75100 (table default.Table_9a9b1ced3bef4e648088e0781851128e [id=ae544735623e4cfd894dc9340aca177c]). Expected version 1 got 0
I20250113 02:51:27.706074 891 tablet_service.cc:1467] Processing CreateTablet for tablet 23476dc111ae430abdce3855d0ef54fe (DEFAULT_TABLE table=default.Table_db6d505f52eb4388a2004fdd604f20ea [id=bf30415202434a3c8708c908ddd3e55b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:27.706460 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23476dc111ae430abdce3855d0ef54fe. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:27.706874 439 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.104.1:40095, user_credentials={real_user=slave}} blocked reactor thread for 104164us
W20250113 02:51:27.707123 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 58053845187f466599750f2a62f4c3e1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 58053845187f466599750f2a62f4c3e1 already in progress: creating tablet
W20250113 02:51:27.707410 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 130f8540b9a6477e84bb9595e7c39994 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 130f8540b9a6477e84bb9595e7c39994 already in progress: creating tablet
W20250113 02:51:27.707614 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 30ca8a716bfc4ab789b216e17567785e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 30ca8a716bfc4ab789b216e17567785e already in progress: creating tablet
I20250113 02:51:27.708283 745 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b3f012b88c04cb68ed7a1848147b70b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:18 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:27.709097 1195 tablet_replica.cc:331] T 5b3f012b88c04cb68ed7a1848147b70b P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:27.709940 1195 raft_consensus.cc:2238] T 5b3f012b88c04cb68ed7a1848147b70b P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:27.710160 1195 raft_consensus.cc:2267] T 5b3f012b88c04cb68ed7a1848147b70b P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:27.710440 886 tablet_service.cc:1467] Processing CreateTablet for tablet 0411f2aab6c34f09bc4f24e0a49a744a (DEFAULT_TABLE table=default.Table_2302c6aa93864d4c98218c4b2299c03d [id=31df3e62a9244b22acaa799ad187bff0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:27.710798 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0411f2aab6c34f09bc4f24e0a49a744a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:27.701016 1029 tablet_service.cc:1467] Processing CreateTablet for tablet 23476dc111ae430abdce3855d0ef54fe (DEFAULT_TABLE table=default.Table_db6d505f52eb4388a2004fdd604f20ea [id=bf30415202434a3c8708c908ddd3e55b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:27.716274 899 tablet_service.cc:1467] Processing CreateTablet for tablet 46ed22e392db441a894dceeedd2911f4 (DEFAULT_TABLE table=default.Table_2c0aebce064848d7ace63d6a47b5daf7 [id=08372bbba1ae47dc9fe2c0a0f20f0849]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:27.716449 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23476dc111ae430abdce3855d0ef54fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:27.716634 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46ed22e392db441a894dceeedd2911f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:27.718461 1195 ts_tablet_manager.cc:1905] T 5b3f012b88c04cb68ed7a1848147b70b P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:27.701470 1031 tablet_service.cc:1467] Processing CreateTablet for tablet 0411f2aab6c34f09bc4f24e0a49a744a (DEFAULT_TABLE table=default.Table_2302c6aa93864d4c98218c4b2299c03d [id=31df3e62a9244b22acaa799ad187bff0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:27.701851 1040 tablet_service.cc:1467] Processing CreateTablet for tablet 46ed22e392db441a894dceeedd2911f4 (DEFAULT_TABLE table=default.Table_2c0aebce064848d7ace63d6a47b5daf7 [id=08372bbba1ae47dc9fe2c0a0f20f0849]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:27.702497 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet 601c71d5f7624fbb87b365d2e775d33e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:19 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:27.702855 1025 tablet_service.cc:1467] Processing CreateTablet for tablet 87e6a4e4a8434bbcbbe70ddf634accba (DEFAULT_TABLE table=default.Table_a85aae70bdd1465786bae1f4505e333c [id=7d573d7f32a54c4f8f1d0acf0063d4ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:27.703634 1033 tablet_service.cc:1467] Processing CreateTablet for tablet ab73866613c841ee85f3a4a2c4a0733b (DEFAULT_TABLE table=default.Table_76a8b4c723fb4930bcda1dafea1b055c [id=1a5ac67592ce47cf8fe89aa4d71fb829]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:27.704212 1041 tablet_service.cc:1467] Processing CreateTablet for tablet 92f80f6b7bcf426ba5b41f6790efd3c6 (DEFAULT_TABLE table=default.Table_0785edda3a764ff6a35bd4d6276d6e40 [id=af4c068aefdb45ab9baf7ef09a5e09ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:27.705627 1027 tablet_service.cc:1467] Processing CreateTablet for tablet ef078624960843eead2edd1e39b10118 (DEFAULT_TABLE table=default.Table_e32cce9feadb4573a31e4b3ddf77e835 [id=15252277aaaf40f497f4e3c50f1c19ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:27.715868 1727 raft_consensus.cc:491] T ee7582ee04494c0db1179359bcb311c8 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:27.720121 880 tablet_service.cc:1467] Processing CreateTablet for tablet ab73866613c841ee85f3a4a2c4a0733b (DEFAULT_TABLE table=default.Table_76a8b4c723fb4930bcda1dafea1b055c [id=1a5ac67592ce47cf8fe89aa4d71fb829]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:27.720487 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab73866613c841ee85f3a4a2c4a0733b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:27.720626 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef078624960843eead2edd1e39b10118. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:27.719214 1727 raft_consensus.cc:513] T ee7582ee04494c0db1179359bcb311c8 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:27.723963 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee7582ee04494c0db1179359bcb311c8" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
I20250113 02:51:27.724293 895 tablet_service.cc:1467] Processing CreateTablet for tablet 92f80f6b7bcf426ba5b41f6790efd3c6 (DEFAULT_TABLE table=default.Table_0785edda3a764ff6a35bd4d6276d6e40 [id=af4c068aefdb45ab9baf7ef09a5e09ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:27.724643 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 92f80f6b7bcf426ba5b41f6790efd3c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:27.725083 1734 raft_consensus.cc:491] T 601c71d5f7624fbb87b365d2e775d33e P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:27.725245 1734 raft_consensus.cc:513] T 601c71d5f7624fbb87b365d2e775d33e P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:27.726367 1734 leader_election.cc:290] T 601c71d5f7624fbb87b365d2e775d33e P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
W20250113 02:51:27.726491 995 leader_election.cc:343] T ee7582ee04494c0db1179359bcb311c8 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:27.726785 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0411f2aab6c34f09bc4f24e0a49a744a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:27.727790 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 87e6a4e4a8434bbcbbe70ddf634accba. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:27.728242 881 tablet_service.cc:1467] Processing CreateTablet for tablet ef078624960843eead2edd1e39b10118 (DEFAULT_TABLE table=default.Table_e32cce9feadb4573a31e4b3ddf77e835 [id=15252277aaaf40f497f4e3c50f1c19ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:27.728379 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 92f80f6b7bcf426ba5b41f6790efd3c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:27.728572 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef078624960843eead2edd1e39b10118. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:27.728978 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab73866613c841ee85f3a4a2c4a0733b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:27.729581 1040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46ed22e392db441a894dceeedd2911f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:27.730434 1727 leader_election.cc:290] T ee7582ee04494c0db1179359bcb311c8 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
I20250113 02:51:27.731104 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "601c71d5f7624fbb87b365d2e775d33e" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
W20250113 02:51:27.731853 995 leader_election.cc:343] T 601c71d5f7624fbb87b365d2e775d33e P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:27.733572 884 tablet_service.cc:1467] Processing CreateTablet for tablet 87e6a4e4a8434bbcbbe70ddf634accba (DEFAULT_TABLE table=default.Table_a85aae70bdd1465786bae1f4505e333c [id=7d573d7f32a54c4f8f1d0acf0063d4ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:27.734032 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 87e6a4e4a8434bbcbbe70ddf634accba. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:27.738955 1195 ts_tablet_manager.cc:1918] T 5b3f012b88c04cb68ed7a1848147b70b P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:27.739100 1195 log.cc:1198] T 5b3f012b88c04cb68ed7a1848147b70b P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/5b3f012b88c04cb68ed7a1848147b70b
I20250113 02:51:27.739457 1195 ts_tablet_manager.cc:1939] T 5b3f012b88c04cb68ed7a1848147b70b P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:27.740506 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee7582ee04494c0db1179359bcb311c8" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:27.740931 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "601c71d5f7624fbb87b365d2e775d33e" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:27.742096 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9b2d0aa6f0e9456d95f4e48c360ab6e8 consensus state has no leader
I20250113 02:51:27.746498 745 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a26e2b5e1054aa6afe7f44bf8d43108 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:21 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:27.747738 745 tablet_service.cc:1467] Processing CreateTablet for tablet 23476dc111ae430abdce3855d0ef54fe (DEFAULT_TABLE table=default.Table_db6d505f52eb4388a2004fdd604f20ea [id=bf30415202434a3c8708c908ddd3e55b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:27.748198 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23476dc111ae430abdce3855d0ef54fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:27.751340 1195 tablet_replica.cc:331] T 1a26e2b5e1054aa6afe7f44bf8d43108 P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:27.753521 1195 raft_consensus.cc:2238] T 1a26e2b5e1054aa6afe7f44bf8d43108 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:27.754277 1195 raft_consensus.cc:2267] T 1a26e2b5e1054aa6afe7f44bf8d43108 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:27.760785 1195 ts_tablet_manager.cc:1905] T 1a26e2b5e1054aa6afe7f44bf8d43108 P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:27.760843 449 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_db6d505f52eb4388a2004fdd604f20ea" } modify_external_catalogs: true
I20250113 02:51:27.761081 446 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_d1b93410e6504e71a32397040f50b464" } modify_external_catalogs: true
I20250113 02:51:27.761103 449 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_db6d505f52eb4388a2004fdd604f20ea" } modify_external_catalogs: true
I20250113 02:51:27.761304 446 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_d1b93410e6504e71a32397040f50b464" } modify_external_catalogs: true
W20250113 02:51:27.761729 998 leader_election.cc:343] T ee7582ee04494c0db1179359bcb311c8 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:27.761889 998 leader_election.cc:304] T ee7582ee04494c0db1179359bcb311c8 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4216a7f853d642b7bc3e5cff458f3aca; no voters: 1186fa869eb14812832fa6562531f196, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:27.762244 889 tablet_service.cc:1514] Processing DeleteTablet for tablet ba61c86743e24666b30580c36b9228c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:19 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:27.762480 887 tablet_service.cc:1514] Processing DeleteTablet for tablet d5fbf97536a2472fa091168363bf9549 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:18 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:27.762684 888 tablet_service.cc:1514] Processing DeleteTablet for tablet a6055d77d168495d9b756a1a56c1a920 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:19 UTC) from {username='slave'} at 127.0.0.1:37366
W20250113 02:51:27.762836 998 leader_election.cc:343] T 601c71d5f7624fbb87b365d2e775d33e P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:27.762974 998 leader_election.cc:304] T 601c71d5f7624fbb87b365d2e775d33e P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4216a7f853d642b7bc3e5cff458f3aca; no voters: 1186fa869eb14812832fa6562531f196, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:27.763391 1634 raft_consensus.cc:2743] T 601c71d5f7624fbb87b365d2e775d33e P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:27.763379 454 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_131966d78f11450584893c018d784b84" } new_table_name: "default.Table_0e1d06e34184490d8c82c4c47f22fef4" modify_external_catalogs: true
I20250113 02:51:27.764513 1727 raft_consensus.cc:2743] T ee7582ee04494c0db1179359bcb311c8 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:27.765025 1727 consensus_queue.cc:1035] T 06f36d09bbb1443bafce906907795404 P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 02:51:27.765170 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_24ea41f0ca3d4b4b919da4716b6c08a2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:27.765714 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_24ea41f0ca3d4b4b919da4716b6c08a2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:27.767928 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_52010496bf7f4c85a99f61e12fb965a9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:27.768612 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_52010496bf7f4c85a99f61e12fb965a9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:27.770407 453 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_766a63106d8a4fd29bb338e76bd96089" } modify_external_catalogs: true
I20250113 02:51:27.770601 453 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_766a63106d8a4fd29bb338e76bd96089" } modify_external_catalogs: true
I20250113 02:51:27.773491 1198 ts_tablet_manager.cc:1918] T 5b1076a5a17b4fa495975285616401cf P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:27.773650 1198 log.cc:1198] T 5b1076a5a17b4fa495975285616401cf P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/5b1076a5a17b4fa495975285616401cf
I20250113 02:51:27.774236 1198 ts_tablet_manager.cc:1939] T 5b1076a5a17b4fa495975285616401cf P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:27.775689 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet 5b1076a5a17b4fa495975285616401cf (table default.Table_f5459a21934542d9b62792bcd109f836 [id=bc2aa39c74b04c7ca71a6d0acd780853]) successfully deleted
I20250113 02:51:27.771929 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_7157800e639b45d8bb947e53d4319cfc" } modify_external_catalogs: true
I20250113 02:51:27.776664 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_7157800e639b45d8bb947e53d4319cfc" } modify_external_catalogs: true
W20250113 02:51:27.777716 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 11950e996c814590bc9f3f22c0354902 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 11950e996c814590bc9f3f22c0354902 already in progress: creating tablet
I20250113 02:51:27.780989 1198 tablet_replica.cc:331] T 7d53de6379b84103ba905effb0835676 P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:27.781090 918 raft_consensus.cc:3054] T 06f36d09bbb1443bafce906907795404 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:27.781241 1198 raft_consensus.cc:2238] T 7d53de6379b84103ba905effb0835676 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:27.781417 1198 raft_consensus.cc:2267] T 7d53de6379b84103ba905effb0835676 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:27.782013 1198 ts_tablet_manager.cc:1905] T 7d53de6379b84103ba905effb0835676 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:27.792316 1727 consensus_queue.cc:1035] T 06f36d09bbb1443bafce906907795404 P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 02:51:27.807248 1192 ts_tablet_manager.cc:1397] T 258fad818b7b4aa4b337f9949e20aa46 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.091s user 0.004s sys 0.008s
I20250113 02:51:27.808352 1192 raft_consensus.cc:357] T 258fad818b7b4aa4b337f9949e20aa46 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:27.810665 1196 tablet_replica.cc:331] T ba61c86743e24666b30580c36b9228c3 P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:27.810909 1196 raft_consensus.cc:2238] T ba61c86743e24666b30580c36b9228c3 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:27.811090 1196 raft_consensus.cc:2267] T ba61c86743e24666b30580c36b9228c3 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:27.811864 1196 ts_tablet_manager.cc:1905] T ba61c86743e24666b30580c36b9228c3 P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:27.812733 739 tablet_service.cc:1467] Processing CreateTablet for tablet 46ed22e392db441a894dceeedd2911f4 (DEFAULT_TABLE table=default.Table_2c0aebce064848d7ace63d6a47b5daf7 [id=08372bbba1ae47dc9fe2c0a0f20f0849]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:27.813189 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46ed22e392db441a894dceeedd2911f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:27.813606 755 tablet_service.cc:1467] Processing CreateTablet for tablet 92f80f6b7bcf426ba5b41f6790efd3c6 (DEFAULT_TABLE table=default.Table_0785edda3a764ff6a35bd4d6276d6e40 [id=af4c068aefdb45ab9baf7ef09a5e09ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:27.814005 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 92f80f6b7bcf426ba5b41f6790efd3c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:27.816159 1195 ts_tablet_manager.cc:1918] T 1a26e2b5e1054aa6afe7f44bf8d43108 P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:27.816298 1195 log.cc:1198] T 1a26e2b5e1054aa6afe7f44bf8d43108 P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/1a26e2b5e1054aa6afe7f44bf8d43108
I20250113 02:51:27.816324 742 tablet_service.cc:1467] Processing CreateTablet for tablet 87e6a4e4a8434bbcbbe70ddf634accba (DEFAULT_TABLE table=default.Table_a85aae70bdd1465786bae1f4505e333c [id=7d573d7f32a54c4f8f1d0acf0063d4ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:27.816702 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 87e6a4e4a8434bbcbbe70ddf634accba. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:27.816748 1195 ts_tablet_manager.cc:1939] T 1a26e2b5e1054aa6afe7f44bf8d43108 P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:27.820293 736 tablet_service.cc:1467] Processing CreateTablet for tablet ab73866613c841ee85f3a4a2c4a0733b (DEFAULT_TABLE table=default.Table_76a8b4c723fb4930bcda1dafea1b055c [id=1a5ac67592ce47cf8fe89aa4d71fb829]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:27.820680 736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab73866613c841ee85f3a4a2c4a0733b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:27.822569 745 tablet_service.cc:1467] Processing CreateTablet for tablet 0411f2aab6c34f09bc4f24e0a49a744a (DEFAULT_TABLE table=default.Table_2302c6aa93864d4c98218c4b2299c03d [id=31df3e62a9244b22acaa799ad187bff0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:27.823005 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0411f2aab6c34f09bc4f24e0a49a744a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:27.830199 755 tablet_service.cc:1467] Processing CreateTablet for tablet ef078624960843eead2edd1e39b10118 (DEFAULT_TABLE table=default.Table_e32cce9feadb4573a31e4b3ddf77e835 [id=15252277aaaf40f497f4e3c50f1c19ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:27.830947 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef078624960843eead2edd1e39b10118. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:27.836144 1192 raft_consensus.cc:383] T 258fad818b7b4aa4b337f9949e20aa46 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:27.836263 1192 raft_consensus.cc:738] T 258fad818b7b4aa4b337f9949e20aa46 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:27.836598 1192 consensus_queue.cc:260] T 258fad818b7b4aa4b337f9949e20aa46 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:27.837585 1669 raft_consensus.cc:695] T 572e4eed1e924381bada27f8389821a7 P 1186fa869eb14812832fa6562531f196 [term 1 LEADER]: Becoming Leader. State: Replica: 1186fa869eb14812832fa6562531f196, State: Running, Role: LEADER
I20250113 02:51:27.837970 1669 consensus_queue.cc:237] T 572e4eed1e924381bada27f8389821a7 P 1186fa869eb14812832fa6562531f196 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:27.846159 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet 5b3f012b88c04cb68ed7a1848147b70b (table default.Table_ce0f77e091564c29bf54791dda458d78 [id=86a46eed92bc4bddad870e392232cb51]) successfully deleted
I20250113 02:51:27.853952 1192 ts_tablet_manager.cc:1428] T 258fad818b7b4aa4b337f9949e20aa46 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.047s user 0.002s sys 0.000s
I20250113 02:51:27.848984 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet c6820d6331e44656a27e23afe973e2e2
W20250113 02:51:27.849372 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1014e51f1c184d729552a5b34292f232 consensus state has no leader
I20250113 02:51:27.849745 455 catalog_manager.cc:5618] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467) does not have the latest schema for tablet 06f36d09bbb1443bafce906907795404 (table default.Table_2a627a757ab0406686564083ecc36c07 [id=423021f304964246bd29e4daaf470154]). Expected version 2 got 0
I20250113 02:51:27.859303 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_7e37949b2b354eb39b0d00057ee5dc2b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:27.859550 451 catalog_manager.cc:5618] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587) does not have the latest schema for tablet 06f36d09bbb1443bafce906907795404 (table default.Table_2a627a757ab0406686564083ecc36c07 [id=423021f304964246bd29e4daaf470154]). Expected version 2 got 0
W20250113 02:51:27.860060 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7e37949b2b354eb39b0d00057ee5dc2b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:27.862316 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet 1a26e2b5e1054aa6afe7f44bf8d43108 (table default.Table_b5da750f5930453fa3951ffba96c34d4 [id=fa5c22ac7c024f0aba13091163742391]) successfully deleted
I20250113 02:51:27.863602 739 tablet_service.cc:1514] Processing DeleteTablet for tablet c6820d6331e44656a27e23afe973e2e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:27 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:27.864786 1192 tablet_bootstrap.cc:492] T 55ebe9c044d944bc813396c699ba9aa1 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
W20250113 02:51:27.865921 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet c6820d6331e44656a27e23afe973e2e2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c6820d6331e44656a27e23afe973e2e2 already in progress: creating tablet
I20250113 02:51:27.866719 1192 tablet_bootstrap.cc:654] T 55ebe9c044d944bc813396c699ba9aa1 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:27.872056 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet 258fad818b7b4aa4b337f9949e20aa46 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:19 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:27.876459 455 master_service.cc:946] ReplaceTablet: received request to replace tablet 1014e51f1c184d729552a5b34292f232 from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:27.880606 1191 tablet_bootstrap.cc:492] T 64cda8796dff46ce95c37439395c77ec P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:27.880954 1191 ts_tablet_manager.cc:1397] T 64cda8796dff46ce95c37439395c77ec P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.296s user 0.000s sys 0.005s
I20250113 02:51:27.882145 1191 raft_consensus.cc:357] T 64cda8796dff46ce95c37439395c77ec P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:27.882383 1191 raft_consensus.cc:383] T 64cda8796dff46ce95c37439395c77ec P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:27.883394 1191 raft_consensus.cc:738] T 64cda8796dff46ce95c37439395c77ec P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:27.884351 1191 consensus_queue.cc:260] T 64cda8796dff46ce95c37439395c77ec P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:27.856715 1198 ts_tablet_manager.cc:1918] T 7d53de6379b84103ba905effb0835676 P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:27.858086 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet c6820d6331e44656a27e23afe973e2e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:27 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:27.885342 1198 log.cc:1198] T 7d53de6379b84103ba905effb0835676 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/7d53de6379b84103ba905effb0835676
I20250113 02:51:27.885860 1198 ts_tablet_manager.cc:1939] T 7d53de6379b84103ba905effb0835676 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:27.885882 1191 ts_tablet_manager.cc:1428] T 64cda8796dff46ce95c37439395c77ec P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20250113 02:51:27.887284 1191 tablet_bootstrap.cc:492] T b2a35fd891c645d68d79c61a2c832ba7 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
W20250113 02:51:27.890525 454 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0f428c555442401285fb4eab8280b10b consensus state has no leader
I20250113 02:51:27.891289 1191 tablet_bootstrap.cc:654] T b2a35fd891c645d68d79c61a2c832ba7 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:27.891949 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet ed68a2b7ebcf4841bef54798dd3ed80d
I20250113 02:51:27.895797 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet e570465a3d4046b1ac19a3a034fe91f1
I20250113 02:51:27.896211 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet 7d53de6379b84103ba905effb0835676 (table default.Table_71699c9b0b1a46cda2db76dee82b90b4 [id=d1b0bb18a7b84473803cd1294b0829ac]) successfully deleted
I20250113 02:51:27.899565 755 tablet_service.cc:1514] Processing DeleteTablet for tablet e570465a3d4046b1ac19a3a034fe91f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:27 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:27.901433 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet e570465a3d4046b1ac19a3a034fe91f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:27 UTC) from {username='slave'} at 127.0.0.1:60230
W20250113 02:51:27.901463 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet e570465a3d4046b1ac19a3a034fe91f1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e570465a3d4046b1ac19a3a034fe91f1 already in progress: creating tablet
I20250113 02:51:27.902540 450 catalog_manager.cc:5618] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587) does not have the latest schema for tablet 06f36d09bbb1443bafce906907795404 (table default.Table_2a627a757ab0406686564083ecc36c07 [id=423021f304964246bd29e4daaf470154]). Expected version 2 got 0
I20250113 02:51:27.902691 453 catalog_manager.cc:5618] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095) does not have the latest schema for tablet 7f08b72301dc431396cf8cc7790d56d4 (table default.Table_6de3bf6640dc47f9b977f608c5ec8ef9 [id=bd14ce0e2b0a4a60a63022be2bc80fef]). Expected version 3 got 0
I20250113 02:51:27.912717 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet ed68a2b7ebcf4841bef54798dd3ed80d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:27 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:27.914310 1769 tablet.cc:1721] T 06f36d09bbb1443bafce906907795404 P 1186fa869eb14812832fa6562531f196: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250113 02:51:27.929558 1769 tablet.cc:1721] T 7f08b72301dc431396cf8cc7790d56d4 P 1186fa869eb14812832fa6562531f196: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250113 02:51:27.938753 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_acfb94f30f9c4f888bd0a4f28ff1a79e" } new_table_name: "default.Table_8d2d5a8abea04c0a993b9c0cc55d7c3a" modify_external_catalogs: true
I20250113 02:51:27.939139 1768 tablet.cc:1721] T 06f36d09bbb1443bafce906907795404 P 4216a7f853d642b7bc3e5cff458f3aca: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250113 02:51:27.940497 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_d018642ee8b5413b90b86e9e114353a3" } new_table_name: "default.Table_1eb2867afbff468cbbca66486afd7075" modify_external_catalogs: true
I20250113 02:51:27.940804 1770 tablet.cc:1721] T 7f08b72301dc431396cf8cc7790d56d4 P 4216a7f853d642b7bc3e5cff458f3aca: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250113 02:51:27.942646 745 tablet_service.cc:1514] Processing DeleteTablet for tablet ed68a2b7ebcf4841bef54798dd3ed80d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:27 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:27.943953 1198 tablet_replica.cc:331] T ee7582ee04494c0db1179359bcb311c8 P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
W20250113 02:51:27.944386 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 130f8540b9a6477e84bb9595e7c39994 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 130f8540b9a6477e84bb9595e7c39994 already in progress: creating tablet
W20250113 02:51:27.945505 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet ed68a2b7ebcf4841bef54798dd3ed80d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ed68a2b7ebcf4841bef54798dd3ed80d already in progress: creating tablet
I20250113 02:51:27.946365 1198 raft_consensus.cc:2238] T ee7582ee04494c0db1179359bcb311c8 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:27.946628 1198 raft_consensus.cc:2267] T ee7582ee04494c0db1179359bcb311c8 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:27.946106 1192 tablet_bootstrap.cc:492] T 55ebe9c044d944bc813396c699ba9aa1 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:27.948522 1192 ts_tablet_manager.cc:1397] T 55ebe9c044d944bc813396c699ba9aa1 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.084s user 0.003s sys 0.001s
I20250113 02:51:27.949440 1192 raft_consensus.cc:357] T 55ebe9c044d944bc813396c699ba9aa1 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:27.947945 1191 tablet_bootstrap.cc:492] T b2a35fd891c645d68d79c61a2c832ba7 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:27.949908 1192 raft_consensus.cc:383] T 55ebe9c044d944bc813396c699ba9aa1 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:27.949999 1192 raft_consensus.cc:738] T 55ebe9c044d944bc813396c699ba9aa1 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:27.949819 1198 ts_tablet_manager.cc:1905] T ee7582ee04494c0db1179359bcb311c8 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:27.950546 1191 ts_tablet_manager.cc:1397] T b2a35fd891c645d68d79c61a2c832ba7 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.063s user 0.004s sys 0.002s
I20250113 02:51:27.950620 1192 consensus_queue.cc:260] T 55ebe9c044d944bc813396c699ba9aa1 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:27.951181 1192 ts_tablet_manager.cc:1428] T 55ebe9c044d944bc813396c699ba9aa1 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250113 02:51:27.951592 1191 raft_consensus.cc:357] T b2a35fd891c645d68d79c61a2c832ba7 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:27.954382 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_8fa472c153f94ebf8fe8de2e9a7e7d2b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:27.955024 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8fa472c153f94ebf8fe8de2e9a7e7d2b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:27.955600 1191 raft_consensus.cc:383] T b2a35fd891c645d68d79c61a2c832ba7 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:27.955691 1191 raft_consensus.cc:738] T b2a35fd891c645d68d79c61a2c832ba7 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:27.955945 1191 consensus_queue.cc:260] T b2a35fd891c645d68d79c61a2c832ba7 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:27.956472 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_69d7dc3f1d7d4161b0a63c85367de424"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:27.956789 1192 tablet_bootstrap.cc:492] T f760fc1e4bed433dbb987a42c12a1ee6 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
W20250113 02:51:27.956938 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_69d7dc3f1d7d4161b0a63c85367de424 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:27.957432 1191 ts_tablet_manager.cc:1428] T b2a35fd891c645d68d79c61a2c832ba7 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20250113 02:51:27.957845 1191 tablet_bootstrap.cc:492] T ee7582ee04494c0db1179359bcb311c8 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:27.958177 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_9cc116d6861b4480b32eb7fa206ae7c0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:27.958629 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9cc116d6861b4480b32eb7fa206ae7c0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:27.960016 1191 tablet_bootstrap.cc:654] T ee7582ee04494c0db1179359bcb311c8 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:27.964488 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_a8d0f4a9e13b48de8ef840a9b6d690c5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:27.964826 1192 tablet_bootstrap.cc:654] T f760fc1e4bed433dbb987a42c12a1ee6 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
W20250113 02:51:27.964984 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a8d0f4a9e13b48de8ef840a9b6d690c5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:27.973804 1192 tablet_bootstrap.cc:492] T f760fc1e4bed433dbb987a42c12a1ee6 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:27.974006 1192 ts_tablet_manager.cc:1397] T f760fc1e4bed433dbb987a42c12a1ee6 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.001s
I20250113 02:51:27.974351 886 tablet_service.cc:1514] Processing DeleteTablet for tablet e570465a3d4046b1ac19a3a034fe91f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:27 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:27.974917 1192 raft_consensus.cc:357] T f760fc1e4bed433dbb987a42c12a1ee6 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:27.975116 884 tablet_service.cc:1514] Processing DeleteTablet for tablet c6820d6331e44656a27e23afe973e2e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:27 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:27.975195 1192 raft_consensus.cc:383] T f760fc1e4bed433dbb987a42c12a1ee6 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:27.975278 1192 raft_consensus.cc:738] T f760fc1e4bed433dbb987a42c12a1ee6 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:27.975570 1192 consensus_queue.cc:260] T f760fc1e4bed433dbb987a42c12a1ee6 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:27.979949 453 catalog_manager.cc:5526] T 572e4eed1e924381bada27f8389821a7 P 1186fa869eb14812832fa6562531f196 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1186fa869eb14812832fa6562531f196 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "1186fa869eb14812832fa6562531f196" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } health_report { overall_health: UNKNOWN } } }
I20250113 02:51:27.980486 453 catalog_manager.cc:5618] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095) does not have the latest schema for tablet 572e4eed1e924381bada27f8389821a7 (table default.Table_1d910c3d2b244474a576fc4088301143 [id=75cb0745bbb84e1c883cb13b546ecb11]). Expected version 4 got 0
I20250113 02:51:27.980805 453 catalog_manager.cc:5618] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095) does not have the latest schema for tablet 06f36d09bbb1443bafce906907795404 (table default.Table_2a627a757ab0406686564083ecc36c07 [id=423021f304964246bd29e4daaf470154]). Expected version 2 got 0
I20250113 02:51:27.981112 1192 ts_tablet_manager.cc:1428] T f760fc1e4bed433dbb987a42c12a1ee6 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20250113 02:51:27.982149 745 tablet_service.cc:1467] Processing CreateTablet for tablet 306163486d294a7e8348ec6506ed3ed7 (DEFAULT_TABLE table=default.Table_ba61ec17b9374461a322a08a44edecf7 [id=169c6bc1e9f443798937fb45f49265a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:27.982584 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 306163486d294a7e8348ec6506ed3ed7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:27.984642 1192 tablet_bootstrap.cc:492] T e8ab4516789e4585a101c8766d338af3 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
W20250113 02:51:27.986218 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet afcb5b1977c447d3a51de9cb72d7196a consensus state has no leader
I20250113 02:51:27.986510 1191 tablet_bootstrap.cc:492] T ee7582ee04494c0db1179359bcb311c8 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:27.986653 1191 ts_tablet_manager.cc:1397] T ee7582ee04494c0db1179359bcb311c8 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.029s user 0.005s sys 0.001s
I20250113 02:51:27.987442 1192 tablet_bootstrap.cc:654] T e8ab4516789e4585a101c8766d338af3 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:27.988708 736 tablet_service.cc:1467] Processing CreateTablet for tablet a04cec5ed9f34a16a80e112e5c9671e6 (DEFAULT_TABLE table=default.Table_9e44dff41935489aa1306e2e5d02bf85 [id=2c51fbdb672d4b01a5932fade2beb3d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:27.989590 736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a04cec5ed9f34a16a80e112e5c9671e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:27.992178 1027 tablet_service.cc:1467] Processing CreateTablet for tablet 306163486d294a7e8348ec6506ed3ed7 (DEFAULT_TABLE table=default.Table_ba61ec17b9374461a322a08a44edecf7 [id=169c6bc1e9f443798937fb45f49265a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:27.992218 742 tablet_service.cc:1467] Processing CreateTablet for tablet 00d4381876234f16b19336523640bbc8 (DEFAULT_TABLE table=default.Table_003cb5a672084be4b29d515fa4485747 [id=2e275f7f92ac440ca3ea6e29c5566f00]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:27.992605 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 306163486d294a7e8348ec6506ed3ed7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:27.992820 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00d4381876234f16b19336523640bbc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:27.995700 1040 tablet_service.cc:1467] Processing CreateTablet for tablet cd6dcfecf4324542b241f909926756ea (DEFAULT_TABLE table=default.Table_47a3c6a8e5d44fc6b95c32c776da62df [id=f0fb9432740d4ff38e277d4995da33c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:27.996106 1040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cd6dcfecf4324542b241f909926756ea. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:27.997362 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 42a8e01bb71f49b09c83a8f5c905f38d consensus state has no leader
I20250113 02:51:27.999133 741 tablet_service.cc:1467] Processing CreateTablet for tablet db486978b33a45df90025988b084aecd (DEFAULT_TABLE table=default.Table_e1594759ceea473eb9f01fff92b93b10 [id=5bde153e88f648c39af2af655867bed5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:27.999411 881 tablet_service.cc:1514] Processing DeleteTablet for tablet ed68a2b7ebcf4841bef54798dd3ed80d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:27 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:27.999505 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet db486978b33a45df90025988b084aecd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:28.000259 1041 tablet_service.cc:1467] Processing CreateTablet for tablet 00d4381876234f16b19336523640bbc8 (DEFAULT_TABLE table=default.Table_003cb5a672084be4b29d515fa4485747 [id=2e275f7f92ac440ca3ea6e29c5566f00]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:28.000600 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00d4381876234f16b19336523640bbc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:28.000620 1669 raft_consensus.cc:491] T b2a35fd891c645d68d79c61a2c832ba7 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:28.000767 1669 raft_consensus.cc:513] T b2a35fd891c645d68d79c61a2c832ba7 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:28.001538 1669 leader_election.cc:290] T b2a35fd891c645d68d79c61a2c832ba7 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:28.002374 1191 raft_consensus.cc:357] T ee7582ee04494c0db1179359bcb311c8 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:28.002594 1191 raft_consensus.cc:383] T ee7582ee04494c0db1179359bcb311c8 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:28.002681 1191 raft_consensus.cc:738] T ee7582ee04494c0db1179359bcb311c8 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:28.002931 1191 consensus_queue.cc:260] T ee7582ee04494c0db1179359bcb311c8 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:28.003224 884 tablet_service.cc:1467] Processing CreateTablet for tablet a6503d0e37cf4675b5d721975c1245b7 (DEFAULT_TABLE table=default.Table_771ecccb82dd4b5ba2f748c98f40eeff [id=f1fac711caa14b7bad7ebc6c5e583e42]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:28.003516 1191 ts_tablet_manager.cc:1428] T ee7582ee04494c0db1179359bcb311c8 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.017s user 0.002s sys 0.000s
I20250113 02:51:28.003611 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6503d0e37cf4675b5d721975c1245b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:28.003863 1191 tablet_bootstrap.cc:492] T 5b1076a5a17b4fa495975285616401cf P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:28.005322 1025 tablet_service.cc:1467] Processing CreateTablet for tablet a04cec5ed9f34a16a80e112e5c9671e6 (DEFAULT_TABLE table=default.Table_9e44dff41935489aa1306e2e5d02bf85 [id=2c51fbdb672d4b01a5932fade2beb3d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:28.005724 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a04cec5ed9f34a16a80e112e5c9671e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:28.005735 1191 tablet_bootstrap.cc:654] T 5b1076a5a17b4fa495975285616401cf P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:28.006747 886 tablet_service.cc:1467] Processing CreateTablet for tablet cd6dcfecf4324542b241f909926756ea (DEFAULT_TABLE table=default.Table_47a3c6a8e5d44fc6b95c32c776da62df [id=f0fb9432740d4ff38e277d4995da33c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:28.007174 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cd6dcfecf4324542b241f909926756ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:28.010048 755 tablet_service.cc:1467] Processing CreateTablet for tablet a6503d0e37cf4675b5d721975c1245b7 (DEFAULT_TABLE table=default.Table_771ecccb82dd4b5ba2f748c98f40eeff [id=f1fac711caa14b7bad7ebc6c5e583e42]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:28.010409 739 tablet_service.cc:1467] Processing CreateTablet for tablet cd6dcfecf4324542b241f909926756ea (DEFAULT_TABLE table=default.Table_47a3c6a8e5d44fc6b95c32c776da62df [id=f0fb9432740d4ff38e277d4995da33c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:28.010947 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6503d0e37cf4675b5d721975c1245b7. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:28.011531 709 leader_election.cc:343] T b2a35fd891c645d68d79c61a2c832ba7 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Not found: Tablet not found: b2a35fd891c645d68d79c61a2c832ba7
I20250113 02:51:28.011794 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cd6dcfecf4324542b241f909926756ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:28.016515 1029 tablet_service.cc:1467] Processing CreateTablet for tablet a6503d0e37cf4675b5d721975c1245b7 (DEFAULT_TABLE table=default.Table_771ecccb82dd4b5ba2f748c98f40eeff [id=f1fac711caa14b7bad7ebc6c5e583e42]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:28.017252 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6503d0e37cf4675b5d721975c1245b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:28.020254 881 tablet_service.cc:1467] Processing CreateTablet for tablet 306163486d294a7e8348ec6506ed3ed7 (DEFAULT_TABLE table=default.Table_ba61ec17b9374461a322a08a44edecf7 [id=169c6bc1e9f443798937fb45f49265a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:28.021844 891 tablet_service.cc:1467] Processing CreateTablet for tablet 00d4381876234f16b19336523640bbc8 (DEFAULT_TABLE table=default.Table_003cb5a672084be4b29d515fa4485747 [id=2e275f7f92ac440ca3ea6e29c5566f00]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:28.022452 880 tablet_service.cc:1467] Processing CreateTablet for tablet a04cec5ed9f34a16a80e112e5c9671e6 (DEFAULT_TABLE table=default.Table_9e44dff41935489aa1306e2e5d02bf85 [id=2c51fbdb672d4b01a5932fade2beb3d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:28.024277 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 306163486d294a7e8348ec6506ed3ed7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:28.027202 1198 ts_tablet_manager.cc:1918] T ee7582ee04494c0db1179359bcb311c8 P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:28.029538 1191 tablet_bootstrap.cc:492] T 5b1076a5a17b4fa495975285616401cf P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:28.030661 1191 ts_tablet_manager.cc:1397] T 5b1076a5a17b4fa495975285616401cf P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.027s user 0.004s sys 0.000s
I20250113 02:51:28.030798 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2a35fd891c645d68d79c61a2c832ba7" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:28.031857 711 leader_election.cc:343] T b2a35fd891c645d68d79c61a2c832ba7 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:28.032033 711 leader_election.cc:304] T b2a35fd891c645d68d79c61a2c832ba7 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:28.032447 1669 raft_consensus.cc:2743] T b2a35fd891c645d68d79c61a2c832ba7 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:28.032980 1194 tablet_bootstrap.cc:654] T bcb59573e82743858bd39bde64627d6d P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:28.033246 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00d4381876234f16b19336523640bbc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:28.033763 1191 raft_consensus.cc:357] T 5b1076a5a17b4fa495975285616401cf P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:28.034173 1198 log.cc:1198] T ee7582ee04494c0db1179359bcb311c8 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/ee7582ee04494c0db1179359bcb311c8
I20250113 02:51:28.034250 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a04cec5ed9f34a16a80e112e5c9671e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:28.034543 1191 raft_consensus.cc:383] T 5b1076a5a17b4fa495975285616401cf P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:28.035076 1198 ts_tablet_manager.cc:1939] T ee7582ee04494c0db1179359bcb311c8 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:28.035352 1191 raft_consensus.cc:738] T 5b1076a5a17b4fa495975285616401cf P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:28.035914 1191 consensus_queue.cc:260] T 5b1076a5a17b4fa495975285616401cf P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:28.048243 1669 raft_consensus.cc:491] T 64cda8796dff46ce95c37439395c77ec P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:28.048422 1669 raft_consensus.cc:513] T 64cda8796dff46ce95c37439395c77ec P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:28.049181 1669 leader_election.cc:290] T 64cda8796dff46ce95c37439395c77ec P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
W20250113 02:51:28.052663 709 leader_election.cc:343] T 64cda8796dff46ce95c37439395c77ec P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Not found: Tablet not found: 64cda8796dff46ce95c37439395c77ec
I20250113 02:51:28.057005 1191 ts_tablet_manager.cc:1428] T 5b1076a5a17b4fa495975285616401cf P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.026s user 0.002s sys 0.000s
I20250113 02:51:28.058728 1029 tablet_service.cc:1467] Processing CreateTablet for tablet db486978b33a45df90025988b084aecd (DEFAULT_TABLE table=default.Table_e1594759ceea473eb9f01fff92b93b10 [id=5bde153e88f648c39af2af655867bed5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:28.059235 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet db486978b33a45df90025988b084aecd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:28.063158 1191 tablet_bootstrap.cc:492] T 7d53de6379b84103ba905effb0835676 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:28.064862 1191 tablet_bootstrap.cc:654] T 7d53de6379b84103ba905effb0835676 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:28.064869 1040 tablet_service.cc:1467] Processing CreateTablet for tablet 85a82ad6e93346ecb1a0cb38b62f5786 (DEFAULT_TABLE table=default.Table_ff26c7d175d24168a34825fdee4c72a0 [id=af981d4d813541c493f8f90f0197470e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:28.065289 1040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 85a82ad6e93346ecb1a0cb38b62f5786. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:28.065887 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_b8ded36063364816988685dfc77391d2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:28.066543 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b8ded36063364816988685dfc77391d2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:28.068239 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64cda8796dff46ce95c37439395c77ec" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:28.068285 1027 tablet_service.cc:1467] Processing CreateTablet for tablet 5f7c91d8c018455e847e5969f7b5e1cc (DEFAULT_TABLE table=default.Table_65c5f2ca0aea479b9edd500ceb8c01bc [id=9eaf97b701fd4e79847dce3a9d259beb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:28.068763 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f7c91d8c018455e847e5969f7b5e1cc. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:28.068918 711 leader_election.cc:343] T 64cda8796dff46ce95c37439395c77ec P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:28.069059 711 leader_election.cc:304] T 64cda8796dff46ce95c37439395c77ec P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:28.070039 1669 raft_consensus.cc:2743] T 64cda8796dff46ce95c37439395c77ec P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:28.072616 1041 tablet_service.cc:1467] Processing CreateTablet for tablet 432bcbc7ba6a44ba98cfe588f8292e32 (DEFAULT_TABLE table=default.Table_daf955e3f47f4bdf8c76884ea4054463 [id=bc2a1270521e43d7941d1e2b421bd4ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:28.072703 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet ee7582ee04494c0db1179359bcb311c8 (table default.Table_c8bea6af909a424c9101c8934d1aee7b [id=bd9d66f2d35446e9bc4ed5e589f23012]) successfully deleted
W20250113 02:51:28.073026 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 1d2fdc75d00f4e749d322ac6cd5fc410 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1d2fdc75d00f4e749d322ac6cd5fc410 already in progress: creating tablet
I20250113 02:51:28.073077 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 432bcbc7ba6a44ba98cfe588f8292e32. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:28.075594 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 0539cbbe47e64c039ad39c4b52edcc4b
I20250113 02:51:28.076440 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_44352829b8da471388d4d10f849b0cd7" } new_table_name: "default.Table_e5b4d614e4b743d7982b025ca6d297e5" modify_external_catalogs: true
I20250113 02:51:28.076542 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_5473033c0c4a464785b6f224feec7506"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:28.077085 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5473033c0c4a464785b6f224feec7506 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:28.077899 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_0f4e49c7b953418aba49a768de9b199b" } modify_external_catalogs: true
I20250113 02:51:28.078104 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_0f4e49c7b953418aba49a768de9b199b" } modify_external_catalogs: true
I20250113 02:51:28.075791 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_f568fbf7cc51459a98545f97e4745407"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:28.078558 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f568fbf7cc51459a98545f97e4745407 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:28.078881 741 tablet_service.cc:1467] Processing CreateTablet for tablet 85a82ad6e93346ecb1a0cb38b62f5786 (DEFAULT_TABLE table=default.Table_ff26c7d175d24168a34825fdee4c72a0 [id=af981d4d813541c493f8f90f0197470e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:28.079339 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 85a82ad6e93346ecb1a0cb38b62f5786. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:28.080523 448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_0e85f1c98fbc41c29ac21fbc44f01c93" } new_table_name: "default.Table_99031e0341fd4ad8a3bb04a6e222a1d0" modify_external_catalogs: true
I20250113 02:51:28.081820 745 tablet_service.cc:1467] Processing CreateTablet for tablet 5f7c91d8c018455e847e5969f7b5e1cc (DEFAULT_TABLE table=default.Table_65c5f2ca0aea479b9edd500ceb8c01bc [id=9eaf97b701fd4e79847dce3a9d259beb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:28.082768 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f7c91d8c018455e847e5969f7b5e1cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:28.083021 899 tablet_service.cc:1467] Processing CreateTablet for tablet db486978b33a45df90025988b084aecd (DEFAULT_TABLE table=default.Table_e1594759ceea473eb9f01fff92b93b10 [id=5bde153e88f648c39af2af655867bed5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:28.083465 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet db486978b33a45df90025988b084aecd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:28.088310 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 23476dc111ae430abdce3855d0ef54fe
I20250113 02:51:28.088323 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 1014e51f1c184d729552a5b34292f232
I20250113 02:51:28.088935 455 catalog_manager.cc:6596] ReplaceTablet: tablet 1014e51f1c184d729552a5b34292f232 deleted and replaced by tablet ec8196505dab4e05bb6487b566cf3144
I20250113 02:51:28.089298 755 tablet_service.cc:1467] Processing CreateTablet for tablet 432bcbc7ba6a44ba98cfe588f8292e32 (DEFAULT_TABLE table=default.Table_daf955e3f47f4bdf8c76884ea4054463 [id=bc2a1270521e43d7941d1e2b421bd4ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:28.089691 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 432bcbc7ba6a44ba98cfe588f8292e32. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:28.090266 452 catalog_manager.cc:6071] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: default.Table_6de3bf6640dc47f9b977f608c5ec8ef9 [id=bd14ce0e2b0a4a60a63022be2bc80fef] alter complete (version 3)
I20250113 02:51:28.091169 1041 tablet_service.cc:1514] Processing DeleteTablet for tablet 0539cbbe47e64c039ad39c4b52edcc4b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:27 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:28.092337 736 tablet_service.cc:1467] Processing CreateTablet for tablet f86318637d5b4e378ce47e89a5d52b07 (DEFAULT_TABLE table=default.Table_0ba8bb338353424c992f00c5940f3fbc [id=4c47c713b81e411a9c9a838e2588071c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:28.092729 736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f86318637d5b4e378ce47e89a5d52b07. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:28.093205 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 58053845187f466599750f2a62f4c3e1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 58053845187f466599750f2a62f4c3e1 already in progress: creating tablet
W20250113 02:51:28.093458 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 30ca8a716bfc4ab789b216e17567785e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 30ca8a716bfc4ab789b216e17567785e already in progress: creating tablet
I20250113 02:51:28.094097 895 tablet_service.cc:1467] Processing CreateTablet for tablet 85a82ad6e93346ecb1a0cb38b62f5786 (DEFAULT_TABLE table=default.Table_ff26c7d175d24168a34825fdee4c72a0 [id=af981d4d813541c493f8f90f0197470e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:28.095678 742 tablet_service.cc:1514] Processing DeleteTablet for tablet 23476dc111ae430abdce3855d0ef54fe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:27 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:28.096344 739 tablet_service.cc:1514] Processing DeleteTablet for tablet 0539cbbe47e64c039ad39c4b52edcc4b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:27 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:28.096771 888 tablet_service.cc:1467] Processing CreateTablet for tablet 5f7c91d8c018455e847e5969f7b5e1cc (DEFAULT_TABLE table=default.Table_65c5f2ca0aea479b9edd500ceb8c01bc [id=9eaf97b701fd4e79847dce3a9d259beb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:28.096839 742 tablet_service.cc:1514] Processing DeleteTablet for tablet 1014e51f1c184d729552a5b34292f232 with delete_type TABLET_DATA_DELETED (replaced by tablet ec8196505dab4e05bb6487b566cf3144) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:28.097301 742 tablet_service.cc:1514] Processing DeleteTablet for tablet 64cda8796dff46ce95c37439395c77ec with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:20 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:28.097802 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 85a82ad6e93346ecb1a0cb38b62f5786. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:28.098106 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet 1014e51f1c184d729552a5b34292f232 with delete_type TABLET_DATA_DELETED (replaced by tablet ec8196505dab4e05bb6487b566cf3144) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:28.098320 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet 23476dc111ae430abdce3855d0ef54fe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:27 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:28.098501 1198 tablet_replica.cc:331] T 1a26e2b5e1054aa6afe7f44bf8d43108 P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:28.098701 1198 raft_consensus.cc:2238] T 1a26e2b5e1054aa6afe7f44bf8d43108 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:28.098872 1198 raft_consensus.cc:2267] T 1a26e2b5e1054aa6afe7f44bf8d43108 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:28.099447 1198 ts_tablet_manager.cc:1905] T 1a26e2b5e1054aa6afe7f44bf8d43108 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:28.099951 1195 tablet_replica.cc:331] T 64cda8796dff46ce95c37439395c77ec P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:28.100036 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f7c91d8c018455e847e5969f7b5e1cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:28.100157 1195 raft_consensus.cc:2238] T 64cda8796dff46ce95c37439395c77ec P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:28.100323 1195 raft_consensus.cc:2267] T 64cda8796dff46ce95c37439395c77ec P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:28.101550 1195 ts_tablet_manager.cc:1905] T 64cda8796dff46ce95c37439395c77ec P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250113 02:51:28.103724 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 23476dc111ae430abdce3855d0ef54fe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 23476dc111ae430abdce3855d0ef54fe already in progress: creating tablet
W20250113 02:51:28.104254 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 0539cbbe47e64c039ad39c4b52edcc4b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0539cbbe47e64c039ad39c4b52edcc4b already in progress: creating tablet
W20250113 02:51:28.104552 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 1014e51f1c184d729552a5b34292f232 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1014e51f1c184d729552a5b34292f232 already in progress: creating tablet
W20250113 02:51:28.105108 439 consensus_peers.cc:487] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9 -> Peer 45e9d43ecac0401fb2c18fe5eaf089b5 (127.0.104.60:42067): Couldn't send request to peer 45e9d43ecac0401fb2c18fe5eaf089b5. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.60:42067: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250113 02:51:28.106163 1191 tablet_bootstrap.cc:492] T 7d53de6379b84103ba905effb0835676 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:28.106714 1191 ts_tablet_manager.cc:1397] T 7d53de6379b84103ba905effb0835676 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.044s user 0.001s sys 0.002s
I20250113 02:51:28.107744 1191 raft_consensus.cc:357] T 7d53de6379b84103ba905effb0835676 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:28.108760 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_4e3ba1ddebc74043b121c831b4abacd8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:28.109687 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4e3ba1ddebc74043b121c831b4abacd8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:28.110510 1191 raft_consensus.cc:383] T 7d53de6379b84103ba905effb0835676 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:28.110677 1191 raft_consensus.cc:738] T 7d53de6379b84103ba905effb0835676 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:28.111096 1198 ts_tablet_manager.cc:1918] T 1a26e2b5e1054aa6afe7f44bf8d43108 P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:28.111027 1191 consensus_queue.cc:260] T 7d53de6379b84103ba905effb0835676 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:28.111820 1198 log.cc:1198] T 1a26e2b5e1054aa6afe7f44bf8d43108 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/1a26e2b5e1054aa6afe7f44bf8d43108
I20250113 02:51:28.111831 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_fc22714e47dd45e6890bf2935057344e" } new_table_name: "default.Table_642669d7ef62419bb9a386144ab4e234" modify_external_catalogs: true
I20250113 02:51:28.112524 1198 ts_tablet_manager.cc:1939] T 1a26e2b5e1054aa6afe7f44bf8d43108 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:28.113483 449 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_064659c197854029a5d7af1798def6d9" } modify_external_catalogs: true
I20250113 02:51:28.113638 449 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_064659c197854029a5d7af1798def6d9" } modify_external_catalogs: true
I20250113 02:51:28.114959 1198 tablet_replica.cc:331] T 501d4e1ae84e4795913035f8a19cc32d P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:28.115424 1198 raft_consensus.cc:2238] T 501d4e1ae84e4795913035f8a19cc32d P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:28.115968 1198 raft_consensus.cc:2267] T 501d4e1ae84e4795913035f8a19cc32d P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:28.116226 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet 1a26e2b5e1054aa6afe7f44bf8d43108 (table default.Table_b5da750f5930453fa3951ffba96c34d4 [id=fa5c22ac7c024f0aba13091163742391]) successfully deleted
I20250113 02:51:28.116979 1198 ts_tablet_manager.cc:1905] T 501d4e1ae84e4795913035f8a19cc32d P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:28.123070 1195 ts_tablet_manager.cc:1918] T 64cda8796dff46ce95c37439395c77ec P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:28.123234 1195 log.cc:1198] T 64cda8796dff46ce95c37439395c77ec P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/64cda8796dff46ce95c37439395c77ec
I20250113 02:51:28.129233 1025 tablet_service.cc:1467] Processing CreateTablet for tablet f86318637d5b4e378ce47e89a5d52b07 (DEFAULT_TABLE table=default.Table_0ba8bb338353424c992f00c5940f3fbc [id=4c47c713b81e411a9c9a838e2588071c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:28.130177 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f86318637d5b4e378ce47e89a5d52b07. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:28.131294 741 tablet_service.cc:1467] Processing CreateTablet for tablet 316cb2863ed645158a8f72e3cb0a94fd (DEFAULT_TABLE table=default.Table_e392f2cbf2f14aa39320676634a32819 [id=973261b5e9fb4f7cb76c87508947c977]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:28.134174 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 316cb2863ed645158a8f72e3cb0a94fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:28.131897 1198 ts_tablet_manager.cc:1918] T 501d4e1ae84e4795913035f8a19cc32d P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:28.134477 1198 log.cc:1198] T 501d4e1ae84e4795913035f8a19cc32d P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/501d4e1ae84e4795913035f8a19cc32d
I20250113 02:51:28.134975 1198 ts_tablet_manager.cc:1939] T 501d4e1ae84e4795913035f8a19cc32d P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:28.136389 1198 tablet_replica.cc:331] T 601c71d5f7624fbb87b365d2e775d33e P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:28.137049 1198 raft_consensus.cc:2238] T 601c71d5f7624fbb87b365d2e775d33e P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:28.137761 1198 raft_consensus.cc:2267] T 601c71d5f7624fbb87b365d2e775d33e P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:28.139063 1198 ts_tablet_manager.cc:1905] T 601c71d5f7624fbb87b365d2e775d33e P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:28.133762 1195 ts_tablet_manager.cc:1939] T 64cda8796dff46ce95c37439395c77ec P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:28.131783 1191 ts_tablet_manager.cc:1428] T 7d53de6379b84103ba905effb0835676 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.025s user 0.001s sys 0.001s
I20250113 02:51:28.142760 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet 501d4e1ae84e4795913035f8a19cc32d (table default.Table_8a5b45e248e94efb9b16df971fb9755c [id=bd54b5a058a44d718071be41b1e29a8c]) successfully deleted
I20250113 02:51:28.143168 1027 tablet_service.cc:1467] Processing CreateTablet for tablet 316cb2863ed645158a8f72e3cb0a94fd (DEFAULT_TABLE table=default.Table_e392f2cbf2f14aa39320676634a32819 [id=973261b5e9fb4f7cb76c87508947c977]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:28.143524 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 316cb2863ed645158a8f72e3cb0a94fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:28.144156 745 tablet_service.cc:1467] Processing CreateTablet for tablet d95efbdafa354795835cc7a48e4b3052 (DEFAULT_TABLE table=default.Table_73371ec3d0f94477b851fc6ce3028d8e [id=55164122fb9a44e096b1e792e39bfce8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:28.144711 1191 tablet_bootstrap.cc:492] T 501d4e1ae84e4795913035f8a19cc32d P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:28.146473 1191 tablet_bootstrap.cc:654] T 501d4e1ae84e4795913035f8a19cc32d P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:28.150426 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d95efbdafa354795835cc7a48e4b3052. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:28.152824 1029 tablet_service.cc:1467] Processing CreateTablet for tablet d95efbdafa354795835cc7a48e4b3052 (DEFAULT_TABLE table=default.Table_73371ec3d0f94477b851fc6ce3028d8e [id=55164122fb9a44e096b1e792e39bfce8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:28.153227 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d95efbdafa354795835cc7a48e4b3052. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:28.154778 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet b5205b359d234babb89707a54a7f968f
I20250113 02:51:28.156497 755 tablet_service.cc:1514] Processing DeleteTablet for tablet b5205b359d234babb89707a54a7f968f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:28 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:28.157024 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet 64cda8796dff46ce95c37439395c77ec (table default.Table_45ad8711e23e44f0b7b83b7a6128c427 [id=983d3b296c064375a1f2747ce556f01d]) successfully deleted
W20250113 02:51:28.157732 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet b5205b359d234babb89707a54a7f968f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b5205b359d234babb89707a54a7f968f already in progress: creating tablet
I20250113 02:51:28.158084 1041 tablet_service.cc:1514] Processing DeleteTablet for tablet b5205b359d234babb89707a54a7f968f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:28 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:28.158294 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_7e37949b2b354eb39b0d00057ee5dc2b" } modify_external_catalogs: true
I20250113 02:51:28.158460 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_7e37949b2b354eb39b0d00057ee5dc2b" } modify_external_catalogs: true
I20250113 02:51:28.159716 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_770cceffb7e940c7a4dfad0fcab73863"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:28.160427 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_770cceffb7e940c7a4dfad0fcab73863 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:28.168915 1191 tablet_bootstrap.cc:492] T 501d4e1ae84e4795913035f8a19cc32d P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:28.169152 1191 ts_tablet_manager.cc:1397] T 501d4e1ae84e4795913035f8a19cc32d P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.024s user 0.002s sys 0.002s
I20250113 02:51:28.169925 887 tablet_service.cc:1467] Processing CreateTablet for tablet 432bcbc7ba6a44ba98cfe588f8292e32 (DEFAULT_TABLE table=default.Table_daf955e3f47f4bdf8c76884ea4054463 [id=bc2a1270521e43d7941d1e2b421bd4ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:28.169946 1191 raft_consensus.cc:357] T 501d4e1ae84e4795913035f8a19cc32d P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:28.170156 1191 raft_consensus.cc:383] T 501d4e1ae84e4795913035f8a19cc32d P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:28.170212 1191 raft_consensus.cc:738] T 501d4e1ae84e4795913035f8a19cc32d P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:28.170432 1191 consensus_queue.cc:260] T 501d4e1ae84e4795913035f8a19cc32d P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:28.170969 1191 ts_tablet_manager.cc:1428] T 501d4e1ae84e4795913035f8a19cc32d P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250113 02:51:28.171377 1669 raft_consensus.cc:491] T ee7582ee04494c0db1179359bcb311c8 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:28.171536 1669 raft_consensus.cc:513] T ee7582ee04494c0db1179359bcb311c8 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:28.172271 1669 leader_election.cc:290] T ee7582ee04494c0db1179359bcb311c8 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:28.172746 1191 tablet_bootstrap.cc:492] T 8adfa738c17a4169a5d99f6ceca75100 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:28.174757 1191 tablet_bootstrap.cc:654] T 8adfa738c17a4169a5d99f6ceca75100 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:28.175850 1198 ts_tablet_manager.cc:1918] T 601c71d5f7624fbb87b365d2e775d33e P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250113 02:51:28.175911 709 leader_election.cc:343] T ee7582ee04494c0db1179359bcb311c8 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Not found: Tablet not found: ee7582ee04494c0db1179359bcb311c8
I20250113 02:51:28.175999 1198 log.cc:1198] T 601c71d5f7624fbb87b365d2e775d33e P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/601c71d5f7624fbb87b365d2e775d33e
I20250113 02:51:28.176323 1198 ts_tablet_manager.cc:1939] T 601c71d5f7624fbb87b365d2e775d33e P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:28.182854 889 tablet_service.cc:1467] Processing CreateTablet for tablet f86318637d5b4e378ce47e89a5d52b07 (DEFAULT_TABLE table=default.Table_0ba8bb338353424c992f00c5940f3fbc [id=4c47c713b81e411a9c9a838e2588071c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:28.183300 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 432bcbc7ba6a44ba98cfe588f8292e32. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:28.183599 1198 tablet_replica.cc:331] T 258fad818b7b4aa4b337f9949e20aa46 P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:28.183782 1196 ts_tablet_manager.cc:1918] T ba61c86743e24666b30580c36b9228c3 P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:28.184128 1198 raft_consensus.cc:2238] T 258fad818b7b4aa4b337f9949e20aa46 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:28.184638 1198 raft_consensus.cc:2267] T 258fad818b7b4aa4b337f9949e20aa46 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:28.185740 1198 ts_tablet_manager.cc:1905] T 258fad818b7b4aa4b337f9949e20aa46 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:28.186126 1191 tablet_bootstrap.cc:492] T 8adfa738c17a4169a5d99f6ceca75100 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:28.186336 1191 ts_tablet_manager.cc:1397] T 8adfa738c17a4169a5d99f6ceca75100 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.005s
I20250113 02:51:28.186538 1196 log.cc:1198] T ba61c86743e24666b30580c36b9228c3 P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/ba61c86743e24666b30580c36b9228c3
I20250113 02:51:28.187836 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f86318637d5b4e378ce47e89a5d52b07. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:28.188160 1191 raft_consensus.cc:357] T 8adfa738c17a4169a5d99f6ceca75100 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:28.188465 1191 raft_consensus.cc:383] T 8adfa738c17a4169a5d99f6ceca75100 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:28.188583 1191 raft_consensus.cc:738] T 8adfa738c17a4169a5d99f6ceca75100 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:28.188906 1634 raft_consensus.cc:491] T 55ebe9c044d944bc813396c699ba9aa1 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:28.188925 1191 consensus_queue.cc:260] T 8adfa738c17a4169a5d99f6ceca75100 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:28.189069 1634 raft_consensus.cc:513] T 55ebe9c044d944bc813396c699ba9aa1 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:28.189433 1191 ts_tablet_manager.cc:1428] T 8adfa738c17a4169a5d99f6ceca75100 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250113 02:51:28.189734 1191 tablet_bootstrap.cc:492] T 601c71d5f7624fbb87b365d2e775d33e P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:28.189774 1634 leader_election.cc:290] T 55ebe9c044d944bc813396c699ba9aa1 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
I20250113 02:51:28.190477 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55ebe9c044d944bc813396c699ba9aa1" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
W20250113 02:51:28.192165 995 leader_election.cc:343] T 55ebe9c044d944bc813396c699ba9aa1 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:28.193671 1191 tablet_bootstrap.cc:654] T 601c71d5f7624fbb87b365d2e775d33e P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:28.197305 896 tablet_service.cc:1514] Processing DeleteTablet for tablet 0539cbbe47e64c039ad39c4b52edcc4b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:27 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:28.197708 885 tablet_service.cc:1514] Processing DeleteTablet for tablet 23476dc111ae430abdce3855d0ef54fe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:27 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:28.197979 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee7582ee04494c0db1179359bcb311c8" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:28.198694 711 leader_election.cc:343] T ee7582ee04494c0db1179359bcb311c8 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:28.198870 711 leader_election.cc:304] T ee7582ee04494c0db1179359bcb311c8 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:28.199357 1669 raft_consensus.cc:2743] T ee7582ee04494c0db1179359bcb311c8 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:28.199780 1198 ts_tablet_manager.cc:1918] T 258fad818b7b4aa4b337f9949e20aa46 P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:28.200112 1198 log.cc:1198] T 258fad818b7b4aa4b337f9949e20aa46 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/258fad818b7b4aa4b337f9949e20aa46
I20250113 02:51:28.200704 1198 ts_tablet_manager.cc:1939] T 258fad818b7b4aa4b337f9949e20aa46 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
W20250113 02:51:28.204541 441 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet e8ab4516789e4585a101c8766d338af3 (table default.Table_3b76cf9fac334ebf9b1ecb9f8d07ea74 [id=b112ac4e2e58460db083da1eb788ca07], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet e8ab4516789e4585a101c8766d338af3 consensus state has no leader
I20250113 02:51:28.207481 1191 tablet_bootstrap.cc:492] T 601c71d5f7624fbb87b365d2e775d33e P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:28.208097 1196 ts_tablet_manager.cc:1939] T ba61c86743e24666b30580c36b9228c3 P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:28.208241 1191 ts_tablet_manager.cc:1397] T 601c71d5f7624fbb87b365d2e775d33e P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.000s
I20250113 02:51:28.209030 1191 raft_consensus.cc:357] T 601c71d5f7624fbb87b365d2e775d33e P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:28.209165 885 tablet_service.cc:1514] Processing DeleteTablet for tablet 1014e51f1c184d729552a5b34292f232 with delete_type TABLET_DATA_DELETED (replaced by tablet ec8196505dab4e05bb6487b566cf3144) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:28.209339 1191 raft_consensus.cc:383] T 601c71d5f7624fbb87b365d2e775d33e P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:28.209452 1191 raft_consensus.cc:738] T 601c71d5f7624fbb87b365d2e775d33e P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:28.209697 1191 consensus_queue.cc:260] T 601c71d5f7624fbb87b365d2e775d33e P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:28.212198 1191 ts_tablet_manager.cc:1428] T 601c71d5f7624fbb87b365d2e775d33e P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250113 02:51:28.212534 1191 tablet_bootstrap.cc:492] T 258fad818b7b4aa4b337f9949e20aa46 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:28.214761 885 tablet_service.cc:1467] Processing CreateTablet for tablet 316cb2863ed645158a8f72e3cb0a94fd (DEFAULT_TABLE table=default.Table_e392f2cbf2f14aa39320676634a32819 [id=973261b5e9fb4f7cb76c87508947c977]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:28.215617 1191 tablet_bootstrap.cc:654] T 258fad818b7b4aa4b337f9949e20aa46 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:28.223026 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet ba61c86743e24666b30580c36b9228c3 (table default.Table_3f75f23afe2f449fa7a1f9df1c0734fb [id=68a4c317da7143cf841ee2d869d2f754]) successfully deleted
I20250113 02:51:28.224380 896 tablet_service.cc:1467] Processing CreateTablet for tablet d95efbdafa354795835cc7a48e4b3052 (DEFAULT_TABLE table=default.Table_73371ec3d0f94477b851fc6ce3028d8e [id=55164122fb9a44e096b1e792e39bfce8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:28.225036 898 tablet_service.cc:1514] Processing DeleteTablet for tablet b5205b359d234babb89707a54a7f968f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:28 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:28.225880 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 316cb2863ed645158a8f72e3cb0a94fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:28.227214 1634 raft_consensus.cc:491] T 8adfa738c17a4169a5d99f6ceca75100 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:28.227464 1634 raft_consensus.cc:513] T 8adfa738c17a4169a5d99f6ceca75100 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:28.228384 1634 leader_election.cc:290] T 8adfa738c17a4169a5d99f6ceca75100 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
I20250113 02:51:28.229121 1191 tablet_bootstrap.cc:492] T 258fad818b7b4aa4b337f9949e20aa46 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:28.229136 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d95efbdafa354795835cc7a48e4b3052. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:28.229171 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8adfa738c17a4169a5d99f6ceca75100" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
I20250113 02:51:28.229339 1191 ts_tablet_manager.cc:1397] T 258fad818b7b4aa4b337f9949e20aa46 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.002s
I20250113 02:51:28.229413 772 raft_consensus.cc:2463] T 8adfa738c17a4169a5d99f6ceca75100 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 0.
I20250113 02:51:28.229904 995 leader_election.cc:304] T 8adfa738c17a4169a5d99f6ceca75100 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters:
I20250113 02:51:28.230283 1634 raft_consensus.cc:2798] T 8adfa738c17a4169a5d99f6ceca75100 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 02:51:28.230173 1191 raft_consensus.cc:357] T 258fad818b7b4aa4b337f9949e20aa46 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:28.230381 1191 raft_consensus.cc:383] T 258fad818b7b4aa4b337f9949e20aa46 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:28.230381 1634 raft_consensus.cc:491] T 8adfa738c17a4169a5d99f6ceca75100 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 02:51:28.230463 1191 raft_consensus.cc:738] T 258fad818b7b4aa4b337f9949e20aa46 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:28.230487 1634 raft_consensus.cc:3054] T 8adfa738c17a4169a5d99f6ceca75100 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:28.230721 1191 consensus_queue.cc:260] T 258fad818b7b4aa4b337f9949e20aa46 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:28.233513 1191 ts_tablet_manager.cc:1428] T 258fad818b7b4aa4b337f9949e20aa46 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250113 02:51:28.233780 1191 tablet_bootstrap.cc:492] T fdbf03c89bff438a9a25115eddf60be9 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:28.237401 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55ebe9c044d944bc813396c699ba9aa1" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:28.238670 1191 tablet_bootstrap.cc:654] T fdbf03c89bff438a9a25115eddf60be9 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:28.238772 1634 raft_consensus.cc:513] T 8adfa738c17a4169a5d99f6ceca75100 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:28.239784 1634 leader_election.cc:290] T 8adfa738c17a4169a5d99f6ceca75100 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Requested vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
I20250113 02:51:28.240036 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8adfa738c17a4169a5d99f6ceca75100" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196"
I20250113 02:51:28.240249 772 raft_consensus.cc:3054] T 8adfa738c17a4169a5d99f6ceca75100 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:28.242938 772 raft_consensus.cc:2463] T 8adfa738c17a4169a5d99f6ceca75100 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 1.
I20250113 02:51:28.243386 995 leader_election.cc:304] T 8adfa738c17a4169a5d99f6ceca75100 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters:
I20250113 02:51:28.243994 1634 raft_consensus.cc:2798] T 8adfa738c17a4169a5d99f6ceca75100 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Leader election won for term 1
I20250113 02:51:28.244203 1634 raft_consensus.cc:695] T 8adfa738c17a4169a5d99f6ceca75100 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: Becoming Leader. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Running, Role: LEADER
I20250113 02:51:28.245342 1634 consensus_queue.cc:237] T 8adfa738c17a4169a5d99f6ceca75100 P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:28.247272 1196 tablet_replica.cc:331] T d5fbf97536a2472fa091168363bf9549 P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:28.252717 1196 raft_consensus.cc:2238] T d5fbf97536a2472fa091168363bf9549 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:28.263224 1191 tablet_bootstrap.cc:492] T fdbf03c89bff438a9a25115eddf60be9 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:28.264739 1191 ts_tablet_manager.cc:1397] T fdbf03c89bff438a9a25115eddf60be9 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.031s user 0.001s sys 0.009s
I20250113 02:51:28.265699 1191 raft_consensus.cc:357] T fdbf03c89bff438a9a25115eddf60be9 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
W20250113 02:51:28.266316 998 leader_election.cc:343] T 55ebe9c044d944bc813396c699ba9aa1 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:28.266501 998 leader_election.cc:304] T 55ebe9c044d944bc813396c699ba9aa1 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4216a7f853d642b7bc3e5cff458f3aca; no voters: 1186fa869eb14812832fa6562531f196, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:28.267062 1734 raft_consensus.cc:2743] T 55ebe9c044d944bc813396c699ba9aa1 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:28.268002 1191 raft_consensus.cc:383] T fdbf03c89bff438a9a25115eddf60be9 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:28.268088 1191 raft_consensus.cc:738] T fdbf03c89bff438a9a25115eddf60be9 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:28.268071 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8adfa738c17a4169a5d99f6ceca75100" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f"
I20250113 02:51:28.268390 1191 consensus_queue.cc:260] T fdbf03c89bff438a9a25115eddf60be9 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:28.270131 1191 ts_tablet_manager.cc:1428] T fdbf03c89bff438a9a25115eddf60be9 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20250113 02:51:28.271183 1191 tablet_bootstrap.cc:492] T e8ab4516789e4585a101c8766d338af3 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:28.273420 1191 tablet_bootstrap.cc:654] T e8ab4516789e4585a101c8766d338af3 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
W20250113 02:51:28.274863 998 leader_election.cc:343] T 8adfa738c17a4169a5d99f6ceca75100 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:28.270969 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8adfa738c17a4169a5d99f6ceca75100" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:28.278434 998 leader_election.cc:343] T 8adfa738c17a4169a5d99f6ceca75100 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:28.282632 1438 raft_consensus.cc:491] T a6055d77d168495d9b756a1a56c1a920 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:28.286897 1196 raft_consensus.cc:2267] T d5fbf97536a2472fa091168363bf9549 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:28.287276 1438 raft_consensus.cc:513] T a6055d77d168495d9b756a1a56c1a920 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:28.289474 1438 leader_election.cc:290] T a6055d77d168495d9b756a1a56c1a920 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
W20250113 02:51:28.290732 852 leader_election.cc:343] T a6055d77d168495d9b756a1a56c1a920 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Not found: Tablet not found: a6055d77d168495d9b756a1a56c1a920
I20250113 02:51:28.292618 1191 tablet_bootstrap.cc:492] T e8ab4516789e4585a101c8766d338af3 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:28.292762 1191 ts_tablet_manager.cc:1397] T e8ab4516789e4585a101c8766d338af3 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.022s user 0.000s sys 0.005s
I20250113 02:51:28.293478 1191 raft_consensus.cc:357] T e8ab4516789e4585a101c8766d338af3 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:28.293658 1191 raft_consensus.cc:383] T e8ab4516789e4585a101c8766d338af3 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:28.293730 1191 raft_consensus.cc:738] T e8ab4516789e4585a101c8766d338af3 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:28.294313 1191 consensus_queue.cc:260] T e8ab4516789e4585a101c8766d338af3 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:28.296712 1191 ts_tablet_manager.cc:1428] T e8ab4516789e4585a101c8766d338af3 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250113 02:51:28.297031 1191 tablet_bootstrap.cc:492] T 2c399784af46426b8327d175c7a6cce8 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:28.298708 1191 tablet_bootstrap.cc:654] T 2c399784af46426b8327d175c7a6cce8 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:28.300185 1669 raft_consensus.cc:491] T 5b1076a5a17b4fa495975285616401cf P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:28.300312 1669 raft_consensus.cc:513] T 5b1076a5a17b4fa495975285616401cf P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:28.300837 1669 leader_election.cc:290] T 5b1076a5a17b4fa495975285616401cf P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
W20250113 02:51:28.301895 709 leader_election.cc:343] T 5b1076a5a17b4fa495975285616401cf P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Not found: Tablet not found: 5b1076a5a17b4fa495975285616401cf
I20250113 02:51:28.305124 1196 ts_tablet_manager.cc:1905] T d5fbf97536a2472fa091168363bf9549 P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:28.311029 1191 tablet_bootstrap.cc:492] T 2c399784af46426b8327d175c7a6cce8 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:28.311192 1191 ts_tablet_manager.cc:1397] T 2c399784af46426b8327d175c7a6cce8 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.004s
W20250113 02:51:28.311380 851 leader_election.cc:343] T a6055d77d168495d9b756a1a56c1a920 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Not found: Tablet not found: a6055d77d168495d9b756a1a56c1a920
I20250113 02:51:28.311928 1191 raft_consensus.cc:357] T 2c399784af46426b8327d175c7a6cce8 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:28.312165 1191 raft_consensus.cc:383] T 2c399784af46426b8327d175c7a6cce8 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:28.312235 1191 raft_consensus.cc:738] T 2c399784af46426b8327d175c7a6cce8 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:28.312335 851 leader_election.cc:304] T a6055d77d168495d9b756a1a56c1a920 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df69f48656b44bb890f76587ee69cc0f; no voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca
I20250113 02:51:28.312440 1191 consensus_queue.cc:260] T 2c399784af46426b8327d175c7a6cce8 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:28.312927 1191 ts_tablet_manager.cc:1428] T 2c399784af46426b8327d175c7a6cce8 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250113 02:51:28.313194 1191 tablet_bootstrap.cc:492] T 19f14b3db0b64847b5422bfd5ed323d0 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:28.314802 1191 tablet_bootstrap.cc:654] T 19f14b3db0b64847b5422bfd5ed323d0 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:28.315228 1438 raft_consensus.cc:2743] T a6055d77d168495d9b756a1a56c1a920 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:28.316478 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1076a5a17b4fa495975285616401cf" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:28.322814 711 leader_election.cc:343] T 5b1076a5a17b4fa495975285616401cf P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:28.323046 711 leader_election.cc:304] T 5b1076a5a17b4fa495975285616401cf P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:28.323660 1669 raft_consensus.cc:2743] T 5b1076a5a17b4fa495975285616401cf P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:28.324128 1734 raft_consensus.cc:491] T f760fc1e4bed433dbb987a42c12a1ee6 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250113 02:51:28.319886 1706 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
I20250113 02:51:28.324345 1734 raft_consensus.cc:513] T f760fc1e4bed433dbb987a42c12a1ee6 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
W20250113 02:51:28.324599 1706 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250113 02:51:28.324738 1706 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20250113 02:51:28.325230 1734 leader_election.cc:290] T f760fc1e4bed433dbb987a42c12a1ee6 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
I20250113 02:51:28.325929 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f760fc1e4bed433dbb987a42c12a1ee6" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
W20250113 02:51:28.326514 995 leader_election.cc:343] T f760fc1e4bed433dbb987a42c12a1ee6 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:28.327116 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f760fc1e4bed433dbb987a42c12a1ee6" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:28.328179 998 leader_election.cc:343] T f760fc1e4bed433dbb987a42c12a1ee6 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:28.328337 998 leader_election.cc:304] T f760fc1e4bed433dbb987a42c12a1ee6 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4216a7f853d642b7bc3e5cff458f3aca; no voters: 1186fa869eb14812832fa6562531f196, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:28.328985 1734 raft_consensus.cc:2743] T f760fc1e4bed433dbb987a42c12a1ee6 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:28.329142 1191 tablet_bootstrap.cc:492] T 19f14b3db0b64847b5422bfd5ed323d0 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:28.329288 1191 ts_tablet_manager.cc:1397] T 19f14b3db0b64847b5422bfd5ed323d0 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.001s
I20250113 02:51:28.330017 1191 raft_consensus.cc:357] T 19f14b3db0b64847b5422bfd5ed323d0 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:28.330212 1191 raft_consensus.cc:383] T 19f14b3db0b64847b5422bfd5ed323d0 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:28.330296 1191 raft_consensus.cc:738] T 19f14b3db0b64847b5422bfd5ed323d0 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:28.330543 1191 consensus_queue.cc:260] T 19f14b3db0b64847b5422bfd5ed323d0 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:28.338029 1191 ts_tablet_manager.cc:1428] T 19f14b3db0b64847b5422bfd5ed323d0 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250113 02:51:28.338442 1191 tablet_bootstrap.cc:492] T 55ebe9c044d944bc813396c699ba9aa1 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:28.340176 1191 tablet_bootstrap.cc:654] T 55ebe9c044d944bc813396c699ba9aa1 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
W20250113 02:51:28.371695 1706 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250113 02:51:28.372175 1706 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250113 02:51:28.372581 1706 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250113 02:51:28.372850 1706 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250113 02:51:28.385486 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet 601c71d5f7624fbb87b365d2e775d33e (table default.Table_7a9c9cc8678e4f7fa12237d3f8c9c3de [id=97a27e2641d2468aac5c827fad4e2091]) successfully deleted
I20250113 02:51:28.385804 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet 258fad818b7b4aa4b337f9949e20aa46 (table default.Table_e44b0cdc7f4640ec867d92338d94426b [id=9538429e42bb48c7b7fbae43644f2b2d]) successfully deleted
W20250113 02:51:28.386054 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet c6820d6331e44656a27e23afe973e2e2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c6820d6331e44656a27e23afe973e2e2 already in progress: creating tablet
W20250113 02:51:28.386307 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet e570465a3d4046b1ac19a3a034fe91f1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e570465a3d4046b1ac19a3a034fe91f1 already in progress: creating tablet
W20250113 02:51:28.422277 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet ed68a2b7ebcf4841bef54798dd3ed80d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ed68a2b7ebcf4841bef54798dd3ed80d already in progress: creating tablet
W20250113 02:51:28.422600 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 0539cbbe47e64c039ad39c4b52edcc4b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0539cbbe47e64c039ad39c4b52edcc4b already in progress: creating tablet
W20250113 02:51:28.422837 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 1014e51f1c184d729552a5b34292f232 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1014e51f1c184d729552a5b34292f232 already in progress: creating tablet
W20250113 02:51:28.423074 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 23476dc111ae430abdce3855d0ef54fe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 23476dc111ae430abdce3855d0ef54fe already in progress: creating tablet
W20250113 02:51:28.423302 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet b5205b359d234babb89707a54a7f968f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b5205b359d234babb89707a54a7f968f already in progress: creating tablet
I20250113 02:51:28.433936 1706 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:45573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/master-2/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:44217,127.0.104.61:35271,127.0.104.60:42067
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.60:42067
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.104.60
--webserver_port=38411
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 13 Jan 2025 02:43:23 UTC on 24a791456cd2
build id 4151
ASAN enabled
W20250113 02:51:28.444825 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 938cc1ccd8dd449996836188ba9e64ff consensus state has no leader
I20250113 02:51:28.448139 1706 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250113 02:51:28.448231 454 master_service.cc:946] ReplaceTablet: received request to replace tablet c6820d6331e44656a27e23afe973e2e2 from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:28.449618 1194 tablet_bootstrap.cc:492] T bcb59573e82743858bd39bde64627d6d P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:28.449858 1194 ts_tablet_manager.cc:1397] T bcb59573e82743858bd39bde64627d6d P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.766s user 0.004s sys 0.009s
I20250113 02:51:28.455837 1669 raft_consensus.cc:491] T 19f14b3db0b64847b5422bfd5ed323d0 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:28.455884 1706 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250113 02:51:28.456064 1669 raft_consensus.cc:513] T 19f14b3db0b64847b5422bfd5ed323d0 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:28.456988 1669 leader_election.cc:290] T 19f14b3db0b64847b5422bfd5ed323d0 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:28.451622 1787 tablet.cc:1721] T 06f36d09bbb1443bafce906907795404 P df69f48656b44bb890f76587ee69cc0f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250113 02:51:28.458567 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19f14b3db0b64847b5422bfd5ed323d0" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:28.466252 711 leader_election.cc:343] T 19f14b3db0b64847b5422bfd5ed323d0 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:28.470379 1191 tablet_bootstrap.cc:492] T 55ebe9c044d944bc813396c699ba9aa1 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:28.470559 1191 ts_tablet_manager.cc:1397] T 55ebe9c044d944bc813396c699ba9aa1 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.132s user 0.004s sys 0.030s
I20250113 02:51:28.472189 446 catalog_manager.cc:5618] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467) does not have the latest schema for tablet 06f36d09bbb1443bafce906907795404 (table default.Table_2a627a757ab0406686564083ecc36c07 [id=423021f304964246bd29e4daaf470154]). Expected version 2 got 0
I20250113 02:51:28.472544 1191 raft_consensus.cc:357] T 55ebe9c044d944bc813396c699ba9aa1 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:28.472764 1191 raft_consensus.cc:383] T 55ebe9c044d944bc813396c699ba9aa1 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:28.472836 1191 raft_consensus.cc:738] T 55ebe9c044d944bc813396c699ba9aa1 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
W20250113 02:51:28.473385 448 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b5c1882125b5491ea79a29a6dac6771e consensus state has no leader
I20250113 02:51:28.473866 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_e6cbe3930484474596b0ef6111c7b419"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:28.474236 454 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_5e92e31ef2494f34a1fab99e7c6bd9f7" } new_table_name: "default.Table_a70bef6ff5714d398f83133e99b5d9d5" modify_external_catalogs: true
W20250113 02:51:28.474506 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e6cbe3930484474596b0ef6111c7b419 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:28.479414 1191 consensus_queue.cc:260] T 55ebe9c044d944bc813396c699ba9aa1 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:28.483821 1191 ts_tablet_manager.cc:1428] T 55ebe9c044d944bc813396c699ba9aa1 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.013s user 0.001s sys 0.001s
I20250113 02:51:28.485409 1191 tablet_bootstrap.cc:492] T f760fc1e4bed433dbb987a42c12a1ee6 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:28.487247 1191 tablet_bootstrap.cc:654] T f760fc1e4bed433dbb987a42c12a1ee6 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
W20250113 02:51:28.494946 440 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.61:35271, user_credentials={real_user=slave}} blocked reactor thread for 71345.4us
I20250113 02:51:28.499781 1191 tablet_bootstrap.cc:492] T f760fc1e4bed433dbb987a42c12a1ee6 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:28.501219 1191 ts_tablet_manager.cc:1397] T f760fc1e4bed433dbb987a42c12a1ee6 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.003s
I20250113 02:51:28.502511 1191 raft_consensus.cc:357] T f760fc1e4bed433dbb987a42c12a1ee6 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:28.504773 1191 raft_consensus.cc:383] T f760fc1e4bed433dbb987a42c12a1ee6 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:28.505149 1191 raft_consensus.cc:738] T f760fc1e4bed433dbb987a42c12a1ee6 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:28.505780 1191 consensus_queue.cc:260] T f760fc1e4bed433dbb987a42c12a1ee6 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:28.508270 1196 ts_tablet_manager.cc:1918] T d5fbf97536a2472fa091168363bf9549 P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:28.508437 1196 log.cc:1198] T d5fbf97536a2472fa091168363bf9549 P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/d5fbf97536a2472fa091168363bf9549
I20250113 02:51:28.509097 1196 ts_tablet_manager.cc:1939] T d5fbf97536a2472fa091168363bf9549 P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:28.509598 1196 tablet_replica.cc:331] T a6055d77d168495d9b756a1a56c1a920 P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:28.510620 1191 ts_tablet_manager.cc:1428] T f760fc1e4bed433dbb987a42c12a1ee6 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20250113 02:51:28.510764 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet d5fbf97536a2472fa091168363bf9549 (table default.Table_abea3b2ec5144f1fadd95a623c48917c [id=cdb71df855274f13a405028421724d37]) successfully deleted
I20250113 02:51:28.511852 1191 tablet_bootstrap.cc:492] T f31dd916336a4d8fb3ab015af7de8ccb P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:28.519611 1196 raft_consensus.cc:2238] T a6055d77d168495d9b756a1a56c1a920 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:28.519862 1196 raft_consensus.cc:2267] T a6055d77d168495d9b756a1a56c1a920 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:28.522899 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_874095afd8f14f9b97268aafb4a38bee" } new_table_name: "default.Table_c22142b5375a4837bbc42af6f77d880f" modify_external_catalogs: true
W20250113 02:51:28.524291 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c4761d692adc405e88d45ca6e748140b consensus state has no leader
I20250113 02:51:28.524338 1191 tablet_bootstrap.cc:654] T f31dd916336a4d8fb3ab015af7de8ccb P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
W20250113 02:51:28.525727 1797 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 02:51:28.526100 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 98bfe9620e284ab1954433ba9a036392
I20250113 02:51:28.527251 741 tablet_service.cc:1514] Processing DeleteTablet for tablet 98bfe9620e284ab1954433ba9a036392 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:28 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:28.529038 1706 server_base.cc:1034] running on GCE node
I20250113 02:51:28.530077 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_e9dd2f1b0ade4fc6a1db103979189b11"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:28.530791 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19f14b3db0b64847b5422bfd5ed323d0" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:28.531462 1191 tablet_bootstrap.cc:492] T f31dd916336a4d8fb3ab015af7de8ccb P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:28.531627 1191 ts_tablet_manager.cc:1397] T f31dd916336a4d8fb3ab015af7de8ccb P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.001s
I20250113 02:51:28.531934 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_9a4ca43a32a04beabb2108aae12c3b96"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:28.532472 1191 raft_consensus.cc:357] T f31dd916336a4d8fb3ab015af7de8ccb P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
W20250113 02:51:28.532588 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9a4ca43a32a04beabb2108aae12c3b96 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:28.532688 1191 raft_consensus.cc:383] T f31dd916336a4d8fb3ab015af7de8ccb P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:28.532761 1191 raft_consensus.cc:738] T f31dd916336a4d8fb3ab015af7de8ccb P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
W20250113 02:51:28.533476 709 leader_election.cc:343] T 19f14b3db0b64847b5422bfd5ed323d0 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:28.533412 1191 consensus_queue.cc:260] T f31dd916336a4d8fb3ab015af7de8ccb P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:28.533636 709 leader_election.cc:304] T 19f14b3db0b64847b5422bfd5ed323d0 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:28.534011 1191 ts_tablet_manager.cc:1428] T f31dd916336a4d8fb3ab015af7de8ccb P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250113 02:51:28.534041 1757 raft_consensus.cc:2743] T 19f14b3db0b64847b5422bfd5ed323d0 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250113 02:51:28.534992 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e9dd2f1b0ade4fc6a1db103979189b11 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250113 02:51:28.535735 1796 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 02:51:28.510493 1194 raft_consensus.cc:357] T bcb59573e82743858bd39bde64627d6d P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:28.536165 1191 tablet_bootstrap.cc:492] T c215ffe61b9b426286cc5e808463078d P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:28.536701 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_d43add0d74a8498f835de5cd0acf0a25" } modify_external_catalogs: true
I20250113 02:51:28.536976 1194 raft_consensus.cc:383] T bcb59573e82743858bd39bde64627d6d P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:28.537086 1194 raft_consensus.cc:738] T bcb59573e82743858bd39bde64627d6d P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:28.537338 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_d43add0d74a8498f835de5cd0acf0a25" } modify_external_catalogs: true
I20250113 02:51:28.537484 1194 consensus_queue.cc:260] T bcb59573e82743858bd39bde64627d6d P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:28.537947 1191 tablet_bootstrap.cc:654] T c215ffe61b9b426286cc5e808463078d P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:28.538031 1194 ts_tablet_manager.cc:1428] T bcb59573e82743858bd39bde64627d6d P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.088s user 0.003s sys 0.000s
I20250113 02:51:28.538524 1194 tablet_bootstrap.cc:492] T 64cda8796dff46ce95c37439395c77ec P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:28.539129 1196 ts_tablet_manager.cc:1905] T a6055d77d168495d9b756a1a56c1a920 P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:28.539599 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet 98bfe9620e284ab1954433ba9a036392 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:28 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:28.540634 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_7f3a0392b4234e48835b8c3c5ec34b15"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:28.525300 452 catalog_manager.cc:6071] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: default.Table_2a627a757ab0406686564083ecc36c07 [id=423021f304964246bd29e4daaf470154] alter complete (version 2)
W20250113 02:51:28.541594 1192 thread.cc:635] log-alloc [worker] (thread pool) Time spent creating pthread: real 0.545s user 0.145s sys 0.179s
W20250113 02:51:28.540331 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 98bfe9620e284ab1954433ba9a036392 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 98bfe9620e284ab1954433ba9a036392 already in progress: creating tablet
W20250113 02:51:28.542150 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7f3a0392b4234e48835b8c3c5ec34b15 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:28.545683 745 tablet_service.cc:1514] Processing DeleteTablet for tablet ee7582ee04494c0db1179359bcb311c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:19 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:28.546088 1195 tablet_replica.cc:331] T ee7582ee04494c0db1179359bcb311c8 P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:28.546372 1195 raft_consensus.cc:2238] T ee7582ee04494c0db1179359bcb311c8 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:28.546595 1195 raft_consensus.cc:2267] T ee7582ee04494c0db1179359bcb311c8 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:28.547219 1195 ts_tablet_manager.cc:1905] T ee7582ee04494c0db1179359bcb311c8 P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:28.550139 741 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b1076a5a17b4fa495975285616401cf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:27 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:28.551106 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_ad3bb183b90444b8a9ca111bbc2d4d2a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:28.551743 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ad3bb183b90444b8a9ca111bbc2d4d2a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:28.554401 1194 tablet_bootstrap.cc:654] T 64cda8796dff46ce95c37439395c77ec P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:28.555397 741 tablet_service.cc:1514] Processing DeleteTablet for tablet b2a35fd891c645d68d79c61a2c832ba7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:20 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:28.564637 885 tablet_service.cc:1514] Processing DeleteTablet for tablet 98bfe9620e284ab1954433ba9a036392 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:28 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:28.567214 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet 55ebe9c044d944bc813396c699ba9aa1 with delete_type TABLET_DATA_DELETED (replaced by tablet 41fcde988f72496b824271fa36c1a226) from {username='slave'} at 127.0.0.1:60230
W20250113 02:51:28.567291 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 98bfe9620e284ab1954433ba9a036392 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 98bfe9620e284ab1954433ba9a036392 already in progress: creating tablet
I20250113 02:51:28.567626 1198 tablet_replica.cc:331] T 55ebe9c044d944bc813396c699ba9aa1 P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:28.572197 1198 raft_consensus.cc:2238] T 55ebe9c044d944bc813396c699ba9aa1 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:28.572405 1198 raft_consensus.cc:2267] T 55ebe9c044d944bc813396c699ba9aa1 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus is shut down!
W20250113 02:51:28.572960 1192 thread.cc:601] log-alloc [worker] (thread pool) Time spent starting thread: real 0.553s user 0.145s sys 0.180s
I20250113 02:51:28.574160 1194 tablet_bootstrap.cc:492] T 64cda8796dff46ce95c37439395c77ec P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:28.574359 1194 ts_tablet_manager.cc:1397] T 64cda8796dff46ce95c37439395c77ec P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.036s user 0.002s sys 0.002s
I20250113 02:51:28.575424 1194 raft_consensus.cc:357] T 64cda8796dff46ce95c37439395c77ec P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:28.575703 1194 raft_consensus.cc:383] T 64cda8796dff46ce95c37439395c77ec P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:28.575830 1194 raft_consensus.cc:738] T 64cda8796dff46ce95c37439395c77ec P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:28.576195 1194 consensus_queue.cc:260] T 64cda8796dff46ce95c37439395c77ec P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:28.576794 1194 ts_tablet_manager.cc:1428] T 64cda8796dff46ce95c37439395c77ec P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20250113 02:51:28.577507 1196 ts_tablet_manager.cc:1918] T a6055d77d168495d9b756a1a56c1a920 P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:28.577654 1196 log.cc:1198] T a6055d77d168495d9b756a1a56c1a920 P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/a6055d77d168495d9b756a1a56c1a920
I20250113 02:51:28.577840 1198 ts_tablet_manager.cc:1905] T 55ebe9c044d944bc813396c699ba9aa1 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:28.578686 1196 ts_tablet_manager.cc:1939] T a6055d77d168495d9b756a1a56c1a920 P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:28.578778 1195 ts_tablet_manager.cc:1918] T ee7582ee04494c0db1179359bcb311c8 P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:28.578905 1195 log.cc:1198] T ee7582ee04494c0db1179359bcb311c8 P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/ee7582ee04494c0db1179359bcb311c8
I20250113 02:51:28.579319 1195 ts_tablet_manager.cc:1939] T ee7582ee04494c0db1179359bcb311c8 P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:28.579674 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet f760fc1e4bed433dbb987a42c12a1ee6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:19 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:28.582717 1194 tablet_bootstrap.cc:492] T 5b1076a5a17b4fa495975285616401cf P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:28.583782 1192 tablet_bootstrap.cc:492] T e8ab4516789e4585a101c8766d338af3 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:28.583990 1192 ts_tablet_manager.cc:1397] T e8ab4516789e4585a101c8766d338af3 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.599s user 0.147s sys 0.181s
I20250113 02:51:28.584574 1194 tablet_bootstrap.cc:654] T 5b1076a5a17b4fa495975285616401cf P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:28.584954 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet ee7582ee04494c0db1179359bcb311c8 (table default.Table_c8bea6af909a424c9101c8934d1aee7b [id=bd9d66f2d35446e9bc4ed5e589f23012]) successfully deleted
I20250113 02:51:28.585561 1191 tablet_bootstrap.cc:492] T c215ffe61b9b426286cc5e808463078d P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:28.585708 1191 ts_tablet_manager.cc:1397] T c215ffe61b9b426286cc5e808463078d P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.050s user 0.021s sys 0.017s
I20250113 02:51:28.586298 1195 tablet_replica.cc:331] T 5b1076a5a17b4fa495975285616401cf P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:28.586581 1195 raft_consensus.cc:2238] T 5b1076a5a17b4fa495975285616401cf P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:28.586822 1195 raft_consensus.cc:2267] T 5b1076a5a17b4fa495975285616401cf P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:28.587401 1195 ts_tablet_manager.cc:1905] T 5b1076a5a17b4fa495975285616401cf P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:28.588160 1191 raft_consensus.cc:357] T c215ffe61b9b426286cc5e808463078d P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:28.588431 1191 raft_consensus.cc:383] T c215ffe61b9b426286cc5e808463078d P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:28.588521 1191 raft_consensus.cc:738] T c215ffe61b9b426286cc5e808463078d P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:28.590767 1191 consensus_queue.cc:260] T c215ffe61b9b426286cc5e808463078d P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:28.592706 1191 ts_tablet_manager.cc:1428] T c215ffe61b9b426286cc5e808463078d P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20250113 02:51:28.594077 1191 tablet_bootstrap.cc:492] T 32cba8f3efc54e78b4df41c15df82eea P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:28.596238 1191 tablet_bootstrap.cc:654] T 32cba8f3efc54e78b4df41c15df82eea P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:28.596747 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet a6055d77d168495d9b756a1a56c1a920 (table default.Table_44f1c4e8cc2b4e83adce8d411ae53448 [id=a9e0aae9f5af42b796d29a75140f9c29]) successfully deleted
W20250113 02:51:28.597445 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet e570465a3d4046b1ac19a3a034fe91f1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e570465a3d4046b1ac19a3a034fe91f1 already in progress: creating tablet
W20250113 02:51:28.598120 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet c6820d6331e44656a27e23afe973e2e2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c6820d6331e44656a27e23afe973e2e2 already in progress: creating tablet
W20250113 02:51:28.598690 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet ed68a2b7ebcf4841bef54798dd3ed80d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ed68a2b7ebcf4841bef54798dd3ed80d already in progress: creating tablet
W20250113 02:51:28.599299 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 0539cbbe47e64c039ad39c4b52edcc4b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0539cbbe47e64c039ad39c4b52edcc4b already in progress: creating tablet
W20250113 02:51:28.599963 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 23476dc111ae430abdce3855d0ef54fe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 23476dc111ae430abdce3855d0ef54fe already in progress: creating tablet
W20250113 02:51:28.602751 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 1014e51f1c184d729552a5b34292f232 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1014e51f1c184d729552a5b34292f232 already in progress: creating tablet
W20250113 02:51:28.603330 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet b5205b359d234babb89707a54a7f968f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b5205b359d234babb89707a54a7f968f already in progress: creating tablet
W20250113 02:51:28.604415 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 98bfe9620e284ab1954433ba9a036392 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 98bfe9620e284ab1954433ba9a036392 already in progress: creating tablet
I20250113 02:51:28.604342 1192 raft_consensus.cc:357] T e8ab4516789e4585a101c8766d338af3 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:28.604588 1192 raft_consensus.cc:383] T e8ab4516789e4585a101c8766d338af3 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:28.604696 1192 raft_consensus.cc:738] T e8ab4516789e4585a101c8766d338af3 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:28.606069 451 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet 9781db6bbaf243e2a602a8db9e9fbdc5
I20250113 02:51:28.610064 1192 consensus_queue.cc:260] T e8ab4516789e4585a101c8766d338af3 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:28.610713 1192 ts_tablet_manager.cc:1428] T e8ab4516789e4585a101c8766d338af3 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.027s user 0.002s sys 0.000s
I20250113 02:51:28.612104 1192 tablet_bootstrap.cc:492] T 19f14b3db0b64847b5422bfd5ed323d0 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:28.614241 1192 tablet_bootstrap.cc:654] T 19f14b3db0b64847b5422bfd5ed323d0 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:28.618474 448 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_2d2546458fac45b79308ff4e7f043292" } modify_external_catalogs: true
I20250113 02:51:28.618654 448 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_2d2546458fac45b79308ff4e7f043292" } modify_external_catalogs: true
I20250113 02:51:28.621661 1194 tablet_bootstrap.cc:492] T 5b1076a5a17b4fa495975285616401cf P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:28.621903 1194 ts_tablet_manager.cc:1397] T 5b1076a5a17b4fa495975285616401cf P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.039s user 0.004s sys 0.000s
I20250113 02:51:28.623167 1194 raft_consensus.cc:357] T 5b1076a5a17b4fa495975285616401cf P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:28.623672 1194 raft_consensus.cc:383] T 5b1076a5a17b4fa495975285616401cf P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:28.623786 1194 raft_consensus.cc:738] T 5b1076a5a17b4fa495975285616401cf P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:28.624159 884 tablet_service.cc:1514] Processing DeleteTablet for tablet bcb59573e82743858bd39bde64627d6d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:20 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:28.624130 1194 consensus_queue.cc:260] T 5b1076a5a17b4fa495975285616401cf P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:28.624537 1198 ts_tablet_manager.cc:1918] T 55ebe9c044d944bc813396c699ba9aa1 P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:28.625416 1194 ts_tablet_manager.cc:1428] T 5b1076a5a17b4fa495975285616401cf P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20250113 02:51:28.627686 1195 ts_tablet_manager.cc:1918] T 5b1076a5a17b4fa495975285616401cf P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:28.627844 1198 log.cc:1198] T 55ebe9c044d944bc813396c699ba9aa1 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/55ebe9c044d944bc813396c699ba9aa1
I20250113 02:51:28.628377 1198 ts_tablet_manager.cc:1939] T 55ebe9c044d944bc813396c699ba9aa1 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
W20250113 02:51:28.629179 454 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6d5148cf8ff24c92aedf3faee5ae8723 consensus state has no leader
I20250113 02:51:28.631548 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_ea693041c5824d818e3bd15e28c7a3f5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:28.632215 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet 55ebe9c044d944bc813396c699ba9aa1 (table default.Table_2caed919541e43c78a858513a2128f50 [id=26ba87b9bd384b99a016b29fe5b5babd]) successfully deleted
I20250113 02:51:28.632874 1438 raft_consensus.cc:491] T 64cda8796dff46ce95c37439395c77ec P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:28.633029 1438 raft_consensus.cc:513] T 64cda8796dff46ce95c37439395c77ec P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
W20250113 02:51:28.633427 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ea693041c5824d818e3bd15e28c7a3f5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:28.633708 1438 leader_election.cc:290] T 64cda8796dff46ce95c37439395c77ec P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587), 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
W20250113 02:51:28.634668 852 leader_election.cc:343] T 64cda8796dff46ce95c37439395c77ec P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Not found: Tablet not found: 64cda8796dff46ce95c37439395c77ec
W20250113 02:51:28.635639 851 leader_election.cc:343] T 64cda8796dff46ce95c37439395c77ec P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Not found: Tablet not found: 64cda8796dff46ce95c37439395c77ec
I20250113 02:51:28.635774 851 leader_election.cc:304] T 64cda8796dff46ce95c37439395c77ec P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df69f48656b44bb890f76587ee69cc0f; no voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca
I20250113 02:51:28.636183 1438 raft_consensus.cc:2743] T 64cda8796dff46ce95c37439395c77ec P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:28.637033 884 tablet_service.cc:1514] Processing DeleteTablet for tablet 64cda8796dff46ce95c37439395c77ec with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:20 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:28.637176 1198 tablet_replica.cc:331] T f760fc1e4bed433dbb987a42c12a1ee6 P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:28.637369 1198 raft_consensus.cc:2238] T f760fc1e4bed433dbb987a42c12a1ee6 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:28.637569 1196 tablet_replica.cc:331] T bcb59573e82743858bd39bde64627d6d P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:28.637786 1196 raft_consensus.cc:2238] T bcb59573e82743858bd39bde64627d6d P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:28.638008 1196 raft_consensus.cc:2267] T bcb59573e82743858bd39bde64627d6d P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:28.638511 1194 tablet_bootstrap.cc:492] T 7f08b72301dc431396cf8cc7790d56d4 P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:28.640929 1194 tablet_bootstrap.cc:654] T 7f08b72301dc431396cf8cc7790d56d4 P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:28.642002 1196 ts_tablet_manager.cc:1905] T bcb59573e82743858bd39bde64627d6d P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250113 02:51:28.646358 1799 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 02:51:28.648586 455 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_7bb48c9e5f084d96980d7efdd395f685" } modify_external_catalogs: true
I20250113 02:51:28.648792 455 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_7bb48c9e5f084d96980d7efdd395f685" } modify_external_catalogs: true
I20250113 02:51:28.649658 884 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b1076a5a17b4fa495975285616401cf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:27 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:28.650461 1706 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250113 02:51:28.661193 1198 raft_consensus.cc:2267] T f760fc1e4bed433dbb987a42c12a1ee6 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:28.662235 1198 ts_tablet_manager.cc:1905] T f760fc1e4bed433dbb987a42c12a1ee6 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:28.641058 1669 raft_consensus.cc:491] T 55ebe9c044d944bc813396c699ba9aa1 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:28.662509 1194 tablet_bootstrap.cc:492] T 7f08b72301dc431396cf8cc7790d56d4 P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:28.662689 1194 ts_tablet_manager.cc:1397] T 7f08b72301dc431396cf8cc7790d56d4 P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.024s user 0.003s sys 0.000s
I20250113 02:51:28.662618 1669 raft_consensus.cc:513] T 55ebe9c044d944bc813396c699ba9aa1 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:28.663353 1669 leader_election.cc:290] T 55ebe9c044d944bc813396c699ba9aa1 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:28.660213 1757 raft_consensus.cc:491] T 2c399784af46426b8327d175c7a6cce8 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:28.663550 1757 raft_consensus.cc:513] T 2c399784af46426b8327d175c7a6cce8 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:28.663712 1194 raft_consensus.cc:357] T 7f08b72301dc431396cf8cc7790d56d4 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:28.664198 1194 raft_consensus.cc:383] T 7f08b72301dc431396cf8cc7790d56d4 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:28.664222 1757 leader_election.cc:290] T 2c399784af46426b8327d175c7a6cce8 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:28.664300 1194 raft_consensus.cc:738] T 7f08b72301dc431396cf8cc7790d56d4 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:28.664633 1194 consensus_queue.cc:260] T 7f08b72301dc431396cf8cc7790d56d4 P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
W20250113 02:51:28.665033 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 32cba8f3efc54e78b4df41c15df82eea consensus state has no leader
I20250113 02:51:28.665081 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55ebe9c044d944bc813396c699ba9aa1" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:28.665340 1194 ts_tablet_manager.cc:1428] T 7f08b72301dc431396cf8cc7790d56d4 P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250113 02:51:28.665472 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c399784af46426b8327d175c7a6cce8" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:28.666041 711 leader_election.cc:343] T 55ebe9c044d944bc813396c699ba9aa1 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
W20250113 02:51:28.666273 711 leader_election.cc:343] T 2c399784af46426b8327d175c7a6cce8 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:28.666851 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_ed2ecde421f24146b87838f8e9009663"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:28.667420 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ed2ecde421f24146b87838f8e9009663 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250113 02:51:28.667690 709 leader_election.cc:343] T 55ebe9c044d944bc813396c699ba9aa1 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Not found: Tablet not found: 55ebe9c044d944bc813396c699ba9aa1
I20250113 02:51:28.667793 709 leader_election.cc:304] T 55ebe9c044d944bc813396c699ba9aa1 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:28.668352 1757 raft_consensus.cc:2743] T 55ebe9c044d944bc813396c699ba9aa1 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:28.670071 1194 tablet_bootstrap.cc:492] T b2a35fd891c645d68d79c61a2c832ba7 P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:28.672009 1194 tablet_bootstrap.cc:654] T b2a35fd891c645d68d79c61a2c832ba7 P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:28.672223 1195 log.cc:1198] T 5b1076a5a17b4fa495975285616401cf P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/5b1076a5a17b4fa495975285616401cf
I20250113 02:51:28.673828 1195 ts_tablet_manager.cc:1939] T 5b1076a5a17b4fa495975285616401cf P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:28.667104 1062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c399784af46426b8327d175c7a6cce8" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
W20250113 02:51:28.675067 709 leader_election.cc:343] T 2c399784af46426b8327d175c7a6cce8 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:28.675287 709 leader_election.cc:304] T 2c399784af46426b8327d175c7a6cce8 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:28.675652 1757 raft_consensus.cc:2743] T 2c399784af46426b8327d175c7a6cce8 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:28.677597 1706 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250113 02:51:28.678694 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet 5b1076a5a17b4fa495975285616401cf (table default.Table_f5459a21934542d9b62792bcd109f836 [id=bc2aa39c74b04c7ca71a6d0acd780853]) successfully deleted
I20250113 02:51:28.679168 1195 tablet_replica.cc:331] T b2a35fd891c645d68d79c61a2c832ba7 P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:28.679368 1195 raft_consensus.cc:2238] T b2a35fd891c645d68d79c61a2c832ba7 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:28.679543 1195 raft_consensus.cc:2267] T b2a35fd891c645d68d79c61a2c832ba7 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:28.680066 1195 ts_tablet_manager.cc:1905] T b2a35fd891c645d68d79c61a2c832ba7 P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:28.683523 449 master_service.cc:946] ReplaceTablet: received request to replace tablet 3025935f9743442eb3e6f2db22eb52ad from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:28.688069 1198 ts_tablet_manager.cc:1918] T f760fc1e4bed433dbb987a42c12a1ee6 P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:28.688227 1198 log.cc:1198] T f760fc1e4bed433dbb987a42c12a1ee6 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/f760fc1e4bed433dbb987a42c12a1ee6
I20250113 02:51:28.688819 1198 ts_tablet_manager.cc:1939] T f760fc1e4bed433dbb987a42c12a1ee6 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:28.689203 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet a3506ba26c9c4f0295cb1b2798b4a001
I20250113 02:51:28.689865 450 master_service.cc:946] ReplaceTablet: received request to replace tablet b16dd8451f32478daf2ae118dbb9b304 from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:28.689965 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet f760fc1e4bed433dbb987a42c12a1ee6 (table default.Table_c45864b4578e45b59a53d2d913da12f9 [id=f02cfa6472eb42de956f8591417eb7b4]) successfully deleted
I20250113 02:51:28.690104 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet a3506ba26c9c4f0295cb1b2798b4a001 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:28 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:28.690629 741 tablet_service.cc:1514] Processing DeleteTablet for tablet a3506ba26c9c4f0295cb1b2798b4a001 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:28 UTC) from {username='slave'} at 127.0.0.1:55398
W20250113 02:51:28.693854 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet a3506ba26c9c4f0295cb1b2798b4a001 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a3506ba26c9c4f0295cb1b2798b4a001 already in progress: creating tablet
I20250113 02:51:28.694542 1191 tablet_bootstrap.cc:492] T 32cba8f3efc54e78b4df41c15df82eea P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:28.694733 1191 ts_tablet_manager.cc:1397] T 32cba8f3efc54e78b4df41c15df82eea P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.102s user 0.049s sys 0.026s
I20250113 02:51:28.695107 884 tablet_service.cc:1514] Processing DeleteTablet for tablet a3506ba26c9c4f0295cb1b2798b4a001 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:28 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:28.695565 1191 raft_consensus.cc:357] T 32cba8f3efc54e78b4df41c15df82eea P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:28.695888 1191 raft_consensus.cc:383] T 32cba8f3efc54e78b4df41c15df82eea P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:28.695963 1191 raft_consensus.cc:738] T 32cba8f3efc54e78b4df41c15df82eea P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:28.696256 1195 ts_tablet_manager.cc:1918] T b2a35fd891c645d68d79c61a2c832ba7 P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:28.696460 1195 log.cc:1198] T b2a35fd891c645d68d79c61a2c832ba7 P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/b2a35fd891c645d68d79c61a2c832ba7
I20250113 02:51:28.696816 1195 ts_tablet_manager.cc:1939] T b2a35fd891c645d68d79c61a2c832ba7 P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:28.698319 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet b2a35fd891c645d68d79c61a2c832ba7 (table default.Table_266561dfe36245ecb8230fdf1904cb02 [id=60925cf1a1224988b0d349b621ab5368]) successfully deleted
W20250113 02:51:28.698568 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet a3506ba26c9c4f0295cb1b2798b4a001 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a3506ba26c9c4f0295cb1b2798b4a001 already in progress: creating tablet
I20250113 02:51:28.700063 1757 raft_consensus.cc:491] T fdbf03c89bff438a9a25115eddf60be9 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:28.700209 1757 raft_consensus.cc:513] T fdbf03c89bff438a9a25115eddf60be9 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:28.700814 1757 leader_election.cc:290] T fdbf03c89bff438a9a25115eddf60be9 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:28.701576 1062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdbf03c89bff438a9a25115eddf60be9" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:28.701766 1192 tablet_bootstrap.cc:492] T 19f14b3db0b64847b5422bfd5ed323d0 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:28.701923 1192 ts_tablet_manager.cc:1397] T 19f14b3db0b64847b5422bfd5ed323d0 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.090s user 0.007s sys 0.003s
I20250113 02:51:28.701996 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdbf03c89bff438a9a25115eddf60be9" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:28.702160 709 leader_election.cc:343] T fdbf03c89bff438a9a25115eddf60be9 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
W20250113 02:51:28.702557 711 leader_election.cc:343] T fdbf03c89bff438a9a25115eddf60be9 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:28.702710 711 leader_election.cc:304] T fdbf03c89bff438a9a25115eddf60be9 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:28.703150 1757 raft_consensus.cc:2743] T fdbf03c89bff438a9a25115eddf60be9 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:28.703328 1192 raft_consensus.cc:357] T 19f14b3db0b64847b5422bfd5ed323d0 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:28.703536 1192 raft_consensus.cc:383] T 19f14b3db0b64847b5422bfd5ed323d0 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:28.696280 1191 consensus_queue.cc:260] T 32cba8f3efc54e78b4df41c15df82eea P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:28.704656 1191 ts_tablet_manager.cc:1428] T 32cba8f3efc54e78b4df41c15df82eea P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.010s user 0.000s sys 0.002s
I20250113 02:51:28.705567 1191 tablet_bootstrap.cc:492] T 63efcb6b2e194c469fb30a197bd6565b P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:28.710924 1191 tablet_bootstrap.cc:654] T 63efcb6b2e194c469fb30a197bd6565b P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:28.711800 1757 raft_consensus.cc:491] T 258fad818b7b4aa4b337f9949e20aa46 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:28.711998 1757 raft_consensus.cc:513] T 258fad818b7b4aa4b337f9949e20aa46 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:28.719424 1757 leader_election.cc:290] T 258fad818b7b4aa4b337f9949e20aa46 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
W20250113 02:51:28.719802 709 leader_election.cc:343] T 258fad818b7b4aa4b337f9949e20aa46 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Not found: Tablet not found: 258fad818b7b4aa4b337f9949e20aa46
I20250113 02:51:28.724951 1191 tablet_bootstrap.cc:492] T 63efcb6b2e194c469fb30a197bd6565b P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:28.725292 1191 ts_tablet_manager.cc:1397] T 63efcb6b2e194c469fb30a197bd6565b P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.004s
I20250113 02:51:28.727540 1191 raft_consensus.cc:357] T 63efcb6b2e194c469fb30a197bd6565b P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:28.728132 1191 raft_consensus.cc:383] T 63efcb6b2e194c469fb30a197bd6565b P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:28.728363 1191 raft_consensus.cc:738] T 63efcb6b2e194c469fb30a197bd6565b P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:28.728782 1191 consensus_queue.cc:260] T 63efcb6b2e194c469fb30a197bd6565b P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:28.729852 1191 ts_tablet_manager.cc:1428] T 63efcb6b2e194c469fb30a197bd6565b P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20250113 02:51:28.730509 1191 tablet_bootstrap.cc:492] T f68f9f4076d9475d938e795e72bdd468 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:28.733954 1191 tablet_bootstrap.cc:654] T f68f9f4076d9475d938e795e72bdd468 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:28.738262 1196 ts_tablet_manager.cc:1918] T bcb59573e82743858bd39bde64627d6d P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:28.738443 1196 log.cc:1198] T bcb59573e82743858bd39bde64627d6d P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/bcb59573e82743858bd39bde64627d6d
I20250113 02:51:28.738957 1196 ts_tablet_manager.cc:1939] T bcb59573e82743858bd39bde64627d6d P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:28.739512 1196 tablet_replica.cc:331] T 64cda8796dff46ce95c37439395c77ec P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:28.739574 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet bcb59573e82743858bd39bde64627d6d (table default.Table_9cd2b585fc704fb6a2d0da56a774b8bd [id=7b5976daebfb49eeb72f919006fa149b]) successfully deleted
I20250113 02:51:28.740196 1706 hybrid_clock.cc:648] HybridClock initialized: now 1736736688739957 us; error 211 us; skew 500 ppm
I20250113 02:51:28.740705 1706 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
W20250113 02:51:28.741097 711 consensus_peers.cc:487] T 572e4eed1e924381bada27f8389821a7 P 1186fa869eb14812832fa6562531f196 -> Peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Couldn't send request to peer df69f48656b44bb890f76587ee69cc0f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250113 02:51:28.746922 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "258fad818b7b4aa4b337f9949e20aa46" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:28.747278 1196 raft_consensus.cc:2238] T 64cda8796dff46ce95c37439395c77ec P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:28.747470 1196 raft_consensus.cc:2267] T 64cda8796dff46ce95c37439395c77ec P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:28.748417 1196 ts_tablet_manager.cc:1905] T 64cda8796dff46ce95c37439395c77ec P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:28.750455 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_911794a03fd1463787a098ca7e61dbc8" } new_table_name: "default.Table_639e38f37be1466e9c2ce92fa1c2d890" modify_external_catalogs: true
I20250113 02:51:28.752864 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_67b9f9f43ada4495a29ed9b8e8a6bf63"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:28.753546 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_67b9f9f43ada4495a29ed9b8e8a6bf63 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:28.755297 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_4dae6fdf9a1e4400b85b31bf8fcc172c" } new_table_name: "default.Table_ccae53f816ca40deb917dce54de4698a" modify_external_catalogs: true
I20250113 02:51:28.756167 1438 raft_consensus.cc:491] T 7f08b72301dc431396cf8cc7790d56d4 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:28.756307 1438 raft_consensus.cc:513] T 7f08b72301dc431396cf8cc7790d56d4 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:28.757002 1438 leader_election.cc:290] T 7f08b72301dc431396cf8cc7790d56d4 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587), 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
I20250113 02:51:28.757592 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet b9ec434644c14acd95c72efbef8ded59
I20250113 02:51:28.758234 741 tablet_service.cc:1514] Processing DeleteTablet for tablet b9ec434644c14acd95c72efbef8ded59 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:28 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:28.758389 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_1a482bb1752b4f4a8fc993b60179f50b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:28.758775 1062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f08b72301dc431396cf8cc7790d56d4" candidate_uuid: "df69f48656b44bb890f76587ee69cc0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:28.759110 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet b9ec434644c14acd95c72efbef8ded59 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:28 UTC) from {username='slave'} at 127.0.0.1:60230
W20250113 02:51:28.759343 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet b9ec434644c14acd95c72efbef8ded59 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b9ec434644c14acd95c72efbef8ded59 already in progress: creating tablet
W20250113 02:51:28.759939 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet b9ec434644c14acd95c72efbef8ded59 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b9ec434644c14acd95c72efbef8ded59 already in progress: creating tablet
I20250113 02:51:28.760061 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f08b72301dc431396cf8cc7790d56d4" candidate_uuid: "df69f48656b44bb890f76587ee69cc0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
I20250113 02:51:28.761463 852 leader_election.cc:304] T 7f08b72301dc431396cf8cc7790d56d4 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df69f48656b44bb890f76587ee69cc0f; no voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca
I20250113 02:51:28.761518 884 tablet_service.cc:1514] Processing DeleteTablet for tablet b9ec434644c14acd95c72efbef8ded59 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:28 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:28.761955 1438 raft_consensus.cc:2743] T 7f08b72301dc431396cf8cc7790d56d4 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250113 02:51:28.768314 711 leader_election.cc:343] T 258fad818b7b4aa4b337f9949e20aa46 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:28.768483 711 leader_election.cc:304] T 258fad818b7b4aa4b337f9949e20aa46 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:28.769001 1757 raft_consensus.cc:2743] T 258fad818b7b4aa4b337f9949e20aa46 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250113 02:51:28.769376 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1a482bb1752b4f4a8fc993b60179f50b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:28.776759 1196 ts_tablet_manager.cc:1918] T 64cda8796dff46ce95c37439395c77ec P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:28.776907 1196 log.cc:1198] T 64cda8796dff46ce95c37439395c77ec P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/64cda8796dff46ce95c37439395c77ec
I20250113 02:51:28.777305 1196 ts_tablet_manager.cc:1939] T 64cda8796dff46ce95c37439395c77ec P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:28.777881 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet 64cda8796dff46ce95c37439395c77ec (table default.Table_45ad8711e23e44f0b7b83b7a6128c427 [id=983d3b296c064375a1f2747ce556f01d]) successfully deleted
I20250113 02:51:28.778211 1196 tablet_replica.cc:331] T 5b1076a5a17b4fa495975285616401cf P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:28.778385 1196 raft_consensus.cc:2238] T 5b1076a5a17b4fa495975285616401cf P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:28.778486 1757 raft_consensus.cc:491] T 501d4e1ae84e4795913035f8a19cc32d P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:28.778556 1196 raft_consensus.cc:2267] T 5b1076a5a17b4fa495975285616401cf P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:28.778693 1757 raft_consensus.cc:513] T 501d4e1ae84e4795913035f8a19cc32d P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:28.779220 1196 ts_tablet_manager.cc:1905] T 5b1076a5a17b4fa495975285616401cf P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:28.779570 1757 leader_election.cc:290] T 501d4e1ae84e4795913035f8a19cc32d P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
W20250113 02:51:28.780762 709 leader_election.cc:343] T 501d4e1ae84e4795913035f8a19cc32d P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Not found: Tablet not found: 501d4e1ae84e4795913035f8a19cc32d
I20250113 02:51:28.781644 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "501d4e1ae84e4795913035f8a19cc32d" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:28.782305 711 leader_election.cc:343] T 501d4e1ae84e4795913035f8a19cc32d P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:28.782446 711 leader_election.cc:304] T 501d4e1ae84e4795913035f8a19cc32d P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:28.782974 1757 raft_consensus.cc:2743] T 501d4e1ae84e4795913035f8a19cc32d P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:28.792376 1757 raft_consensus.cc:491] T e8ab4516789e4585a101c8766d338af3 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:28.792558 1757 raft_consensus.cc:513] T e8ab4516789e4585a101c8766d338af3 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:28.793331 1757 leader_election.cc:290] T e8ab4516789e4585a101c8766d338af3 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:28.794049 1062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8ab4516789e4585a101c8766d338af3" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:28.794289 1062 raft_consensus.cc:2463] T e8ab4516789e4585a101c8766d338af3 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1186fa869eb14812832fa6562531f196 in term 0.
I20250113 02:51:28.794817 709 leader_election.cc:304] T e8ab4516789e4585a101c8766d338af3 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters:
I20250113 02:51:28.795709 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8ab4516789e4585a101c8766d338af3" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:28.796703 711 leader_election.cc:343] T e8ab4516789e4585a101c8766d338af3 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:28.797149 1757 raft_consensus.cc:2798] T e8ab4516789e4585a101c8766d338af3 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 02:51:28.797269 1757 raft_consensus.cc:491] T e8ab4516789e4585a101c8766d338af3 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 02:51:28.797366 1757 raft_consensus.cc:3054] T e8ab4516789e4585a101c8766d338af3 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:28.801048 1194 tablet_bootstrap.cc:492] T b2a35fd891c645d68d79c61a2c832ba7 P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:28.801223 1194 ts_tablet_manager.cc:1397] T b2a35fd891c645d68d79c61a2c832ba7 P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.131s user 0.007s sys 0.017s
I20250113 02:51:28.801992 1194 raft_consensus.cc:357] T b2a35fd891c645d68d79c61a2c832ba7 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:28.802219 1194 raft_consensus.cc:383] T b2a35fd891c645d68d79c61a2c832ba7 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:28.802289 1194 raft_consensus.cc:738] T b2a35fd891c645d68d79c61a2c832ba7 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:28.802561 1194 consensus_queue.cc:260] T b2a35fd891c645d68d79c61a2c832ba7 P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:28.805811 1194 ts_tablet_manager.cc:1428] T b2a35fd891c645d68d79c61a2c832ba7 P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250113 02:51:28.806301 1194 tablet_bootstrap.cc:492] T 572e4eed1e924381bada27f8389821a7 P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:28.807709 1757 raft_consensus.cc:513] T e8ab4516789e4585a101c8766d338af3 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:28.808998 1194 tablet_bootstrap.cc:654] T 572e4eed1e924381bada27f8389821a7 P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:28.809113 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8ab4516789e4585a101c8766d338af3" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f"
W20250113 02:51:28.809851 711 leader_election.cc:343] T e8ab4516789e4585a101c8766d338af3 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:28.810736 1062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8ab4516789e4585a101c8766d338af3" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca"
I20250113 02:51:28.810976 1062 raft_consensus.cc:3054] T e8ab4516789e4585a101c8766d338af3 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:28.812877 1196 ts_tablet_manager.cc:1918] T 5b1076a5a17b4fa495975285616401cf P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:28.813040 1196 log.cc:1198] T 5b1076a5a17b4fa495975285616401cf P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/5b1076a5a17b4fa495975285616401cf
I20250113 02:51:28.813445 1196 ts_tablet_manager.cc:1939] T 5b1076a5a17b4fa495975285616401cf P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:28.814234 1062 raft_consensus.cc:2463] T e8ab4516789e4585a101c8766d338af3 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1186fa869eb14812832fa6562531f196 in term 1.
I20250113 02:51:28.814539 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet 5b1076a5a17b4fa495975285616401cf (table default.Table_f5459a21934542d9b62792bcd109f836 [id=bc2aa39c74b04c7ca71a6d0acd780853]) successfully deleted
I20250113 02:51:28.814818 709 leader_election.cc:304] T e8ab4516789e4585a101c8766d338af3 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters: df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:28.815207 1669 raft_consensus.cc:2798] T e8ab4516789e4585a101c8766d338af3 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Leader election won for term 1
W20250113 02:51:28.815502 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet a3506ba26c9c4f0295cb1b2798b4a001 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a3506ba26c9c4f0295cb1b2798b4a001 already in progress: creating tablet
W20250113 02:51:28.816263 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet b9ec434644c14acd95c72efbef8ded59 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b9ec434644c14acd95c72efbef8ded59 already in progress: creating tablet
I20250113 02:51:28.819157 1757 leader_election.cc:290] T e8ab4516789e4585a101c8766d338af3 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 election: Requested vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:28.819960 1194 tablet_bootstrap.cc:492] T 572e4eed1e924381bada27f8389821a7 P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:28.820140 1194 ts_tablet_manager.cc:1397] T 572e4eed1e924381bada27f8389821a7 P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.001s
I20250113 02:51:28.820982 1194 raft_consensus.cc:357] T 572e4eed1e924381bada27f8389821a7 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:28.821203 1194 raft_consensus.cc:383] T 572e4eed1e924381bada27f8389821a7 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:28.821300 1194 raft_consensus.cc:738] T 572e4eed1e924381bada27f8389821a7 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:28.821525 1194 consensus_queue.cc:260] T 572e4eed1e924381bada27f8389821a7 P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:28.822242 1194 ts_tablet_manager.cc:1428] T 572e4eed1e924381bada27f8389821a7 P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250113 02:51:28.822532 1194 tablet_bootstrap.cc:492] T ee7582ee04494c0db1179359bcb311c8 P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:28.827584 1194 tablet_bootstrap.cc:654] T ee7582ee04494c0db1179359bcb311c8 P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
W20250113 02:51:28.831044 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 01a455eda0ee488fb3d45a6a09c43833 consensus state has no leader
I20250113 02:51:28.831478 453 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_03b7491a47674f7187f4952feb9b44e2" } modify_external_catalogs: true
I20250113 02:51:28.831707 453 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_03b7491a47674f7187f4952feb9b44e2" } modify_external_catalogs: true
I20250113 02:51:28.833786 451 catalog_manager.cc:5618] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095) does not have the latest schema for tablet e8ab4516789e4585a101c8766d338af3 (table default.Table_3b76cf9fac334ebf9b1ecb9f8d07ea74 [id=b112ac4e2e58460db083da1eb788ca07]). Expected version 2 got 0
W20250113 02:51:28.834012 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 42fad91fe1b64a21a2c8412a2585af47 consensus state has no leader
I20250113 02:51:28.834218 451 catalog_manager.cc:5618] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095) does not have the latest schema for tablet 8adfa738c17a4169a5d99f6ceca75100 (table default.Table_9a9b1ced3bef4e648088e0781851128e [id=ae544735623e4cfd894dc9340aca177c]). Expected version 1 got 0
I20250113 02:51:28.834364 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_e392f2cbf2f14aa39320676634a32819" } new_table_name: "default.Table_7c964c948c4f44298d9bae9fc5e608c6" modify_external_catalogs: true
I20250113 02:51:28.837168 454 master_service.cc:946] ReplaceTablet: received request to replace tablet 46ed22e392db441a894dceeedd2911f4 from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:28.837283 447 catalog_manager.cc:5526] T 8adfa738c17a4169a5d99f6ceca75100 P 4216a7f853d642b7bc3e5cff458f3aca reported cstate change: term changed from 0 to 1, leader changed from <none> to 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3). New cstate: current_term: 1 leader_uuid: "4216a7f853d642b7bc3e5cff458f3aca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } health_report { overall_health: UNKNOWN } } }
I20250113 02:51:28.837726 447 catalog_manager.cc:5618] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587) does not have the latest schema for tablet 8adfa738c17a4169a5d99f6ceca75100 (table default.Table_9a9b1ced3bef4e648088e0781851128e [id=ae544735623e4cfd894dc9340aca177c]). Expected version 1 got 0
I20250113 02:51:28.838478 1669 raft_consensus.cc:695] T e8ab4516789e4585a101c8766d338af3 P 1186fa869eb14812832fa6562531f196 [term 1 LEADER]: Becoming Leader. State: Replica: 1186fa869eb14812832fa6562531f196, State: Running, Role: LEADER
I20250113 02:51:28.838831 1669 consensus_queue.cc:237] T e8ab4516789e4585a101c8766d338af3 P 1186fa869eb14812832fa6562531f196 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:28.841825 741 tablet_service.cc:1467] Processing CreateTablet for tablet b16ff51101e0452ebc05374f0d1f4917 (DEFAULT_TABLE table=default.Table_b8ded36063364816988685dfc77391d2 [id=fff02d5a50df47f1acceaf4454ed85c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:28.842300 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b16ff51101e0452ebc05374f0d1f4917. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:28.839955 446 catalog_manager.cc:5618] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467) does not have the latest schema for tablet 7f08b72301dc431396cf8cc7790d56d4 (table default.Table_6de3bf6640dc47f9b977f608c5ec8ef9 [id=bd14ce0e2b0a4a60a63022be2bc80fef]). Expected version 3 got 0
I20250113 02:51:28.844347 1706 webserver.cc:458] Webserver started at http://127.0.104.60:38411/ using document root <none> and password file <none>
I20250113 02:51:28.845057 1029 tablet_service.cc:1467] Processing CreateTablet for tablet 960a7078181f4621a5ff4f3eb3050de3 (DEFAULT_TABLE table=default.Table_9cc116d6861b4480b32eb7fa206ae7c0 [id=3b891cf1daf44b229ae53c4c524f06f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:28.845448 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 960a7078181f4621a5ff4f3eb3050de3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:28.845441 1706 fs_manager.cc:362] Metadata directory not provided
I20250113 02:51:28.845587 1706 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 02:51:28.848235 1041 tablet_service.cc:1467] Processing CreateTablet for tablet d7193e2799a54950902c4ceac65b663b (DEFAULT_TABLE table=default.Table_24ea41f0ca3d4b4b919da4716b6c08a2 [id=63464d10972f4bab8454fb09eb7a029b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:28.848558 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7193e2799a54950902c4ceac65b663b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:28.848992 745 tablet_service.cc:1467] Processing CreateTablet for tablet 1292edfc0ff04cc4bb1ea2fb914ef368 (DEFAULT_TABLE table=default.Table_8fa472c153f94ebf8fe8de2e9a7e7d2b [id=8c9b2e36ee5842209c1c8a9e436e4265]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:28.849372 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1292edfc0ff04cc4bb1ea2fb914ef368. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:28.852353 755 tablet_service.cc:1467] Processing CreateTablet for tablet 960a7078181f4621a5ff4f3eb3050de3 (DEFAULT_TABLE table=default.Table_9cc116d6861b4480b32eb7fa206ae7c0 [id=3b891cf1daf44b229ae53c4c524f06f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:28.852757 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 960a7078181f4621a5ff4f3eb3050de3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:28.852952 1025 tablet_service.cc:1467] Processing CreateTablet for tablet 1292edfc0ff04cc4bb1ea2fb914ef368 (DEFAULT_TABLE table=default.Table_8fa472c153f94ebf8fe8de2e9a7e7d2b [id=8c9b2e36ee5842209c1c8a9e436e4265]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:28.853353 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1292edfc0ff04cc4bb1ea2fb914ef368. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:28.856233 1706 fs_manager.cc:714] Time spent opening directory manager: real 0.008s user 0.003s sys 0.001s
I20250113 02:51:28.845067 1027 tablet_service.cc:1467] Processing CreateTablet for tablet b16ff51101e0452ebc05374f0d1f4917 (DEFAULT_TABLE table=default.Table_b8ded36063364816988685dfc77391d2 [id=fff02d5a50df47f1acceaf4454ed85c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:28.856716 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b16ff51101e0452ebc05374f0d1f4917. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:28.859735 1669 raft_consensus.cc:491] T 7d53de6379b84103ba905effb0835676 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:28.859895 1669 raft_consensus.cc:513] T 7d53de6379b84103ba905effb0835676 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:28.860621 1669 leader_election.cc:290] T 7d53de6379b84103ba905effb0835676 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:28.861364 446 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_bbda194bd2cd4df2b7e7d429700e5f3c" } modify_external_catalogs: true
W20250113 02:51:28.861537 709 leader_election.cc:343] T 7d53de6379b84103ba905effb0835676 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Not found: Tablet not found: 7d53de6379b84103ba905effb0835676
I20250113 02:51:28.861577 446 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_bbda194bd2cd4df2b7e7d429700e5f3c" } modify_external_catalogs: true
I20250113 02:51:28.866127 451 master_service.cc:946] ReplaceTablet: received request to replace tablet acd83024592e431898e9675c17f50d4f from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:28.868959 736 tablet_service.cc:1514] Processing DeleteTablet for tablet f760fc1e4bed433dbb987a42c12a1ee6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:19 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:28.869243 742 tablet_service.cc:1514] Processing DeleteTablet for tablet 19f14b3db0b64847b5422bfd5ed323d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:19 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:28.869452 739 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c399784af46426b8327d175c7a6cce8 with delete_type TABLET_DATA_DELETED (replaced by tablet 9e736211eb3c4536b9fd1d90d2d83838) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:28.869621 738 tablet_service.cc:1514] Processing DeleteTablet for tablet 601c71d5f7624fbb87b365d2e775d33e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:19 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:28.872172 748 tablet_service.cc:1514] Processing DeleteTablet for tablet 501d4e1ae84e4795913035f8a19cc32d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:18 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:28.872502 1195 tablet_replica.cc:331] T f760fc1e4bed433dbb987a42c12a1ee6 P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:28.872701 1195 raft_consensus.cc:2238] T f760fc1e4bed433dbb987a42c12a1ee6 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:28.872859 1195 raft_consensus.cc:2267] T f760fc1e4bed433dbb987a42c12a1ee6 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:28.873164 1191 tablet_bootstrap.cc:492] T f68f9f4076d9475d938e795e72bdd468 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:28.873281 751 tablet_service.cc:1514] Processing DeleteTablet for tablet fdbf03c89bff438a9a25115eddf60be9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:19 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:28.873610 746 tablet_service.cc:1514] Processing DeleteTablet for tablet 258fad818b7b4aa4b337f9949e20aa46 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:19 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:28.873759 743 tablet_service.cc:1514] Processing DeleteTablet for tablet 55ebe9c044d944bc813396c699ba9aa1 with delete_type TABLET_DATA_DELETED (replaced by tablet 41fcde988f72496b824271fa36c1a226) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:28.873889 750 tablet_service.cc:1514] Processing DeleteTablet for tablet f31dd916336a4d8fb3ab015af7de8ccb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:23 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:28.874024 1195 ts_tablet_manager.cc:1905] T f760fc1e4bed433dbb987a42c12a1ee6 P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:28.874337 1192 raft_consensus.cc:738] T 19f14b3db0b64847b5422bfd5ed323d0 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:28.873378 1191 ts_tablet_manager.cc:1397] T f68f9f4076d9475d938e795e72bdd468 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.143s user 0.013s sys 0.043s
I20250113 02:51:28.875562 1191 raft_consensus.cc:357] T f68f9f4076d9475d938e795e72bdd468 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:28.875855 1191 raft_consensus.cc:383] T f68f9f4076d9475d938e795e72bdd468 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:28.875928 1191 raft_consensus.cc:738] T f68f9f4076d9475d938e795e72bdd468 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:28.876552 1192 consensus_queue.cc:260] T 19f14b3db0b64847b5422bfd5ed323d0 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:28.880254 1192 ts_tablet_manager.cc:1428] T 19f14b3db0b64847b5422bfd5ed323d0 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.178s user 0.024s sys 0.000s
I20250113 02:51:28.881263 752 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d53de6379b84103ba905effb0835676 with delete_type TABLET_DATA_DELETED (replaced by tablet fd0b8592997d4a61b8ca854ecbb67c55) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:28.881392 1192 tablet_bootstrap.cc:492] T f31dd916336a4d8fb3ab015af7de8ccb P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:28.883260 1192 tablet_bootstrap.cc:654] T f31dd916336a4d8fb3ab015af7de8ccb P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:28.876175 1191 consensus_queue.cc:260] T f68f9f4076d9475d938e795e72bdd468 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:28.883916 1195 ts_tablet_manager.cc:1918] T f760fc1e4bed433dbb987a42c12a1ee6 P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:28.884075 1195 log.cc:1198] T f760fc1e4bed433dbb987a42c12a1ee6 P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/f760fc1e4bed433dbb987a42c12a1ee6
I20250113 02:51:28.884465 1195 ts_tablet_manager.cc:1939] T f760fc1e4bed433dbb987a42c12a1ee6 P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:28.885358 1195 tablet_replica.cc:331] T 19f14b3db0b64847b5422bfd5ed323d0 P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:28.887184 1191 ts_tablet_manager.cc:1428] T f68f9f4076d9475d938e795e72bdd468 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.013s user 0.000s sys 0.002s
I20250113 02:51:28.887423 1195 raft_consensus.cc:2238] T 19f14b3db0b64847b5422bfd5ed323d0 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:28.887516 1191 tablet_bootstrap.cc:492] T b4806b18a4194f9c95350d89ba62e13a P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:28.887681 1195 raft_consensus.cc:2267] T 19f14b3db0b64847b5422bfd5ed323d0 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:28.888273 1195 ts_tablet_manager.cc:1905] T 19f14b3db0b64847b5422bfd5ed323d0 P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:28.889899 1191 tablet_bootstrap.cc:654] T b4806b18a4194f9c95350d89ba62e13a P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:28.890076 1814 raft_consensus.cc:491] T c215ffe61b9b426286cc5e808463078d P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:28.890197 1814 raft_consensus.cc:513] T c215ffe61b9b426286cc5e808463078d P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:28.893154 1814 leader_election.cc:290] T c215ffe61b9b426286cc5e808463078d P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:28.894178 1062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c215ffe61b9b426286cc5e808463078d" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
W20250113 02:51:28.894747 709 leader_election.cc:343] T c215ffe61b9b426286cc5e808463078d P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:28.898550 1040 tablet_service.cc:1467] Processing CreateTablet for tablet 3c623bd0fbbf4922aedda031a88b79eb (DEFAULT_TABLE table=default.Table_52010496bf7f4c85a99f61e12fb965a9 [id=a8d48640818f440886c6d34d58e33c34]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:28.898942 1040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c623bd0fbbf4922aedda031a88b79eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:28.899706 1033 tablet_service.cc:1467] Processing CreateTablet for tablet ec8196505dab4e05bb6487b566cf3144 (DEFAULT_TABLE table=default.Table_2abef959c5c34459a8dfa3034e207995 [id=f7e15a7c1fe848d591ba3232281c45ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:28.900022 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ec8196505dab4e05bb6487b566cf3144. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:28.907562 1195 ts_tablet_manager.cc:1918] T 19f14b3db0b64847b5422bfd5ed323d0 P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:28.886088 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet f760fc1e4bed433dbb987a42c12a1ee6 (table default.Table_c45864b4578e45b59a53d2d913da12f9 [id=f02cfa6472eb42de956f8591417eb7b4]) successfully deleted
I20250113 02:51:28.910118 1195 log.cc:1198] T 19f14b3db0b64847b5422bfd5ed323d0 P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/19f14b3db0b64847b5422bfd5ed323d0
I20250113 02:51:28.910544 1195 ts_tablet_manager.cc:1939] T 19f14b3db0b64847b5422bfd5ed323d0 P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:28.911072 752 tablet_service.cc:1467] Processing CreateTablet for tablet d7193e2799a54950902c4ceac65b663b (DEFAULT_TABLE table=default.Table_24ea41f0ca3d4b4b919da4716b6c08a2 [id=63464d10972f4bab8454fb09eb7a029b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:28.911551 750 tablet_service.cc:1467] Processing CreateTablet for tablet e6e8966da70746d7a10c7243ba2f6f87 (DEFAULT_TABLE table=default.Table_517c2399a90d4b36925a3b7fb36d5963 [id=f12ab779419245fe80589a9480e127aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:28.912202 746 tablet_service.cc:1467] Processing CreateTablet for tablet 3c623bd0fbbf4922aedda031a88b79eb (DEFAULT_TABLE table=default.Table_52010496bf7f4c85a99f61e12fb965a9 [id=a8d48640818f440886c6d34d58e33c34]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:28.912626 751 tablet_service.cc:1467] Processing CreateTablet for tablet ec8196505dab4e05bb6487b566cf3144 (DEFAULT_TABLE table=default.Table_2abef959c5c34459a8dfa3034e207995 [id=f7e15a7c1fe848d591ba3232281c45ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:28.912987 743 tablet_service.cc:1467] Processing CreateTablet for tablet f3771f0fe7ff4d54b433640c76bc07ae (DEFAULT_TABLE table=default.Table_69d7dc3f1d7d4161b0a63c85367de424 [id=e30935f621b74364ba0c83cc41971f17]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:28.913395 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7193e2799a54950902c4ceac65b663b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:28.913993 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c623bd0fbbf4922aedda031a88b79eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:28.914556 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ec8196505dab4e05bb6487b566cf3144. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:28.914989 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e6e8966da70746d7a10c7243ba2f6f87. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:28.915429 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3771f0fe7ff4d54b433640c76bc07ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:28.919117 1040 tablet_service.cc:1467] Processing CreateTablet for tablet e266a9083e2f46a0a43861b7708dcdd5 (DEFAULT_TABLE table=default.Table_a8d0f4a9e13b48de8ef840a9b6d690c5 [id=50501154ceba45cba2ffe3cf0d5df998]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:28.919461 1040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e266a9083e2f46a0a43861b7708dcdd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:28.924299 1027 tablet_service.cc:1467] Processing CreateTablet for tablet e6e8966da70746d7a10c7243ba2f6f87 (DEFAULT_TABLE table=default.Table_517c2399a90d4b36925a3b7fb36d5963 [id=f12ab779419245fe80589a9480e127aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:28.924618 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e6e8966da70746d7a10c7243ba2f6f87. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:28.928067 1025 tablet_service.cc:1467] Processing CreateTablet for tablet f3771f0fe7ff4d54b433640c76bc07ae (DEFAULT_TABLE table=default.Table_69d7dc3f1d7d4161b0a63c85367de424 [id=e30935f621b74364ba0c83cc41971f17]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:28.928370 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3771f0fe7ff4d54b433640c76bc07ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:28.933323 748 tablet_service.cc:1467] Processing CreateTablet for tablet e266a9083e2f46a0a43861b7708dcdd5 (DEFAULT_TABLE table=default.Table_a8d0f4a9e13b48de8ef840a9b6d690c5 [id=50501154ceba45cba2ffe3cf0d5df998]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:28.933696 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e266a9083e2f46a0a43861b7708dcdd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:28.945930 884 tablet_service.cc:1467] Processing CreateTablet for tablet b16ff51101e0452ebc05374f0d1f4917 (DEFAULT_TABLE table=default.Table_b8ded36063364816988685dfc77391d2 [id=fff02d5a50df47f1acceaf4454ed85c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:28.946404 885 tablet_service.cc:1467] Processing CreateTablet for tablet 1292edfc0ff04cc4bb1ea2fb914ef368 (DEFAULT_TABLE table=default.Table_8fa472c153f94ebf8fe8de2e9a7e7d2b [id=8c9b2e36ee5842209c1c8a9e436e4265]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:28.946775 880 tablet_service.cc:1467] Processing CreateTablet for tablet 960a7078181f4621a5ff4f3eb3050de3 (DEFAULT_TABLE table=default.Table_9cc116d6861b4480b32eb7fa206ae7c0 [id=3b891cf1daf44b229ae53c4c524f06f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:28.947129 887 tablet_service.cc:1467] Processing CreateTablet for tablet d7193e2799a54950902c4ceac65b663b (DEFAULT_TABLE table=default.Table_24ea41f0ca3d4b4b919da4716b6c08a2 [id=63464d10972f4bab8454fb09eb7a029b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:28.947484 886 tablet_service.cc:1467] Processing CreateTablet for tablet e6e8966da70746d7a10c7243ba2f6f87 (DEFAULT_TABLE table=default.Table_517c2399a90d4b36925a3b7fb36d5963 [id=f12ab779419245fe80589a9480e127aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:28.947847 896 tablet_service.cc:1467] Processing CreateTablet for tablet f3771f0fe7ff4d54b433640c76bc07ae (DEFAULT_TABLE table=default.Table_69d7dc3f1d7d4161b0a63c85367de424 [id=e30935f621b74364ba0c83cc41971f17]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:28.948225 889 tablet_service.cc:1467] Processing CreateTablet for tablet 3c623bd0fbbf4922aedda031a88b79eb (DEFAULT_TABLE table=default.Table_52010496bf7f4c85a99f61e12fb965a9 [id=a8d48640818f440886c6d34d58e33c34]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:28.948563 881 tablet_service.cc:1467] Processing CreateTablet for tablet ec8196505dab4e05bb6487b566cf3144 (DEFAULT_TABLE table=default.Table_2abef959c5c34459a8dfa3034e207995 [id=f7e15a7c1fe848d591ba3232281c45ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:28.948918 888 tablet_service.cc:1467] Processing CreateTablet for tablet e266a9083e2f46a0a43861b7708dcdd5 (DEFAULT_TABLE table=default.Table_a8d0f4a9e13b48de8ef840a9b6d690c5 [id=50501154ceba45cba2ffe3cf0d5df998]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:28.949887 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b16ff51101e0452ebc05374f0d1f4917. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:28.952316 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c215ffe61b9b426286cc5e808463078d" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:28.952857 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d53de6379b84103ba905effb0835676" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:28.953512 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e266a9083e2f46a0a43861b7708dcdd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:28.956532 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7193e2799a54950902c4ceac65b663b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:28.960376 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c623bd0fbbf4922aedda031a88b79eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:28.964551 1195 tablet_replica.cc:331] T 2c399784af46426b8327d175c7a6cce8 P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:28.964731 1195 raft_consensus.cc:2238] T 2c399784af46426b8327d175c7a6cce8 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:28.964886 1195 raft_consensus.cc:2267] T 2c399784af46426b8327d175c7a6cce8 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:28.965364 1195 ts_tablet_manager.cc:1905] T 2c399784af46426b8327d175c7a6cce8 P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:28.966022 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 960a7078181f4621a5ff4f3eb3050de3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:28.971890 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e6e8966da70746d7a10c7243ba2f6f87. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:28.972115 1191 tablet_bootstrap.cc:492] T b4806b18a4194f9c95350d89ba62e13a P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:28.972280 1191 ts_tablet_manager.cc:1397] T b4806b18a4194f9c95350d89ba62e13a P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.085s user 0.013s sys 0.028s
W20250113 02:51:28.973331 711 leader_election.cc:343] T c215ffe61b9b426286cc5e808463078d P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:28.973278 1191 raft_consensus.cc:357] T b4806b18a4194f9c95350d89ba62e13a P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:28.973441 711 leader_election.cc:304] T c215ffe61b9b426286cc5e808463078d P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:28.973730 1814 raft_consensus.cc:2743] T c215ffe61b9b426286cc5e808463078d P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250113 02:51:28.974102 711 leader_election.cc:343] T 7d53de6379b84103ba905effb0835676 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:28.974231 711 leader_election.cc:304] T 7d53de6379b84103ba905effb0835676 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:28.974846 1757 raft_consensus.cc:2743] T 7d53de6379b84103ba905effb0835676 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:28.974831 1195 ts_tablet_manager.cc:1918] T 2c399784af46426b8327d175c7a6cce8 P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:28.975111 1195 log.cc:1198] T 2c399784af46426b8327d175c7a6cce8 P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/2c399784af46426b8327d175c7a6cce8
I20250113 02:51:28.975116 1191 raft_consensus.cc:383] T b4806b18a4194f9c95350d89ba62e13a P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:28.975237 1191 raft_consensus.cc:738] T b4806b18a4194f9c95350d89ba62e13a P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:28.975401 1195 ts_tablet_manager.cc:1939] T 2c399784af46426b8327d175c7a6cce8 P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:28.975706 1191 consensus_queue.cc:260] T b4806b18a4194f9c95350d89ba62e13a P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:28.976423 1191 ts_tablet_manager.cc:1428] T b4806b18a4194f9c95350d89ba62e13a P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20250113 02:51:28.977137 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ec8196505dab4e05bb6487b566cf3144. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:28.977480 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1292edfc0ff04cc4bb1ea2fb914ef368. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:28.977701 1191 tablet_bootstrap.cc:492] T 9fff52e99c64462a8a73fccc3c22ca63 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:28.977856 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3771f0fe7ff4d54b433640c76bc07ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:28.978962 1757 raft_consensus.cc:491] T f31dd916336a4d8fb3ab015af7de8ccb P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:28.979087 1757 raft_consensus.cc:513] T f31dd916336a4d8fb3ab015af7de8ccb P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:28.979640 1757 leader_election.cc:290] T f31dd916336a4d8fb3ab015af7de8ccb P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:28.979745 1757 raft_consensus.cc:491] T 32cba8f3efc54e78b4df41c15df82eea P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:28.979831 1757 raft_consensus.cc:513] T 32cba8f3efc54e78b4df41c15df82eea P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:28.980515 1062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f31dd916336a4d8fb3ab015af7de8ccb" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
W20250113 02:51:28.980963 709 leader_election.cc:343] T f31dd916336a4d8fb3ab015af7de8ccb P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:28.981077 1191 tablet_bootstrap.cc:654] T 9fff52e99c64462a8a73fccc3c22ca63 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:28.981495 1757 leader_election.cc:290] T 32cba8f3efc54e78b4df41c15df82eea P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:28.981915 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f31dd916336a4d8fb3ab015af7de8ccb" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:28.982568 1062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32cba8f3efc54e78b4df41c15df82eea" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:28.982816 1195 tablet_replica.cc:331] T 501d4e1ae84e4795913035f8a19cc32d P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:28.983669 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32cba8f3efc54e78b4df41c15df82eea" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:28.983521 711 leader_election.cc:343] T f31dd916336a4d8fb3ab015af7de8ccb P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:28.985167 711 leader_election.cc:304] T f31dd916336a4d8fb3ab015af7de8ccb P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
W20250113 02:51:28.983116 709 leader_election.cc:343] T 32cba8f3efc54e78b4df41c15df82eea P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
W20250113 02:51:28.985867 711 leader_election.cc:343] T 32cba8f3efc54e78b4df41c15df82eea P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:28.985965 711 leader_election.cc:304] T 32cba8f3efc54e78b4df41c15df82eea P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:28.986292 1757 raft_consensus.cc:2743] T f31dd916336a4d8fb3ab015af7de8ccb P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:28.986423 1757 raft_consensus.cc:2743] T 32cba8f3efc54e78b4df41c15df82eea P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:28.982990 1195 raft_consensus.cc:2238] T 501d4e1ae84e4795913035f8a19cc32d P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:28.986868 1191 tablet_bootstrap.cc:492] T 9fff52e99c64462a8a73fccc3c22ca63 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:28.986891 1195 raft_consensus.cc:2267] T 501d4e1ae84e4795913035f8a19cc32d P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:28.987046 1191 ts_tablet_manager.cc:1397] T 9fff52e99c64462a8a73fccc3c22ca63 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.002s
I20250113 02:51:28.987455 1195 ts_tablet_manager.cc:1905] T 501d4e1ae84e4795913035f8a19cc32d P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:28.987993 1191 raft_consensus.cc:357] T 9fff52e99c64462a8a73fccc3c22ca63 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:28.990134 1191 raft_consensus.cc:383] T 9fff52e99c64462a8a73fccc3c22ca63 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:28.990217 1191 raft_consensus.cc:738] T 9fff52e99c64462a8a73fccc3c22ca63 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:28.990453 1191 consensus_queue.cc:260] T 9fff52e99c64462a8a73fccc3c22ca63 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:28.991128 1191 ts_tablet_manager.cc:1428] T 9fff52e99c64462a8a73fccc3c22ca63 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250113 02:51:28.992987 1191 tablet_bootstrap.cc:492] T 378d59d314a84ddb84cf2ab041c51cf1 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:28.993741 1195 ts_tablet_manager.cc:1918] T 501d4e1ae84e4795913035f8a19cc32d P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:28.993918 1195 log.cc:1198] T 501d4e1ae84e4795913035f8a19cc32d P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/501d4e1ae84e4795913035f8a19cc32d
I20250113 02:51:28.997800 1191 tablet_bootstrap.cc:654] T 378d59d314a84ddb84cf2ab041c51cf1 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:28.999778 1195 ts_tablet_manager.cc:1939] T 501d4e1ae84e4795913035f8a19cc32d P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:29.000710 1195 tablet_replica.cc:331] T fdbf03c89bff438a9a25115eddf60be9 P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:29.000945 1195 raft_consensus.cc:2238] T fdbf03c89bff438a9a25115eddf60be9 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:29.001132 1195 raft_consensus.cc:2267] T fdbf03c89bff438a9a25115eddf60be9 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:29.001650 1195 ts_tablet_manager.cc:1905] T fdbf03c89bff438a9a25115eddf60be9 P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:29.006494 1191 tablet_bootstrap.cc:492] T 378d59d314a84ddb84cf2ab041c51cf1 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:29.006656 1191 ts_tablet_manager.cc:1397] T 378d59d314a84ddb84cf2ab041c51cf1 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.003s
I20250113 02:51:29.007542 1195 ts_tablet_manager.cc:1918] T fdbf03c89bff438a9a25115eddf60be9 P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:29.007714 1195 log.cc:1198] T fdbf03c89bff438a9a25115eddf60be9 P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/fdbf03c89bff438a9a25115eddf60be9
I20250113 02:51:29.008118 1195 ts_tablet_manager.cc:1939] T fdbf03c89bff438a9a25115eddf60be9 P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:29.008972 1195 tablet_replica.cc:331] T 258fad818b7b4aa4b337f9949e20aa46 P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:29.009196 1195 raft_consensus.cc:2238] T 258fad818b7b4aa4b337f9949e20aa46 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:29.007539 1191 raft_consensus.cc:357] T 378d59d314a84ddb84cf2ab041c51cf1 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:29.009388 1191 raft_consensus.cc:383] T 378d59d314a84ddb84cf2ab041c51cf1 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:29.009438 1195 raft_consensus.cc:2267] T 258fad818b7b4aa4b337f9949e20aa46 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:29.009464 1191 raft_consensus.cc:738] T 378d59d314a84ddb84cf2ab041c51cf1 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:29.010036 1195 ts_tablet_manager.cc:1905] T 258fad818b7b4aa4b337f9949e20aa46 P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:29.010257 1191 consensus_queue.cc:260] T 378d59d314a84ddb84cf2ab041c51cf1 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:29.010927 1191 ts_tablet_manager.cc:1428] T 378d59d314a84ddb84cf2ab041c51cf1 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250113 02:51:29.011258 1191 tablet_bootstrap.cc:492] T 6d63f0e5835540a78c82020b20a07dac P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:29.014073 1191 tablet_bootstrap.cc:654] T 6d63f0e5835540a78c82020b20a07dac P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:29.016333 1195 ts_tablet_manager.cc:1918] T 258fad818b7b4aa4b337f9949e20aa46 P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:29.016492 1195 log.cc:1198] T 258fad818b7b4aa4b337f9949e20aa46 P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/258fad818b7b4aa4b337f9949e20aa46
I20250113 02:51:29.016855 1195 ts_tablet_manager.cc:1939] T 258fad818b7b4aa4b337f9949e20aa46 P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:29.017671 1195 tablet_replica.cc:331] T 55ebe9c044d944bc813396c699ba9aa1 P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:29.017884 1195 raft_consensus.cc:2238] T 55ebe9c044d944bc813396c699ba9aa1 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:29.018071 1195 raft_consensus.cc:2267] T 55ebe9c044d944bc813396c699ba9aa1 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:29.018560 1195 ts_tablet_manager.cc:1905] T 55ebe9c044d944bc813396c699ba9aa1 P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:29.021250 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet 19f14b3db0b64847b5422bfd5ed323d0 (table default.Table_c7093afcc0ae47c78e7c5cf85219481c [id=141489c57ce44e96a33a1140a8c2b077]) successfully deleted
I20250113 02:51:29.021500 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet 2c399784af46426b8327d175c7a6cce8 (table default.Table_f8af5d3e0dc64d3fae054f82a7b33ccf [id=4b739228564344b8b394a4718750dbb0]) successfully deleted
I20250113 02:51:29.021714 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet 501d4e1ae84e4795913035f8a19cc32d (table default.Table_8a5b45e248e94efb9b16df971fb9755c [id=bd54b5a058a44d718071be41b1e29a8c]) successfully deleted
I20250113 02:51:29.021914 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet fdbf03c89bff438a9a25115eddf60be9 (table default.Table_732e990e6f154d6486889304bbf88cef [id=48c7c3f6b4a840ee9b9e3f3844add13d]) successfully deleted
I20250113 02:51:29.022140 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet 258fad818b7b4aa4b337f9949e20aa46 (table default.Table_e44b0cdc7f4640ec867d92338d94426b [id=9538429e42bb48c7b7fbae43644f2b2d]) successfully deleted
I20250113 02:51:29.025208 1191 tablet_bootstrap.cc:492] T 6d63f0e5835540a78c82020b20a07dac P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:29.025362 1191 ts_tablet_manager.cc:1397] T 6d63f0e5835540a78c82020b20a07dac P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.000s
I20250113 02:51:29.025799 1814 consensus_queue.cc:1035] T 572e4eed1e924381bada27f8389821a7 P 1186fa869eb14812832fa6562531f196 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 02:51:29.026329 1191 raft_consensus.cc:357] T 6d63f0e5835540a78c82020b20a07dac P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:29.026577 1191 raft_consensus.cc:383] T 6d63f0e5835540a78c82020b20a07dac P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:29.026665 1191 raft_consensus.cc:738] T 6d63f0e5835540a78c82020b20a07dac P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:29.028318 1195 ts_tablet_manager.cc:1918] T 55ebe9c044d944bc813396c699ba9aa1 P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:29.028460 1195 log.cc:1198] T 55ebe9c044d944bc813396c699ba9aa1 P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/55ebe9c044d944bc813396c699ba9aa1
I20250113 02:51:29.028810 1195 ts_tablet_manager.cc:1939] T 55ebe9c044d944bc813396c699ba9aa1 P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:29.029786 1195 tablet_replica.cc:331] T f31dd916336a4d8fb3ab015af7de8ccb P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:29.029798 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet 55ebe9c044d944bc813396c699ba9aa1 (table default.Table_2caed919541e43c78a858513a2128f50 [id=26ba87b9bd384b99a016b29fe5b5babd]) successfully deleted
I20250113 02:51:29.030174 915 raft_consensus.cc:3054] T 7f08b72301dc431396cf8cc7790d56d4 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:29.030035 1191 consensus_queue.cc:260] T 6d63f0e5835540a78c82020b20a07dac P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:29.030650 1195 raft_consensus.cc:2238] T f31dd916336a4d8fb3ab015af7de8ccb P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:29.030906 1195 raft_consensus.cc:2267] T f31dd916336a4d8fb3ab015af7de8ccb P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:29.033524 1191 ts_tablet_manager.cc:1428] T 6d63f0e5835540a78c82020b20a07dac P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20250113 02:51:29.033855 1191 tablet_bootstrap.cc:492] T 092892a36e0e47cb8d2184c2bbde95d7 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:29.036064 1191 tablet_bootstrap.cc:654] T 092892a36e0e47cb8d2184c2bbde95d7 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:29.047164 1669 raft_consensus.cc:491] T 601c71d5f7624fbb87b365d2e775d33e P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:29.047328 1669 raft_consensus.cc:513] T 601c71d5f7624fbb87b365d2e775d33e P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:29.047884 1669 leader_election.cc:290] T 601c71d5f7624fbb87b365d2e775d33e P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
W20250113 02:51:29.049659 709 leader_election.cc:343] T 601c71d5f7624fbb87b365d2e775d33e P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Not found: Tablet not found: 601c71d5f7624fbb87b365d2e775d33e
I20250113 02:51:29.049968 1195 ts_tablet_manager.cc:1905] T f31dd916336a4d8fb3ab015af7de8ccb P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:29.056690 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "601c71d5f7624fbb87b365d2e775d33e" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:29.057437 711 leader_election.cc:343] T 601c71d5f7624fbb87b365d2e775d33e P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:29.057564 711 leader_election.cc:304] T 601c71d5f7624fbb87b365d2e775d33e P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:29.057981 1669 raft_consensus.cc:2743] T 601c71d5f7624fbb87b365d2e775d33e P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:29.066732 1195 ts_tablet_manager.cc:1918] T f31dd916336a4d8fb3ab015af7de8ccb P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:29.067575 1195 log.cc:1198] T f31dd916336a4d8fb3ab015af7de8ccb P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/f31dd916336a4d8fb3ab015af7de8ccb
I20250113 02:51:29.067965 1195 ts_tablet_manager.cc:1939] T f31dd916336a4d8fb3ab015af7de8ccb P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:29.068715 1191 tablet_bootstrap.cc:492] T 092892a36e0e47cb8d2184c2bbde95d7 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:29.068889 1191 ts_tablet_manager.cc:1397] T 092892a36e0e47cb8d2184c2bbde95d7 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.035s user 0.003s sys 0.011s
I20250113 02:51:29.069178 1195 tablet_replica.cc:331] T 7d53de6379b84103ba905effb0835676 P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:29.069109 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet f31dd916336a4d8fb3ab015af7de8ccb (table default.Table_37b1d9d091924f8d9a8a2cbf1d06af1a [id=4d85a53a49d642b798af44eef60b3f78]) successfully deleted
I20250113 02:51:29.069415 1195 raft_consensus.cc:2238] T 7d53de6379b84103ba905effb0835676 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:29.069613 1195 raft_consensus.cc:2267] T 7d53de6379b84103ba905effb0835676 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:29.070920 1195 ts_tablet_manager.cc:1905] T 7d53de6379b84103ba905effb0835676 P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:29.071768 1191 raft_consensus.cc:357] T 092892a36e0e47cb8d2184c2bbde95d7 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:29.072053 1191 raft_consensus.cc:383] T 092892a36e0e47cb8d2184c2bbde95d7 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:29.072119 1191 raft_consensus.cc:738] T 092892a36e0e47cb8d2184c2bbde95d7 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:29.073165 1191 consensus_queue.cc:260] T 092892a36e0e47cb8d2184c2bbde95d7 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:29.075181 1191 ts_tablet_manager.cc:1428] T 092892a36e0e47cb8d2184c2bbde95d7 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20250113 02:51:29.075469 1191 tablet_bootstrap.cc:492] T a0ca2e3a3ce94f3aa20df64c89c7cf95 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:29.079483 1191 tablet_bootstrap.cc:654] T a0ca2e3a3ce94f3aa20df64c89c7cf95 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:29.080369 1195 ts_tablet_manager.cc:1918] T 7d53de6379b84103ba905effb0835676 P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:29.080534 1195 log.cc:1198] T 7d53de6379b84103ba905effb0835676 P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/7d53de6379b84103ba905effb0835676
I20250113 02:51:29.080900 1195 ts_tablet_manager.cc:1939] T 7d53de6379b84103ba905effb0835676 P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:29.082109 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet 7d53de6379b84103ba905effb0835676 (table default.Table_71699c9b0b1a46cda2db76dee82b90b4 [id=d1b0bb18a7b84473803cd1294b0829ac]) successfully deleted
I20250113 02:51:29.082444 1195 tablet_replica.cc:331] T 601c71d5f7624fbb87b365d2e775d33e P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:29.082661 1195 raft_consensus.cc:2238] T 601c71d5f7624fbb87b365d2e775d33e P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:29.082866 1195 raft_consensus.cc:2267] T 601c71d5f7624fbb87b365d2e775d33e P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:29.083420 1195 ts_tablet_manager.cc:1905] T 601c71d5f7624fbb87b365d2e775d33e P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:29.094313 448 catalog_manager.cc:5618] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095) does not have the latest schema for tablet 32cba8f3efc54e78b4df41c15df82eea (table default.Table_c22142b5375a4837bbc42af6f77d880f [id=8aff13d6d3c5483eadb596b705fe5151]). Expected version 4 got 0
I20250113 02:51:29.094561 448 catalog_manager.cc:5526] T e8ab4516789e4585a101c8766d338af3 P 1186fa869eb14812832fa6562531f196 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1186fa869eb14812832fa6562531f196 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "1186fa869eb14812832fa6562531f196" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } health_report { overall_health: HEALTHY } } }
W20250113 02:51:29.095129 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 316cb2863ed645158a8f72e3cb0a94fd consensus state has no leader
I20250113 02:51:29.095376 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 6666922d677a4c678f80d7c6c47fd2a5
I20250113 02:51:29.096307 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_e621a0cb47054f84898824e0343e3a6a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:29.096518 452 catalog_manager.cc:5618] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467) does not have the latest schema for tablet 7f08b72301dc431396cf8cc7790d56d4 (table default.Table_6de3bf6640dc47f9b977f608c5ec8ef9 [id=bd14ce0e2b0a4a60a63022be2bc80fef]). Expected version 3 got 0
I20250113 02:51:29.096697 452 catalog_manager.cc:5618] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467) does not have the latest schema for tablet 572e4eed1e924381bada27f8389821a7 (table default.Table_1d910c3d2b244474a576fc4088301143 [id=75cb0745bbb84e1c883cb13b546ecb11]). Expected version 4 got 0
W20250113 02:51:29.096952 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e621a0cb47054f84898824e0343e3a6a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:29.097817 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_07e3e48b34b040ebbcf9a5dffdbb6bd9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:29.098320 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_07e3e48b34b040ebbcf9a5dffdbb6bd9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:29.099428 738 tablet_service.cc:1514] Processing DeleteTablet for tablet 6666922d677a4c678f80d7c6c47fd2a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:28 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:29.098596 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 46ed22e392db441a894dceeedd2911f4
I20250113 02:51:29.100926 454 catalog_manager.cc:6596] ReplaceTablet: tablet 46ed22e392db441a894dceeedd2911f4 deleted and replaced by tablet 524b4bd640a04123b4abbd716e7d40f2
I20250113 02:51:29.101291 454 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_72783655bdee4fcf8c2fff2186fe0a60" } new_table_name: "default.Table_a752553e5fcc4ed1a56d9d5197d6b885" modify_external_catalogs: true
I20250113 02:51:29.103283 1033 tablet_service.cc:1514] Processing DeleteTablet for tablet 46ed22e392db441a894dceeedd2911f4 with delete_type TABLET_DATA_DELETED (replaced by tablet 524b4bd640a04123b4abbd716e7d40f2) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:29.103510 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet 6666922d677a4c678f80d7c6c47fd2a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:28 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:29.099109 447 master_service.cc:946] ReplaceTablet: received request to replace tablet 8547b63c40ec4ccebd962b5a3b2c5e0f from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:29.104450 738 tablet_service.cc:1514] Processing DeleteTablet for tablet 46ed22e392db441a894dceeedd2911f4 with delete_type TABLET_DATA_DELETED (replaced by tablet 524b4bd640a04123b4abbd716e7d40f2) from {username='slave'} at 127.0.0.1:55398
W20250113 02:51:29.104679 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 46ed22e392db441a894dceeedd2911f4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 46ed22e392db441a894dceeedd2911f4 already in progress: creating tablet
I20250113 02:51:29.099398 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 32d70f20f9f7440b89e861edb5aa57e1
W20250113 02:51:29.105082 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 6666922d677a4c678f80d7c6c47fd2a5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6666922d677a4c678f80d7c6c47fd2a5 already in progress: creating tablet
I20250113 02:51:29.105620 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet 32d70f20f9f7440b89e861edb5aa57e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:28 UTC) from {username='slave'} at 127.0.0.1:60230
W20250113 02:51:29.106087 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 32d70f20f9f7440b89e861edb5aa57e1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 32d70f20f9f7440b89e861edb5aa57e1 already in progress: creating tablet
I20250113 02:51:29.100026 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 3025935f9743442eb3e6f2db22eb52ad
I20250113 02:51:29.106913 449 catalog_manager.cc:6596] ReplaceTablet: tablet 3025935f9743442eb3e6f2db22eb52ad deleted and replaced by tablet 5d2d14548b734b1fbe57d68c0ea3e2f8
I20250113 02:51:29.107201 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet 3025935f9743442eb3e6f2db22eb52ad with delete_type TABLET_DATA_DELETED (replaced by tablet 5d2d14548b734b1fbe57d68c0ea3e2f8) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:29.107323 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_2f896daff51e4eb78a833f59d07ea44d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:29.107760 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 3025935f9743442eb3e6f2db22eb52ad because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3025935f9743442eb3e6f2db22eb52ad already in progress: creating tablet
W20250113 02:51:29.107813 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2f896daff51e4eb78a833f59d07ea44d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:29.100289 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 5d4912055c5d4c8d8be8b65121f53200
I20250113 02:51:29.100354 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet b16dd8451f32478daf2ae118dbb9b304
I20250113 02:51:29.108325 450 catalog_manager.cc:6596] ReplaceTablet: tablet b16dd8451f32478daf2ae118dbb9b304 deleted and replaced by tablet ea51f6e169ae49bd815a9496a1830d0a
I20250113 02:51:29.108475 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_dbec5dd582c846b893ff6250a548bad9" } new_table_name: "default.Table_9c2b9596afed4532858a877b5e954b4d" modify_external_catalogs: true
I20250113 02:51:29.108662 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_0146ad4fe4424350a582eb66ed609eba"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:29.109144 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0146ad4fe4424350a582eb66ed609eba in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:29.095083 448 catalog_manager.cc:5618] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095) does not have the latest schema for tablet e8ab4516789e4585a101c8766d338af3 (table default.Table_3b76cf9fac334ebf9b1ecb9f8d07ea74 [id=b112ac4e2e58460db083da1eb788ca07]). Expected version 2 got 0
I20250113 02:51:29.109961 752 tablet_service.cc:1514] Processing DeleteTablet for tablet 3025935f9743442eb3e6f2db22eb52ad with delete_type TABLET_DATA_DELETED (replaced by tablet 5d2d14548b734b1fbe57d68c0ea3e2f8) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:29.110167 750 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d4912055c5d4c8d8be8b65121f53200 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:28 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:29.110344 743 tablet_service.cc:1514] Processing DeleteTablet for tablet b16dd8451f32478daf2ae118dbb9b304 with delete_type TABLET_DATA_DELETED (replaced by tablet ea51f6e169ae49bd815a9496a1830d0a) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:29.110487 738 tablet_service.cc:1514] Processing DeleteTablet for tablet 32d70f20f9f7440b89e861edb5aa57e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:28 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:29.111258 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d4912055c5d4c8d8be8b65121f53200 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:28 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:29.111740 1033 tablet_service.cc:1514] Processing DeleteTablet for tablet b16dd8451f32478daf2ae118dbb9b304 with delete_type TABLET_DATA_DELETED (replaced by tablet ea51f6e169ae49bd815a9496a1830d0a) from {username='slave'} at 127.0.0.1:60230
W20250113 02:51:29.112075 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 5d4912055c5d4c8d8be8b65121f53200 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5d4912055c5d4c8d8be8b65121f53200 already in progress: creating tablet
W20250113 02:51:29.112493 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet b16dd8451f32478daf2ae118dbb9b304 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b16dd8451f32478daf2ae118dbb9b304 already in progress: creating tablet
I20250113 02:51:29.117893 886 tablet_service.cc:1514] Processing DeleteTablet for tablet 32d70f20f9f7440b89e861edb5aa57e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:28 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:29.118212 887 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d4912055c5d4c8d8be8b65121f53200 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:28 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:29.118420 896 tablet_service.cc:1514] Processing DeleteTablet for tablet b2a35fd891c645d68d79c61a2c832ba7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:20 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:29.118636 889 tablet_service.cc:1514] Processing DeleteTablet for tablet 46ed22e392db441a894dceeedd2911f4 with delete_type TABLET_DATA_DELETED (replaced by tablet 524b4bd640a04123b4abbd716e7d40f2) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:29.118685 1196 tablet_replica.cc:331] T b2a35fd891c645d68d79c61a2c832ba7 P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:29.118855 888 tablet_service.cc:1514] Processing DeleteTablet for tablet b16dd8451f32478daf2ae118dbb9b304 with delete_type TABLET_DATA_DELETED (replaced by tablet ea51f6e169ae49bd815a9496a1830d0a) from {username='slave'} at 127.0.0.1:37366
W20250113 02:51:29.119258 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 32d70f20f9f7440b89e861edb5aa57e1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 32d70f20f9f7440b89e861edb5aa57e1 already in progress: creating tablet
I20250113 02:51:29.118911 1196 raft_consensus.cc:2238] T b2a35fd891c645d68d79c61a2c832ba7 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
W20250113 02:51:29.119563 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 5d4912055c5d4c8d8be8b65121f53200 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5d4912055c5d4c8d8be8b65121f53200 already in progress: creating tablet
I20250113 02:51:29.119673 884 tablet_service.cc:1514] Processing DeleteTablet for tablet 3025935f9743442eb3e6f2db22eb52ad with delete_type TABLET_DATA_DELETED (replaced by tablet 5d2d14548b734b1fbe57d68c0ea3e2f8) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:29.119676 1196 raft_consensus.cc:2267] T b2a35fd891c645d68d79c61a2c832ba7 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:29.119889 885 tablet_service.cc:1514] Processing DeleteTablet for tablet 6666922d677a4c678f80d7c6c47fd2a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:28 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:29.120471 1194 tablet_bootstrap.cc:492] T ee7582ee04494c0db1179359bcb311c8 P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:29.120613 1196 ts_tablet_manager.cc:1905] T b2a35fd891c645d68d79c61a2c832ba7 P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:29.120846 1194 ts_tablet_manager.cc:1397] T ee7582ee04494c0db1179359bcb311c8 P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.298s user 0.004s sys 0.000s
I20250113 02:51:29.124737 1194 raft_consensus.cc:357] T ee7582ee04494c0db1179359bcb311c8 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:29.129848 1194 raft_consensus.cc:383] T ee7582ee04494c0db1179359bcb311c8 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:29.129957 1194 raft_consensus.cc:738] T ee7582ee04494c0db1179359bcb311c8 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:29.130224 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet acd83024592e431898e9675c17f50d4f
I20250113 02:51:29.130223 1194 consensus_queue.cc:260] T ee7582ee04494c0db1179359bcb311c8 P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:29.131002 885 tablet_service.cc:1514] Processing DeleteTablet for tablet acd83024592e431898e9675c17f50d4f with delete_type TABLET_DATA_DELETED (replaced by tablet 18716752335043e89038f01b4d65d711) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:29.131269 738 tablet_service.cc:1514] Processing DeleteTablet for tablet acd83024592e431898e9675c17f50d4f with delete_type TABLET_DATA_DELETED (replaced by tablet 18716752335043e89038f01b4d65d711) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:29.131531 451 catalog_manager.cc:6596] ReplaceTablet: tablet acd83024592e431898e9675c17f50d4f deleted and replaced by tablet 18716752335043e89038f01b4d65d711
I20250113 02:51:29.131824 1194 ts_tablet_manager.cc:1428] T ee7582ee04494c0db1179359bcb311c8 P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.011s user 0.000s sys 0.002s
I20250113 02:51:29.131913 1033 tablet_service.cc:1514] Processing DeleteTablet for tablet acd83024592e431898e9675c17f50d4f with delete_type TABLET_DATA_DELETED (replaced by tablet 18716752335043e89038f01b4d65d711) from {username='slave'} at 127.0.0.1:60230
W20250113 02:51:29.132972 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet acd83024592e431898e9675c17f50d4f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet acd83024592e431898e9675c17f50d4f already in progress: creating tablet
I20250113 02:51:29.136150 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_f82ef9176e6043d28f8d6a337ac4bce3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:29.136735 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f82ef9176e6043d28f8d6a337ac4bce3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:29.138085 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_82d6d0a936524f35beab62f451e5eb99" } modify_external_catalogs: true
I20250113 02:51:29.138324 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_82d6d0a936524f35beab62f451e5eb99" } modify_external_catalogs: true
I20250113 02:51:29.138368 1196 ts_tablet_manager.cc:1918] T b2a35fd891c645d68d79c61a2c832ba7 P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:29.138504 1194 tablet_bootstrap.cc:492] T 7d53de6379b84103ba905effb0835676 P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:29.140789 1194 tablet_bootstrap.cc:654] T 7d53de6379b84103ba905effb0835676 P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:29.141405 1196 log.cc:1198] T b2a35fd891c645d68d79c61a2c832ba7 P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/b2a35fd891c645d68d79c61a2c832ba7
I20250113 02:51:29.141883 1196 ts_tablet_manager.cc:1939] T b2a35fd891c645d68d79c61a2c832ba7 P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:29.143182 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet b2a35fd891c645d68d79c61a2c832ba7 (table default.Table_266561dfe36245ecb8230fdf1904cb02 [id=60925cf1a1224988b0d349b621ab5368]) successfully deleted
W20250113 02:51:29.145782 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 46ed22e392db441a894dceeedd2911f4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 46ed22e392db441a894dceeedd2911f4 already in progress: creating tablet
W20250113 02:51:29.146708 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet b16dd8451f32478daf2ae118dbb9b304 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b16dd8451f32478daf2ae118dbb9b304 already in progress: creating tablet
W20250113 02:51:29.147521 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 3025935f9743442eb3e6f2db22eb52ad because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3025935f9743442eb3e6f2db22eb52ad already in progress: creating tablet
W20250113 02:51:29.148154 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 6666922d677a4c678f80d7c6c47fd2a5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6666922d677a4c678f80d7c6c47fd2a5 already in progress: creating tablet
W20250113 02:51:29.148895 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet acd83024592e431898e9675c17f50d4f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet acd83024592e431898e9675c17f50d4f already in progress: creating tablet
W20250113 02:51:29.149758 454 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8f2fb58fc24d4692a3fe9a652e076aed consensus state has no leader
I20250113 02:51:29.150239 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_19f32aa03bc941cebef094ca3896e0c3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:29.150746 1191 tablet_bootstrap.cc:492] T a0ca2e3a3ce94f3aa20df64c89c7cf95 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
W20250113 02:51:29.150863 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_19f32aa03bc941cebef094ca3896e0c3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:29.150919 1191 ts_tablet_manager.cc:1397] T a0ca2e3a3ce94f3aa20df64c89c7cf95 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.076s user 0.007s sys 0.014s
I20250113 02:51:29.151726 1191 raft_consensus.cc:357] T a0ca2e3a3ce94f3aa20df64c89c7cf95 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:29.151925 1191 raft_consensus.cc:383] T a0ca2e3a3ce94f3aa20df64c89c7cf95 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:29.151968 1191 raft_consensus.cc:738] T a0ca2e3a3ce94f3aa20df64c89c7cf95 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:29.156879 1191 consensus_queue.cc:260] T a0ca2e3a3ce94f3aa20df64c89c7cf95 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:29.159180 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 8547b63c40ec4ccebd962b5a3b2c5e0f
I20250113 02:51:29.159655 447 catalog_manager.cc:6596] ReplaceTablet: tablet 8547b63c40ec4ccebd962b5a3b2c5e0f deleted and replaced by tablet 573cffe308cf4774a2108a7390f08e90
I20250113 02:51:29.160004 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_4e3230947e8b4b71992b241c34d4e6ad"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:29.160347 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet 8547b63c40ec4ccebd962b5a3b2c5e0f with delete_type TABLET_DATA_DELETED (replaced by tablet 573cffe308cf4774a2108a7390f08e90) from {username='slave'} at 127.0.0.1:60230
W20250113 02:51:29.160563 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4e3230947e8b4b71992b241c34d4e6ad in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:29.160607 1198 tablet_replica.cc:331] T 8547b63c40ec4ccebd962b5a3b2c5e0f P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:29.160686 1191 ts_tablet_manager.cc:1428] T a0ca2e3a3ce94f3aa20df64c89c7cf95 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.010s user 0.003s sys 0.000s
I20250113 02:51:29.160822 1198 raft_consensus.cc:2238] T 8547b63c40ec4ccebd962b5a3b2c5e0f P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:29.160985 1198 raft_consensus.cc:2267] T 8547b63c40ec4ccebd962b5a3b2c5e0f P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:29.160998 1191 tablet_bootstrap.cc:492] T 2778d7dc7d9c4843bbc9eb705cec71cd P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:29.161531 1198 ts_tablet_manager.cc:1905] T 8547b63c40ec4ccebd962b5a3b2c5e0f P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:29.162089 738 tablet_service.cc:1514] Processing DeleteTablet for tablet 8547b63c40ec4ccebd962b5a3b2c5e0f with delete_type TABLET_DATA_DELETED (replaced by tablet 573cffe308cf4774a2108a7390f08e90) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:29.163149 1191 tablet_bootstrap.cc:654] T 2778d7dc7d9c4843bbc9eb705cec71cd P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:29.163272 885 tablet_service.cc:1514] Processing DeleteTablet for tablet 8547b63c40ec4ccebd962b5a3b2c5e0f with delete_type TABLET_DATA_DELETED (replaced by tablet 573cffe308cf4774a2108a7390f08e90) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:29.163540 1196 tablet_replica.cc:331] T 8547b63c40ec4ccebd962b5a3b2c5e0f P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:29.163714 1196 raft_consensus.cc:2238] T 8547b63c40ec4ccebd962b5a3b2c5e0f P df69f48656b44bb890f76587ee69cc0f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:29.163950 1196 raft_consensus.cc:2267] T 8547b63c40ec4ccebd962b5a3b2c5e0f P df69f48656b44bb890f76587ee69cc0f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:29.165184 1196 ts_tablet_manager.cc:1905] T 8547b63c40ec4ccebd962b5a3b2c5e0f P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:29.169960 1198 ts_tablet_manager.cc:1918] T 8547b63c40ec4ccebd962b5a3b2c5e0f P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250113 02:51:29.164052 1833 tablet.cc:1721] T 7f08b72301dc431396cf8cc7790d56d4 P df69f48656b44bb890f76587ee69cc0f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250113 02:51:29.172315 1194 tablet_bootstrap.cc:492] T 7d53de6379b84103ba905effb0835676 P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:29.174889 1195 ts_tablet_manager.cc:1918] T 601c71d5f7624fbb87b365d2e775d33e P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:29.175004 1195 log.cc:1198] T 601c71d5f7624fbb87b365d2e775d33e P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/601c71d5f7624fbb87b365d2e775d33e
I20250113 02:51:29.175328 1195 ts_tablet_manager.cc:1939] T 601c71d5f7624fbb87b365d2e775d33e P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:29.176401 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet 601c71d5f7624fbb87b365d2e775d33e (table default.Table_7a9c9cc8678e4f7fa12237d3f8c9c3de [id=97a27e2641d2468aac5c827fad4e2091]) successfully deleted
I20250113 02:51:29.183468 1191 tablet_bootstrap.cc:492] T 2778d7dc7d9c4843bbc9eb705cec71cd P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:29.183588 1191 ts_tablet_manager.cc:1397] T 2778d7dc7d9c4843bbc9eb705cec71cd P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.023s user 0.002s sys 0.003s
I20250113 02:51:29.183611 448 catalog_manager.cc:5618] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467) does not have the latest schema for tablet 7f08b72301dc431396cf8cc7790d56d4 (table default.Table_6de3bf6640dc47f9b977f608c5ec8ef9 [id=bd14ce0e2b0a4a60a63022be2bc80fef]). Expected version 3 got 0
I20250113 02:51:29.184208 455 catalog_manager.cc:5618] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587) does not have the latest schema for tablet e8ab4516789e4585a101c8766d338af3 (table default.Table_3b76cf9fac334ebf9b1ecb9f8d07ea74 [id=b112ac4e2e58460db083da1eb788ca07]). Expected version 2 got 0
I20250113 02:51:29.184433 455 catalog_manager.cc:5618] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587) does not have the latest schema for tablet 572e4eed1e924381bada27f8389821a7 (table default.Table_1d910c3d2b244474a576fc4088301143 [id=75cb0745bbb84e1c883cb13b546ecb11]). Expected version 4 got 0
I20250113 02:51:29.184779 1191 raft_consensus.cc:357] T 2778d7dc7d9c4843bbc9eb705cec71cd P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:29.185671 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_49cae5c8551e4a9cae1e7dfba2b4e97a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:29.184948 1191 raft_consensus.cc:383] T 2778d7dc7d9c4843bbc9eb705cec71cd P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:29.187224 1191 raft_consensus.cc:738] T 2778d7dc7d9c4843bbc9eb705cec71cd P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:29.188202 1194 ts_tablet_manager.cc:1397] T 7d53de6379b84103ba905effb0835676 P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.056s user 0.009s sys 0.002s
I20250113 02:51:29.187849 1191 consensus_queue.cc:260] T 2778d7dc7d9c4843bbc9eb705cec71cd P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:29.188766 1191 ts_tablet_manager.cc:1428] T 2778d7dc7d9c4843bbc9eb705cec71cd P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.005s user 0.001s sys 0.002s
I20250113 02:51:29.189038 1191 tablet_bootstrap.cc:492] T ddf79eec41214693859f526265a55492 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:29.189072 743 tablet_service.cc:1514] Processing DeleteTablet for tablet c215ffe61b9b426286cc5e808463078d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:20 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:29.189419 738 tablet_service.cc:1514] Processing DeleteTablet for tablet 63efcb6b2e194c469fb30a197bd6565b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:20 UTC) from {username='slave'} at 127.0.0.1:55398
W20250113 02:51:29.190622 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_49cae5c8551e4a9cae1e7dfba2b4e97a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:29.192059 1191 tablet_bootstrap.cc:654] T ddf79eec41214693859f526265a55492 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:29.192359 448 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_74ce8ff3e2c64e088ac82c48ec722b39" } modify_external_catalogs: true
I20250113 02:51:29.192554 448 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_74ce8ff3e2c64e088ac82c48ec722b39" } modify_external_catalogs: true
I20250113 02:51:29.172562 1196 ts_tablet_manager.cc:1918] T 8547b63c40ec4ccebd962b5a3b2c5e0f P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250113 02:51:29.194270 1196 log.cc:1198] T 8547b63c40ec4ccebd962b5a3b2c5e0f P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/8547b63c40ec4ccebd962b5a3b2c5e0f
I20250113 02:51:29.194954 1196 ts_tablet_manager.cc:1939] T 8547b63c40ec4ccebd962b5a3b2c5e0f P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:29.196075 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet 8547b63c40ec4ccebd962b5a3b2c5e0f (table default.Table_3d5fd97116b343b0a6c798e1f22327c2 [id=1396c4482ee646d18330d200fd3f9816]) successfully deleted
I20250113 02:51:29.197315 1819 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 02:51:29.197407 1191 tablet_bootstrap.cc:492] T ddf79eec41214693859f526265a55492 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:29.197528 1191 ts_tablet_manager.cc:1397] T ddf79eec41214693859f526265a55492 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.003s
I20250113 02:51:29.198232 1191 raft_consensus.cc:357] T ddf79eec41214693859f526265a55492 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:29.198413 1191 raft_consensus.cc:383] T ddf79eec41214693859f526265a55492 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:29.198463 1191 raft_consensus.cc:738] T ddf79eec41214693859f526265a55492 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:29.198655 1191 consensus_queue.cc:260] T ddf79eec41214693859f526265a55492 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:29.199091 1191 ts_tablet_manager.cc:1428] T ddf79eec41214693859f526265a55492 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250113 02:51:29.199371 1191 tablet_bootstrap.cc:492] T 401c4f5dbd304950877b59a865f417b3 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:29.203276 1191 tablet_bootstrap.cc:654] T 401c4f5dbd304950877b59a865f417b3 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:29.203680 1706 fs_manager.cc:730] Time spent opening block manager: real 0.345s user 0.056s sys 0.071s
W20250113 02:51:29.204154 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 6666922d677a4c678f80d7c6c47fd2a5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6666922d677a4c678f80d7c6c47fd2a5 already in progress: creating tablet
W20250113 02:51:29.204811 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 46ed22e392db441a894dceeedd2911f4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 46ed22e392db441a894dceeedd2911f4 already in progress: creating tablet
I20250113 02:51:29.203891 1706 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/master-2/data,/tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/master-2/wal
uuid: "45e9d43ecac0401fb2c18fe5eaf089b5"
format_stamp: "Formatted at 2025-01-13 02:51:13 on dist-test-slave-v0t5"
I20250113 02:51:29.175545 1634 raft_consensus.cc:491] T 19f14b3db0b64847b5422bfd5ed323d0 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:29.205240 1706 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 02:51:29.205328 1634 raft_consensus.cc:513] T 19f14b3db0b64847b5422bfd5ed323d0 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:29.206043 1634 leader_election.cc:290] T 19f14b3db0b64847b5422bfd5ed323d0 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
W20250113 02:51:29.206507 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 3025935f9743442eb3e6f2db22eb52ad because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3025935f9743442eb3e6f2db22eb52ad already in progress: creating tablet
I20250113 02:51:29.209326 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19f14b3db0b64847b5422bfd5ed323d0" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:29.185948 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet 19f14b3db0b64847b5422bfd5ed323d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:19 UTC) from {username='slave'} at 127.0.0.1:60230
W20250113 02:51:29.209638 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 5d4912055c5d4c8d8be8b65121f53200 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5d4912055c5d4c8d8be8b65121f53200 already in progress: creating tablet
W20250113 02:51:29.209878 998 leader_election.cc:343] T 19f14b3db0b64847b5422bfd5ed323d0 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:29.199093 1192 tablet_bootstrap.cc:492] T f31dd916336a4d8fb3ab015af7de8ccb P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:29.210139 1192 ts_tablet_manager.cc:1397] T f31dd916336a4d8fb3ab015af7de8ccb P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.330s user 0.043s sys 0.076s
W20250113 02:51:29.210629 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet b16dd8451f32478daf2ae118dbb9b304 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b16dd8451f32478daf2ae118dbb9b304 already in progress: creating tablet
I20250113 02:51:29.210865 1192 raft_consensus.cc:357] T f31dd916336a4d8fb3ab015af7de8ccb P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:29.211064 1192 raft_consensus.cc:383] T f31dd916336a4d8fb3ab015af7de8ccb P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:29.211136 1192 raft_consensus.cc:738] T f31dd916336a4d8fb3ab015af7de8ccb P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
W20250113 02:51:29.211223 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 32d70f20f9f7440b89e861edb5aa57e1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 32d70f20f9f7440b89e861edb5aa57e1 already in progress: creating tablet
I20250113 02:51:29.211335 1192 consensus_queue.cc:260] T f31dd916336a4d8fb3ab015af7de8ccb P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:29.214073 1192 ts_tablet_manager.cc:1428] T f31dd916336a4d8fb3ab015af7de8ccb P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250113 02:51:29.214341 1192 tablet_bootstrap.cc:492] T fdbf03c89bff438a9a25115eddf60be9 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:29.215118 1814 raft_consensus.cc:491] T a0ca2e3a3ce94f3aa20df64c89c7cf95 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:29.215269 1814 raft_consensus.cc:513] T a0ca2e3a3ce94f3aa20df64c89c7cf95 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:29.216006 1814 leader_election.cc:290] T a0ca2e3a3ce94f3aa20df64c89c7cf95 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:29.216042 1192 tablet_bootstrap.cc:654] T fdbf03c89bff438a9a25115eddf60be9 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
W20250113 02:51:29.216272 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9fa6a1c4f6ef4f98b4de114210af08c5 consensus state has no leader
I20250113 02:51:29.216626 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_322cee4c328e4e2ca3dcfed1ac77749d" } new_table_name: "default.Table_cf3667be494e4680bd98beb9c0e1fcf2" modify_external_catalogs: true
I20250113 02:51:29.217002 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0ca2e3a3ce94f3aa20df64c89c7cf95" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:29.217721 711 leader_election.cc:343] T a0ca2e3a3ce94f3aa20df64c89c7cf95 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:29.218539 885 tablet_service.cc:1514] Processing DeleteTablet for tablet ee7582ee04494c0db1179359bcb311c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:19 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:29.219012 1062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0ca2e3a3ce94f3aa20df64c89c7cf95" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
W20250113 02:51:29.219347 439 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 8f2fb58fc24d4692a3fe9a652e076aed (table default.Table_a752553e5fcc4ed1a56d9d5197d6b885 [id=8edee52f05e24c03b088ce58f4b3f4d9], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 8f2fb58fc24d4692a3fe9a652e076aed consensus state has no leader
W20250113 02:51:29.219777 709 leader_election.cc:343] T a0ca2e3a3ce94f3aa20df64c89c7cf95 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:29.219894 709 leader_election.cc:304] T a0ca2e3a3ce94f3aa20df64c89c7cf95 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:29.221052 1198 log.cc:1198] T 8547b63c40ec4ccebd962b5a3b2c5e0f P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/8547b63c40ec4ccebd962b5a3b2c5e0f
I20250113 02:51:29.204842 1194 raft_consensus.cc:357] T 7d53de6379b84103ba905effb0835676 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:29.221801 1814 raft_consensus.cc:2743] T a0ca2e3a3ce94f3aa20df64c89c7cf95 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:29.221743 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_c2be66cb47964190b4a1a67e3a9f6787" } new_table_name: "default.Table_810d62357c794aeaa18da712725a338f" modify_external_catalogs: true
I20250113 02:51:29.221850 1194 raft_consensus.cc:383] T 7d53de6379b84103ba905effb0835676 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:29.221956 1194 raft_consensus.cc:738] T 7d53de6379b84103ba905effb0835676 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:29.222213 1194 consensus_queue.cc:260] T 7d53de6379b84103ba905effb0835676 P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:29.223937 1198 ts_tablet_manager.cc:1939] T 8547b63c40ec4ccebd962b5a3b2c5e0f P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:29.225502 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet 8547b63c40ec4ccebd962b5a3b2c5e0f (table default.Table_3d5fd97116b343b0a6c798e1f22327c2 [id=1396c4482ee646d18330d200fd3f9816]) successfully deleted
I20250113 02:51:29.226348 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet f31dd916336a4d8fb3ab015af7de8ccb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:23 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:29.227509 1194 ts_tablet_manager.cc:1428] T 7d53de6379b84103ba905effb0835676 P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.034s user 0.002s sys 0.000s
I20250113 02:51:29.229669 1198 tablet_replica.cc:331] T 19f14b3db0b64847b5422bfd5ed323d0 P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:29.230171 885 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d53de6379b84103ba905effb0835676 with delete_type TABLET_DATA_DELETED (replaced by tablet fd0b8592997d4a61b8ca854ecbb67c55) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:29.230616 1194 tablet_bootstrap.cc:492] T 5b3f012b88c04cb68ed7a1848147b70b P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:29.231459 1196 tablet_replica.cc:331] T ee7582ee04494c0db1179359bcb311c8 P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:29.231942 1198 raft_consensus.cc:2238] T 19f14b3db0b64847b5422bfd5ed323d0 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:29.232143 1198 raft_consensus.cc:2267] T 19f14b3db0b64847b5422bfd5ed323d0 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:29.231827 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_8db01e4eaddd49aeb8f955e75cabb671"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:29.232970 1198 ts_tablet_manager.cc:1905] T 19f14b3db0b64847b5422bfd5ed323d0 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250113 02:51:29.233063 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8db01e4eaddd49aeb8f955e75cabb671 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:29.234946 1191 tablet_bootstrap.cc:492] T 401c4f5dbd304950877b59a865f417b3 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:29.235095 1191 ts_tablet_manager.cc:1397] T 401c4f5dbd304950877b59a865f417b3 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.036s user 0.001s sys 0.008s
I20250113 02:51:29.235653 1196 raft_consensus.cc:2238] T ee7582ee04494c0db1179359bcb311c8 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:29.235836 1191 raft_consensus.cc:357] T 401c4f5dbd304950877b59a865f417b3 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:29.236059 1191 raft_consensus.cc:383] T 401c4f5dbd304950877b59a865f417b3 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:29.235953 1196 raft_consensus.cc:2267] T ee7582ee04494c0db1179359bcb311c8 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:29.236135 1191 raft_consensus.cc:738] T 401c4f5dbd304950877b59a865f417b3 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:29.236383 1191 consensus_queue.cc:260] T 401c4f5dbd304950877b59a865f417b3 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:29.237000 1196 ts_tablet_manager.cc:1905] T ee7582ee04494c0db1179359bcb311c8 P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250113 02:51:29.237006 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet acd83024592e431898e9675c17f50d4f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet acd83024592e431898e9675c17f50d4f already in progress: creating tablet
I20250113 02:51:29.238895 1195 tablet_replica.cc:331] T 8547b63c40ec4ccebd962b5a3b2c5e0f P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:29.239308 1195 raft_consensus.cc:2238] T 8547b63c40ec4ccebd962b5a3b2c5e0f P 1186fa869eb14812832fa6562531f196 [term 1 LEADER]: Raft consensus shutting down.
W20250113 02:51:29.239413 995 leader_election.cc:343] T 19f14b3db0b64847b5422bfd5ed323d0 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Not found: Tablet not found: 19f14b3db0b64847b5422bfd5ed323d0
I20250113 02:51:29.239526 995 leader_election.cc:304] T 19f14b3db0b64847b5422bfd5ed323d0 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4216a7f853d642b7bc3e5cff458f3aca; no voters: 1186fa869eb14812832fa6562531f196, df69f48656b44bb890f76587ee69cc0f
W20250113 02:51:29.239693 995 raft_consensus.cc:2681] T 19f14b3db0b64847b5422bfd5ed323d0 P 4216a7f853d642b7bc3e5cff458f3aca: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250113 02:51:29.240314 1634 consensus_queue.cc:1035] T 8adfa738c17a4169a5d99f6ceca75100 P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 02:51:29.241164 1194 tablet_bootstrap.cc:654] T 5b3f012b88c04cb68ed7a1848147b70b P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:29.241231 1191 ts_tablet_manager.cc:1428] T 401c4f5dbd304950877b59a865f417b3 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.006s user 0.002s sys 0.001s
I20250113 02:51:29.241510 1191 tablet_bootstrap.cc:492] T 11950e996c814590bc9f3f22c0354902 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
W20250113 02:51:29.241665 998 consensus_peers.cc:487] T 8adfa738c17a4169a5d99f6ceca75100 P 4216a7f853d642b7bc3e5cff458f3aca -> Peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Couldn't send request to peer df69f48656b44bb890f76587ee69cc0f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250113 02:51:29.239708 1195 raft_consensus.cc:2267] T 8547b63c40ec4ccebd962b5a3b2c5e0f P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:29.244987 1195 ts_tablet_manager.cc:1905] T 8547b63c40ec4ccebd962b5a3b2c5e0f P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:29.251425 1196 ts_tablet_manager.cc:1918] T ee7582ee04494c0db1179359bcb311c8 P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:29.252233 1196 log.cc:1198] T ee7582ee04494c0db1179359bcb311c8 P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/ee7582ee04494c0db1179359bcb311c8
I20250113 02:51:29.252852 1196 ts_tablet_manager.cc:1939] T ee7582ee04494c0db1179359bcb311c8 P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:29.254381 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet ee7582ee04494c0db1179359bcb311c8 (table default.Table_c8bea6af909a424c9101c8934d1aee7b [id=bd9d66f2d35446e9bc4ed5e589f23012]) successfully deleted
I20250113 02:51:29.255728 1196 tablet_replica.cc:331] T 7d53de6379b84103ba905effb0835676 P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:29.256245 1196 raft_consensus.cc:2238] T 7d53de6379b84103ba905effb0835676 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:29.256803 1196 raft_consensus.cc:2267] T 7d53de6379b84103ba905effb0835676 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:29.257839 1196 ts_tablet_manager.cc:1905] T 7d53de6379b84103ba905effb0835676 P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:29.260689 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 80c647a3b5a1411e9130461115c53550
I20250113 02:51:29.261626 885 tablet_service.cc:1514] Processing DeleteTablet for tablet 80c647a3b5a1411e9130461115c53550 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:29 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:29.261888 738 tablet_service.cc:1514] Processing DeleteTablet for tablet 80c647a3b5a1411e9130461115c53550 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:29 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:29.245545 1191 tablet_bootstrap.cc:654] T 11950e996c814590bc9f3f22c0354902 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:29.267235 1194 tablet_bootstrap.cc:492] T 5b3f012b88c04cb68ed7a1848147b70b P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:29.267447 1194 ts_tablet_manager.cc:1397] T 5b3f012b88c04cb68ed7a1848147b70b P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.039s user 0.001s sys 0.003s
I20250113 02:51:29.268111 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet 80c647a3b5a1411e9130461115c53550 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:29 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:29.268991 1194 raft_consensus.cc:357] T 5b3f012b88c04cb68ed7a1848147b70b P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:29.269230 1194 raft_consensus.cc:383] T 5b3f012b88c04cb68ed7a1848147b70b P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:29.269315 1194 raft_consensus.cc:738] T 5b3f012b88c04cb68ed7a1848147b70b P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:29.269558 1194 consensus_queue.cc:260] T 5b3f012b88c04cb68ed7a1848147b70b P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:29.270092 1194 ts_tablet_manager.cc:1428] T 5b3f012b88c04cb68ed7a1848147b70b P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20250113 02:51:29.270421 1194 tablet_bootstrap.cc:492] T 1a26e2b5e1054aa6afe7f44bf8d43108 P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:29.274837 1194 tablet_bootstrap.cc:654] T 1a26e2b5e1054aa6afe7f44bf8d43108 P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:29.278867 1196 ts_tablet_manager.cc:1918] T 7d53de6379b84103ba905effb0835676 P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:29.279067 1196 log.cc:1198] T 7d53de6379b84103ba905effb0835676 P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/7d53de6379b84103ba905effb0835676
I20250113 02:51:29.279954 885 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b3f012b88c04cb68ed7a1848147b70b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:18 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:29.280187 1196 ts_tablet_manager.cc:1939] T 7d53de6379b84103ba905effb0835676 P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:29.281230 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet 7d53de6379b84103ba905effb0835676 (table default.Table_71699c9b0b1a46cda2db76dee82b90b4 [id=d1b0bb18a7b84473803cd1294b0829ac]) successfully deleted
W20250113 02:51:29.282227 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 80c647a3b5a1411e9130461115c53550 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 80c647a3b5a1411e9130461115c53550 already in progress: creating tablet
I20250113 02:51:29.282579 1196 tablet_replica.cc:331] T 5b3f012b88c04cb68ed7a1848147b70b P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:29.282833 1196 raft_consensus.cc:2238] T 5b3f012b88c04cb68ed7a1848147b70b P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:29.283077 1196 raft_consensus.cc:2267] T 5b3f012b88c04cb68ed7a1848147b70b P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
W20250113 02:51:29.284169 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f24cda32b6e947c7983ae2f1f9a64c23 consensus state has no leader
W20250113 02:51:29.284649 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c508724edea147e5bf6e20bca2f50397 consensus state has no leader
I20250113 02:51:29.285158 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet aa1ebdb92a5f4c92a5195b614689c100
I20250113 02:51:29.285238 1196 ts_tablet_manager.cc:1905] T 5b3f012b88c04cb68ed7a1848147b70b P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:29.286218 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet aa1ebdb92a5f4c92a5195b614689c100 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:29 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:29.286325 738 tablet_service.cc:1514] Processing DeleteTablet for tablet aa1ebdb92a5f4c92a5195b614689c100 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:29 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:29.287566 885 tablet_service.cc:1514] Processing DeleteTablet for tablet aa1ebdb92a5f4c92a5195b614689c100 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:29 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:29.288503 1194 tablet_bootstrap.cc:492] T 1a26e2b5e1054aa6afe7f44bf8d43108 P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:29.288456 447 catalog_manager.cc:5618] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587) does not have the latest schema for tablet 8adfa738c17a4169a5d99f6ceca75100 (table default.Table_9a9b1ced3bef4e648088e0781851128e [id=ae544735623e4cfd894dc9340aca177c]). Expected version 1 got 0
I20250113 02:51:29.288735 1194 ts_tablet_manager.cc:1397] T 1a26e2b5e1054aa6afe7f44bf8d43108 P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.018s user 0.000s sys 0.004s
I20250113 02:51:29.289654 1194 raft_consensus.cc:357] T 1a26e2b5e1054aa6afe7f44bf8d43108 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:29.289924 1194 raft_consensus.cc:383] T 1a26e2b5e1054aa6afe7f44bf8d43108 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:29.290444 1194 raft_consensus.cc:738] T 1a26e2b5e1054aa6afe7f44bf8d43108 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:29.290752 1194 consensus_queue.cc:260] T 1a26e2b5e1054aa6afe7f44bf8d43108 P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:29.291388 1194 ts_tablet_manager.cc:1428] T 1a26e2b5e1054aa6afe7f44bf8d43108 P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250113 02:51:29.291692 1194 tablet_bootstrap.cc:492] T 0fb75c551b4c424186ef2021db742e8e P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:29.291079 1195 ts_tablet_manager.cc:1918] T 8547b63c40ec4ccebd962b5a3b2c5e0f P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250113 02:51:29.291932 1195 log.cc:1198] T 8547b63c40ec4ccebd962b5a3b2c5e0f P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/8547b63c40ec4ccebd962b5a3b2c5e0f
I20250113 02:51:29.292497 1195 ts_tablet_manager.cc:1939] T 8547b63c40ec4ccebd962b5a3b2c5e0f P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:29.293810 1194 tablet_bootstrap.cc:654] T 0fb75c551b4c424186ef2021db742e8e P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:29.294965 885 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a26e2b5e1054aa6afe7f44bf8d43108 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:21 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:29.293444 1814 raft_consensus.cc:491] T 2778d7dc7d9c4843bbc9eb705cec71cd P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:29.295253 1814 raft_consensus.cc:513] T 2778d7dc7d9c4843bbc9eb705cec71cd P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:29.296133 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet 8547b63c40ec4ccebd962b5a3b2c5e0f (table default.Table_3d5fd97116b343b0a6c798e1f22327c2 [id=1396c4482ee646d18330d200fd3f9816]) successfully deleted
I20250113 02:51:29.296921 1062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2778d7dc7d9c4843bbc9eb705cec71cd" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:29.297467 1198 ts_tablet_manager.cc:1918] T 19f14b3db0b64847b5422bfd5ed323d0 P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:29.297583 1198 log.cc:1198] T 19f14b3db0b64847b5422bfd5ed323d0 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/19f14b3db0b64847b5422bfd5ed323d0
W20250113 02:51:29.297555 709 leader_election.cc:343] T 2778d7dc7d9c4843bbc9eb705cec71cd P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:29.297806 1814 leader_election.cc:290] T 2778d7dc7d9c4843bbc9eb705cec71cd P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:29.297883 1198 ts_tablet_manager.cc:1939] T 19f14b3db0b64847b5422bfd5ed323d0 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:29.298103 1195 tablet_replica.cc:331] T c215ffe61b9b426286cc5e808463078d P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:29.298327 1195 raft_consensus.cc:2238] T c215ffe61b9b426286cc5e808463078d P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:29.298491 1195 raft_consensus.cc:2267] T c215ffe61b9b426286cc5e808463078d P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:29.298435 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2778d7dc7d9c4843bbc9eb705cec71cd" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:29.299187 1195 ts_tablet_manager.cc:1905] T c215ffe61b9b426286cc5e808463078d P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250113 02:51:29.299261 711 leader_election.cc:343] T 2778d7dc7d9c4843bbc9eb705cec71cd P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:29.299412 711 leader_election.cc:304] T 2778d7dc7d9c4843bbc9eb705cec71cd P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:29.299847 1814 raft_consensus.cc:2743] T 2778d7dc7d9c4843bbc9eb705cec71cd P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:29.300595 1191 tablet_bootstrap.cc:492] T 11950e996c814590bc9f3f22c0354902 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:29.300627 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet 19f14b3db0b64847b5422bfd5ed323d0 (table default.Table_c7093afcc0ae47c78e7c5cf85219481c [id=141489c57ce44e96a33a1140a8c2b077]) successfully deleted
I20250113 02:51:29.300736 1191 ts_tablet_manager.cc:1397] T 11950e996c814590bc9f3f22c0354902 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.059s user 0.003s sys 0.003s
I20250113 02:51:29.301506 1192 tablet_bootstrap.cc:492] T fdbf03c89bff438a9a25115eddf60be9 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:29.301626 1192 ts_tablet_manager.cc:1397] T fdbf03c89bff438a9a25115eddf60be9 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.087s user 0.002s sys 0.010s
I20250113 02:51:29.301738 1191 raft_consensus.cc:357] T 11950e996c814590bc9f3f22c0354902 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:29.301956 1191 raft_consensus.cc:383] T 11950e996c814590bc9f3f22c0354902 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:29.302018 1191 raft_consensus.cc:738] T 11950e996c814590bc9f3f22c0354902 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:29.302227 1191 consensus_queue.cc:260] T 11950e996c814590bc9f3f22c0354902 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:29.302330 1192 raft_consensus.cc:357] T fdbf03c89bff438a9a25115eddf60be9 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:29.302489 1192 raft_consensus.cc:383] T fdbf03c89bff438a9a25115eddf60be9 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:29.302573 1192 raft_consensus.cc:738] T fdbf03c89bff438a9a25115eddf60be9 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:29.302718 1191 ts_tablet_manager.cc:1428] T 11950e996c814590bc9f3f22c0354902 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250113 02:51:29.302985 1191 tablet_bootstrap.cc:492] T c94eedb5dce44e6d896192c93127903a P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:29.302763 1192 consensus_queue.cc:260] T fdbf03c89bff438a9a25115eddf60be9 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:29.307037 1192 ts_tablet_manager.cc:1428] T fdbf03c89bff438a9a25115eddf60be9 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250113 02:51:29.307055 447 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_47a3c6a8e5d44fc6b95c32c776da62df" } modify_external_catalogs: true
I20250113 02:51:29.307251 447 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_47a3c6a8e5d44fc6b95c32c776da62df" } modify_external_catalogs: true
I20250113 02:51:29.307535 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_580e03ac0ded4964948e4809ddc91945"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:29.307821 448 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_3a73b4672b154b99af1a24e962007ea0" } modify_external_catalogs: true
I20250113 02:51:29.308010 448 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_3a73b4672b154b99af1a24e962007ea0" } modify_external_catalogs: true
I20250113 02:51:29.308393 1192 tablet_bootstrap.cc:492] T 2c399784af46426b8327d175c7a6cce8 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:29.308518 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_81ec0fd5c30248a5bdef2311e0164b36" } new_table_name: "default.Table_d79dc368b229418d9ba63683c93bbcb4" modify_external_catalogs: true
I20250113 02:51:29.309968 1192 tablet_bootstrap.cc:654] T 2c399784af46426b8327d175c7a6cce8 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:29.311156 1438 raft_consensus.cc:491] T 1a26e2b5e1054aa6afe7f44bf8d43108 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:29.311350 1438 raft_consensus.cc:513] T 1a26e2b5e1054aa6afe7f44bf8d43108 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:29.311683 1191 tablet_bootstrap.cc:654] T c94eedb5dce44e6d896192c93127903a P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:29.312083 1438 leader_election.cc:290] T 1a26e2b5e1054aa6afe7f44bf8d43108 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587), 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
I20250113 02:51:29.312294 446 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_c52b68d0d3d74003bf10bc8ff205d25b" } modify_external_catalogs: true
I20250113 02:51:29.312460 446 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_c52b68d0d3d74003bf10bc8ff205d25b" } modify_external_catalogs: true
I20250113 02:51:29.315678 1836 raft_consensus.cc:491] T fdbf03c89bff438a9a25115eddf60be9 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:29.315821 1836 raft_consensus.cc:513] T fdbf03c89bff438a9a25115eddf60be9 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:29.316440 1836 leader_election.cc:290] T fdbf03c89bff438a9a25115eddf60be9 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
W20250113 02:51:29.317567 851 leader_election.cc:343] T 1a26e2b5e1054aa6afe7f44bf8d43108 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Not found: Tablet not found: 1a26e2b5e1054aa6afe7f44bf8d43108
W20250113 02:51:29.317914 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_580e03ac0ded4964948e4809ddc91945 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:29.321132 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdbf03c89bff438a9a25115eddf60be9" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:29.323089 998 leader_election.cc:343] T fdbf03c89bff438a9a25115eddf60be9 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:29.325170 1196 ts_tablet_manager.cc:1918] T 5b3f012b88c04cb68ed7a1848147b70b P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:29.325333 1196 log.cc:1198] T 5b3f012b88c04cb68ed7a1848147b70b P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/5b3f012b88c04cb68ed7a1848147b70b
I20250113 02:51:29.325448 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet fdbf03c89bff438a9a25115eddf60be9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:19 UTC) from {username='slave'} at 127.0.0.1:60230
W20250113 02:51:29.325703 995 leader_election.cc:343] T fdbf03c89bff438a9a25115eddf60be9 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Not found: Tablet not found: fdbf03c89bff438a9a25115eddf60be9
I20250113 02:51:29.325841 995 leader_election.cc:304] T fdbf03c89bff438a9a25115eddf60be9 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4216a7f853d642b7bc3e5cff458f3aca; no voters: 1186fa869eb14812832fa6562531f196, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:29.326349 1814 raft_consensus.cc:491] T 63efcb6b2e194c469fb30a197bd6565b P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:29.326494 1814 raft_consensus.cc:513] T 63efcb6b2e194c469fb30a197bd6565b P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:29.326841 1836 raft_consensus.cc:2743] T fdbf03c89bff438a9a25115eddf60be9 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:29.327134 1814 leader_election.cc:290] T 63efcb6b2e194c469fb30a197bd6565b P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
W20250113 02:51:29.328095 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 93c3a547f2674149a89fd0a68abd1d61 consensus state has no leader
I20250113 02:51:29.328459 1062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63efcb6b2e194c469fb30a197bd6565b" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
W20250113 02:51:29.329051 709 leader_election.cc:343] T 63efcb6b2e194c469fb30a197bd6565b P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
W20250113 02:51:29.326550 852 leader_election.cc:343] T 1a26e2b5e1054aa6afe7f44bf8d43108 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Not found: Tablet not found: 1a26e2b5e1054aa6afe7f44bf8d43108
I20250113 02:51:29.330415 1194 tablet_bootstrap.cc:492] T 0fb75c551b4c424186ef2021db742e8e P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:29.333967 1194 ts_tablet_manager.cc:1397] T 0fb75c551b4c424186ef2021db742e8e P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.042s user 0.002s sys 0.003s
I20250113 02:51:29.334867 1194 raft_consensus.cc:357] T 0fb75c551b4c424186ef2021db742e8e P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:29.335080 1194 raft_consensus.cc:383] T 0fb75c551b4c424186ef2021db742e8e P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:29.335155 1194 raft_consensus.cc:738] T 0fb75c551b4c424186ef2021db742e8e P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:29.335433 1194 consensus_queue.cc:260] T 0fb75c551b4c424186ef2021db742e8e P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:29.338977 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_95e76bedf8924501b57f7a26d66a9d33"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:29.339664 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_95e76bedf8924501b57f7a26d66a9d33 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:29.345160 450 catalog_manager.cc:5618] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095) does not have the latest schema for tablet 572e4eed1e924381bada27f8389821a7 (table default.Table_1d910c3d2b244474a576fc4088301143 [id=75cb0745bbb84e1c883cb13b546ecb11]). Expected version 4 got 0
I20250113 02:51:29.345427 450 catalog_manager.cc:5618] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095) does not have the latest schema for tablet 378d59d314a84ddb84cf2ab041c51cf1 (table default.Table_2794f9d2a3cc49f78de33a775c1619f1 [id=0c01441cec7043b687d6744e72a0e45d]). Expected version 1 got 0
I20250113 02:51:29.345590 450 catalog_manager.cc:5618] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095) does not have the latest schema for tablet 9fff52e99c64462a8a73fccc3c22ca63 (table default.Table_9ef18ee71ad843c5921750a2f8478855 [id=bd8bd572ed524f2cae01f820f07e3c36]). Expected version 2 got 0
I20250113 02:51:29.347153 1195 ts_tablet_manager.cc:1918] T c215ffe61b9b426286cc5e808463078d P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:29.347884 738 tablet_service.cc:1514] Processing DeleteTablet for tablet a0ca2e3a3ce94f3aa20df64c89c7cf95 with delete_type TABLET_DATA_DELETED (replaced by tablet 2470b010b571444eba3f132323038f33) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:29.348080 743 tablet_service.cc:1514] Processing DeleteTablet for tablet 2778d7dc7d9c4843bbc9eb705cec71cd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:20 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:29.351056 750 tablet_service.cc:1514] Processing DeleteTablet for tablet 092892a36e0e47cb8d2184c2bbde95d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:22 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:29.351259 752 tablet_service.cc:1514] Processing DeleteTablet for tablet b4806b18a4194f9c95350d89ba62e13a with delete_type TABLET_DATA_DELETED (replaced by tablet 146dad5c977f464496f7d21611030dad) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:29.351522 743 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d63f0e5835540a78c82020b20a07dac with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:23 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:29.351713 738 tablet_service.cc:1514] Processing DeleteTablet for tablet f68f9f4076d9475d938e795e72bdd468 with delete_type TABLET_DATA_DELETED (replaced by tablet f2c6c8c17d6e499d84ad7c0c23889cc4) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:29.356046 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 3d85539cf0074a03b167d3279e940079
I20250113 02:51:29.356338 1194 ts_tablet_manager.cc:1428] T 0fb75c551b4c424186ef2021db742e8e P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.022s user 0.000s sys 0.002s
I20250113 02:51:29.357057 738 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d85539cf0074a03b167d3279e940079 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:29 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:29.357367 1198 tablet_replica.cc:331] T f31dd916336a4d8fb3ab015af7de8ccb P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:29.357581 1198 raft_consensus.cc:2238] T f31dd916336a4d8fb3ab015af7de8ccb P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:29.357750 1198 raft_consensus.cc:2267] T f31dd916336a4d8fb3ab015af7de8ccb P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:29.358654 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d85539cf0074a03b167d3279e940079 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:29 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:29.358672 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_a70bef6ff5714d398f83133e99b5d9d5" } new_table_name: "default.Table_867ef33b763b4f52834e49855a545a8b" modify_external_catalogs: true
I20250113 02:51:29.360491 448 catalog_manager.cc:5618] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095) does not have the latest schema for tablet e8ab4516789e4585a101c8766d338af3 (table default.Table_3b76cf9fac334ebf9b1ecb9f8d07ea74 [id=b112ac4e2e58460db083da1eb788ca07]). Expected version 2 got 0
I20250113 02:51:29.360759 448 catalog_manager.cc:5618] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095) does not have the latest schema for tablet 572e4eed1e924381bada27f8389821a7 (table default.Table_1d910c3d2b244474a576fc4088301143 [id=75cb0745bbb84e1c883cb13b546ecb11]). Expected version 4 got 0
I20250113 02:51:29.360960 1027 tablet_service.cc:1467] Processing CreateTablet for tablet be2ca8d2ac6547189a0edeff12413330 (DEFAULT_TABLE table=default.Table_67b9f9f43ada4495a29ed9b8e8a6bf63 [id=1a7a18b5f2c9468b945be581670951aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:29.361083 448 catalog_manager.cc:5618] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095) does not have the latest schema for tablet d304b65eba8c40c6922333506a60540c (table default.Table_e03d954cc20048e4b0d598282689643d [id=e62f8e4c51b04729903b1943275c5f7f]). Expected version 1 got 0
I20250113 02:51:29.361335 448 catalog_manager.cc:5618] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095) does not have the latest schema for tablet 8adfa738c17a4169a5d99f6ceca75100 (table default.Table_9a9b1ced3bef4e648088e0781851128e [id=ae544735623e4cfd894dc9340aca177c]). Expected version 1 got 0
I20250113 02:51:29.361416 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be2ca8d2ac6547189a0edeff12413330. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:29.361534 448 catalog_manager.cc:5618] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095) does not have the latest schema for tablet 401c4f5dbd304950877b59a865f417b3 (table default.Table_c991f9dbd70d4c4682664b8711f05db4 [id=2c64da64ecdb4bf48e11b523533902e8]). Expected version 1 got 0
I20250113 02:51:29.363997 738 tablet_service.cc:1514] Processing DeleteTablet for tablet ddf79eec41214693859f526265a55492 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:23 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:29.364794 738 tablet_service.cc:1514] Processing DeleteTablet for tablet 11950e996c814590bc9f3f22c0354902 with delete_type TABLET_DATA_DELETED (replaced by tablet 00d4381876234f16b19336523640bbc8) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:29.364759 885 tablet_service.cc:1467] Processing CreateTablet for tablet be2ca8d2ac6547189a0edeff12413330 (DEFAULT_TABLE table=default.Table_67b9f9f43ada4495a29ed9b8e8a6bf63 [id=1a7a18b5f2c9468b945be581670951aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:29.365178 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be2ca8d2ac6547189a0edeff12413330. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:29.365163 1706 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 02:51:29.365896 884 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d85539cf0074a03b167d3279e940079 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:29 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:29.366226 1706 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250113 02:51:29.366624 1706 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 02:51:29.367707 1198 ts_tablet_manager.cc:1905] T f31dd916336a4d8fb3ab015af7de8ccb P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:29.372983 738 tablet_service.cc:1467] Processing CreateTablet for tablet be2ca8d2ac6547189a0edeff12413330 (DEFAULT_TABLE table=default.Table_67b9f9f43ada4495a29ed9b8e8a6bf63 [id=1a7a18b5f2c9468b945be581670951aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:29.373382 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be2ca8d2ac6547189a0edeff12413330. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:29.373497 1192 tablet_bootstrap.cc:492] T 2c399784af46426b8327d175c7a6cce8 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:29.373665 1192 ts_tablet_manager.cc:1397] T 2c399784af46426b8327d175c7a6cce8 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.065s user 0.003s sys 0.000s
I20250113 02:51:29.374497 1192 raft_consensus.cc:357] T 2c399784af46426b8327d175c7a6cce8 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:29.374720 1192 raft_consensus.cc:383] T 2c399784af46426b8327d175c7a6cce8 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:29.374819 1192 raft_consensus.cc:738] T 2c399784af46426b8327d175c7a6cce8 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:29.375069 1192 consensus_queue.cc:260] T 2c399784af46426b8327d175c7a6cce8 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:29.375695 1194 tablet_bootstrap.cc:492] T 601c71d5f7624fbb87b365d2e775d33e P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:29.378180 1192 ts_tablet_manager.cc:1428] T 2c399784af46426b8327d175c7a6cce8 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250113 02:51:29.379153 1192 tablet_bootstrap.cc:492] T 63efcb6b2e194c469fb30a197bd6565b P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:29.380121 1040 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c399784af46426b8327d175c7a6cce8 with delete_type TABLET_DATA_DELETED (replaced by tablet 9e736211eb3c4536b9fd1d90d2d83838) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:29.381093 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet cd6dcfecf4324542b241f909926756ea
I20250113 02:51:29.382037 1040 tablet_service.cc:1514] Processing DeleteTablet for tablet cd6dcfecf4324542b241f909926756ea with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:29 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:29.383687 1192 tablet_bootstrap.cc:654] T 63efcb6b2e194c469fb30a197bd6565b P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:29.384420 752 tablet_service.cc:1467] Processing CreateTablet for tablet ee289c3236834dc48d922062041be3b5 (DEFAULT_TABLE table=default.Table_ad3bb183b90444b8a9ca111bbc2d4d2a [id=9be51932b831417fbbfa71e0af27dc9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:29.384778 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee289c3236834dc48d922062041be3b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:29.388162 743 tablet_service.cc:1514] Processing DeleteTablet for tablet cd6dcfecf4324542b241f909926756ea with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:29 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:29.390544 746 tablet_service.cc:1467] Processing CreateTablet for tablet 51a7cb7e7e2c49949c3be328005ce468 (DEFAULT_TABLE table=default.Table_9a4ca43a32a04beabb2108aae12c3b96 [id=7969bb22bccd463a919b636a9274da19]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:29.390738 1025 tablet_service.cc:1467] Processing CreateTablet for tablet 6e208f83f3384a78a959909b4ac690c0 (DEFAULT_TABLE table=default.Table_7f3a0392b4234e48835b8c3c5ec34b15 [id=0e5e42fb02d2488ca4e9ae1e191c67c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:29.390875 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 51a7cb7e7e2c49949c3be328005ce468. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:29.391076 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e208f83f3384a78a959909b4ac690c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:29.391350 1033 tablet_service.cc:1467] Processing CreateTablet for tablet 905542fb0dce4bbb8852b74d94c7b2a6 (DEFAULT_TABLE table=default.Table_e9dd2f1b0ade4fc6a1db103979189b11 [id=ced6cbaf17fd403e807452f63b4107eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:29.391841 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 905542fb0dce4bbb8852b74d94c7b2a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:29.394176 748 tablet_service.cc:1467] Processing CreateTablet for tablet 49647eca15dc4770afc8ba0c7f2d412e (DEFAULT_TABLE table=default.Table_e6cbe3930484474596b0ef6111c7b419 [id=cb0d822b2a90483a95e079e27ea27a3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:29.394475 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49647eca15dc4770afc8ba0c7f2d412e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:29.396023 1041 tablet_service.cc:1467] Processing CreateTablet for tablet 49647eca15dc4770afc8ba0c7f2d412e (DEFAULT_TABLE table=default.Table_e6cbe3930484474596b0ef6111c7b419 [id=cb0d822b2a90483a95e079e27ea27a3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:29.396380 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49647eca15dc4770afc8ba0c7f2d412e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:29.396823 750 tablet_service.cc:1467] Processing CreateTablet for tablet 905542fb0dce4bbb8852b74d94c7b2a6 (DEFAULT_TABLE table=default.Table_e9dd2f1b0ade4fc6a1db103979189b11 [id=ced6cbaf17fd403e807452f63b4107eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:29.396955 1029 tablet_service.cc:1467] Processing CreateTablet for tablet 51a7cb7e7e2c49949c3be328005ce468 (DEFAULT_TABLE table=default.Table_9a4ca43a32a04beabb2108aae12c3b96 [id=7969bb22bccd463a919b636a9274da19]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:29.397140 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 905542fb0dce4bbb8852b74d94c7b2a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:29.397395 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 51a7cb7e7e2c49949c3be328005ce468. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:29.400007 751 tablet_service.cc:1467] Processing CreateTablet for tablet 9798a2b066ea4caaa6761d62644b34e9 (DEFAULT_TABLE table=default.Table_770cceffb7e940c7a4dfad0fcab73863 [id=2f31a2d950af4988a2ace1af4a92b49e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:29.400305 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9798a2b066ea4caaa6761d62644b34e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:29.400370 1040 tablet_service.cc:1467] Processing CreateTablet for tablet ee289c3236834dc48d922062041be3b5 (DEFAULT_TABLE table=default.Table_ad3bb183b90444b8a9ca111bbc2d4d2a [id=9be51932b831417fbbfa71e0af27dc9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:29.400666 1040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee289c3236834dc48d922062041be3b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:29.396574 1031 tablet_service.cc:1467] Processing CreateTablet for tablet 9798a2b066ea4caaa6761d62644b34e9 (DEFAULT_TABLE table=default.Table_770cceffb7e940c7a4dfad0fcab73863 [id=2f31a2d950af4988a2ace1af4a92b49e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:29.403752 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9798a2b066ea4caaa6761d62644b34e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:29.405064 1041 tablet_service.cc:1467] Processing CreateTablet for tablet 5d2d14548b734b1fbe57d68c0ea3e2f8 (DEFAULT_TABLE table=default.Table_ccebddd68b0448a49d408b9922a78a4d [id=c0b628d230fd4eae83a20530cd998993]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:29.405329 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d2d14548b734b1fbe57d68c0ea3e2f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:29.409029 743 tablet_service.cc:1467] Processing CreateTablet for tablet 6e208f83f3384a78a959909b4ac690c0 (DEFAULT_TABLE table=default.Table_7f3a0392b4234e48835b8c3c5ec34b15 [id=0e5e42fb02d2488ca4e9ae1e191c67c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:29.409389 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e208f83f3384a78a959909b4ac690c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:29.414544 1192 tablet_bootstrap.cc:492] T 63efcb6b2e194c469fb30a197bd6565b P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:29.414750 1192 ts_tablet_manager.cc:1397] T 63efcb6b2e194c469fb30a197bd6565b P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.036s user 0.000s sys 0.003s
I20250113 02:51:29.414482 1033 tablet_service.cc:1467] Processing CreateTablet for tablet f3a8c7a25aca4496919386396626de00 (DEFAULT_TABLE table=default.Table_ed2ecde421f24146b87838f8e9009663 [id=cd224e27938a44d1a61c74a21762cff0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:29.415571 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet e468b652f71f44e3945f8d919ac0e1a8
I20250113 02:51:29.415500 1192 raft_consensus.cc:357] T 63efcb6b2e194c469fb30a197bd6565b P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:29.415694 1192 raft_consensus.cc:383] T 63efcb6b2e194c469fb30a197bd6565b P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:29.415807 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3a8c7a25aca4496919386396626de00. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:29.415745 1025 tablet_service.cc:1467] Processing CreateTablet for tablet 4175a624a220440897e0ff8d6ea38f1b (DEFAULT_TABLE table=default.Table_f568fbf7cc51459a98545f97e4745407 [id=0dbad90f05e540ff835a13b600c3b317]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:29.415825 1192 raft_consensus.cc:738] T 63efcb6b2e194c469fb30a197bd6565b P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:29.417225 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet e468b652f71f44e3945f8d919ac0e1a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:29 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:29.417268 1192 consensus_queue.cc:260] T 63efcb6b2e194c469fb30a197bd6565b P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:29.418089 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_d41198149c094b9cb0f1fc8c7cc6a799"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:29.418630 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d41198149c094b9cb0f1fc8c7cc6a799 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:29.418740 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4175a624a220440897e0ff8d6ea38f1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:29.419201 1192 ts_tablet_manager.cc:1428] T 63efcb6b2e194c469fb30a197bd6565b P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20250113 02:51:29.419474 1192 tablet_bootstrap.cc:492] T c215ffe61b9b426286cc5e808463078d P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:29.421113 1192 tablet_bootstrap.cc:654] T c215ffe61b9b426286cc5e808463078d P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:29.422629 884 tablet_service.cc:1467] Processing CreateTablet for tablet ee289c3236834dc48d922062041be3b5 (DEFAULT_TABLE table=default.Table_ad3bb183b90444b8a9ca111bbc2d4d2a [id=9be51932b831417fbbfa71e0af27dc9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:29.423859 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet 63efcb6b2e194c469fb30a197bd6565b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:20 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:29.424615 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_216f6b44551d41d49fa21b862e08aee1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:29.425159 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_216f6b44551d41d49fa21b862e08aee1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:29.425549 1196 ts_tablet_manager.cc:1939] T 5b3f012b88c04cb68ed7a1848147b70b P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:29.426885 755 tablet_service.cc:1467] Processing CreateTablet for tablet 4175a624a220440897e0ff8d6ea38f1b (DEFAULT_TABLE table=default.Table_f568fbf7cc51459a98545f97e4745407 [id=0dbad90f05e540ff835a13b600c3b317]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:29.427083 888 tablet_service.cc:1514] Processing DeleteTablet for tablet cd6dcfecf4324542b241f909926756ea with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:29 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:29.427253 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4175a624a220440897e0ff8d6ea38f1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:29.427443 889 tablet_service.cc:1467] Processing CreateTablet for tablet 6e208f83f3384a78a959909b4ac690c0 (DEFAULT_TABLE table=default.Table_7f3a0392b4234e48835b8c3c5ec34b15 [id=0e5e42fb02d2488ca4e9ae1e191c67c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:29.427806 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e208f83f3384a78a959909b4ac690c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:29.428210 745 tablet_service.cc:1467] Processing CreateTablet for tablet 5d2d14548b734b1fbe57d68c0ea3e2f8 (DEFAULT_TABLE table=default.Table_ccebddd68b0448a49d408b9922a78a4d [id=c0b628d230fd4eae83a20530cd998993]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:29.428490 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d2d14548b734b1fbe57d68c0ea3e2f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:29.428844 896 tablet_service.cc:1467] Processing CreateTablet for tablet 905542fb0dce4bbb8852b74d94c7b2a6 (DEFAULT_TABLE table=default.Table_e9dd2f1b0ade4fc6a1db103979189b11 [id=ced6cbaf17fd403e807452f63b4107eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:29.429214 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 905542fb0dce4bbb8852b74d94c7b2a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:29.430052 887 tablet_service.cc:1467] Processing CreateTablet for tablet 51a7cb7e7e2c49949c3be328005ce468 (DEFAULT_TABLE table=default.Table_9a4ca43a32a04beabb2108aae12c3b96 [id=7969bb22bccd463a919b636a9274da19]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:29.430333 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 51a7cb7e7e2c49949c3be328005ce468. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:29.430960 886 tablet_service.cc:1467] Processing CreateTablet for tablet 49647eca15dc4770afc8ba0c7f2d412e (DEFAULT_TABLE table=default.Table_e6cbe3930484474596b0ef6111c7b419 [id=cb0d822b2a90483a95e079e27ea27a3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:29.431245 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49647eca15dc4770afc8ba0c7f2d412e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:29.432030 881 tablet_service.cc:1467] Processing CreateTablet for tablet 9798a2b066ea4caaa6761d62644b34e9 (DEFAULT_TABLE table=default.Table_770cceffb7e940c7a4dfad0fcab73863 [id=2f31a2d950af4988a2ace1af4a92b49e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:29.432276 741 tablet_service.cc:1467] Processing CreateTablet for tablet f3a8c7a25aca4496919386396626de00 (DEFAULT_TABLE table=default.Table_ed2ecde421f24146b87838f8e9009663 [id=cd224e27938a44d1a61c74a21762cff0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:29.432340 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9798a2b066ea4caaa6761d62644b34e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:29.432557 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3a8c7a25aca4496919386396626de00. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:29.433147 880 tablet_service.cc:1467] Processing CreateTablet for tablet 4175a624a220440897e0ff8d6ea38f1b (DEFAULT_TABLE table=default.Table_f568fbf7cc51459a98545f97e4745407 [id=0dbad90f05e540ff835a13b600c3b317]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:29.433472 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4175a624a220440897e0ff8d6ea38f1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:29.434033 739 tablet_service.cc:1514] Processing DeleteTablet for tablet e468b652f71f44e3945f8d919ac0e1a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:29 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:29.434139 899 tablet_service.cc:1467] Processing CreateTablet for tablet 5d2d14548b734b1fbe57d68c0ea3e2f8 (DEFAULT_TABLE table=default.Table_ccebddd68b0448a49d408b9922a78a4d [id=c0b628d230fd4eae83a20530cd998993]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:29.434510 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d2d14548b734b1fbe57d68c0ea3e2f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:29.435328 895 tablet_service.cc:1467] Processing CreateTablet for tablet f3a8c7a25aca4496919386396626de00 (DEFAULT_TABLE table=default.Table_ed2ecde421f24146b87838f8e9009663 [id=cd224e27938a44d1a61c74a21762cff0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:29.435684 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3a8c7a25aca4496919386396626de00. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:29.436092 898 tablet_service.cc:1514] Processing DeleteTablet for tablet e468b652f71f44e3945f8d919ac0e1a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:29 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:29.436471 891 tablet_service.cc:1467] Processing CreateTablet for tablet 58b0e9391d614b578974f0e29b020c16 (DEFAULT_TABLE table=default.Table_ea693041c5824d818e3bd15e28c7a3f5 [id=48a21ba87d76443eaab0dcdd6cf55147]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:29.438962 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_c22142b5375a4837bbc42af6f77d880f" } new_table_name: "default.Table_27e1ad9a94f8455f806f48768ef4ddac" modify_external_catalogs: true
I20250113 02:51:29.440989 898 tablet_service.cc:1467] Processing CreateTablet for tablet aa85362980bb4c4db18d5d55a545046a (DEFAULT_TABLE table=default.Table_5473033c0c4a464785b6f224feec7506 [id=8ccf171b4d6a4909837a6fe30377d365]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:29.442564 739 tablet_service.cc:1467] Processing CreateTablet for tablet 58b0e9391d614b578974f0e29b020c16 (DEFAULT_TABLE table=default.Table_ea693041c5824d818e3bd15e28c7a3f5 [id=48a21ba87d76443eaab0dcdd6cf55147]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:29.442921 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 58b0e9391d614b578974f0e29b020c16. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:29.443380 888 tablet_service.cc:1467] Processing CreateTablet for tablet f362b8ab6c464159a7dc2164ac92d7c1 (DEFAULT_TABLE table=default.Table_4e3ba1ddebc74043b121c831b4abacd8 [id=088596b1d7014c399feac5aee67048e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:29.446830 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_adf2c40363104ff9a3343f5a7659917f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:29.447315 892 tablet_service.cc:1467] Processing CreateTablet for tablet ea51f6e169ae49bd815a9496a1830d0a (DEFAULT_TABLE table=default.Table_677784013a07442b9963695510a0a055 [id=06a7fc0682504214990e9a6c830d15ea]), partition=RANGE (key) PARTITION UNBOUNDED
W20250113 02:51:29.447542 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_adf2c40363104ff9a3343f5a7659917f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:29.447547 883 tablet_service.cc:1467] Processing CreateTablet for tablet 7ff9b30d502a4aa1a553035bbc992ad2 (DEFAULT_TABLE table=default.Table_1a482bb1752b4f4a8fc993b60179f50b [id=8aeb11a9e3fa471893d42f2c60c652f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:29.450554 897 tablet_service.cc:1467] Processing CreateTablet for tablet 524b4bd640a04123b4abbd716e7d40f2 (DEFAULT_TABLE table=default.Table_2c0aebce064848d7ace63d6a47b5daf7 [id=08372bbba1ae47dc9fe2c0a0f20f0849]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:29.451243 742 tablet_service.cc:1467] Processing CreateTablet for tablet aa85362980bb4c4db18d5d55a545046a (DEFAULT_TABLE table=default.Table_5473033c0c4a464785b6f224feec7506 [id=8ccf171b4d6a4909837a6fe30377d365]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:29.451581 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa85362980bb4c4db18d5d55a545046a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:29.455152 736 tablet_service.cc:1467] Processing CreateTablet for tablet f362b8ab6c464159a7dc2164ac92d7c1 (DEFAULT_TABLE table=default.Table_4e3ba1ddebc74043b121c831b4abacd8 [id=088596b1d7014c399feac5aee67048e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:29.455471 736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f362b8ab6c464159a7dc2164ac92d7c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:29.460062 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_f9117b338db445fba012e659c3b51f66"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:29.460649 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f9117b338db445fba012e659c3b51f66 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:29.461143 740 tablet_service.cc:1467] Processing CreateTablet for tablet ea51f6e169ae49bd815a9496a1830d0a (DEFAULT_TABLE table=default.Table_677784013a07442b9963695510a0a055 [id=06a7fc0682504214990e9a6c830d15ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:29.461460 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea51f6e169ae49bd815a9496a1830d0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:29.464250 454 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_4e3230947e8b4b71992b241c34d4e6ad" } modify_external_catalogs: true
I20250113 02:51:29.464933 747 tablet_service.cc:1467] Processing CreateTablet for tablet 7ff9b30d502a4aa1a553035bbc992ad2 (DEFAULT_TABLE table=default.Table_1a482bb1752b4f4a8fc993b60179f50b [id=8aeb11a9e3fa471893d42f2c60c652f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:29.465245 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ff9b30d502a4aa1a553035bbc992ad2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:29.468147 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 58b0e9391d614b578974f0e29b020c16. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:29.468369 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa85362980bb4c4db18d5d55a545046a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:29.468539 754 tablet_service.cc:1467] Processing CreateTablet for tablet 524b4bd640a04123b4abbd716e7d40f2 (DEFAULT_TABLE table=default.Table_2c0aebce064848d7ace63d6a47b5daf7 [id=08372bbba1ae47dc9fe2c0a0f20f0849]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:29.468858 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 524b4bd640a04123b4abbd716e7d40f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:29.472081 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f362b8ab6c464159a7dc2164ac92d7c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:29.472326 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea51f6e169ae49bd815a9496a1830d0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:29.472688 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ff9b30d502a4aa1a553035bbc992ad2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:29.474241 454 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_4e3230947e8b4b71992b241c34d4e6ad" } modify_external_catalogs: true
I20250113 02:51:29.475481 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_abe5b12ac76741b194dceecebd1e173c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:29.476023 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_abe5b12ac76741b194dceecebd1e173c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:29.476083 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 524b4bd640a04123b4abbd716e7d40f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:29.477176 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_9b57c1f76abd4ef1b85f88572215474a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:29.477617 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9b57c1f76abd4ef1b85f88572215474a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:29.478533 1814 raft_consensus.cc:491] T f68f9f4076d9475d938e795e72bdd468 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:29.478655 1814 raft_consensus.cc:513] T f68f9f4076d9475d938e795e72bdd468 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:29.479306 1814 leader_election.cc:290] T f68f9f4076d9475d938e795e72bdd468 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:29.480089 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee289c3236834dc48d922062041be3b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:29.480003 1062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f68f9f4076d9475d938e795e72bdd468" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:29.480103 1836 raft_consensus.cc:491] T 2c399784af46426b8327d175c7a6cce8 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:29.480208 1836 raft_consensus.cc:513] T 2c399784af46426b8327d175c7a6cce8 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
W20250113 02:51:29.480559 709 leader_election.cc:343] T f68f9f4076d9475d938e795e72bdd468 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:29.480764 1836 leader_election.cc:290] T 2c399784af46426b8327d175c7a6cce8 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:29.481853 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c399784af46426b8327d175c7a6cce8" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:29.482371 998 leader_election.cc:343] T 2c399784af46426b8327d175c7a6cce8 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
W20250113 02:51:29.493804 995 leader_election.cc:343] T 2c399784af46426b8327d175c7a6cce8 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Not found: Tablet not found: 2c399784af46426b8327d175c7a6cce8
I20250113 02:51:29.493945 995 leader_election.cc:304] T 2c399784af46426b8327d175c7a6cce8 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4216a7f853d642b7bc3e5cff458f3aca; no voters: 1186fa869eb14812832fa6562531f196, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:29.494156 1033 tablet_service.cc:1467] Processing CreateTablet for tablet 58b0e9391d614b578974f0e29b020c16 (DEFAULT_TABLE table=default.Table_ea693041c5824d818e3bd15e28c7a3f5 [id=48a21ba87d76443eaab0dcdd6cf55147]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:29.494330 1836 raft_consensus.cc:2743] T 2c399784af46426b8327d175c7a6cce8 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:29.494531 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 58b0e9391d614b578974f0e29b020c16. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:29.496264 1027 tablet_service.cc:1467] Processing CreateTablet for tablet f362b8ab6c464159a7dc2164ac92d7c1 (DEFAULT_TABLE table=default.Table_4e3ba1ddebc74043b121c831b4abacd8 [id=088596b1d7014c399feac5aee67048e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:29.496660 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f362b8ab6c464159a7dc2164ac92d7c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:29.500231 1041 tablet_service.cc:1467] Processing CreateTablet for tablet 524b4bd640a04123b4abbd716e7d40f2 (DEFAULT_TABLE table=default.Table_2c0aebce064848d7ace63d6a47b5daf7 [id=08372bbba1ae47dc9fe2c0a0f20f0849]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:29.500555 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 524b4bd640a04123b4abbd716e7d40f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:29.501672 1836 raft_consensus.cc:491] T 63efcb6b2e194c469fb30a197bd6565b P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:29.501798 1836 raft_consensus.cc:513] T 63efcb6b2e194c469fb30a197bd6565b P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:29.502451 1836 leader_election.cc:290] T 63efcb6b2e194c469fb30a197bd6565b P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
I20250113 02:51:29.504238 1025 tablet_service.cc:1467] Processing CreateTablet for tablet aa85362980bb4c4db18d5d55a545046a (DEFAULT_TABLE table=default.Table_5473033c0c4a464785b6f224feec7506 [id=8ccf171b4d6a4909837a6fe30377d365]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:29.504544 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa85362980bb4c4db18d5d55a545046a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:29.505407 1196 tablet_replica.cc:331] T 1a26e2b5e1054aa6afe7f44bf8d43108 P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:29.505596 1196 raft_consensus.cc:2238] T 1a26e2b5e1054aa6afe7f44bf8d43108 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:29.505769 1196 raft_consensus.cc:2267] T 1a26e2b5e1054aa6afe7f44bf8d43108 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:29.506453 1196 ts_tablet_manager.cc:1905] T 1a26e2b5e1054aa6afe7f44bf8d43108 P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:29.509428 1040 tablet_service.cc:1467] Processing CreateTablet for tablet 7ff9b30d502a4aa1a553035bbc992ad2 (DEFAULT_TABLE table=default.Table_1a482bb1752b4f4a8fc993b60179f50b [id=8aeb11a9e3fa471893d42f2c60c652f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:29.509866 1040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ff9b30d502a4aa1a553035bbc992ad2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:29.515230 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet 5b3f012b88c04cb68ed7a1848147b70b (table default.Table_ce0f77e091564c29bf54791dda458d78 [id=86a46eed92bc4bddad870e392232cb51]) successfully deleted
W20250113 02:51:29.515524 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet aa1ebdb92a5f4c92a5195b614689c100 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet aa1ebdb92a5f4c92a5195b614689c100 already in progress: creating tablet
I20250113 02:51:29.516399 1029 tablet_service.cc:1467] Processing CreateTablet for tablet ea51f6e169ae49bd815a9496a1830d0a (DEFAULT_TABLE table=default.Table_677784013a07442b9963695510a0a055 [id=06a7fc0682504214990e9a6c830d15ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:29.516767 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea51f6e169ae49bd815a9496a1830d0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:29.524160 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63efcb6b2e194c469fb30a197bd6565b" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:29.525617 998 leader_election.cc:343] T 63efcb6b2e194c469fb30a197bd6565b P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:29.530329 1194 tablet_bootstrap.cc:654] T 601c71d5f7624fbb87b365d2e775d33e P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:29.537664 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63efcb6b2e194c469fb30a197bd6565b" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
I20250113 02:51:29.537897 773 raft_consensus.cc:2463] T 63efcb6b2e194c469fb30a197bd6565b P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 0.
I20250113 02:51:29.539521 995 leader_election.cc:304] T 63efcb6b2e194c469fb30a197bd6565b P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters: df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:29.539840 1194 tablet_bootstrap.cc:492] T 601c71d5f7624fbb87b365d2e775d33e P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:29.540009 1194 ts_tablet_manager.cc:1397] T 601c71d5f7624fbb87b365d2e775d33e P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.183s user 0.004s sys 0.001s
I20250113 02:51:29.539958 1836 raft_consensus.cc:2798] T 63efcb6b2e194c469fb30a197bd6565b P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 02:51:29.540079 1836 raft_consensus.cc:491] T 63efcb6b2e194c469fb30a197bd6565b P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 02:51:29.540146 1836 raft_consensus.cc:3054] T 63efcb6b2e194c469fb30a197bd6565b P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:29.540768 1194 raft_consensus.cc:357] T 601c71d5f7624fbb87b365d2e775d33e P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:29.540961 1194 raft_consensus.cc:383] T 601c71d5f7624fbb87b365d2e775d33e P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:29.541018 1194 raft_consensus.cc:738] T 601c71d5f7624fbb87b365d2e775d33e P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:29.541225 1194 consensus_queue.cc:260] T 601c71d5f7624fbb87b365d2e775d33e P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:29.542430 1194 ts_tablet_manager.cc:1428] T 601c71d5f7624fbb87b365d2e775d33e P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250113 02:51:29.542569 1836 raft_consensus.cc:513] T 63efcb6b2e194c469fb30a197bd6565b P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:29.543081 1194 tablet_bootstrap.cc:492] T 8adfa738c17a4169a5d99f6ceca75100 P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:29.543198 1836 leader_election.cc:290] T 63efcb6b2e194c469fb30a197bd6565b P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Requested vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
I20250113 02:51:29.543968 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63efcb6b2e194c469fb30a197bd6565b" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f"
W20250113 02:51:29.544469 998 leader_election.cc:343] T 63efcb6b2e194c469fb30a197bd6565b P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:29.546139 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63efcb6b2e194c469fb30a197bd6565b" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196"
I20250113 02:51:29.546322 773 raft_consensus.cc:3054] T 63efcb6b2e194c469fb30a197bd6565b P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:29.549068 773 raft_consensus.cc:2463] T 63efcb6b2e194c469fb30a197bd6565b P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 1.
I20250113 02:51:29.549507 995 leader_election.cc:304] T 63efcb6b2e194c469fb30a197bd6565b P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters: df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:29.549928 1634 raft_consensus.cc:2798] T 63efcb6b2e194c469fb30a197bd6565b P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Leader election won for term 1
I20250113 02:51:29.550029 1634 raft_consensus.cc:695] T 63efcb6b2e194c469fb30a197bd6565b P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: Becoming Leader. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Running, Role: LEADER
I20250113 02:51:29.550256 1634 consensus_queue.cc:237] T 63efcb6b2e194c469fb30a197bd6565b P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:29.551891 852 leader_election.cc:304] T 1a26e2b5e1054aa6afe7f44bf8d43108 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df69f48656b44bb890f76587ee69cc0f; no voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca
W20250113 02:51:29.552130 852 raft_consensus.cc:2681] T 1a26e2b5e1054aa6afe7f44bf8d43108 P df69f48656b44bb890f76587ee69cc0f: unable to run election callback: Service unavailable: Thread pool token was shut down
W20250113 02:51:29.552314 852 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.3:42587, user_credentials={real_user=slave}} blocked reactor thread for 225778us
I20250113 02:51:29.552974 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63efcb6b2e194c469fb30a197bd6565b" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:29.554963 915 raft_consensus.cc:3054] T 572e4eed1e924381bada27f8389821a7 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Advancing to term 1
W20250113 02:51:29.555351 711 leader_election.cc:343] T 63efcb6b2e194c469fb30a197bd6565b P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:29.555526 711 leader_election.cc:304] T 63efcb6b2e194c469fb30a197bd6565b P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:29.556090 1814 raft_consensus.cc:2743] T 63efcb6b2e194c469fb30a197bd6565b P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:29.560813 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f68f9f4076d9475d938e795e72bdd468" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:29.564647 1192 tablet_bootstrap.cc:492] T c215ffe61b9b426286cc5e808463078d P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:29.564814 1192 ts_tablet_manager.cc:1397] T c215ffe61b9b426286cc5e808463078d P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.145s user 0.015s sys 0.019s
I20250113 02:51:29.565551 1192 raft_consensus.cc:357] T c215ffe61b9b426286cc5e808463078d P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:29.565768 1192 raft_consensus.cc:383] T c215ffe61b9b426286cc5e808463078d P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:29.565832 1192 raft_consensus.cc:738] T c215ffe61b9b426286cc5e808463078d P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
W20250113 02:51:29.570328 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6d5148cf8ff24c92aedf3faee5ae8723 consensus state has no leader
W20250113 02:51:29.573686 448 catalog_manager.cc:5718] Failed to send AlterTable RPC for tablet 401c4f5dbd304950877b59a865f417b3 (table default.Table_c991f9dbd70d4c4682664b8711f05db4 [id=2c64da64ecdb4bf48e11b523533902e8], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 401c4f5dbd304950877b59a865f417b3 consensus state has no leader
I20250113 02:51:29.574832 448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_6431cb4cd5374cbcaceda152cbc19e5b" } new_table_name: "default.Table_69a9d7c8990e42c68549f893afb31c50" modify_external_catalogs: true
I20250113 02:51:29.576890 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_b89481a4a7be4e49b13f3260d8a31d5d" } modify_external_catalogs: true
I20250113 02:51:29.577098 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_b89481a4a7be4e49b13f3260d8a31d5d" } modify_external_catalogs: true
I20250113 02:51:29.578056 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_43fca7e946094914badafec91de2b53e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:29.578642 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_43fca7e946094914badafec91de2b53e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:29.580377 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_2f84c2362a2b4da7b088342232749760"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:29.580930 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2f84c2362a2b4da7b088342232749760 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:29.585098 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_9e44dff41935489aa1306e2e5d02bf85" } new_table_name: "default.Table_93d51ec3a124475ea63b71ae5744e3c0" modify_external_catalogs: true
I20250113 02:51:29.586548 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_69621fd76ebb45e5803d1a54f9023251"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:29.587066 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_69621fd76ebb45e5803d1a54f9023251 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:29.588963 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_c59e24fdd40849b48b0b75b0123cd6ed"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:29.589505 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c59e24fdd40849b48b0b75b0123cd6ed in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250113 02:51:29.590878 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 32cba8f3efc54e78b4df41c15df82eea consensus state has no leader
I20250113 02:51:29.591274 446 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_3c89afa5af12403b9a48046ee6c81da3" } modify_external_catalogs: true
I20250113 02:51:29.591449 446 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_3c89afa5af12403b9a48046ee6c81da3" } modify_external_catalogs: true
I20250113 02:51:29.595245 1198 ts_tablet_manager.cc:1918] T f31dd916336a4d8fb3ab015af7de8ccb P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:29.595449 1198 log.cc:1198] T f31dd916336a4d8fb3ab015af7de8ccb P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/f31dd916336a4d8fb3ab015af7de8ccb
I20250113 02:51:29.566037 1192 consensus_queue.cc:260] T c215ffe61b9b426286cc5e808463078d P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:29.596343 1198 ts_tablet_manager.cc:1939] T f31dd916336a4d8fb3ab015af7de8ccb P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:29.597524 1192 ts_tablet_manager.cc:1428] T c215ffe61b9b426286cc5e808463078d P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.033s user 0.001s sys 0.001s
I20250113 02:51:29.601403 454 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet 69dd1a5c96d24acbb1e822877732cfb5
I20250113 02:51:29.601800 454 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_c9a2c2b1c62849babefee677197da844" } modify_external_catalogs: true
I20250113 02:51:29.602079 454 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_c9a2c2b1c62849babefee677197da844" } modify_external_catalogs: true
I20250113 02:51:29.604879 886 tablet_service.cc:1514] Processing DeleteTablet for tablet 0fb75c551b4c424186ef2021db742e8e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:20 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:29.605688 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_2a627a757ab0406686564083ecc36c07" } new_table_name: "default.Table_b016b8047ff84e33959561a580efd594" modify_external_catalogs: true
W20250113 02:51:29.609182 711 leader_election.cc:343] T f68f9f4076d9475d938e795e72bdd468 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:29.609390 711 leader_election.cc:304] T f68f9f4076d9475d938e795e72bdd468 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:29.621644 1814 raft_consensus.cc:2743] T f68f9f4076d9475d938e795e72bdd468 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:29.637202 1195 log.cc:1198] T c215ffe61b9b426286cc5e808463078d P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/c215ffe61b9b426286cc5e808463078d
I20250113 02:51:29.637743 1195 ts_tablet_manager.cc:1939] T c215ffe61b9b426286cc5e808463078d P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
W20250113 02:51:29.640754 448 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 17fc320eb01f4ac2b9361aeab5b452fc consensus state has no leader
I20250113 02:51:29.643302 448 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_657257f5b9604a1994dc736b6e69330c" } modify_external_catalogs: true
I20250113 02:51:29.643635 448 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_657257f5b9604a1994dc736b6e69330c" } modify_external_catalogs: true
I20250113 02:51:29.645488 1191 tablet_bootstrap.cc:492] T c94eedb5dce44e6d896192c93127903a P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:29.645750 1191 ts_tablet_manager.cc:1397] T c94eedb5dce44e6d896192c93127903a P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.343s user 0.005s sys 0.005s
I20250113 02:51:29.650153 1192 tablet_bootstrap.cc:492] T ddf79eec41214693859f526265a55492 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:29.650545 1196 ts_tablet_manager.cc:1918] T 1a26e2b5e1054aa6afe7f44bf8d43108 P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:29.650676 1196 log.cc:1198] T 1a26e2b5e1054aa6afe7f44bf8d43108 P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/1a26e2b5e1054aa6afe7f44bf8d43108
I20250113 02:51:29.655068 1196 ts_tablet_manager.cc:1939] T 1a26e2b5e1054aa6afe7f44bf8d43108 P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:29.655112 1192 tablet_bootstrap.cc:654] T ddf79eec41214693859f526265a55492 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:29.655254 1195 tablet_replica.cc:331] T 63efcb6b2e194c469fb30a197bd6565b P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:29.655884 1195 raft_consensus.cc:2238] T 63efcb6b2e194c469fb30a197bd6565b P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:29.655963 1196 tablet_replica.cc:331] T 0fb75c551b4c424186ef2021db742e8e P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:29.656157 1196 raft_consensus.cc:2238] T 0fb75c551b4c424186ef2021db742e8e P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:29.655663 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet f31dd916336a4d8fb3ab015af7de8ccb (table default.Table_37b1d9d091924f8d9a8a2cbf1d06af1a [id=4d85a53a49d642b798af44eef60b3f78]) successfully deleted
W20250113 02:51:29.657409 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 80c647a3b5a1411e9130461115c53550 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 80c647a3b5a1411e9130461115c53550 already in progress: creating tablet
W20250113 02:51:29.657645 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet aa1ebdb92a5f4c92a5195b614689c100 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet aa1ebdb92a5f4c92a5195b614689c100 already in progress: creating tablet
I20250113 02:51:29.658142 1195 raft_consensus.cc:2267] T 63efcb6b2e194c469fb30a197bd6565b P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:29.658388 1198 tablet_replica.cc:331] T fdbf03c89bff438a9a25115eddf60be9 P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:29.659016 1195 ts_tablet_manager.cc:1905] T 63efcb6b2e194c469fb30a197bd6565b P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:29.659273 1198 raft_consensus.cc:2238] T fdbf03c89bff438a9a25115eddf60be9 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:29.659473 1198 raft_consensus.cc:2267] T fdbf03c89bff438a9a25115eddf60be9 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:29.654552 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet c215ffe61b9b426286cc5e808463078d (table default.Table_d4146353239a45b281b482d6e9537502 [id=20b69dfe61d841829e1804153c0d0184]) successfully deleted
I20250113 02:51:29.661150 1198 ts_tablet_manager.cc:1905] T fdbf03c89bff438a9a25115eddf60be9 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250113 02:51:29.656839 711 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:38467, user_credentials={real_user=slave}} blocked reactor thread for 47673us
W20250113 02:51:29.664472 451 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a04cec5ed9f34a16a80e112e5c9671e6 consensus state has no leader
I20250113 02:51:29.656709 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet 1a26e2b5e1054aa6afe7f44bf8d43108 (table default.Table_b5da750f5930453fa3951ffba96c34d4 [id=fa5c22ac7c024f0aba13091163742391]) successfully deleted
W20250113 02:51:29.667445 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 3d85539cf0074a03b167d3279e940079 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3d85539cf0074a03b167d3279e940079 already in progress: creating tablet
W20250113 02:51:29.667654 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet cd6dcfecf4324542b241f909926756ea because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cd6dcfecf4324542b241f909926756ea already in progress: creating tablet
W20250113 02:51:29.667843 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet e468b652f71f44e3945f8d919ac0e1a8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e468b652f71f44e3945f8d919ac0e1a8 already in progress: creating tablet
I20250113 02:51:29.668418 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet fc9189a9d1e4418e9f9205386f473232
I20250113 02:51:29.670245 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet d7920c0544c0466bad6bd8c09aae1a7d
I20250113 02:51:29.671761 896 tablet_service.cc:1514] Processing DeleteTablet for tablet d7920c0544c0466bad6bd8c09aae1a7d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:29 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:29.671751 886 tablet_service.cc:1514] Processing DeleteTablet for tablet fc9189a9d1e4418e9f9205386f473232 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:29 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:29.673317 1814 raft_consensus.cc:491] T 092892a36e0e47cb8d2184c2bbde95d7 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:29.673482 1814 raft_consensus.cc:513] T 092892a36e0e47cb8d2184c2bbde95d7 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:29.674048 1814 leader_election.cc:290] T 092892a36e0e47cb8d2184c2bbde95d7 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:29.678210 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "092892a36e0e47cb8d2184c2bbde95d7" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:29.678992 711 leader_election.cc:343] T 092892a36e0e47cb8d2184c2bbde95d7 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:29.685472 1194 tablet_bootstrap.cc:654] T 8adfa738c17a4169a5d99f6ceca75100 P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:29.666752 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet c215ffe61b9b426286cc5e808463078d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:20 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:29.669445 1040 tablet_service.cc:1514] Processing DeleteTablet for tablet fc9189a9d1e4418e9f9205386f473232 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:29 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:29.670842 1033 tablet_service.cc:1514] Processing DeleteTablet for tablet d7920c0544c0466bad6bd8c09aae1a7d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:29 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:29.675371 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "092892a36e0e47cb8d2184c2bbde95d7" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
W20250113 02:51:29.687304 709 leader_election.cc:343] T 092892a36e0e47cb8d2184c2bbde95d7 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:29.687466 709 leader_election.cc:304] T 092892a36e0e47cb8d2184c2bbde95d7 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:29.688082 1814 raft_consensus.cc:2743] T 092892a36e0e47cb8d2184c2bbde95d7 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:29.688052 1196 raft_consensus.cc:2267] T 0fb75c551b4c424186ef2021db742e8e P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:29.689353 1196 ts_tablet_manager.cc:1905] T 0fb75c551b4c424186ef2021db742e8e P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:29.697446 1196 ts_tablet_manager.cc:1918] T 0fb75c551b4c424186ef2021db742e8e P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:29.697562 1196 log.cc:1198] T 0fb75c551b4c424186ef2021db742e8e P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/0fb75c551b4c424186ef2021db742e8e
I20250113 02:51:29.660104 1191 raft_consensus.cc:357] T c94eedb5dce44e6d896192c93127903a P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:29.697803 1191 raft_consensus.cc:383] T c94eedb5dce44e6d896192c93127903a P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:29.698261 1191 raft_consensus.cc:738] T c94eedb5dce44e6d896192c93127903a P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:29.701381 1191 consensus_queue.cc:260] T c94eedb5dce44e6d896192c93127903a P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:29.702374 739 tablet_service.cc:1514] Processing DeleteTablet for tablet fc9189a9d1e4418e9f9205386f473232 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:29 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:29.702961 742 tablet_service.cc:1514] Processing DeleteTablet for tablet d7920c0544c0466bad6bd8c09aae1a7d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:29 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:29.705825 1873 tablet.cc:1721] T d304b65eba8c40c6922333506a60540c P df69f48656b44bb890f76587ee69cc0f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250113 02:51:29.708456 450 master_service.cc:946] ReplaceTablet: received request to replace tablet 542df00229f645788f1b821c0ce00c46 from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:29.710009 451 catalog_manager.cc:5618] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467) does not have the latest schema for tablet 572e4eed1e924381bada27f8389821a7 (table default.Table_1d910c3d2b244474a576fc4088301143 [id=75cb0745bbb84e1c883cb13b546ecb11]). Expected version 4 got 0
I20250113 02:51:29.711126 742 tablet_service.cc:1514] Processing DeleteTablet for tablet c94eedb5dce44e6d896192c93127903a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:20 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:29.711401 886 tablet_service.cc:1514] Processing DeleteTablet for tablet 601c71d5f7624fbb87b365d2e775d33e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:19 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:29.713291 1192 tablet_bootstrap.cc:492] T ddf79eec41214693859f526265a55492 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:29.713449 1192 ts_tablet_manager.cc:1397] T ddf79eec41214693859f526265a55492 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.063s user 0.014s sys 0.002s
I20250113 02:51:29.714243 1192 raft_consensus.cc:357] T ddf79eec41214693859f526265a55492 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:29.714450 1192 raft_consensus.cc:383] T ddf79eec41214693859f526265a55492 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:29.714521 1192 raft_consensus.cc:738] T ddf79eec41214693859f526265a55492 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:29.727003 1192 consensus_queue.cc:260] T ddf79eec41214693859f526265a55492 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:29.727777 451 catalog_manager.cc:5618] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467) does not have the latest schema for tablet 572e4eed1e924381bada27f8389821a7 (table default.Table_1d910c3d2b244474a576fc4088301143 [id=75cb0745bbb84e1c883cb13b546ecb11]). Expected version 4 got 0
I20250113 02:51:29.733613 1033 tablet_service.cc:1514] Processing DeleteTablet for tablet ddf79eec41214693859f526265a55492 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:23 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:29.734464 1196 ts_tablet_manager.cc:1939] T 0fb75c551b4c424186ef2021db742e8e P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:29.736042 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet 0fb75c551b4c424186ef2021db742e8e (table default.Table_2e62961b127b4bb3ac9b5d30f408c299 [id=e2dbf349032c4342818c5f8a891b7f99]) successfully deleted
I20250113 02:51:29.737414 1191 ts_tablet_manager.cc:1428] T c94eedb5dce44e6d896192c93127903a P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.091s user 0.003s sys 0.000s
I20250113 02:51:29.739121 1196 tablet_replica.cc:331] T d7920c0544c0466bad6bd8c09aae1a7d P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:29.742964 1196 raft_consensus.cc:2238] T d7920c0544c0466bad6bd8c09aae1a7d P df69f48656b44bb890f76587ee69cc0f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:29.743279 1196 raft_consensus.cc:2267] T d7920c0544c0466bad6bd8c09aae1a7d P df69f48656b44bb890f76587ee69cc0f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:29.744035 1196 ts_tablet_manager.cc:1905] T d7920c0544c0466bad6bd8c09aae1a7d P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:29.745445 1192 ts_tablet_manager.cc:1428] T ddf79eec41214693859f526265a55492 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.032s user 0.001s sys 0.001s
I20250113 02:51:29.735097 1194 tablet_bootstrap.cc:492] T 8adfa738c17a4169a5d99f6ceca75100 P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:29.746140 1194 ts_tablet_manager.cc:1397] T 8adfa738c17a4169a5d99f6ceca75100 P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.203s user 0.000s sys 0.005s
I20250113 02:51:29.747154 1878 tablet.cc:1721] T 06f36d09bbb1443bafce906907795404 P 4216a7f853d642b7bc3e5cff458f3aca: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250113 02:51:29.748947 1192 tablet_bootstrap.cc:492] T f68f9f4076d9475d938e795e72bdd468 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:29.751295 1194 raft_consensus.cc:357] T 8adfa738c17a4169a5d99f6ceca75100 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:29.751483 1194 raft_consensus.cc:383] T 8adfa738c17a4169a5d99f6ceca75100 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:29.751546 1194 raft_consensus.cc:738] T 8adfa738c17a4169a5d99f6ceca75100 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:29.751780 1194 consensus_queue.cc:260] T 8adfa738c17a4169a5d99f6ceca75100 P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:29.752492 1192 tablet_bootstrap.cc:654] T f68f9f4076d9475d938e795e72bdd468 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:29.755035 1191 tablet_bootstrap.cc:492] T 1cfbcd682b944c89847b40fe69f0a75e P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:29.761548 1194 ts_tablet_manager.cc:1428] T 8adfa738c17a4169a5d99f6ceca75100 P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.015s user 0.001s sys 0.000s
I20250113 02:51:29.763628 1192 tablet_bootstrap.cc:492] T f68f9f4076d9475d938e795e72bdd468 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:29.764034 1192 ts_tablet_manager.cc:1397] T f68f9f4076d9475d938e795e72bdd468 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.003s
I20250113 02:51:29.764356 1194 tablet_bootstrap.cc:492] T 501d4e1ae84e4795913035f8a19cc32d P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:29.765791 1192 raft_consensus.cc:357] T f68f9f4076d9475d938e795e72bdd468 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:29.767652 1192 raft_consensus.cc:383] T f68f9f4076d9475d938e795e72bdd468 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:29.767869 1192 raft_consensus.cc:738] T f68f9f4076d9475d938e795e72bdd468 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:29.768105 1873 tablet.cc:1721] T 06f36d09bbb1443bafce906907795404 P df69f48656b44bb890f76587ee69cc0f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250113 02:51:29.768352 1192 consensus_queue.cc:260] T f68f9f4076d9475d938e795e72bdd468 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:29.768494 1194 tablet_bootstrap.cc:654] T 501d4e1ae84e4795913035f8a19cc32d P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:29.768930 1872 tablet.cc:1721] T d304b65eba8c40c6922333506a60540c P 1186fa869eb14812832fa6562531f196: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250113 02:51:29.772336 1879 tablet.cc:1721] T 06f36d09bbb1443bafce906907795404 P 1186fa869eb14812832fa6562531f196: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250113 02:51:29.773545 1033 tablet_service.cc:1514] Processing DeleteTablet for tablet f68f9f4076d9475d938e795e72bdd468 with delete_type TABLET_DATA_DELETED (replaced by tablet f2c6c8c17d6e499d84ad7c0c23889cc4) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:29.774554 1192 ts_tablet_manager.cc:1428] T f68f9f4076d9475d938e795e72bdd468 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20250113 02:51:29.775621 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_d1f76dd334eb4d5890cceb429ed27e9f" } new_table_name: "default.Table_d6dd014b2a3f4de5aaca5efe047b5819" modify_external_catalogs: true
I20250113 02:51:29.776319 447 master_service.cc:946] ReplaceTablet: received request to replace tablet 5d4912055c5d4c8d8be8b65121f53200 from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:29.778434 1192 tablet_bootstrap.cc:492] T b4806b18a4194f9c95350d89ba62e13a P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
W20250113 02:51:29.780850 709 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.3:42587, user_credentials={real_user=slave}} blocked reactor thread for 60211.1us
I20250113 02:51:29.783138 1192 tablet_bootstrap.cc:654] T b4806b18a4194f9c95350d89ba62e13a P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:29.785424 1191 tablet_bootstrap.cc:654] T 1cfbcd682b944c89847b40fe69f0a75e P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:29.787635 1196 ts_tablet_manager.cc:1918] T d7920c0544c0466bad6bd8c09aae1a7d P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250113 02:51:29.792598 1196 log.cc:1198] T d7920c0544c0466bad6bd8c09aae1a7d P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/d7920c0544c0466bad6bd8c09aae1a7d
I20250113 02:51:29.795992 454 catalog_manager.cc:6071] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: default.Table_e03d954cc20048e4b0d598282689643d [id=e62f8e4c51b04729903b1943275c5f7f] alter complete (version 1)
I20250113 02:51:29.796442 1192 tablet_bootstrap.cc:492] T b4806b18a4194f9c95350d89ba62e13a P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:29.796609 1192 ts_tablet_manager.cc:1397] T b4806b18a4194f9c95350d89ba62e13a P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.003s
I20250113 02:51:29.797600 1192 raft_consensus.cc:357] T b4806b18a4194f9c95350d89ba62e13a P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:29.797678 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_22ace32627b84d33b3fe01731f8d2c07"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:29.797897 1192 raft_consensus.cc:383] T b4806b18a4194f9c95350d89ba62e13a P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:29.797978 1192 raft_consensus.cc:738] T b4806b18a4194f9c95350d89ba62e13a P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
W20250113 02:51:29.798321 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_22ace32627b84d33b3fe01731f8d2c07 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:29.799482 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_eb8e889492074afe8bdb891316a92caf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:29.799960 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_eb8e889492074afe8bdb891316a92caf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:29.800264 1033 tablet_service.cc:1467] Processing CreateTablet for tablet 6ee2ea47a519435ebd2a5babe0b9a980 (DEFAULT_TABLE table=default.Table_580e03ac0ded4964948e4809ddc91945 [id=cc445fb420564eb1ad2e09df08bc0fac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:29.800457 1192 consensus_queue.cc:260] T b4806b18a4194f9c95350d89ba62e13a P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:29.801092 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_73f84e2870c44b50b38f55848ef2bbbe"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:29.801492 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_73f84e2870c44b50b38f55848ef2bbbe in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:29.802086 1040 tablet_service.cc:1467] Processing CreateTablet for tablet e85d7f31bb46458798474115ac2aae8e (DEFAULT_TABLE table=default.Table_49cae5c8551e4a9cae1e7dfba2b4e97a [id=4e32ea43ff68411092b2b6facec91b18]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:29.802560 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_54127c30dcd54c1dbe1f089100b2a4b6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:29.802970 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_54127c30dcd54c1dbe1f089100b2a4b6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:29.803668 1029 tablet_service.cc:1467] Processing CreateTablet for tablet 032002b007424c028b4c671e1b0fb2ac (DEFAULT_TABLE table=default.Table_19f32aa03bc941cebef094ca3896e0c3 [id=472bccdfda7d496985b91af213b56025]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:29.805045 1041 tablet_service.cc:1467] Processing CreateTablet for tablet 368e878e89b54694823863a49074c11e (DEFAULT_TABLE table=default.Table_f82ef9176e6043d28f8d6a337ac4bce3 [id=4a7bb26a78be4f7a9eb4f895d7f746a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:29.806041 1192 ts_tablet_manager.cc:1428] T b4806b18a4194f9c95350d89ba62e13a P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20250113 02:51:29.807511 886 tablet_service.cc:1467] Processing CreateTablet for tablet 6ee2ea47a519435ebd2a5babe0b9a980 (DEFAULT_TABLE table=default.Table_580e03ac0ded4964948e4809ddc91945 [id=cc445fb420564eb1ad2e09df08bc0fac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:29.807897 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ee2ea47a519435ebd2a5babe0b9a980. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:29.809005 1192 tablet_bootstrap.cc:492] T 378d59d314a84ddb84cf2ab041c51cf1 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:29.812207 1192 tablet_bootstrap.cc:654] T 378d59d314a84ddb84cf2ab041c51cf1 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:29.812228 896 tablet_service.cc:1467] Processing CreateTablet for tablet e85d7f31bb46458798474115ac2aae8e (DEFAULT_TABLE table=default.Table_49cae5c8551e4a9cae1e7dfba2b4e97a [id=4e32ea43ff68411092b2b6facec91b18]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:29.812531 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e85d7f31bb46458798474115ac2aae8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:29.814299 1040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e85d7f31bb46458798474115ac2aae8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:29.814723 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 368e878e89b54694823863a49074c11e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:29.815119 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 032002b007424c028b4c671e1b0fb2ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:29.815572 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ee2ea47a519435ebd2a5babe0b9a980. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:29.816233 899 tablet_service.cc:1467] Processing CreateTablet for tablet 032002b007424c028b4c671e1b0fb2ac (DEFAULT_TABLE table=default.Table_19f32aa03bc941cebef094ca3896e0c3 [id=472bccdfda7d496985b91af213b56025]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:29.816375 1192 tablet_bootstrap.cc:492] T 378d59d314a84ddb84cf2ab041c51cf1 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:29.816525 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 032002b007424c028b4c671e1b0fb2ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:29.816548 1192 ts_tablet_manager.cc:1397] T 378d59d314a84ddb84cf2ab041c51cf1 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.000s
I20250113 02:51:29.818881 1192 raft_consensus.cc:357] T 378d59d314a84ddb84cf2ab041c51cf1 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:29.819133 1192 raft_consensus.cc:383] T 378d59d314a84ddb84cf2ab041c51cf1 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:29.819216 1192 raft_consensus.cc:738] T 378d59d314a84ddb84cf2ab041c51cf1 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:29.820266 881 tablet_service.cc:1467] Processing CreateTablet for tablet 368e878e89b54694823863a49074c11e (DEFAULT_TABLE table=default.Table_f82ef9176e6043d28f8d6a337ac4bce3 [id=4a7bb26a78be4f7a9eb4f895d7f746a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:29.820565 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 368e878e89b54694823863a49074c11e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:29.822706 1192 consensus_queue.cc:260] T 378d59d314a84ddb84cf2ab041c51cf1 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:29.823779 1192 ts_tablet_manager.cc:1428] T 378d59d314a84ddb84cf2ab041c51cf1 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20250113 02:51:29.824966 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_8bca38c8f95a4ea8aaec77a9e6c57f96"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:29.825461 1192 tablet_bootstrap.cc:492] T 9fff52e99c64462a8a73fccc3c22ca63 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
W20250113 02:51:29.825570 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8bca38c8f95a4ea8aaec77a9e6c57f96 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:29.834873 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_3fe958090174491c874058061bc4c252"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:29.835628 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3fe958090174491c874058061bc4c252 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:29.838987 742 tablet_service.cc:1467] Processing CreateTablet for tablet 6ee2ea47a519435ebd2a5babe0b9a980 (DEFAULT_TABLE table=default.Table_580e03ac0ded4964948e4809ddc91945 [id=cc445fb420564eb1ad2e09df08bc0fac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:29.839449 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ee2ea47a519435ebd2a5babe0b9a980. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:29.841356 1192 tablet_bootstrap.cc:654] T 9fff52e99c64462a8a73fccc3c22ca63 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:29.843271 1198 ts_tablet_manager.cc:1918] T fdbf03c89bff438a9a25115eddf60be9 P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:29.843403 1198 log.cc:1198] T fdbf03c89bff438a9a25115eddf60be9 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/fdbf03c89bff438a9a25115eddf60be9
I20250113 02:51:29.843763 1198 ts_tablet_manager.cc:1939] T fdbf03c89bff438a9a25115eddf60be9 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:29.845327 1192 tablet_bootstrap.cc:492] T 9fff52e99c64462a8a73fccc3c22ca63 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:29.845485 1192 ts_tablet_manager.cc:1397] T 9fff52e99c64462a8a73fccc3c22ca63 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.000s
I20250113 02:51:29.846344 742 tablet_service.cc:1467] Processing CreateTablet for tablet e85d7f31bb46458798474115ac2aae8e (DEFAULT_TABLE table=default.Table_49cae5c8551e4a9cae1e7dfba2b4e97a [id=4e32ea43ff68411092b2b6facec91b18]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:29.847327 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e85d7f31bb46458798474115ac2aae8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:29.847944 1192 raft_consensus.cc:357] T 9fff52e99c64462a8a73fccc3c22ca63 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:29.848184 1192 raft_consensus.cc:383] T 9fff52e99c64462a8a73fccc3c22ca63 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:29.848287 1192 raft_consensus.cc:738] T 9fff52e99c64462a8a73fccc3c22ca63 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:29.849567 1192 consensus_queue.cc:260] T 9fff52e99c64462a8a73fccc3c22ca63 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:29.852361 1192 ts_tablet_manager.cc:1428] T 9fff52e99c64462a8a73fccc3c22ca63 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20250113 02:51:29.853017 739 tablet_service.cc:1467] Processing CreateTablet for tablet 032002b007424c028b4c671e1b0fb2ac (DEFAULT_TABLE table=default.Table_19f32aa03bc941cebef094ca3896e0c3 [id=472bccdfda7d496985b91af213b56025]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:29.853942 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 032002b007424c028b4c671e1b0fb2ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:29.857609 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet b4806b18a4194f9c95350d89ba62e13a with delete_type TABLET_DATA_DELETED (replaced by tablet 146dad5c977f464496f7d21611030dad) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:29.860929 1195 ts_tablet_manager.cc:1918] T 63efcb6b2e194c469fb30a197bd6565b P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:29.861102 1195 log.cc:1198] T 63efcb6b2e194c469fb30a197bd6565b P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/63efcb6b2e194c469fb30a197bd6565b
I20250113 02:51:29.861454 1195 ts_tablet_manager.cc:1939] T 63efcb6b2e194c469fb30a197bd6565b P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:29.861760 1873 tablet.cc:1721] T 572e4eed1e924381bada27f8389821a7 P df69f48656b44bb890f76587ee69cc0f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20250113 02:51:29.862041 1194 tablet_bootstrap.cc:492] T 501d4e1ae84e4795913035f8a19cc32d P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:29.862156 1194 ts_tablet_manager.cc:1397] T 501d4e1ae84e4795913035f8a19cc32d P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.098s user 0.003s sys 0.001s
I20250113 02:51:29.862519 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet 63efcb6b2e194c469fb30a197bd6565b (table default.Table_a6447976a51a4e2f93d22bd1c657c49d [id=0485ff4fda424e97bb71b77d657ab5e6]) successfully deleted
I20250113 02:51:29.862852 1195 tablet_replica.cc:331] T 2778d7dc7d9c4843bbc9eb705cec71cd P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:29.863080 1195 raft_consensus.cc:2238] T 2778d7dc7d9c4843bbc9eb705cec71cd P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:29.863268 1195 raft_consensus.cc:2267] T 2778d7dc7d9c4843bbc9eb705cec71cd P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:29.863317 1191 tablet_bootstrap.cc:492] T 1cfbcd682b944c89847b40fe69f0a75e P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:29.863461 1191 ts_tablet_manager.cc:1397] T 1cfbcd682b944c89847b40fe69f0a75e P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.108s user 0.002s sys 0.001s
I20250113 02:51:29.863548 1194 raft_consensus.cc:357] T 501d4e1ae84e4795913035f8a19cc32d P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:29.863735 1194 raft_consensus.cc:383] T 501d4e1ae84e4795913035f8a19cc32d P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:29.863782 1194 raft_consensus.cc:738] T 501d4e1ae84e4795913035f8a19cc32d P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:29.863842 1195 ts_tablet_manager.cc:1905] T 2778d7dc7d9c4843bbc9eb705cec71cd P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:29.863960 1194 consensus_queue.cc:260] T 501d4e1ae84e4795913035f8a19cc32d P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:29.864704 1191 raft_consensus.cc:357] T 1cfbcd682b944c89847b40fe69f0a75e P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:29.864954 1191 raft_consensus.cc:383] T 1cfbcd682b944c89847b40fe69f0a75e P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:29.865037 1191 raft_consensus.cc:738] T 1cfbcd682b944c89847b40fe69f0a75e P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:29.865283 1191 consensus_queue.cc:260] T 1cfbcd682b944c89847b40fe69f0a75e P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:29.865876 1191 ts_tablet_manager.cc:1428] T 1cfbcd682b944c89847b40fe69f0a75e P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250113 02:51:29.866262 1194 ts_tablet_manager.cc:1428] T 501d4e1ae84e4795913035f8a19cc32d P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20250113 02:51:29.866901 1194 tablet_bootstrap.cc:492] T fdbf03c89bff438a9a25115eddf60be9 P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:29.868088 886 tablet_service.cc:1467] Processing CreateTablet for tablet 16f547e3cec54baf86b96ad40e3afb6a (DEFAULT_TABLE table=default.Table_07e3e48b34b040ebbcf9a5dffdbb6bd9 [id=52b66a1e8bc04f549d90e830088c64e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:29.868453 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 16f547e3cec54baf86b96ad40e3afb6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:29.870267 1029 tablet_service.cc:1467] Processing CreateTablet for tablet 16f547e3cec54baf86b96ad40e3afb6a (DEFAULT_TABLE table=default.Table_07e3e48b34b040ebbcf9a5dffdbb6bd9 [id=52b66a1e8bc04f549d90e830088c64e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:29.871126 1195 ts_tablet_manager.cc:1918] T 2778d7dc7d9c4843bbc9eb705cec71cd P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:29.871260 1195 log.cc:1198] T 2778d7dc7d9c4843bbc9eb705cec71cd P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/2778d7dc7d9c4843bbc9eb705cec71cd
I20250113 02:51:29.872247 1195 ts_tablet_manager.cc:1939] T 2778d7dc7d9c4843bbc9eb705cec71cd P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:29.873426 1195 tablet_replica.cc:331] T a0ca2e3a3ce94f3aa20df64c89c7cf95 P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:29.873708 1195 raft_consensus.cc:2238] T a0ca2e3a3ce94f3aa20df64c89c7cf95 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:29.873929 1195 raft_consensus.cc:2267] T a0ca2e3a3ce94f3aa20df64c89c7cf95 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:29.874529 1195 ts_tablet_manager.cc:1905] T a0ca2e3a3ce94f3aa20df64c89c7cf95 P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:29.875706 1438 raft_consensus.cc:491] T 501d4e1ae84e4795913035f8a19cc32d P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:29.875815 1438 raft_consensus.cc:513] T 501d4e1ae84e4795913035f8a19cc32d P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:29.876480 1438 leader_election.cc:290] T 501d4e1ae84e4795913035f8a19cc32d P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587), 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
I20250113 02:51:29.877350 899 tablet_service.cc:1467] Processing CreateTablet for tablet a3adda5baa9a480ab1985f33103b8485 (DEFAULT_TABLE table=default.Table_e621a0cb47054f84898824e0343e3a6a [id=14db6cf509664e639542ec7bbccb071e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:29.877840 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3adda5baa9a480ab1985f33103b8485. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:29.880653 1195 ts_tablet_manager.cc:1918] T a0ca2e3a3ce94f3aa20df64c89c7cf95 P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:29.880872 1195 log.cc:1198] T a0ca2e3a3ce94f3aa20df64c89c7cf95 P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/a0ca2e3a3ce94f3aa20df64c89c7cf95
I20250113 02:51:29.881332 1195 ts_tablet_manager.cc:1939] T a0ca2e3a3ce94f3aa20df64c89c7cf95 P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:29.882669 1195 tablet_replica.cc:331] T 092892a36e0e47cb8d2184c2bbde95d7 P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:29.883080 1191 tablet_bootstrap.cc:492] T 4c615b0b83d74fcfae0c51047dbfbbf4 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:29.885201 881 tablet_service.cc:1467] Processing CreateTablet for tablet 573cffe308cf4774a2108a7390f08e90 (DEFAULT_TABLE table=default.Table_3d5fd97116b343b0a6c798e1f22327c2 [id=1396c4482ee646d18330d200fd3f9816]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:29.885449 1191 tablet_bootstrap.cc:654] T 4c615b0b83d74fcfae0c51047dbfbbf4 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:29.885708 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 573cffe308cf4774a2108a7390f08e90. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:29.888203 1033 tablet_service.cc:1467] Processing CreateTablet for tablet ef99c223dcf04b11bf785703df55f877 (DEFAULT_TABLE table=default.Table_2f896daff51e4eb78a833f59d07ea44d [id=bef0886b34494716856620bf19d3f6a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:29.888760 1025 tablet_service.cc:1467] Processing CreateTablet for tablet 573cffe308cf4774a2108a7390f08e90 (DEFAULT_TABLE table=default.Table_3d5fd97116b343b0a6c798e1f22327c2 [id=1396c4482ee646d18330d200fd3f9816]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:29.890159 1192 tablet_bootstrap.cc:492] T 401c4f5dbd304950877b59a865f417b3 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:29.891510 880 tablet_service.cc:1467] Processing CreateTablet for tablet 35a13d9c065348e08de66044c04d221d (DEFAULT_TABLE table=default.Table_95e76bedf8924501b57f7a26d66a9d33 [id=9d9f708a75c648ec8b5dfb7e25d4ba92]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:29.891876 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35a13d9c065348e08de66044c04d221d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:29.891844 896 tablet_service.cc:1467] Processing CreateTablet for tablet 54aae7c131404facba9af3a3d143b4a4 (DEFAULT_TABLE table=default.Table_8db01e4eaddd49aeb8f955e75cabb671 [id=4edd1ca499d949c699cde36faf86cd72]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:29.892187 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 54aae7c131404facba9af3a3d143b4a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:29.892583 1192 tablet_bootstrap.cc:654] T 401c4f5dbd304950877b59a865f417b3 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:29.895783 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 16f547e3cec54baf86b96ad40e3afb6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:29.896322 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef99c223dcf04b11bf785703df55f877. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:29.896471 1878 tablet.cc:1721] T 572e4eed1e924381bada27f8389821a7 P 4216a7f853d642b7bc3e5cff458f3aca: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20250113 02:51:29.896971 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 573cffe308cf4774a2108a7390f08e90. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:29.898600 1887 raft_consensus.cc:491] T 6d63f0e5835540a78c82020b20a07dac P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:29.899489 1887 raft_consensus.cc:513] T 6d63f0e5835540a78c82020b20a07dac P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:29.900729 1887 leader_election.cc:290] T 6d63f0e5835540a78c82020b20a07dac P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:29.905239 1196 ts_tablet_manager.cc:1939] T d7920c0544c0466bad6bd8c09aae1a7d P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:29.906637 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet d7920c0544c0466bad6bd8c09aae1a7d (table default.Table_c9a2c2b1c62849babefee677197da844 [id=05755562cfc24370a3e6fb245a573dd4]) successfully deleted
I20250113 02:51:29.907737 1198 tablet_replica.cc:331] T 2c399784af46426b8327d175c7a6cce8 P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:29.908023 1198 raft_consensus.cc:2238] T 2c399784af46426b8327d175c7a6cce8 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:29.908252 1198 raft_consensus.cc:2267] T 2c399784af46426b8327d175c7a6cce8 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:29.909183 881 tablet_service.cc:1467] Processing CreateTablet for tablet 18716752335043e89038f01b4d65d711 (DEFAULT_TABLE table=default.Table_685277f86f8f48fc9577c4adef84e006 [id=044304e1c4b44cc28b8990d6f28145a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:29.910738 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 18716752335043e89038f01b4d65d711. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:29.908865 1198 ts_tablet_manager.cc:1905] T 2c399784af46426b8327d175c7a6cce8 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:29.910740 1029 tablet_service.cc:1467] Processing CreateTablet for tablet 18716752335043e89038f01b4d65d711 (DEFAULT_TABLE table=default.Table_685277f86f8f48fc9577c4adef84e006 [id=044304e1c4b44cc28b8990d6f28145a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:29.910029 886 tablet_service.cc:1467] Processing CreateTablet for tablet ef99c223dcf04b11bf785703df55f877 (DEFAULT_TABLE table=default.Table_2f896daff51e4eb78a833f59d07ea44d [id=bef0886b34494716856620bf19d3f6a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:29.913745 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 18716752335043e89038f01b4d65d711. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:29.913980 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef99c223dcf04b11bf785703df55f877. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:29.909565 899 tablet_service.cc:1467] Processing CreateTablet for tablet 05b936a3f99b4db7ac32ba4d85c22ca9 (DEFAULT_TABLE table=default.Table_0146ad4fe4424350a582eb66ed609eba [id=f95ee8db63be4d8faaad91b7ffbd7781]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:29.915676 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05b936a3f99b4db7ac32ba4d85c22ca9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:29.916115 1876 raft_consensus.cc:491] T c94eedb5dce44e6d896192c93127903a P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:29.916352 1876 raft_consensus.cc:513] T c94eedb5dce44e6d896192c93127903a P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:29.917099 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 542df00229f645788f1b821c0ce00c46
I20250113 02:51:29.917124 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet fdbf03c89bff438a9a25115eddf60be9 (table default.Table_732e990e6f154d6486889304bbf88cef [id=48c7c3f6b4a840ee9b9e3f3844add13d]) successfully deleted
W20250113 02:51:29.917384 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 3d85539cf0074a03b167d3279e940079 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3d85539cf0074a03b167d3279e940079 already in progress: creating tablet
I20250113 02:51:29.917575 450 catalog_manager.cc:6596] ReplaceTablet: tablet 542df00229f645788f1b821c0ce00c46 deleted and replaced by tablet 2460e7cddcd84788a54dab342e7bb27b
I20250113 02:51:29.917945 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet 542df00229f645788f1b821c0ce00c46 with delete_type TABLET_DATA_DELETED (replaced by tablet 2460e7cddcd84788a54dab342e7bb27b) from {username='slave'} at 127.0.0.1:60230
W20250113 02:51:29.918853 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet fc9189a9d1e4418e9f9205386f473232 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fc9189a9d1e4418e9f9205386f473232 already in progress: creating tablet
I20250113 02:51:29.919524 880 tablet_service.cc:1514] Processing DeleteTablet for tablet 542df00229f645788f1b821c0ce00c46 with delete_type TABLET_DATA_DELETED (replaced by tablet 2460e7cddcd84788a54dab342e7bb27b) from {username='slave'} at 127.0.0.1:37366
W20250113 02:51:29.919871 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 80c647a3b5a1411e9130461115c53550 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 80c647a3b5a1411e9130461115c53550 already in progress: creating tablet
W20250113 02:51:29.920132 439 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.104.1:40095, user_credentials={real_user=slave}} blocked reactor thread for 57154us
I20250113 02:51:29.920287 1041 tablet_service.cc:1467] Processing CreateTablet for tablet a3adda5baa9a480ab1985f33103b8485 (DEFAULT_TABLE table=default.Table_e621a0cb47054f84898824e0343e3a6a [id=14db6cf509664e639542ec7bbccb071e]), partition=RANGE (key) PARTITION UNBOUNDED
W20250113 02:51:29.920428 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet aa1ebdb92a5f4c92a5195b614689c100 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet aa1ebdb92a5f4c92a5195b614689c100 already in progress: creating tablet
W20250113 02:51:29.920498 852 leader_election.cc:343] T 501d4e1ae84e4795913035f8a19cc32d P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Not found: Tablet not found: 501d4e1ae84e4795913035f8a19cc32d
I20250113 02:51:29.920629 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet 2778d7dc7d9c4843bbc9eb705cec71cd (table default.Table_b3284070cce54277bed74120ca8f199b [id=4fafe716b4704c0c96514223a04e480a]) successfully deleted
I20250113 02:51:29.920917 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet a0ca2e3a3ce94f3aa20df64c89c7cf95 (table default.Table_f08525e2a08e48ec85ef55287d0bac49 [id=96ff1ee9327f41a5a1bb2757b188254a]) successfully deleted
I20250113 02:51:29.922559 451 catalog_manager.cc:5618] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467) does not have the latest schema for tablet 8adfa738c17a4169a5d99f6ceca75100 (table default.Table_9a9b1ced3bef4e648088e0781851128e [id=ae544735623e4cfd894dc9340aca177c]). Expected version 1 got 0
I20250113 02:51:29.922693 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3adda5baa9a480ab1985f33103b8485. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:29.923646 450 catalog_manager.cc:5618] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587) does not have the latest schema for tablet 9fff52e99c64462a8a73fccc3c22ca63 (table default.Table_9ef18ee71ad843c5921750a2f8478855 [id=bd8bd572ed524f2cae01f820f07e3c36]). Expected version 2 got 0
I20250113 02:51:29.923839 450 catalog_manager.cc:5618] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587) does not have the latest schema for tablet 378d59d314a84ddb84cf2ab041c51cf1 (table default.Table_2794f9d2a3cc49f78de33a775c1619f1 [id=0c01441cec7043b687d6744e72a0e45d]). Expected version 1 got 0
I20250113 02:51:29.875617 1194 tablet_bootstrap.cc:654] T fdbf03c89bff438a9a25115eddf60be9 P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:29.920838 1040 tablet_service.cc:1467] Processing CreateTablet for tablet 54aae7c131404facba9af3a3d143b4a4 (DEFAULT_TABLE table=default.Table_8db01e4eaddd49aeb8f955e75cabb671 [id=4edd1ca499d949c699cde36faf86cd72]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:29.928275 1040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 54aae7c131404facba9af3a3d143b4a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:29.911931 1196 tablet_replica.cc:331] T 601c71d5f7624fbb87b365d2e775d33e P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:29.928951 1196 raft_consensus.cc:2238] T 601c71d5f7624fbb87b365d2e775d33e P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:29.929141 1196 raft_consensus.cc:2267] T 601c71d5f7624fbb87b365d2e775d33e P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:29.931427 1876 leader_election.cc:290] T c94eedb5dce44e6d896192c93127903a P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:29.921922 1033 tablet_service.cc:1467] Processing CreateTablet for tablet 05b936a3f99b4db7ac32ba4d85c22ca9 (DEFAULT_TABLE table=default.Table_0146ad4fe4424350a582eb66ed609eba [id=f95ee8db63be4d8faaad91b7ffbd7781]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:29.932838 748 tablet_service.cc:1514] Processing DeleteTablet for tablet 542df00229f645788f1b821c0ce00c46 with delete_type TABLET_DATA_DELETED (replaced by tablet 2460e7cddcd84788a54dab342e7bb27b) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:29.933172 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05b936a3f99b4db7ac32ba4d85c22ca9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:29.933070 740 tablet_service.cc:1467] Processing CreateTablet for tablet a3adda5baa9a480ab1985f33103b8485 (DEFAULT_TABLE table=default.Table_e621a0cb47054f84898824e0343e3a6a [id=14db6cf509664e639542ec7bbccb071e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:29.933521 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3adda5baa9a480ab1985f33103b8485. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:29.934126 755 tablet_service.cc:1467] Processing CreateTablet for tablet 54aae7c131404facba9af3a3d143b4a4 (DEFAULT_TABLE table=default.Table_8db01e4eaddd49aeb8f955e75cabb671 [id=4edd1ca499d949c699cde36faf86cd72]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:29.934538 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 54aae7c131404facba9af3a3d143b4a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:29.935422 750 tablet_service.cc:1467] Processing CreateTablet for tablet 573cffe308cf4774a2108a7390f08e90 (DEFAULT_TABLE table=default.Table_3d5fd97116b343b0a6c798e1f22327c2 [id=1396c4482ee646d18330d200fd3f9816]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:29.935611 1196 ts_tablet_manager.cc:1905] T 601c71d5f7624fbb87b365d2e775d33e P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:29.935842 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 573cffe308cf4774a2108a7390f08e90. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:29.936414 736 tablet_service.cc:1467] Processing CreateTablet for tablet 35a13d9c065348e08de66044c04d221d (DEFAULT_TABLE table=default.Table_95e76bedf8924501b57f7a26d66a9d33 [id=9d9f708a75c648ec8b5dfb7e25d4ba92]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:29.936888 736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35a13d9c065348e08de66044c04d221d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:29.936915 1906 tablet.cc:1721] T d304b65eba8c40c6922333506a60540c P 4216a7f853d642b7bc3e5cff458f3aca: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250113 02:51:29.937740 746 tablet_service.cc:1467] Processing CreateTablet for tablet 18716752335043e89038f01b4d65d711 (DEFAULT_TABLE table=default.Table_685277f86f8f48fc9577c4adef84e006 [id=044304e1c4b44cc28b8990d6f28145a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:29.938150 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 18716752335043e89038f01b4d65d711. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:29.938843 1192 tablet_bootstrap.cc:492] T 401c4f5dbd304950877b59a865f417b3 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:29.938993 1192 ts_tablet_manager.cc:1397] T 401c4f5dbd304950877b59a865f417b3 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.050s user 0.006s sys 0.009s
I20250113 02:51:29.939154 747 tablet_service.cc:1467] Processing CreateTablet for tablet 05b936a3f99b4db7ac32ba4d85c22ca9 (DEFAULT_TABLE table=default.Table_0146ad4fe4424350a582eb66ed609eba [id=f95ee8db63be4d8faaad91b7ffbd7781]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:29.939555 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05b936a3f99b4db7ac32ba4d85c22ca9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:29.939749 1192 raft_consensus.cc:357] T 401c4f5dbd304950877b59a865f417b3 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:29.939960 1192 raft_consensus.cc:383] T 401c4f5dbd304950877b59a865f417b3 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:29.940057 1192 raft_consensus.cc:738] T 401c4f5dbd304950877b59a865f417b3 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:29.940301 1192 consensus_queue.cc:260] T 401c4f5dbd304950877b59a865f417b3 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:29.934927 1025 tablet_service.cc:1467] Processing CreateTablet for tablet 35a13d9c065348e08de66044c04d221d (DEFAULT_TABLE table=default.Table_95e76bedf8924501b57f7a26d66a9d33 [id=9d9f708a75c648ec8b5dfb7e25d4ba92]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:29.940797 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35a13d9c065348e08de66044c04d221d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:29.941088 1192 ts_tablet_manager.cc:1428] T 401c4f5dbd304950877b59a865f417b3 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250113 02:51:29.941397 1192 tablet_bootstrap.cc:492] T a0ca2e3a3ce94f3aa20df64c89c7cf95 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:29.941773 745 tablet_service.cc:1467] Processing CreateTablet for tablet ef99c223dcf04b11bf785703df55f877 (DEFAULT_TABLE table=default.Table_2f896daff51e4eb78a833f59d07ea44d [id=bef0886b34494716856620bf19d3f6a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:29.942190 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef99c223dcf04b11bf785703df55f877. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:29.949044 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 64aa8390b1b1475c8743b6491d40674c
I20250113 02:51:29.949910 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_ff26c7d175d24168a34825fdee4c72a0" } modify_external_catalogs: true
I20250113 02:51:29.950047 455 catalog_manager.cc:5618] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095) does not have the latest schema for tablet 572e4eed1e924381bada27f8389821a7 (table default.Table_1d910c3d2b244474a576fc4088301143 [id=75cb0745bbb84e1c883cb13b546ecb11]). Expected version 4 got 0
I20250113 02:51:29.950095 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_ff26c7d175d24168a34825fdee4c72a0" } modify_external_catalogs: true
I20250113 02:51:29.950255 455 catalog_manager.cc:5618] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095) does not have the latest schema for tablet 1cfbcd682b944c89847b40fe69f0a75e (table default.Table_32e88abaae96492fbcce39fb2cd7c221 [id=2b84deb07e5c4ab8b550fe42945d7b71]). Expected version 2 got 0
I20250113 02:51:29.951344 880 tablet_service.cc:1514] Processing DeleteTablet for tablet 64aa8390b1b1475c8743b6491d40674c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:29 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:29.951684 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_016e5998c5ef4112908e6917ccd57d15" } new_table_name: "default.Table_c8b93363989c4e109540af6bef426f73" modify_external_catalogs: true
I20250113 02:51:29.952332 453 master_service.cc:946] ReplaceTablet: received request to replace tablet cd21f10d74c946a2872a84cadccf7950 from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:29.953576 446 master_service.cc:946] ReplaceTablet: received request to replace tablet 572e4eed1e924381bada27f8389821a7 from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:29.956686 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet f4af42b9bae44e3fb009310212cec587
I20250113 02:51:29.957384 448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_50f21224ee254e3bb13748a44e257783" } new_table_name: "default.Table_4eff233920ec4a0cb1c20513a510ce8b" modify_external_catalogs: true
I20250113 02:51:29.959118 880 tablet_service.cc:1514] Processing DeleteTablet for tablet f4af42b9bae44e3fb009310212cec587 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:29 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:29.963198 1195 raft_consensus.cc:2238] T 092892a36e0e47cb8d2184c2bbde95d7 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:29.963516 1195 raft_consensus.cc:2267] T 092892a36e0e47cb8d2184c2bbde95d7 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:29.972680 748 tablet_service.cc:1514] Processing DeleteTablet for tablet 64aa8390b1b1475c8743b6491d40674c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:29 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:29.975522 748 tablet_service.cc:1514] Processing DeleteTablet for tablet f4af42b9bae44e3fb009310212cec587 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:29 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:29.975466 739 tablet_service.cc:1467] Processing CreateTablet for tablet 16f547e3cec54baf86b96ad40e3afb6a (DEFAULT_TABLE table=default.Table_07e3e48b34b040ebbcf9a5dffdbb6bd9 [id=52b66a1e8bc04f549d90e830088c64e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:29.976022 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 16f547e3cec54baf86b96ad40e3afb6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:29.980013 1195 ts_tablet_manager.cc:1905] T 092892a36e0e47cb8d2184c2bbde95d7 P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:29.980013 742 tablet_service.cc:1467] Processing CreateTablet for tablet 368e878e89b54694823863a49074c11e (DEFAULT_TABLE table=default.Table_f82ef9176e6043d28f8d6a337ac4bce3 [id=4a7bb26a78be4f7a9eb4f895d7f746a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:29.980810 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 368e878e89b54694823863a49074c11e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:29.984318 447 master_service.cc:946] ReplaceTablet: received request to replace tablet cf7b8bd6801743f49e9b9473747447c5 from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:29.986521 454 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_65c5f2ca0aea479b9edd500ceb8c01bc" } new_table_name: "default.Table_b7013bb28efb4fd5a1bdb6b64048e437" modify_external_catalogs: true
I20250113 02:51:29.988554 450 catalog_manager.cc:6071] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: default.Table_b016b8047ff84e33959561a580efd594 [id=423021f304964246bd29e4daaf470154] alter complete (version 3)
I20250113 02:51:29.989503 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_2f896daff51e4eb78a833f59d07ea44d" } modify_external_catalogs: true
I20250113 02:51:29.989696 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_2f896daff51e4eb78a833f59d07ea44d" } modify_external_catalogs: true
I20250113 02:51:29.991791 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_fc7ae5b847754786984b5b25591def13"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:29.992058 711 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.2:38467, user_credentials={real_user=slave}} blocked reactor thread for 116231us
W20250113 02:51:29.992362 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fc7ae5b847754786984b5b25591def13 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:29.992691 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d63f0e5835540a78c82020b20a07dac" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:29.994804 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c94eedb5dce44e6d896192c93127903a" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:29.995713 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_5d4d2d0f30f44ba89da01f4af3b1a4c3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:29.996487 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5d4d2d0f30f44ba89da01f4af3b1a4c3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:29.980414 1872 tablet.cc:1721] T 572e4eed1e924381bada27f8389821a7 P 1186fa869eb14812832fa6562531f196: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20250113 02:51:29.998282 1194 tablet_bootstrap.cc:492] T fdbf03c89bff438a9a25115eddf60be9 P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:29.998410 1194 ts_tablet_manager.cc:1397] T fdbf03c89bff438a9a25115eddf60be9 P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.132s user 0.000s sys 0.007s
W20250113 02:51:30.002998 711 leader_election.cc:343] T 6d63f0e5835540a78c82020b20a07dac P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
W20250113 02:51:30.003212 711 leader_election.cc:343] T c94eedb5dce44e6d896192c93127903a P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:30.003376 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet f4af42b9bae44e3fb009310212cec587 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:29 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:30.003712 1192 tablet_bootstrap.cc:654] T a0ca2e3a3ce94f3aa20df64c89c7cf95 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:30.006814 1195 ts_tablet_manager.cc:1918] T 092892a36e0e47cb8d2184c2bbde95d7 P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:30.008173 1191 tablet_bootstrap.cc:492] T 4c615b0b83d74fcfae0c51047dbfbbf4 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:30.010007 1191 ts_tablet_manager.cc:1397] T 4c615b0b83d74fcfae0c51047dbfbbf4 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.127s user 0.003s sys 0.005s
I20250113 02:51:30.009905 1195 log.cc:1198] T 092892a36e0e47cb8d2184c2bbde95d7 P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/092892a36e0e47cb8d2184c2bbde95d7
I20250113 02:51:30.010543 1195 ts_tablet_manager.cc:1939] T 092892a36e0e47cb8d2184c2bbde95d7 P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:30.014389 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet cd21f10d74c946a2872a84cadccf7950
I20250113 02:51:30.015360 739 tablet_service.cc:1514] Processing DeleteTablet for tablet cd21f10d74c946a2872a84cadccf7950 with delete_type TABLET_DATA_DELETED (replaced by tablet 702e03e213b14ef5a1b64a02df1b3bf9) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:30.015494 453 catalog_manager.cc:6596] ReplaceTablet: tablet cd21f10d74c946a2872a84cadccf7950 deleted and replaced by tablet 702e03e213b14ef5a1b64a02df1b3bf9
I20250113 02:51:30.016288 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 572e4eed1e924381bada27f8389821a7
I20250113 02:51:30.016806 446 catalog_manager.cc:6596] ReplaceTablet: tablet 572e4eed1e924381bada27f8389821a7 deleted and replaced by tablet 29b1444efd28483297bafc73a33e7a54
I20250113 02:51:30.017024 739 tablet_service.cc:1514] Processing DeleteTablet for tablet 572e4eed1e924381bada27f8389821a7 with delete_type TABLET_DATA_DELETED (replaced by tablet 29b1444efd28483297bafc73a33e7a54) from {username='slave'} at 127.0.0.1:55398
W20250113 02:51:30.017423 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7527e6e0b65e4a43b7f48899a34cdf7f consensus state has no leader
I20250113 02:51:30.019053 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 85a82ad6e93346ecb1a0cb38b62f5786
W20250113 02:51:30.019284 448 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 36726b40ac5145bfb371871b3fff079a consensus state has no leader
I20250113 02:51:30.019634 448 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_f7097c6cf1fd4839872c42a30fa5f64f" } modify_external_catalogs: true
I20250113 02:51:30.019855 739 tablet_service.cc:1514] Processing DeleteTablet for tablet 85a82ad6e93346ecb1a0cb38b62f5786 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:29 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:30.019851 448 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_f7097c6cf1fd4839872c42a30fa5f64f" } modify_external_catalogs: true
I20250113 02:51:30.019934 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_d4b3b319759f44d4bbe072b1bc97cc94" } new_table_name: "default.Table_c0fd5aef6a3c4695be61ebaa20ca4a61" modify_external_catalogs: true
I20250113 02:51:30.023835 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_5cc38270bc7e429b8491000d00a7a782"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:30.024422 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5cc38270bc7e429b8491000d00a7a782 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:30.025789 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_3196fc458c3740f2a1af55de808edb06"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:30.026242 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3196fc458c3740f2a1af55de808edb06 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:30.036628 1194 raft_consensus.cc:357] T fdbf03c89bff438a9a25115eddf60be9 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:30.036842 1194 raft_consensus.cc:383] T fdbf03c89bff438a9a25115eddf60be9 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:30.036892 1194 raft_consensus.cc:738] T fdbf03c89bff438a9a25115eddf60be9 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
W20250113 02:51:30.037079 709 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.3:42587, user_credentials={real_user=slave}} blocked reactor thread for 153828us
I20250113 02:51:30.037684 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet cd21f10d74c946a2872a84cadccf7950 with delete_type TABLET_DATA_DELETED (replaced by tablet 702e03e213b14ef5a1b64a02df1b3bf9) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:30.044404 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet cf7b8bd6801743f49e9b9473747447c5
I20250113 02:51:30.044934 447 catalog_manager.cc:6596] ReplaceTablet: tablet cf7b8bd6801743f49e9b9473747447c5 deleted and replaced by tablet a8aedbc8d7bb4183bb1ab9902f07bea7
I20250113 02:51:30.040536 1191 raft_consensus.cc:357] T 4c615b0b83d74fcfae0c51047dbfbbf4 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:30.046149 1191 raft_consensus.cc:383] T 4c615b0b83d74fcfae0c51047dbfbbf4 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:30.046238 1191 raft_consensus.cc:738] T 4c615b0b83d74fcfae0c51047dbfbbf4 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:30.046208 880 tablet_service.cc:1514] Processing DeleteTablet for tablet cf7b8bd6801743f49e9b9473747447c5 with delete_type TABLET_DATA_DELETED (replaced by tablet a8aedbc8d7bb4183bb1ab9902f07bea7) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:30.046434 1191 consensus_queue.cc:260] T 4c615b0b83d74fcfae0c51047dbfbbf4 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:30.046952 1191 ts_tablet_manager.cc:1428] T 4c615b0b83d74fcfae0c51047dbfbbf4 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.037s user 0.002s sys 0.000s
I20250113 02:51:30.047225 1191 tablet_bootstrap.cc:492] T de4a58bf63054890808e2e2fe46e43d2 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:30.048854 1191 tablet_bootstrap.cc:654] T de4a58bf63054890808e2e2fe46e43d2 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
W20250113 02:51:30.049140 454 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5f7c91d8c018455e847e5969f7b5e1cc consensus state has no leader
I20250113 02:51:30.049649 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet ef99c223dcf04b11bf785703df55f877
I20250113 02:51:30.051364 739 tablet_service.cc:1514] Processing DeleteTablet for tablet cf7b8bd6801743f49e9b9473747447c5 with delete_type TABLET_DATA_DELETED (replaced by tablet a8aedbc8d7bb4183bb1ab9902f07bea7) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:30.052497 880 tablet_service.cc:1514] Processing DeleteTablet for tablet ef99c223dcf04b11bf785703df55f877 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:29 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:30.052898 755 tablet_service.cc:1514] Processing DeleteTablet for tablet ef99c223dcf04b11bf785703df55f877 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:29 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:30.056159 1904 raft_consensus.cc:491] T 378d59d314a84ddb84cf2ab041c51cf1 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:30.056363 1904 raft_consensus.cc:513] T 378d59d314a84ddb84cf2ab041c51cf1 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:30.057103 1904 leader_election.cc:290] T 378d59d314a84ddb84cf2ab041c51cf1 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:30.057132 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet 092892a36e0e47cb8d2184c2bbde95d7 (table default.Table_03fafac4f68e4eb3be2ea41493d9e006 [id=d4bf205fb0464358af61bb3fb889c687]) successfully deleted
I20250113 02:51:30.057510 1195 tablet_replica.cc:331] T b4806b18a4194f9c95350d89ba62e13a P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:30.057660 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "378d59d314a84ddb84cf2ab041c51cf1" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:30.057755 1195 raft_consensus.cc:2238] T b4806b18a4194f9c95350d89ba62e13a P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:30.057884 1195 raft_consensus.cc:2267] T b4806b18a4194f9c95350d89ba62e13a P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:30.058141 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_770cceffb7e940c7a4dfad0fcab73863" } new_table_name: "default.Table_da161f6412074e37b23da57bd4febeee" modify_external_catalogs: true
I20250113 02:51:30.058367 1195 ts_tablet_manager.cc:1905] T b4806b18a4194f9c95350d89ba62e13a P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250113 02:51:30.059438 711 consensus_peers.cc:487] T e8ab4516789e4585a101c8766d338af3 P 1186fa869eb14812832fa6562531f196 -> Peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Couldn't send request to peer df69f48656b44bb890f76587ee69cc0f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250113 02:51:30.059722 1904 raft_consensus.cc:491] T 11950e996c814590bc9f3f22c0354902 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:30.063300 1904 raft_consensus.cc:513] T 11950e996c814590bc9f3f22c0354902 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:30.063966 1904 leader_election.cc:290] T 11950e996c814590bc9f3f22c0354902 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
W20250113 02:51:30.064277 711 leader_election.cc:343] T 378d59d314a84ddb84cf2ab041c51cf1 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:30.015143 886 tablet_service.cc:1514] Processing DeleteTablet for tablet cd21f10d74c946a2872a84cadccf7950 with delete_type TABLET_DATA_DELETED (replaced by tablet 702e03e213b14ef5a1b64a02df1b3bf9) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:30.016309 1196 ts_tablet_manager.cc:1918] T 601c71d5f7624fbb87b365d2e775d33e P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:30.017216 1438 raft_consensus.cc:491] T 8adfa738c17a4169a5d99f6ceca75100 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:30.067580 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11950e996c814590bc9f3f22c0354902" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:30.067658 1438 raft_consensus.cc:513] T 8adfa738c17a4169a5d99f6ceca75100 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:30.068337 1438 leader_election.cc:290] T 8adfa738c17a4169a5d99f6ceca75100 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587), 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
I20250113 02:51:30.068425 1062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c94eedb5dce44e6d896192c93127903a" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:30.017563 881 tablet_service.cc:1514] Processing DeleteTablet for tablet 572e4eed1e924381bada27f8389821a7 with delete_type TABLET_DATA_DELETED (replaced by tablet 29b1444efd28483297bafc73a33e7a54) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:30.068780 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet cf7b8bd6801743f49e9b9473747447c5 with delete_type TABLET_DATA_DELETED (replaced by tablet a8aedbc8d7bb4183bb1ab9902f07bea7) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:30.020318 899 tablet_service.cc:1514] Processing DeleteTablet for tablet 85a82ad6e93346ecb1a0cb38b62f5786 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:29 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:30.023825 896 tablet_service.cc:1514] Processing DeleteTablet for tablet 501d4e1ae84e4795913035f8a19cc32d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:18 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:30.069206 1196 log.cc:1198] T 601c71d5f7624fbb87b365d2e775d33e P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/601c71d5f7624fbb87b365d2e775d33e
I20250113 02:51:30.069612 1196 ts_tablet_manager.cc:1939] T 601c71d5f7624fbb87b365d2e775d33e P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:30.069933 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet 64aa8390b1b1475c8743b6491d40674c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:29 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:30.070677 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet 601c71d5f7624fbb87b365d2e775d33e (table default.Table_7a9c9cc8678e4f7fa12237d3f8c9c3de [id=97a27e2641d2468aac5c827fad4e2091]) successfully deleted
I20250113 02:51:30.071306 1196 tablet_replica.cc:331] T 572e4eed1e924381bada27f8389821a7 P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:30.071483 1196 raft_consensus.cc:2238] T 572e4eed1e924381bada27f8389821a7 P df69f48656b44bb890f76587ee69cc0f [term 1 FOLLOWER]: Raft consensus shutting down.
W20250113 02:51:30.071475 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 542df00229f645788f1b821c0ce00c46 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 542df00229f645788f1b821c0ce00c46 already in progress: creating tablet
I20250113 02:51:30.071645 1196 raft_consensus.cc:2267] T 572e4eed1e924381bada27f8389821a7 P df69f48656b44bb890f76587ee69cc0f [term 1 FOLLOWER]: Raft consensus is shut down!
W20250113 02:51:30.071671 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 64aa8390b1b1475c8743b6491d40674c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 64aa8390b1b1475c8743b6491d40674c already in progress: creating tablet
W20250113 02:51:30.071846 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet f4af42b9bae44e3fb009310212cec587 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f4af42b9bae44e3fb009310212cec587 already in progress: creating tablet
W20250113 02:51:30.072022 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet cf7b8bd6801743f49e9b9473747447c5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cf7b8bd6801743f49e9b9473747447c5 already in progress: creating tablet
I20250113 02:51:30.072155 1198 ts_tablet_manager.cc:1918] T 2c399784af46426b8327d175c7a6cce8 P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250113 02:51:30.072189 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet ef99c223dcf04b11bf785703df55f877 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ef99c223dcf04b11bf785703df55f877 already in progress: creating tablet
W20250113 02:51:30.072350 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet cd21f10d74c946a2872a84cadccf7950 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cd21f10d74c946a2872a84cadccf7950 already in progress: creating tablet
W20250113 02:51:30.072615 851 leader_election.cc:343] T 501d4e1ae84e4795913035f8a19cc32d P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Not found: Tablet not found: 501d4e1ae84e4795913035f8a19cc32d
I20250113 02:51:30.072724 851 leader_election.cc:304] T 501d4e1ae84e4795913035f8a19cc32d P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df69f48656b44bb890f76587ee69cc0f; no voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca
I20250113 02:51:30.073091 1438 raft_consensus.cc:2743] T 501d4e1ae84e4795913035f8a19cc32d P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:30.073279 1196 ts_tablet_manager.cc:1905] T 572e4eed1e924381bada27f8389821a7 P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:30.073354 1887 raft_consensus.cc:491] T 9fff52e99c64462a8a73fccc3c22ca63 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:30.073464 1887 raft_consensus.cc:513] T 9fff52e99c64462a8a73fccc3c22ca63 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:30.073966 1887 leader_election.cc:290] T 9fff52e99c64462a8a73fccc3c22ca63 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:30.077253 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8adfa738c17a4169a5d99f6ceca75100" candidate_uuid: "df69f48656b44bb890f76587ee69cc0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
I20250113 02:51:30.078457 450 master_service.cc:946] ReplaceTablet: received request to replace tablet 51d9483ecacb413796d0fd9e7560eac1 from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:30.078495 1031 tablet_service.cc:1514] Processing DeleteTablet for tablet 85a82ad6e93346ecb1a0cb38b62f5786 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:29 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:30.080639 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d63f0e5835540a78c82020b20a07dac" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:30.081135 1041 tablet_service.cc:1514] Processing DeleteTablet for tablet 572e4eed1e924381bada27f8389821a7 with delete_type TABLET_DATA_DELETED (replaced by tablet 29b1444efd28483297bafc73a33e7a54) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:30.081872 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet ef99c223dcf04b11bf785703df55f877 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:29 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:30.083406 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fff52e99c64462a8a73fccc3c22ca63" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:30.086283 1191 tablet_bootstrap.cc:492] T de4a58bf63054890808e2e2fe46e43d2 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:30.086458 1191 ts_tablet_manager.cc:1397] T de4a58bf63054890808e2e2fe46e43d2 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.039s user 0.002s sys 0.004s
I20250113 02:51:30.087172 1191 raft_consensus.cc:357] T de4a58bf63054890808e2e2fe46e43d2 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:30.087347 1191 raft_consensus.cc:383] T de4a58bf63054890808e2e2fe46e43d2 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:30.087417 1191 raft_consensus.cc:738] T de4a58bf63054890808e2e2fe46e43d2 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:30.087581 1191 consensus_queue.cc:260] T de4a58bf63054890808e2e2fe46e43d2 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:30.088234 1191 ts_tablet_manager.cc:1428] T de4a58bf63054890808e2e2fe46e43d2 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250113 02:51:30.088496 1191 tablet_bootstrap.cc:492] T 0810b9671f824d0a84741bc1904df247 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:30.091136 1191 tablet_bootstrap.cc:654] T 0810b9671f824d0a84741bc1904df247 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
W20250113 02:51:30.092345 711 leader_election.cc:343] T 11950e996c814590bc9f3f22c0354902 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
W20250113 02:51:30.092615 711 leader_election.cc:343] T 9fff52e99c64462a8a73fccc3c22ca63 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:30.094355 1062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8adfa738c17a4169a5d99f6ceca75100" candidate_uuid: "df69f48656b44bb890f76587ee69cc0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:30.094888 1198 log.cc:1198] T 2c399784af46426b8327d175c7a6cce8 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/2c399784af46426b8327d175c7a6cce8
I20250113 02:51:30.094851 852 leader_election.cc:304] T 8adfa738c17a4169a5d99f6ceca75100 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df69f48656b44bb890f76587ee69cc0f; no voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca
I20250113 02:51:30.095899 1198 ts_tablet_manager.cc:1939] T 2c399784af46426b8327d175c7a6cce8 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:30.098044 1634 raft_consensus.cc:491] T 378d59d314a84ddb84cf2ab041c51cf1 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:30.098218 1634 raft_consensus.cc:513] T 378d59d314a84ddb84cf2ab041c51cf1 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:30.099047 1634 leader_election.cc:290] T 378d59d314a84ddb84cf2ab041c51cf1 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
I20250113 02:51:30.100018 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "378d59d314a84ddb84cf2ab041c51cf1" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:30.100620 998 leader_election.cc:343] T 378d59d314a84ddb84cf2ab041c51cf1 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:30.097630 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11950e996c814590bc9f3f22c0354902" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:30.102128 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "378d59d314a84ddb84cf2ab041c51cf1" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:30.102586 1059 raft_consensus.cc:2463] T 378d59d314a84ddb84cf2ab041c51cf1 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1186fa869eb14812832fa6562531f196 in term 0.
W20250113 02:51:30.105028 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 656a0dda5c0b46309ed472d34a42ff94 consensus state has no leader
I20250113 02:51:30.105269 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet 2c399784af46426b8327d175c7a6cce8 (table default.Table_f8af5d3e0dc64d3fae054f82a7b33ccf [id=4b739228564344b8b394a4718750dbb0]) successfully deleted
I20250113 02:51:30.105358 1198 tablet_replica.cc:331] T 63efcb6b2e194c469fb30a197bd6565b P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:30.105544 1198 raft_consensus.cc:2238] T 63efcb6b2e194c469fb30a197bd6565b P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: Raft consensus shutting down.
W20250113 02:51:30.105546 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet cd6dcfecf4324542b241f909926756ea because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cd6dcfecf4324542b241f909926756ea already in progress: creating tablet
I20250113 02:51:30.105818 1198 pending_rounds.cc:62] T 63efcb6b2e194c469fb30a197bd6565b P 4216a7f853d642b7bc3e5cff458f3aca: Trying to abort 1 pending ops.
W20250113 02:51:30.105829 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet e468b652f71f44e3945f8d919ac0e1a8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e468b652f71f44e3945f8d919ac0e1a8 already in progress: creating tablet
I20250113 02:51:30.105885 1198 pending_rounds.cc:69] T 63efcb6b2e194c469fb30a197bd6565b P 4216a7f853d642b7bc3e5cff458f3aca: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113673480400355328 op_type: NO_OP noop_request { }
I20250113 02:51:30.108307 1198 raft_consensus.cc:2883] T 63efcb6b2e194c469fb30a197bd6565b P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250113 02:51:30.108522 1198 raft_consensus.cc:2267] T 63efcb6b2e194c469fb30a197bd6565b P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:30.109403 1198 ts_tablet_manager.cc:1905] T 63efcb6b2e194c469fb30a197bd6565b P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:30.113214 1917 raft_consensus.cc:2743] T 8adfa738c17a4169a5d99f6ceca75100 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:30.116279 1192 tablet_bootstrap.cc:492] T a0ca2e3a3ce94f3aa20df64c89c7cf95 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:30.116439 1192 ts_tablet_manager.cc:1397] T a0ca2e3a3ce94f3aa20df64c89c7cf95 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.175s user 0.003s sys 0.008s
I20250113 02:51:30.117192 1192 raft_consensus.cc:357] T a0ca2e3a3ce94f3aa20df64c89c7cf95 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:30.117388 1192 raft_consensus.cc:383] T a0ca2e3a3ce94f3aa20df64c89c7cf95 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:30.117471 1192 raft_consensus.cc:738] T a0ca2e3a3ce94f3aa20df64c89c7cf95 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:30.117672 1192 consensus_queue.cc:260] T a0ca2e3a3ce94f3aa20df64c89c7cf95 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:30.118191 1192 ts_tablet_manager.cc:1428] T a0ca2e3a3ce94f3aa20df64c89c7cf95 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250113 02:51:30.118453 1192 tablet_bootstrap.cc:492] T c94eedb5dce44e6d896192c93127903a P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:30.120194 1192 tablet_bootstrap.cc:654] T c94eedb5dce44e6d896192c93127903a P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:30.123955 1194 consensus_queue.cc:260] T fdbf03c89bff438a9a25115eddf60be9 P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
W20250113 02:51:30.125496 709 leader_election.cc:343] T c94eedb5dce44e6d896192c93127903a P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:30.126497 709 leader_election.cc:304] T c94eedb5dce44e6d896192c93127903a P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:30.126725 1438 raft_consensus.cc:491] T fdbf03c89bff438a9a25115eddf60be9 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:30.126843 1438 raft_consensus.cc:513] T fdbf03c89bff438a9a25115eddf60be9 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:30.129398 1194 ts_tablet_manager.cc:1428] T fdbf03c89bff438a9a25115eddf60be9 P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.094s user 0.002s sys 0.000s
I20250113 02:51:30.129824 1887 raft_consensus.cc:2743] T c94eedb5dce44e6d896192c93127903a P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250113 02:51:30.130066 709 leader_election.cc:343] T 6d63f0e5835540a78c82020b20a07dac P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:30.132905 1194 tablet_bootstrap.cc:492] T 258fad818b7b4aa4b337f9949e20aa46 P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:30.136451 896 tablet_service.cc:1514] Processing DeleteTablet for tablet fdbf03c89bff438a9a25115eddf60be9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:19 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:30.137121 1198 ts_tablet_manager.cc:1918] T 63efcb6b2e194c469fb30a197bd6565b P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250113 02:51:30.137297 1198 log.cc:1198] T 63efcb6b2e194c469fb30a197bd6565b P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/63efcb6b2e194c469fb30a197bd6565b
I20250113 02:51:30.137781 1198 ts_tablet_manager.cc:1939] T 63efcb6b2e194c469fb30a197bd6565b P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:30.139964 1438 leader_election.cc:290] T fdbf03c89bff438a9a25115eddf60be9 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587), 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
W20250113 02:51:30.142099 852 leader_election.cc:343] T fdbf03c89bff438a9a25115eddf60be9 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Not found: Tablet not found: fdbf03c89bff438a9a25115eddf60be9
I20250113 02:51:30.143420 1195 ts_tablet_manager.cc:1918] T b4806b18a4194f9c95350d89ba62e13a P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:30.143558 1195 log.cc:1198] T b4806b18a4194f9c95350d89ba62e13a P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/b4806b18a4194f9c95350d89ba62e13a
I20250113 02:51:30.143890 1195 ts_tablet_manager.cc:1939] T b4806b18a4194f9c95350d89ba62e13a P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:30.145051 1195 tablet_replica.cc:331] T 6d63f0e5835540a78c82020b20a07dac P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:30.145310 1195 raft_consensus.cc:2238] T 6d63f0e5835540a78c82020b20a07dac P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus shutting down.
W20250113 02:51:30.149005 451 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9798a2b066ea4caaa6761d62644b34e9 consensus state has no leader
I20250113 02:51:30.149173 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet b4806b18a4194f9c95350d89ba62e13a (table default.Table_4369a143c72643b084df58d25314c6dd [id=36ff239c08a54be99ca307354810fff3]) successfully deleted
I20250113 02:51:30.149914 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 79aaab73797c474bae06831fa308b066
I20250113 02:51:30.151247 1195 raft_consensus.cc:2267] T 6d63f0e5835540a78c82020b20a07dac P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:30.151381 896 tablet_service.cc:1514] Processing DeleteTablet for tablet 79aaab73797c474bae06831fa308b066 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:30 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:30.152019 1195 ts_tablet_manager.cc:1905] T 6d63f0e5835540a78c82020b20a07dac P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:30.145130 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "378d59d314a84ddb84cf2ab041c51cf1" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
I20250113 02:51:30.153200 736 tablet_service.cc:1514] Processing DeleteTablet for tablet 79aaab73797c474bae06831fa308b066 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:30 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:30.153244 772 raft_consensus.cc:2463] T 378d59d314a84ddb84cf2ab041c51cf1 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 0.
I20250113 02:51:30.130204 709 leader_election.cc:304] T 6d63f0e5835540a78c82020b20a07dac P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
W20250113 02:51:30.153964 709 raft_consensus.cc:2681] T 6d63f0e5835540a78c82020b20a07dac P 1186fa869eb14812832fa6562531f196: unable to run election callback: Service unavailable: Thread pool token was shut down
W20250113 02:51:30.154129 851 leader_election.cc:343] T fdbf03c89bff438a9a25115eddf60be9 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Not found: Tablet not found: fdbf03c89bff438a9a25115eddf60be9
I20250113 02:51:30.154237 851 leader_election.cc:304] T fdbf03c89bff438a9a25115eddf60be9 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df69f48656b44bb890f76587ee69cc0f; no voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca
I20250113 02:51:30.152928 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet 79aaab73797c474bae06831fa308b066 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:30 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:30.153750 995 leader_election.cc:304] T 378d59d314a84ddb84cf2ab041c51cf1 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters: df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:30.154620 1438 raft_consensus.cc:2743] T fdbf03c89bff438a9a25115eddf60be9 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:30.154966 1634 raft_consensus.cc:2798] T 378d59d314a84ddb84cf2ab041c51cf1 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 02:51:30.155100 1634 raft_consensus.cc:491] T 378d59d314a84ddb84cf2ab041c51cf1 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 02:51:30.155217 1634 raft_consensus.cc:3054] T 378d59d314a84ddb84cf2ab041c51cf1 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:30.158167 1634 raft_consensus.cc:513] T 378d59d314a84ddb84cf2ab041c51cf1 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:30.158118 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_c85e49def4fb4cbbbb3eebc2aced958e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:30.158836 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c85e49def4fb4cbbbb3eebc2aced958e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:30.158922 1196 ts_tablet_manager.cc:1918] T 572e4eed1e924381bada27f8389821a7 P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250113 02:51:30.159026 1196 log.cc:1198] T 572e4eed1e924381bada27f8389821a7 P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/572e4eed1e924381bada27f8389821a7
I20250113 02:51:30.159200 446 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_216f6b44551d41d49fa21b862e08aee1" } modify_external_catalogs: true
I20250113 02:51:30.159431 1196 ts_tablet_manager.cc:1939] T 572e4eed1e924381bada27f8389821a7 P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:30.159402 446 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_216f6b44551d41d49fa21b862e08aee1" } modify_external_catalogs: true
I20250113 02:51:30.160053 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_720678f42d2849259edbacc2f1a9c24b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:30.160471 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_720678f42d2849259edbacc2f1a9c24b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:30.160979 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_28444afd72804ea5bc5099ae319b1321"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:30.161408 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_28444afd72804ea5bc5099ae319b1321 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:30.161828 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "378d59d314a84ddb84cf2ab041c51cf1" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f"
W20250113 02:51:30.162277 998 leader_election.cc:343] T 378d59d314a84ddb84cf2ab041c51cf1 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:30.162488 447 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_4e6b1d81ffef42c3a7f83bf18c1fbc48" } modify_external_catalogs: true
I20250113 02:51:30.162642 447 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_4e6b1d81ffef42c3a7f83bf18c1fbc48" } modify_external_catalogs: true
I20250113 02:51:30.162837 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet 572e4eed1e924381bada27f8389821a7 (table default.Table_1d910c3d2b244474a576fc4088301143 [id=75cb0745bbb84e1c883cb13b546ecb11]) successfully deleted
W20250113 02:51:30.163101 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 85a82ad6e93346ecb1a0cb38b62f5786 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 85a82ad6e93346ecb1a0cb38b62f5786 already in progress: creating tablet
I20250113 02:51:30.164552 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_b8e04994e6894fe4a4ff012d0050eb6c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:30.165035 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b8e04994e6894fe4a4ff012d0050eb6c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:30.166139 1634 leader_election.cc:290] T 378d59d314a84ddb84cf2ab041c51cf1 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Requested vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
W20250113 02:51:30.166513 709 leader_election.cc:343] T 11950e996c814590bc9f3f22c0354902 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:30.167111 709 leader_election.cc:304] T 11950e996c814590bc9f3f22c0354902 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:30.167605 1887 raft_consensus.cc:2743] T 11950e996c814590bc9f3f22c0354902 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:30.170823 1194 tablet_bootstrap.cc:654] T 258fad818b7b4aa4b337f9949e20aa46 P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:30.168220 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "378d59d314a84ddb84cf2ab041c51cf1" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196"
I20250113 02:51:30.172103 772 raft_consensus.cc:3054] T 378d59d314a84ddb84cf2ab041c51cf1 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:30.172161 1196 tablet_replica.cc:331] T 501d4e1ae84e4795913035f8a19cc32d P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:30.172374 1196 raft_consensus.cc:2238] T 501d4e1ae84e4795913035f8a19cc32d P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:30.172526 1196 raft_consensus.cc:2267] T 501d4e1ae84e4795913035f8a19cc32d P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:30.174628 772 raft_consensus.cc:2463] T 378d59d314a84ddb84cf2ab041c51cf1 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 1.
I20250113 02:51:30.175326 995 leader_election.cc:304] T 378d59d314a84ddb84cf2ab041c51cf1 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters: df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:30.176656 1836 raft_consensus.cc:2798] T 378d59d314a84ddb84cf2ab041c51cf1 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Leader election won for term 1
I20250113 02:51:30.176738 1195 ts_tablet_manager.cc:1918] T 6d63f0e5835540a78c82020b20a07dac P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:30.176877 1195 log.cc:1198] T 6d63f0e5835540a78c82020b20a07dac P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/6d63f0e5835540a78c82020b20a07dac
I20250113 02:51:30.177255 1195 ts_tablet_manager.cc:1939] T 6d63f0e5835540a78c82020b20a07dac P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:30.178206 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet 6d63f0e5835540a78c82020b20a07dac (table default.Table_af3ebf543c9448b7ae473d721a23d47e [id=93c6f37549ce4544bc0cb4a267f44b00]) successfully deleted
I20250113 02:51:30.178539 1195 tablet_replica.cc:331] T f68f9f4076d9475d938e795e72bdd468 P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:30.178768 1195 raft_consensus.cc:2238] T f68f9f4076d9475d938e795e72bdd468 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:30.179020 1195 raft_consensus.cc:2267] T f68f9f4076d9475d938e795e72bdd468 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:30.179684 1195 ts_tablet_manager.cc:1905] T f68f9f4076d9475d938e795e72bdd468 P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:30.184047 1196 ts_tablet_manager.cc:1905] T 501d4e1ae84e4795913035f8a19cc32d P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:30.185719 1195 ts_tablet_manager.cc:1918] T f68f9f4076d9475d938e795e72bdd468 P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:30.187273 1192 tablet_bootstrap.cc:492] T c94eedb5dce44e6d896192c93127903a P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:30.187439 1192 ts_tablet_manager.cc:1397] T c94eedb5dce44e6d896192c93127903a P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.069s user 0.008s sys 0.023s
I20250113 02:51:30.188189 1192 raft_consensus.cc:357] T c94eedb5dce44e6d896192c93127903a P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:30.188422 1192 raft_consensus.cc:383] T c94eedb5dce44e6d896192c93127903a P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:30.188503 1192 raft_consensus.cc:738] T c94eedb5dce44e6d896192c93127903a P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:30.188757 1192 consensus_queue.cc:260] T c94eedb5dce44e6d896192c93127903a P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:30.189445 1194 tablet_bootstrap.cc:492] T 258fad818b7b4aa4b337f9949e20aa46 P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:30.189646 1194 ts_tablet_manager.cc:1397] T 258fad818b7b4aa4b337f9949e20aa46 P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.058s user 0.004s sys 0.000s
I20250113 02:51:30.189671 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet 63efcb6b2e194c469fb30a197bd6565b (table default.Table_a6447976a51a4e2f93d22bd1c657c49d [id=0485ff4fda424e97bb71b77d657ab5e6]) successfully deleted
I20250113 02:51:30.190059 1198 tablet_replica.cc:331] T c215ffe61b9b426286cc5e808463078d P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:30.190389 1194 raft_consensus.cc:357] T 258fad818b7b4aa4b337f9949e20aa46 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:30.190493 1198 raft_consensus.cc:2238] T c215ffe61b9b426286cc5e808463078d P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:30.190593 1194 raft_consensus.cc:383] T 258fad818b7b4aa4b337f9949e20aa46 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:30.190646 1194 raft_consensus.cc:738] T 258fad818b7b4aa4b337f9949e20aa46 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:30.190829 1194 consensus_queue.cc:260] T 258fad818b7b4aa4b337f9949e20aa46 P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:30.167799 709 leader_election.cc:304] T 378d59d314a84ddb84cf2ab041c51cf1 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters: df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:30.192875 1887 raft_consensus.cc:2758] T 378d59d314a84ddb84cf2ab041c51cf1 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20250113 02:51:30.193248 1198 raft_consensus.cc:2267] T c215ffe61b9b426286cc5e808463078d P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:30.193547 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fff52e99c64462a8a73fccc3c22ca63" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:30.193755 1059 raft_consensus.cc:2463] T 9fff52e99c64462a8a73fccc3c22ca63 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1186fa869eb14812832fa6562531f196 in term 0.
I20250113 02:51:30.193815 1887 raft_consensus.cc:491] T ddf79eec41214693859f526265a55492 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:30.193915 1198 ts_tablet_manager.cc:1905] T c215ffe61b9b426286cc5e808463078d P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:30.193930 1887 raft_consensus.cc:513] T ddf79eec41214693859f526265a55492 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:30.194572 1887 leader_election.cc:290] T ddf79eec41214693859f526265a55492 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:30.195433 1904 raft_consensus.cc:491] T 401c4f5dbd304950877b59a865f417b3 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:30.195524 1904 raft_consensus.cc:513] T 401c4f5dbd304950877b59a865f417b3 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:30.196913 1904 leader_election.cc:290] T 401c4f5dbd304950877b59a865f417b3 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:30.197660 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_6df0545f12364f96a61aafaa9e3ff8f5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:30.198364 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6df0545f12364f96a61aafaa9e3ff8f5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:30.200798 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_bbd4259916374725a0e633d94ed73eb4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:30.202481 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bbd4259916374725a0e633d94ed73eb4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:30.204260 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddf79eec41214693859f526265a55492" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:30.204453 1061 raft_consensus.cc:2463] T ddf79eec41214693859f526265a55492 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1186fa869eb14812832fa6562531f196 in term 0.
I20250113 02:51:30.204635 709 leader_election.cc:304] T 9fff52e99c64462a8a73fccc3c22ca63 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters: df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:30.204732 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "401c4f5dbd304950877b59a865f417b3" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:30.204885 1059 raft_consensus.cc:2463] T 401c4f5dbd304950877b59a865f417b3 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1186fa869eb14812832fa6562531f196 in term 0.
I20250113 02:51:30.205313 1906 tablet.cc:1721] T 8adfa738c17a4169a5d99f6ceca75100 P 4216a7f853d642b7bc3e5cff458f3aca: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250113 02:51:30.205410 1194 ts_tablet_manager.cc:1428] T 258fad818b7b4aa4b337f9949e20aa46 P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.016s user 0.001s sys 0.001s
I20250113 02:51:30.205542 1904 raft_consensus.cc:2798] T 9fff52e99c64462a8a73fccc3c22ca63 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 02:51:30.205641 1904 raft_consensus.cc:491] T 9fff52e99c64462a8a73fccc3c22ca63 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 02:51:30.205718 1904 raft_consensus.cc:3054] T 9fff52e99c64462a8a73fccc3c22ca63 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:30.209084 1192 ts_tablet_manager.cc:1428] T c94eedb5dce44e6d896192c93127903a P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.022s user 0.004s sys 0.000s
I20250113 02:51:30.209429 1192 tablet_bootstrap.cc:492] T 32cba8f3efc54e78b4df41c15df82eea P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:30.209492 1887 consensus_queue.cc:1035] T e8ab4516789e4585a101c8766d338af3 P 1186fa869eb14812832fa6562531f196 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 02:51:30.205168 1836 raft_consensus.cc:695] T 378d59d314a84ddb84cf2ab041c51cf1 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: Becoming Leader. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Running, Role: LEADER
I20250113 02:51:30.209926 1836 consensus_queue.cc:237] T 378d59d314a84ddb84cf2ab041c51cf1 P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:30.210702 1192 tablet_bootstrap.cc:654] T 32cba8f3efc54e78b4df41c15df82eea P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:30.211668 709 leader_election.cc:304] T ddf79eec41214693859f526265a55492 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters:
I20250113 02:51:30.211745 1904 raft_consensus.cc:513] T 9fff52e99c64462a8a73fccc3c22ca63 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:30.212572 1904 leader_election.cc:290] T 9fff52e99c64462a8a73fccc3c22ca63 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 election: Requested vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:30.212952 709 leader_election.cc:304] T 401c4f5dbd304950877b59a865f417b3 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters:
I20250113 02:51:30.213481 1887 raft_consensus.cc:2798] T 401c4f5dbd304950877b59a865f417b3 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 02:51:30.213254 1904 raft_consensus.cc:2798] T ddf79eec41214693859f526265a55492 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 02:51:30.213551 1887 raft_consensus.cc:491] T 401c4f5dbd304950877b59a865f417b3 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 02:51:30.213696 1887 raft_consensus.cc:3054] T 401c4f5dbd304950877b59a865f417b3 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:30.217212 1887 raft_consensus.cc:513] T 401c4f5dbd304950877b59a865f417b3 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:30.217526 1904 raft_consensus.cc:491] T ddf79eec41214693859f526265a55492 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 02:51:30.218443 1904 raft_consensus.cc:3054] T ddf79eec41214693859f526265a55492 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:30.220280 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_c85e49def4fb4cbbbb3eebc2aced958e" } new_table_name: "default.Table_ef81f67732f246589e2e2f2c8808dc00" modify_external_catalogs: true
I20250113 02:51:30.221479 448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_9c812746e36d4df38dd75d3a561d7d57" } new_table_name: "default.Table_5627e75046d44ac98b829afa751f42d0" modify_external_catalogs: true
I20250113 02:51:30.222023 1872 tablet.cc:1721] T 4fe54131aafb40ee9d8bc5ea299af193 P 1186fa869eb14812832fa6562531f196: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20250113 02:51:30.222779 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_ad3bb183b90444b8a9ca111bbc2d4d2a" } modify_external_catalogs: true
I20250113 02:51:30.222975 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_ad3bb183b90444b8a9ca111bbc2d4d2a" } modify_external_catalogs: true
I20250113 02:51:30.223212 1926 raft_consensus.cc:491] T c94eedb5dce44e6d896192c93127903a P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:30.223342 1926 raft_consensus.cc:513] T c94eedb5dce44e6d896192c93127903a P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:30.223711 1906 tablet.cc:1721] T 4fe54131aafb40ee9d8bc5ea299af193 P 4216a7f853d642b7bc3e5cff458f3aca: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20250113 02:51:30.223929 1887 leader_election.cc:290] T 401c4f5dbd304950877b59a865f417b3 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 election: Requested vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:30.224651 1926 leader_election.cc:290] T c94eedb5dce44e6d896192c93127903a P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
I20250113 02:51:30.224922 1904 raft_consensus.cc:513] T ddf79eec41214693859f526265a55492 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:30.226804 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fff52e99c64462a8a73fccc3c22ca63" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca"
I20250113 02:51:30.227011 1061 raft_consensus.cc:3054] T 9fff52e99c64462a8a73fccc3c22ca63 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:30.227593 1062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "401c4f5dbd304950877b59a865f417b3" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca"
I20250113 02:51:30.227739 1062 raft_consensus.cc:3054] T 401c4f5dbd304950877b59a865f417b3 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:30.229645 1061 raft_consensus.cc:2463] T 9fff52e99c64462a8a73fccc3c22ca63 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1186fa869eb14812832fa6562531f196 in term 1.
I20250113 02:51:30.230201 1062 raft_consensus.cc:2463] T 401c4f5dbd304950877b59a865f417b3 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1186fa869eb14812832fa6562531f196 in term 1.
W20250113 02:51:30.231488 442 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 401c4f5dbd304950877b59a865f417b3 (table default.Table_c991f9dbd70d4c4682664b8711f05db4 [id=2c64da64ecdb4bf48e11b523533902e8], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 401c4f5dbd304950877b59a865f417b3 consensus state has no leader
I20250113 02:51:30.232015 455 catalog_manager.cc:5618] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587) does not have the latest schema for tablet 9fff52e99c64462a8a73fccc3c22ca63 (table default.Table_9ef18ee71ad843c5921750a2f8478855 [id=bd8bd572ed524f2cae01f820f07e3c36]). Expected version 2 got 0
I20250113 02:51:30.232239 455 catalog_manager.cc:5618] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587) does not have the latest schema for tablet 401c4f5dbd304950877b59a865f417b3 (table default.Table_c991f9dbd70d4c4682664b8711f05db4 [id=2c64da64ecdb4bf48e11b523533902e8]). Expected version 1 got 0
I20250113 02:51:30.232470 1062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddf79eec41214693859f526265a55492" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca"
I20250113 02:51:30.232626 1062 raft_consensus.cc:3054] T ddf79eec41214693859f526265a55492 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:30.233713 454 catalog_manager.cc:5618] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095) does not have the latest schema for tablet 1cfbcd682b944c89847b40fe69f0a75e (table default.Table_32e88abaae96492fbcce39fb2cd7c221 [id=2b84deb07e5c4ab8b550fe42945d7b71]). Expected version 2 got 0
I20250113 02:51:30.228430 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c94eedb5dce44e6d896192c93127903a" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
I20250113 02:51:30.233919 454 catalog_manager.cc:5618] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095) does not have the latest schema for tablet 4c615b0b83d74fcfae0c51047dbfbbf4 (table default.Table_bf689f8eb8884e88a72e9b50598fc389 [id=a956f95f6dd94b578f6bd3cb3b97f9da]). Expected version 2 got 0
I20250113 02:51:30.233973 773 raft_consensus.cc:2463] T c94eedb5dce44e6d896192c93127903a P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 0.
I20250113 02:51:30.234380 995 leader_election.cc:304] T c94eedb5dce44e6d896192c93127903a P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters:
I20250113 02:51:30.235411 1062 raft_consensus.cc:2463] T ddf79eec41214693859f526265a55492 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1186fa869eb14812832fa6562531f196 in term 1.
I20250113 02:51:30.235849 709 leader_election.cc:304] T 9fff52e99c64462a8a73fccc3c22ca63 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters:
I20250113 02:51:30.236301 709 leader_election.cc:304] T 401c4f5dbd304950877b59a865f417b3 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters:
I20250113 02:51:30.236487 1887 raft_consensus.cc:2798] T 9fff52e99c64462a8a73fccc3c22ca63 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Leader election won for term 1
I20250113 02:51:30.236049 1904 leader_election.cc:290] T ddf79eec41214693859f526265a55492 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 election: Requested vote from peers 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:30.236862 1863 raft_consensus.cc:2798] T 401c4f5dbd304950877b59a865f417b3 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Leader election won for term 1
I20250113 02:51:30.236984 1863 raft_consensus.cc:695] T 401c4f5dbd304950877b59a865f417b3 P 1186fa869eb14812832fa6562531f196 [term 1 LEADER]: Becoming Leader. State: Replica: 1186fa869eb14812832fa6562531f196, State: Running, Role: LEADER
I20250113 02:51:30.237341 1863 consensus_queue.cc:237] T 401c4f5dbd304950877b59a865f417b3 P 1186fa869eb14812832fa6562531f196 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:30.205542 1195 log.cc:1198] T f68f9f4076d9475d938e795e72bdd468 P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/f68f9f4076d9475d938e795e72bdd468
I20250113 02:51:30.239078 1195 ts_tablet_manager.cc:1939] T f68f9f4076d9475d938e795e72bdd468 P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
W20250113 02:51:30.239957 448 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 37b7e05f7bbd4a4180cc40e76868469b consensus state has no leader
W20250113 02:51:30.240453 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9fa6f11e7a86410099069389bbdec6c3 has no consensus state
I20250113 02:51:30.241164 896 tablet_service.cc:1467] Processing CreateTablet for tablet 370c53f565df4e6f915e54a2fd2bf2ce (DEFAULT_TABLE table=default.Table_43fca7e946094914badafec91de2b53e [id=751e9d5139104d139fd94b5e4bd35d94]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:30.243466 1887 raft_consensus.cc:695] T 9fff52e99c64462a8a73fccc3c22ca63 P 1186fa869eb14812832fa6562531f196 [term 1 LEADER]: Becoming Leader. State: Replica: 1186fa869eb14812832fa6562531f196, State: Running, Role: LEADER
I20250113 02:51:30.243870 1887 consensus_queue.cc:237] T 9fff52e99c64462a8a73fccc3c22ca63 P 1186fa869eb14812832fa6562531f196 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:30.245079 709 leader_election.cc:304] T ddf79eec41214693859f526265a55492 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters:
I20250113 02:51:30.245731 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 51d9483ecacb413796d0fd9e7560eac1
I20250113 02:51:30.245785 1904 raft_consensus.cc:2798] T ddf79eec41214693859f526265a55492 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Leader election won for term 1
I20250113 02:51:30.246099 1904 raft_consensus.cc:695] T ddf79eec41214693859f526265a55492 P 1186fa869eb14812832fa6562531f196 [term 1 LEADER]: Becoming Leader. State: Replica: 1186fa869eb14812832fa6562531f196, State: Running, Role: LEADER
I20250113 02:51:30.246292 450 catalog_manager.cc:6596] ReplaceTablet: tablet 51d9483ecacb413796d0fd9e7560eac1 deleted and replaced by tablet 8898e1e9efce453488fa4a4562a82127
I20250113 02:51:30.246564 899 tablet_service.cc:1514] Processing DeleteTablet for tablet 51d9483ecacb413796d0fd9e7560eac1 with delete_type TABLET_DATA_DELETED (replaced by tablet 8898e1e9efce453488fa4a4562a82127) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:30.247192 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 65f42cda151744c3bd3b0ec9415253c6
I20250113 02:51:30.247361 1033 tablet_service.cc:1514] Processing DeleteTablet for tablet 51d9483ecacb413796d0fd9e7560eac1 with delete_type TABLET_DATA_DELETED (replaced by tablet 8898e1e9efce453488fa4a4562a82127) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:30.247520 736 tablet_service.cc:1514] Processing DeleteTablet for tablet 51d9483ecacb413796d0fd9e7560eac1 with delete_type TABLET_DATA_DELETED (replaced by tablet 8898e1e9efce453488fa4a4562a82127) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:30.246320 1904 consensus_queue.cc:237] T ddf79eec41214693859f526265a55492 P 1186fa869eb14812832fa6562531f196 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:30.247817 899 tablet_service.cc:1514] Processing DeleteTablet for tablet 65f42cda151744c3bd3b0ec9415253c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:30 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:30.248286 736 tablet_service.cc:1514] Processing DeleteTablet for tablet 65f42cda151744c3bd3b0ec9415253c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:30 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:30.248907 1033 tablet_service.cc:1514] Processing DeleteTablet for tablet 65f42cda151744c3bd3b0ec9415253c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:30 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:30.249258 1025 tablet_service.cc:1467] Processing CreateTablet for tablet 370c53f565df4e6f915e54a2fd2bf2ce (DEFAULT_TABLE table=default.Table_43fca7e946094914badafec91de2b53e [id=751e9d5139104d139fd94b5e4bd35d94]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:30.249976 736 tablet_service.cc:1467] Processing CreateTablet for tablet 370c53f565df4e6f915e54a2fd2bf2ce (DEFAULT_TABLE table=default.Table_43fca7e946094914badafec91de2b53e [id=751e9d5139104d139fd94b5e4bd35d94]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:30.250428 736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 370c53f565df4e6f915e54a2fd2bf2ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:30.250459 899 tablet_service.cc:1467] Processing CreateTablet for tablet a58588b1befe48a8a1418b788a227519 (DEFAULT_TABLE table=default.Table_2f84c2362a2b4da7b088342232749760 [id=8986254e9ec14780874f0e61bb6924ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:30.250826 1196 ts_tablet_manager.cc:1918] T 501d4e1ae84e4795913035f8a19cc32d P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:30.250921 1196 log.cc:1198] T 501d4e1ae84e4795913035f8a19cc32d P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/501d4e1ae84e4795913035f8a19cc32d
I20250113 02:51:30.251204 1196 ts_tablet_manager.cc:1939] T 501d4e1ae84e4795913035f8a19cc32d P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:30.252727 1033 tablet_service.cc:1467] Processing CreateTablet for tablet a58588b1befe48a8a1418b788a227519 (DEFAULT_TABLE table=default.Table_2f84c2362a2b4da7b088342232749760 [id=8986254e9ec14780874f0e61bb6924ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:30.253616 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet f68f9f4076d9475d938e795e72bdd468 (table default.Table_92ea541283614df981d96c5b74362f1d [id=9e58548d1a184cc0b362f148c44005ed]) successfully deleted
W20250113 02:51:30.256732 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 3d85539cf0074a03b167d3279e940079 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3d85539cf0074a03b167d3279e940079 already in progress: creating tablet
I20250113 02:51:30.257115 1195 tablet_replica.cc:331] T ddf79eec41214693859f526265a55492 P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:30.257486 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddf79eec41214693859f526265a55492" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:30.258059 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "401c4f5dbd304950877b59a865f417b3" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:30.258307 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fff52e99c64462a8a73fccc3c22ca63" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f"
I20250113 02:51:30.258513 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "401c4f5dbd304950877b59a865f417b3" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f"
I20250113 02:51:30.258754 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c94eedb5dce44e6d896192c93127903a" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:30.259410 1040 tablet_service.cc:1467] Processing CreateTablet for tablet 1c77033ca393469e9324d2406a65a5e9 (DEFAULT_TABLE table=default.Table_c59e24fdd40849b48b0b75b0123cd6ed [id=095537fb40584e6b9d0a5966ef09d81b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:30.261291 1027 tablet_service.cc:1467] Processing CreateTablet for tablet bf72471311854e01ac4b95a35e770d76 (DEFAULT_TABLE table=default.Table_69621fd76ebb45e5803d1a54f9023251 [id=5a4f2c260b6c48f594de3bc21791b9f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:30.261296 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddf79eec41214693859f526265a55492" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f"
I20250113 02:51:30.261787 745 tablet_service.cc:1467] Processing CreateTablet for tablet a58588b1befe48a8a1418b788a227519 (DEFAULT_TABLE table=default.Table_2f84c2362a2b4da7b088342232749760 [id=8986254e9ec14780874f0e61bb6924ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:30.262141 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a58588b1befe48a8a1418b788a227519. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:30.264233 746 tablet_service.cc:1467] Processing CreateTablet for tablet 1c77033ca393469e9324d2406a65a5e9 (DEFAULT_TABLE table=default.Table_c59e24fdd40849b48b0b75b0123cd6ed [id=095537fb40584e6b9d0a5966ef09d81b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:30.264544 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c77033ca393469e9324d2406a65a5e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:30.265561 1041 tablet_service.cc:1467] Processing CreateTablet for tablet 0741955d74f64f0393444fa64cfe1c90 (DEFAULT_TABLE table=default.Table_d41198149c094b9cb0f1fc8c7cc6a799 [id=b22338491032477cb4d570f7591ba2f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:30.268870 740 tablet_service.cc:1467] Processing CreateTablet for tablet bf72471311854e01ac4b95a35e770d76 (DEFAULT_TABLE table=default.Table_69621fd76ebb45e5803d1a54f9023251 [id=5a4f2c260b6c48f594de3bc21791b9f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:30.269184 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf72471311854e01ac4b95a35e770d76. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:30.269435 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet a0385f2c6aad45748ebf86d2cfdd40d6
I20250113 02:51:30.269757 1031 tablet_service.cc:1467] Processing CreateTablet for tablet c548a79f394d4ab88a309ca6400a05a2 (DEFAULT_TABLE table=default.Table_abe5b12ac76741b194dceecebd1e173c [id=8556512ee1ce43efb49aafae533beeab]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:30.270440 742 tablet_service.cc:1514] Processing DeleteTablet for tablet a0385f2c6aad45748ebf86d2cfdd40d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:30 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:30.270635 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet a0385f2c6aad45748ebf86d2cfdd40d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:30 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:30.270856 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet ee289c3236834dc48d922062041be3b5
I20250113 02:51:30.255290 1198 ts_tablet_manager.cc:1918] T c215ffe61b9b426286cc5e808463078d P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:30.271700 742 tablet_service.cc:1514] Processing DeleteTablet for tablet ee289c3236834dc48d922062041be3b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:30 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:30.271504 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet ee289c3236834dc48d922062041be3b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:30 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:30.271961 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_82f5a202f8d04a2680202b3ab06994c4" } new_table_name: "default.Table_0468611ec68e47109f5357935e50874d" modify_external_catalogs: true
I20250113 02:51:30.272178 747 tablet_service.cc:1467] Processing CreateTablet for tablet 0741955d74f64f0393444fa64cfe1c90 (DEFAULT_TABLE table=default.Table_d41198149c094b9cb0f1fc8c7cc6a799 [id=b22338491032477cb4d570f7591ba2f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:30.272251 1198 log.cc:1198] T c215ffe61b9b426286cc5e808463078d P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/c215ffe61b9b426286cc5e808463078d
I20250113 02:51:30.272444 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0741955d74f64f0393444fa64cfe1c90. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:30.272578 1198 ts_tablet_manager.cc:1939] T c215ffe61b9b426286cc5e808463078d P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:30.273538 446 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_6a3d6ef326e74c4ab0a5ed1e1bec2ecc" } modify_external_catalogs: true
I20250113 02:51:30.273708 446 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_6a3d6ef326e74c4ab0a5ed1e1bec2ecc" } modify_external_catalogs: true
W20250113 02:51:30.276147 711 leader_election.cc:343] T ddf79eec41214693859f526265a55492 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:30.276408 742 tablet_service.cc:1467] Processing CreateTablet for tablet c548a79f394d4ab88a309ca6400a05a2 (DEFAULT_TABLE table=default.Table_abe5b12ac76741b194dceecebd1e173c [id=8556512ee1ce43efb49aafae533beeab]), partition=RANGE (key) PARTITION UNBOUNDED
W20250113 02:51:30.276522 711 leader_election.cc:343] T 401c4f5dbd304950877b59a865f417b3 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:30.276732 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c548a79f394d4ab88a309ca6400a05a2. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:30.276798 711 leader_election.cc:343] T 9fff52e99c64462a8a73fccc3c22ca63 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
W20250113 02:51:30.277019 711 leader_election.cc:343] T 401c4f5dbd304950877b59a865f417b3 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
W20250113 02:51:30.277284 711 leader_election.cc:343] T ddf79eec41214693859f526265a55492 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:30.279397 1029 tablet_service.cc:1467] Processing CreateTablet for tablet 097cc4bd230046e2ac8551ca23d8f3e8 (DEFAULT_TABLE table=default.Table_f9117b338db445fba012e659c3b51f66 [id=94209be479874471b284541de4f72bfa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:30.281055 755 tablet_service.cc:1467] Processing CreateTablet for tablet 097cc4bd230046e2ac8551ca23d8f3e8 (DEFAULT_TABLE table=default.Table_f9117b338db445fba012e659c3b51f66 [id=94209be479874471b284541de4f72bfa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:30.281450 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 097cc4bd230046e2ac8551ca23d8f3e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:30.281527 1035 tablet_service.cc:1467] Processing CreateTablet for tablet 65f42cda151744c3bd3b0ec9415253c6 (DEFAULT_TABLE table=default.Table_216f6b44551d41d49fa21b862e08aee1 [id=5f65d1c759c443b492e50a58290b59c1]), partition=RANGE (key) PARTITION UNBOUNDED
W20250113 02:51:30.284699 998 leader_election.cc:343] T c94eedb5dce44e6d896192c93127903a P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:30.285806 739 tablet_service.cc:1467] Processing CreateTablet for tablet 65f42cda151744c3bd3b0ec9415253c6 (DEFAULT_TABLE table=default.Table_216f6b44551d41d49fa21b862e08aee1 [id=5f65d1c759c443b492e50a58290b59c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:30.286114 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65f42cda151744c3bd3b0ec9415253c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:30.294101 881 tablet_service.cc:1467] Processing CreateTablet for tablet 1c77033ca393469e9324d2406a65a5e9 (DEFAULT_TABLE table=default.Table_c59e24fdd40849b48b0b75b0123cd6ed [id=095537fb40584e6b9d0a5966ef09d81b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:30.294190 1024 tablet_service.cc:1467] Processing CreateTablet for tablet 3c59ae14a6584892bf1ebf6def81d497 (DEFAULT_TABLE table=default.Table_9b57c1f76abd4ef1b85f88572215474a [id=5a72ce8b06c54138817637abfe92be99]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:30.294504 886 tablet_service.cc:1467] Processing CreateTablet for tablet bf72471311854e01ac4b95a35e770d76 (DEFAULT_TABLE table=default.Table_69621fd76ebb45e5803d1a54f9023251 [id=5a4f2c260b6c48f594de3bc21791b9f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:30.294803 880 tablet_service.cc:1467] Processing CreateTablet for tablet 0741955d74f64f0393444fa64cfe1c90 (DEFAULT_TABLE table=default.Table_d41198149c094b9cb0f1fc8c7cc6a799 [id=b22338491032477cb4d570f7591ba2f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:30.295081 883 tablet_service.cc:1467] Processing CreateTablet for tablet c548a79f394d4ab88a309ca6400a05a2 (DEFAULT_TABLE table=default.Table_abe5b12ac76741b194dceecebd1e173c [id=8556512ee1ce43efb49aafae533beeab]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:30.295251 884 tablet_service.cc:1514] Processing DeleteTablet for tablet a0385f2c6aad45748ebf86d2cfdd40d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:30 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:30.295439 895 tablet_service.cc:1514] Processing DeleteTablet for tablet ee289c3236834dc48d922062041be3b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:30 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:30.295732 891 tablet_service.cc:1467] Processing CreateTablet for tablet 097cc4bd230046e2ac8551ca23d8f3e8 (DEFAULT_TABLE table=default.Table_f9117b338db445fba012e659c3b51f66 [id=94209be479874471b284541de4f72bfa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:30.296051 889 tablet_service.cc:1467] Processing CreateTablet for tablet 65f42cda151744c3bd3b0ec9415253c6 (DEFAULT_TABLE table=default.Table_216f6b44551d41d49fa21b862e08aee1 [id=5f65d1c759c443b492e50a58290b59c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:30.296331 898 tablet_service.cc:1467] Processing CreateTablet for tablet 3c59ae14a6584892bf1ebf6def81d497 (DEFAULT_TABLE table=default.Table_9b57c1f76abd4ef1b85f88572215474a [id=5a72ce8b06c54138817637abfe92be99]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:30.296283 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_ae350a6e57174618a0dcf77428d84826"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:30.296507 897 tablet_service.cc:1514] Processing DeleteTablet for tablet 258fad818b7b4aa4b337f9949e20aa46 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:19 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:30.296762 895 tablet_service.cc:1467] Processing CreateTablet for tablet d2699bb529bf4f0f9d7f2668ffe4c9ff (DEFAULT_TABLE table=default.Table_adf2c40363104ff9a3343f5a7659917f [id=d500d9b3e81c4e1f897b1db459dfd157]), partition=RANGE (key) PARTITION UNBOUNDED
W20250113 02:51:30.296830 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ae350a6e57174618a0dcf77428d84826 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:30.298466 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_2abb5d2c250c4d489df2815d6ec71827"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:30.299053 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2abb5d2c250c4d489df2815d6ec71827 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:30.299883 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c77033ca393469e9324d2406a65a5e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:30.301014 742 tablet_service.cc:1467] Processing CreateTablet for tablet 3c59ae14a6584892bf1ebf6def81d497 (DEFAULT_TABLE table=default.Table_9b57c1f76abd4ef1b85f88572215474a [id=5a72ce8b06c54138817637abfe92be99]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:30.301663 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c59ae14a6584892bf1ebf6def81d497. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:30.302713 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c13e9a51c1e74b44b352bade78390ba5 consensus state has no leader
I20250113 02:51:30.303119 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet f21eb3fa0f414816a29e3d4560cdc0fe
I20250113 02:51:30.304486 1042 tablet_service.cc:1514] Processing DeleteTablet for tablet f21eb3fa0f414816a29e3d4560cdc0fe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:30 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:30.305016 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 370c53f565df4e6f915e54a2fd2bf2ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:30.305750 897 tablet_service.cc:1514] Processing DeleteTablet for tablet f21eb3fa0f414816a29e3d4560cdc0fe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:30 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:30.305815 745 tablet_service.cc:1514] Processing DeleteTablet for tablet f21eb3fa0f414816a29e3d4560cdc0fe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:30 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:30.307152 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_0785edda3a764ff6a35bd4d6276d6e40" } new_table_name: "default.Table_8a8cdc09ad8649eab3bb7c934345da0f" modify_external_catalogs: true
W20250113 02:51:30.302632 709 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.3:42587, user_credentials={real_user=slave}} blocked reactor thread for 49820.7us
I20250113 02:51:30.308243 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a58588b1befe48a8a1418b788a227519. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:30.308373 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 370c53f565df4e6f915e54a2fd2bf2ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:30.308465 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf72471311854e01ac4b95a35e770d76. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:30.309355 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a58588b1befe48a8a1418b788a227519. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:30.312170 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0741955d74f64f0393444fa64cfe1c90. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:30.312247 1926 raft_consensus.cc:2798] T c94eedb5dce44e6d896192c93127903a P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 02:51:30.312328 1926 raft_consensus.cc:491] T c94eedb5dce44e6d896192c93127903a P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 02:51:30.312389 1926 raft_consensus.cc:3054] T c94eedb5dce44e6d896192c93127903a P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:30.312664 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c548a79f394d4ab88a309ca6400a05a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:30.313256 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65f42cda151744c3bd3b0ec9415253c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:30.313751 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c59ae14a6584892bf1ebf6def81d497. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:30.314373 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c59ae14a6584892bf1ebf6def81d497. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:30.315337 1926 raft_consensus.cc:513] T c94eedb5dce44e6d896192c93127903a P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:30.318168 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 097cc4bd230046e2ac8551ca23d8f3e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:30.323467 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0741955d74f64f0393444fa64cfe1c90. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:30.325191 452 master_service.cc:946] ReplaceTablet: received request to replace tablet eebad045e7bb43fa9ca9c9af8b3b7ea5 from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:30.328521 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d2699bb529bf4f0f9d7f2668ffe4c9ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:30.329185 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf72471311854e01ac4b95a35e770d76. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:30.332343 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c548a79f394d4ab88a309ca6400a05a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:30.336658 1926 leader_election.cc:290] T c94eedb5dce44e6d896192c93127903a P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Requested vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
I20250113 02:51:30.336706 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65f42cda151744c3bd3b0ec9415253c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:30.337147 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c94eedb5dce44e6d896192c93127903a" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196"
I20250113 02:51:30.337402 773 raft_consensus.cc:3054] T c94eedb5dce44e6d896192c93127903a P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:30.340610 773 raft_consensus.cc:2463] T c94eedb5dce44e6d896192c93127903a P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 1.
I20250113 02:51:30.341065 995 leader_election.cc:304] T c94eedb5dce44e6d896192c93127903a P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters:
I20250113 02:51:30.341372 1040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c77033ca393469e9324d2406a65a5e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:30.342501 1926 raft_consensus.cc:2798] T c94eedb5dce44e6d896192c93127903a P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Leader election won for term 1
I20250113 02:51:30.342594 1926 raft_consensus.cc:695] T c94eedb5dce44e6d896192c93127903a P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: Becoming Leader. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Running, Role: LEADER
I20250113 02:51:30.342803 1926 consensus_queue.cc:237] T c94eedb5dce44e6d896192c93127903a P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:30.350926 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_0236f2bec3074997a4cbfdba55c455bd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:30.351588 447 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_066f136aea8141fa91862f7285fbaac2" } modify_external_catalogs: true
I20250113 02:51:30.351755 447 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_066f136aea8141fa91862f7285fbaac2" } modify_external_catalogs: true
I20250113 02:51:30.353143 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_f68a86b5448042cd898cfd456b3924a7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:30.353633 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f68a86b5448042cd898cfd456b3924a7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250113 02:51:30.354964 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0236f2bec3074997a4cbfdba55c455bd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:30.355827 1873 tablet.cc:1721] T 4fe54131aafb40ee9d8bc5ea299af193 P df69f48656b44bb890f76587ee69cc0f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20250113 02:51:30.356165 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_19f32aa03bc941cebef094ca3896e0c3" } new_table_name: "default.Table_6fffe7a4aeca4314b182f3f0a0cd31e0" modify_external_catalogs: true
I20250113 02:51:30.356441 1194 tablet_bootstrap.cc:492] T 2c399784af46426b8327d175c7a6cce8 P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:30.358652 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c94eedb5dce44e6d896192c93127903a" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f"
W20250113 02:51:30.362023 998 leader_election.cc:343] T c94eedb5dce44e6d896192c93127903a P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
W20250113 02:51:30.363133 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 92f80f6b7bcf426ba5b41f6790efd3c6 consensus state has no leader
I20250113 02:51:30.368081 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 097cc4bd230046e2ac8551ca23d8f3e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:30.373590 1872 tablet.cc:1721] T 8adfa738c17a4169a5d99f6ceca75100 P 1186fa869eb14812832fa6562531f196: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250113 02:51:30.381175 1195 raft_consensus.cc:2238] T ddf79eec41214693859f526265a55492 P 1186fa869eb14812832fa6562531f196 [term 1 LEADER]: Raft consensus shutting down.
I20250113 02:51:30.381580 1195 pending_rounds.cc:62] T ddf79eec41214693859f526265a55492 P 1186fa869eb14812832fa6562531f196: Trying to abort 1 pending ops.
I20250113 02:51:30.381654 1195 pending_rounds.cc:69] T ddf79eec41214693859f526265a55492 P 1186fa869eb14812832fa6562531f196: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113673483259920384 op_type: NO_OP noop_request { }
I20250113 02:51:30.381752 1195 raft_consensus.cc:2883] T ddf79eec41214693859f526265a55492 P 1186fa869eb14812832fa6562531f196 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250113 02:51:30.381829 1195 raft_consensus.cc:2267] T ddf79eec41214693859f526265a55492 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250113 02:51:30.381574 1936 consensus_queue.cc:1159] T ddf79eec41214693859f526265a55492 P 1186fa869eb14812832fa6562531f196 [LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "1186fa869eb14812832fa6562531f196" status { last_received { term: 1 index: 1 } last_committed_idx: 0 last_received_current_leader { term: 1 index: 1 } }
I20250113 02:51:30.384093 1195 ts_tablet_manager.cc:1905] T ddf79eec41214693859f526265a55492 P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:30.391495 449 catalog_manager.cc:5526] T 401c4f5dbd304950877b59a865f417b3 P 1186fa869eb14812832fa6562531f196 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1186fa869eb14812832fa6562531f196 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "1186fa869eb14812832fa6562531f196" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } health_report { overall_health: UNKNOWN } } }
I20250113 02:51:30.391963 449 catalog_manager.cc:5618] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095) does not have the latest schema for tablet 401c4f5dbd304950877b59a865f417b3 (table default.Table_c991f9dbd70d4c4682664b8711f05db4 [id=2c64da64ecdb4bf48e11b523533902e8]). Expected version 1 got 0
I20250113 02:51:30.392076 1195 ts_tablet_manager.cc:1918] T ddf79eec41214693859f526265a55492 P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250113 02:51:30.392179 1195 log.cc:1198] T ddf79eec41214693859f526265a55492 P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/ddf79eec41214693859f526265a55492
I20250113 02:51:30.392350 449 catalog_manager.cc:5526] T 9fff52e99c64462a8a73fccc3c22ca63 P 1186fa869eb14812832fa6562531f196 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1186fa869eb14812832fa6562531f196 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "1186fa869eb14812832fa6562531f196" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } health_report { overall_health: UNKNOWN } } }
I20250113 02:51:30.393106 1195 ts_tablet_manager.cc:1939] T ddf79eec41214693859f526265a55492 P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:30.393169 449 catalog_manager.cc:5618] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095) does not have the latest schema for tablet 9fff52e99c64462a8a73fccc3c22ca63 (table default.Table_9ef18ee71ad843c5921750a2f8478855 [id=bd8bd572ed524f2cae01f820f07e3c36]). Expected version 2 got 0
W20250113 02:51:30.393694 711 consensus_peers.cc:487] T d7920c0544c0466bad6bd8c09aae1a7d P 1186fa869eb14812832fa6562531f196 -> Peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Couldn't send request to peer df69f48656b44bb890f76587ee69cc0f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d7920c0544c0466bad6bd8c09aae1a7d. This is attempt 1: this message will repeat every 5th retry.
I20250113 02:51:30.393811 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 699) took 1029 ms. Trace:
I20250113 02:51:30.396248 446 catalog_manager.cc:5618] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587) does not have the latest schema for tablet e8ab4516789e4585a101c8766d338af3 (table default.Table_3b76cf9fac334ebf9b1ecb9f8d07ea74 [id=b112ac4e2e58460db083da1eb788ca07]). Expected version 2 got 0
I20250113 02:51:30.396660 446 catalog_manager.cc:5526] T 378d59d314a84ddb84cf2ab041c51cf1 P 4216a7f853d642b7bc3e5cff458f3aca reported cstate change: term changed from 0 to 1, leader changed from <none> to 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3). New cstate: current_term: 1 leader_uuid: "4216a7f853d642b7bc3e5cff458f3aca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } health_report { overall_health: UNKNOWN } } }
I20250113 02:51:30.397226 446 catalog_manager.cc:5618] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587) does not have the latest schema for tablet 378d59d314a84ddb84cf2ab041c51cf1 (table default.Table_2794f9d2a3cc49f78de33a775c1619f1 [id=0c01441cec7043b687d6744e72a0e45d]). Expected version 1 got 0
I20250113 02:51:30.397269 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet c215ffe61b9b426286cc5e808463078d (table default.Table_d4146353239a45b281b482d6e9537502 [id=20b69dfe61d841829e1804153c0d0184]) successfully deleted
W20250113 02:51:30.397610 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet fc9189a9d1e4418e9f9205386f473232 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fc9189a9d1e4418e9f9205386f473232 already in progress: creating tablet
I20250113 02:51:30.397660 1198 tablet_replica.cc:331] T d7920c0544c0466bad6bd8c09aae1a7d P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:30.398986 1195 rpcz_store.cc:276] 0113 02:51:29.363871 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:29.363926 (+ 55us) service_pool.cc:225] Handling call
0113 02:51:30.257070 (+893144us) ts_tablet_manager.cc:1145] Deleting tablet ddf79eec41214693859f526265a55492
0113 02:51:30.257079 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:30.381402 (+124323us) spinlock_profiling.cc:243] Waited 124 ms on lock 0x616000212254. stack: 00007f2b88d31fd6 00007f2b88d31d90 00007f2b87d1148d 000000000096d544 000000000096d50d 000000000096ceda 00007f2b945d4ed0 00007f2b958c9463 00007f2b9836deb7 00007f2b983a0584 00007f2b98381690 00007f2b9838141d 0000000000a71ecc 00007f2b88d78b64 00007f2b88d7c18b 00007f2b88d7c008
0113 02:51:30.386978 (+ 5576us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:30.391987 (+ 5009us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:30.393791 (+ 1804us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":284895616,"tablet-delete.queue_time_us":892940}
I20250113 02:51:30.399734 1195 tablet_replica.cc:331] T 11950e996c814590bc9f3f22c0354902 P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:30.399698 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet ddf79eec41214693859f526265a55492 (table default.Table_f5342954ee0548ec90d482ed62198240 [id=888ae0b48950407086e5e2faa7fcf506]) successfully deleted
I20250113 02:51:30.399999 1195 raft_consensus.cc:2238] T 11950e996c814590bc9f3f22c0354902 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:30.400211 1195 raft_consensus.cc:2267] T 11950e996c814590bc9f3f22c0354902 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:30.400235 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet 501d4e1ae84e4795913035f8a19cc32d (table default.Table_8a5b45e248e94efb9b16df971fb9755c [id=bd54b5a058a44d718071be41b1e29a8c]) successfully deleted
I20250113 02:51:30.402962 1196 tablet_replica.cc:331] T fdbf03c89bff438a9a25115eddf60be9 P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:30.403278 1196 raft_consensus.cc:2238] T fdbf03c89bff438a9a25115eddf60be9 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:30.403556 1196 raft_consensus.cc:2267] T fdbf03c89bff438a9a25115eddf60be9 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:30.406229 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_5893047163024e76bd537c27b9b10458"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:30.406426 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_b95dec5993da4c6fb06409ef6cc17a5b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:30.406831 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b95dec5993da4c6fb06409ef6cc17a5b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:30.407828 1198 raft_consensus.cc:2238] T d7920c0544c0466bad6bd8c09aae1a7d P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:30.408123 1198 raft_consensus.cc:2267] T d7920c0544c0466bad6bd8c09aae1a7d P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:30.408038 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_e47dd2ccb95c4d6787808baa4497c77d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:30.408506 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e47dd2ccb95c4d6787808baa4497c77d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250113 02:51:30.409612 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5893047163024e76bd537c27b9b10458 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:30.410364 1198 ts_tablet_manager.cc:1905] T d7920c0544c0466bad6bd8c09aae1a7d P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:30.411067 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_174390617a294f1090362caca3ab703f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:30.411522 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_174390617a294f1090362caca3ab703f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:30.417268 1194 tablet_bootstrap.cc:654] T 2c399784af46426b8327d175c7a6cce8 P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:30.417304 1904 raft_consensus.cc:491] T de4a58bf63054890808e2e2fe46e43d2 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:30.425573 1904 raft_consensus.cc:513] T de4a58bf63054890808e2e2fe46e43d2 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:30.425969 1195 ts_tablet_manager.cc:1905] T 11950e996c814590bc9f3f22c0354902 P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:30.426275 1904 leader_election.cc:290] T de4a58bf63054890808e2e2fe46e43d2 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:30.427549 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de4a58bf63054890808e2e2fe46e43d2" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:30.428095 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de4a58bf63054890808e2e2fe46e43d2" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:30.428671 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 4b07a985897641c3ac48021c2a4daedb
W20250113 02:51:30.429169 711 leader_election.cc:343] T de4a58bf63054890808e2e2fe46e43d2 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
W20250113 02:51:30.429371 709 leader_election.cc:343] T de4a58bf63054890808e2e2fe46e43d2 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:30.429454 709 leader_election.cc:304] T de4a58bf63054890808e2e2fe46e43d2 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:30.429473 1031 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b07a985897641c3ac48021c2a4daedb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:30 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:30.429790 1904 raft_consensus.cc:2743] T de4a58bf63054890808e2e2fe46e43d2 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:30.430924 742 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b07a985897641c3ac48021c2a4daedb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:30 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:30.433506 896 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b07a985897641c3ac48021c2a4daedb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:30 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:30.436100 1191 tablet_bootstrap.cc:492] T 0810b9671f824d0a84741bc1904df247 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:30.436237 1191 ts_tablet_manager.cc:1397] T 0810b9671f824d0a84741bc1904df247 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.348s user 0.012s sys 0.003s
I20250113 02:51:30.436911 1191 raft_consensus.cc:357] T 0810b9671f824d0a84741bc1904df247 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:30.437093 1191 raft_consensus.cc:383] T 0810b9671f824d0a84741bc1904df247 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:30.437145 1191 raft_consensus.cc:738] T 0810b9671f824d0a84741bc1904df247 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:30.437417 1191 consensus_queue.cc:260] T 0810b9671f824d0a84741bc1904df247 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:30.439587 1191 ts_tablet_manager.cc:1428] T 0810b9671f824d0a84741bc1904df247 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250113 02:51:30.440146 1191 tablet_bootstrap.cc:492] T 840773ee80c7447097ea000b106fc89d P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:30.442090 1191 tablet_bootstrap.cc:654] T 840773ee80c7447097ea000b106fc89d P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:30.445731 912 raft_consensus.cc:3054] T 8adfa738c17a4169a5d99f6ceca75100 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:30.460012 1196 ts_tablet_manager.cc:1905] T fdbf03c89bff438a9a25115eddf60be9 P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:30.462376 1031 tablet_service.cc:1467] Processing CreateTablet for tablet d2699bb529bf4f0f9d7f2668ffe4c9ff (DEFAULT_TABLE table=default.Table_adf2c40363104ff9a3343f5a7659917f [id=d500d9b3e81c4e1f897b1db459dfd157]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:30.462788 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d2699bb529bf4f0f9d7f2668ffe4c9ff. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:30.469676 709 consensus_peers.cc:487] T d7920c0544c0466bad6bd8c09aae1a7d P 1186fa869eb14812832fa6562531f196 -> Peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Couldn't send request to peer 4216a7f853d642b7bc3e5cff458f3aca. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250113 02:51:30.473753 1192 tablet_bootstrap.cc:492] T 32cba8f3efc54e78b4df41c15df82eea P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:30.473945 1192 ts_tablet_manager.cc:1397] T 32cba8f3efc54e78b4df41c15df82eea P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.265s user 0.032s sys 0.056s
I20250113 02:51:30.474771 1192 raft_consensus.cc:357] T 32cba8f3efc54e78b4df41c15df82eea P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:30.475049 1192 raft_consensus.cc:383] T 32cba8f3efc54e78b4df41c15df82eea P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:30.475153 1192 raft_consensus.cc:738] T 32cba8f3efc54e78b4df41c15df82eea P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:30.475438 1192 consensus_queue.cc:260] T 32cba8f3efc54e78b4df41c15df82eea P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:30.475944 1192 ts_tablet_manager.cc:1428] T 32cba8f3efc54e78b4df41c15df82eea P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250113 02:51:30.477171 1192 tablet_bootstrap.cc:492] T 6d63f0e5835540a78c82020b20a07dac P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:30.478097 1196 ts_tablet_manager.cc:1918] T fdbf03c89bff438a9a25115eddf60be9 P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:30.478838 1192 tablet_bootstrap.cc:654] T 6d63f0e5835540a78c82020b20a07dac P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:30.483618 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_a71ef07a0c614223916ca517b610739d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:30.484215 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a71ef07a0c614223916ca517b610739d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:30.485777 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_c2171604b32e4fc4b4ff30e22c529c67"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:30.486239 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c2171604b32e4fc4b4ff30e22c529c67 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:30.487632 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_2087b79408c949cc88238ba68aabcaf1" } new_table_name: "default.Table_21fe260f30524201a0367153afecc2d8" modify_external_catalogs: true
I20250113 02:51:30.489593 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_37b52d3cfe0f4b0fb1ef1468117512dd" } modify_external_catalogs: true
I20250113 02:51:30.489786 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_37b52d3cfe0f4b0fb1ef1468117512dd" } modify_external_catalogs: true
I20250113 02:51:30.491571 1191 tablet_bootstrap.cc:492] T 840773ee80c7447097ea000b106fc89d P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:30.488570 1195 ts_tablet_manager.cc:1918] T 11950e996c814590bc9f3f22c0354902 P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250113 02:51:30.492363 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 032002b007424c028b4c671e1b0fb2ac consensus state has no leader
I20250113 02:51:30.492393 1195 log.cc:1198] T 11950e996c814590bc9f3f22c0354902 P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/11950e996c814590bc9f3f22c0354902
I20250113 02:51:30.492615 450 master_service.cc:946] ReplaceTablet: received request to replace tablet 936b8ceaedd34a019a4fb89a80dd2708 from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:30.493012 1195 ts_tablet_manager.cc:1939] T 11950e996c814590bc9f3f22c0354902 P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:30.494318 1191 ts_tablet_manager.cc:1397] T 840773ee80c7447097ea000b106fc89d P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.055s user 0.002s sys 0.001s
I20250113 02:51:30.494609 454 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_a5a1020e5fc44e0e878aa129c40eed5d" } new_table_name: "default.Table_ca4412c5b1a24d739bcdad8874cc84e9" modify_external_catalogs: true
I20250113 02:51:30.495129 1191 raft_consensus.cc:357] T 840773ee80c7447097ea000b106fc89d P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:30.495375 1191 raft_consensus.cc:383] T 840773ee80c7447097ea000b106fc89d P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:30.495448 1191 raft_consensus.cc:738] T 840773ee80c7447097ea000b106fc89d P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:30.496645 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_3318768dfa8c4d4bbc513e660dabfcc5" } modify_external_catalogs: true
I20250113 02:51:30.496829 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_3318768dfa8c4d4bbc513e660dabfcc5" } modify_external_catalogs: true
I20250113 02:51:30.499548 1191 consensus_queue.cc:260] T 840773ee80c7447097ea000b106fc89d P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:30.500540 449 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_639e38f37be1466e9c2ce92fa1c2d890" } modify_external_catalogs: true
I20250113 02:51:30.500595 1926 raft_consensus.cc:491] T f68f9f4076d9475d938e795e72bdd468 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:30.500658 742 tablet_service.cc:1514] Processing DeleteTablet for tablet de4a58bf63054890808e2e2fe46e43d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:20 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:30.500705 449 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_639e38f37be1466e9c2ce92fa1c2d890" } modify_external_catalogs: true
I20250113 02:51:30.500842 736 tablet_service.cc:1514] Processing DeleteTablet for tablet ddf79eec41214693859f526265a55492 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:23 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:30.500767 1926 raft_consensus.cc:513] T f68f9f4076d9475d938e795e72bdd468 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:30.504208 1926 leader_election.cc:290] T f68f9f4076d9475d938e795e72bdd468 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
I20250113 02:51:30.505329 1887 raft_consensus.cc:491] T 0810b9671f824d0a84741bc1904df247 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:30.505458 1887 raft_consensus.cc:513] T 0810b9671f824d0a84741bc1904df247 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:30.506022 1887 leader_election.cc:290] T 0810b9671f824d0a84741bc1904df247 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:30.506048 1191 ts_tablet_manager.cc:1428] T 840773ee80c7447097ea000b106fc89d P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.012s user 0.001s sys 0.001s
W20250113 02:51:30.506222 995 leader_election.cc:343] T f68f9f4076d9475d938e795e72bdd468 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Not found: Tablet not found: f68f9f4076d9475d938e795e72bdd468
I20250113 02:51:30.506315 1191 tablet_bootstrap.cc:492] T ff8ef6290698477391721fd65b9eeb28 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:30.509275 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0810b9671f824d0a84741bc1904df247" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
W20250113 02:51:30.509810 709 leader_election.cc:343] T 0810b9671f824d0a84741bc1904df247 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:30.510768 1191 tablet_bootstrap.cc:654] T ff8ef6290698477391721fd65b9eeb28 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:30.514804 736 tablet_service.cc:1467] Processing CreateTablet for tablet d2699bb529bf4f0f9d7f2668ffe4c9ff (DEFAULT_TABLE table=default.Table_adf2c40363104ff9a3343f5a7659917f [id=d500d9b3e81c4e1f897b1db459dfd157]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:30.515201 736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d2699bb529bf4f0f9d7f2668ffe4c9ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:30.515826 1031 tablet_service.cc:1467] Processing CreateTablet for tablet 2460e7cddcd84788a54dab342e7bb27b (DEFAULT_TABLE table=default.Table_b2598c65e9d04d8aa594a9f3d61d1e5a [id=e4f6f4ec007c454eba94263a37690f56]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:30.516224 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2460e7cddcd84788a54dab342e7bb27b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:30.520772 742 tablet_service.cc:1467] Processing CreateTablet for tablet 2460e7cddcd84788a54dab342e7bb27b (DEFAULT_TABLE table=default.Table_b2598c65e9d04d8aa594a9f3d61d1e5a [id=e4f6f4ec007c454eba94263a37690f56]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:30.521162 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2460e7cddcd84788a54dab342e7bb27b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:30.535655 447 catalog_manager.cc:6071] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: default.Table_d6dd014b2a3f4de5aaca5efe047b5819 [id=70089f757a1c45f6b983f216fb167478] alter complete (version 4)
I20250113 02:51:30.537729 447 catalog_manager.cc:5618] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095) does not have the latest schema for tablet e8ab4516789e4585a101c8766d338af3 (table default.Table_3b76cf9fac334ebf9b1ecb9f8d07ea74 [id=b112ac4e2e58460db083da1eb788ca07]). Expected version 2 got 0
I20250113 02:51:30.539077 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 0953b2f89e35402c9e9e84f3589c23ac
I20250113 02:51:30.539809 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_e47dd2ccb95c4d6787808baa4497c77d" } new_table_name: "default.Table_5b0a22c0da534b649ee7a211ee6718fa" modify_external_catalogs: true
I20250113 02:51:30.539882 736 tablet_service.cc:1514] Processing DeleteTablet for tablet 0953b2f89e35402c9e9e84f3589c23ac with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:30 UTC) from {username='slave'} at 127.0.0.1:55398
W20250113 02:51:30.542058 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 247af1a17df04b27898f86642336d51e consensus state has no leader
I20250113 02:51:30.542254 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 700) took 1177 ms. Trace:
I20250113 02:51:30.542496 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_e3d1ee1c42934e6a92821ecf7a3d1621"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:30.543083 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e3d1ee1c42934e6a92821ecf7a3d1621 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:30.544364 1192 tablet_bootstrap.cc:492] T 6d63f0e5835540a78c82020b20a07dac P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:30.544497 1192 ts_tablet_manager.cc:1397] T 6d63f0e5835540a78c82020b20a07dac P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.067s user 0.019s sys 0.043s
I20250113 02:51:30.545174 1192 raft_consensus.cc:357] T 6d63f0e5835540a78c82020b20a07dac P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:30.545344 1192 raft_consensus.cc:383] T 6d63f0e5835540a78c82020b20a07dac P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:30.545388 1192 raft_consensus.cc:738] T 6d63f0e5835540a78c82020b20a07dac P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:30.544363 1191 tablet_bootstrap.cc:492] T ff8ef6290698477391721fd65b9eeb28 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:30.547039 1191 ts_tablet_manager.cc:1397] T ff8ef6290698477391721fd65b9eeb28 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.041s user 0.002s sys 0.002s
W20250113 02:51:30.547282 454 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c7da2c406e64426faf123c4bddfaf213 consensus state has no leader
I20250113 02:51:30.548292 454 catalog_manager.cc:5618] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467) does not have the latest schema for tablet 8adfa738c17a4169a5d99f6ceca75100 (table default.Table_9a9b1ced3bef4e648088e0781851128e [id=ae544735623e4cfd894dc9340aca177c]). Expected version 1 got 0
I20250113 02:51:30.548377 1191 raft_consensus.cc:357] T ff8ef6290698477391721fd65b9eeb28 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:30.548568 1191 raft_consensus.cc:383] T ff8ef6290698477391721fd65b9eeb28 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:30.548658 1191 raft_consensus.cc:738] T ff8ef6290698477391721fd65b9eeb28 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:30.548907 1191 consensus_queue.cc:260] T ff8ef6290698477391721fd65b9eeb28 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:30.549491 1192 consensus_queue.cc:260] T 6d63f0e5835540a78c82020b20a07dac P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:30.551335 1191 ts_tablet_manager.cc:1428] T ff8ef6290698477391721fd65b9eeb28 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20250113 02:51:30.551584 1192 ts_tablet_manager.cc:1428] T 6d63f0e5835540a78c82020b20a07dac P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20250113 02:51:30.551851 1192 tablet_bootstrap.cc:492] T a77c38a394114d6c8a8da99d7eef7669 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:30.553546 1191 tablet_bootstrap.cc:492] T be4b2a8a3927410eaf12e7592646e4af P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:30.554843 1198 ts_tablet_manager.cc:1918] T d7920c0544c0466bad6bd8c09aae1a7d P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250113 02:51:30.554960 1198 log.cc:1198] T d7920c0544c0466bad6bd8c09aae1a7d P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/d7920c0544c0466bad6bd8c09aae1a7d
I20250113 02:51:30.555078 1191 tablet_bootstrap.cc:654] T be4b2a8a3927410eaf12e7592646e4af P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:30.555341 1198 ts_tablet_manager.cc:1939] T d7920c0544c0466bad6bd8c09aae1a7d P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:30.556140 1192 tablet_bootstrap.cc:654] T a77c38a394114d6c8a8da99d7eef7669 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:30.556118 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 936b8ceaedd34a019a4fb89a80dd2708
I20250113 02:51:30.556602 450 catalog_manager.cc:6596] ReplaceTablet: tablet 936b8ceaedd34a019a4fb89a80dd2708 deleted and replaced by tablet eba80548f52f4c32a098d1540c9cded0
I20250113 02:51:30.556926 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_3d5fd97116b343b0a6c798e1f22327c2" } modify_external_catalogs: true
I20250113 02:51:30.557096 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_3d5fd97116b343b0a6c798e1f22327c2" } modify_external_catalogs: true
I20250113 02:51:30.546917 1195 rpcz_store.cc:276] 0113 02:51:29.364708 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:29.364755 (+ 47us) service_pool.cc:225] Handling call
0113 02:51:30.399687 (+1034932us) ts_tablet_manager.cc:1145] Deleting tablet 11950e996c814590bc9f3f22c0354902
0113 02:51:30.399698 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:30.469954 (+ 70256us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:30.488469 (+ 18515us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:30.542230 (+ 53761us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1,"tablet-delete.queue_time_us":1034792}
I20250113 02:51:30.559082 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 9174df876fe64adc805fba0a36fb4d38
I20250113 02:51:30.559330 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 702) took 1175 ms. Trace:
I20250113 02:51:30.559621 451 master_service.cc:946] ReplaceTablet: received request to replace tablet acae3ff634104b5d931ece75d1eba5d6 from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:30.560477 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet 11950e996c814590bc9f3f22c0354902 (table default.Table_003cb5a672084be4b29d515fa4485747 [id=2e275f7f92ac440ca3ea6e29c5566f00]) successfully deleted
I20250113 02:51:30.561867 742 tablet_service.cc:1514] Processing DeleteTablet for tablet 9174df876fe64adc805fba0a36fb4d38 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:30 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:30.562284 736 tablet_service.cc:1514] Processing DeleteTablet for tablet 936b8ceaedd34a019a4fb89a80dd2708 with delete_type TABLET_DATA_DELETED (replaced by tablet eba80548f52f4c32a098d1540c9cded0) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:30.562783 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet 9174df876fe64adc805fba0a36fb4d38 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:30 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:30.562783 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet 936b8ceaedd34a019a4fb89a80dd2708 with delete_type TABLET_DATA_DELETED (replaced by tablet eba80548f52f4c32a098d1540c9cded0) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:30.559403 1195 rpcz_store.cc:276] 0113 02:51:29.383958 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:29.388109 (+ 4151us) service_pool.cc:225] Handling call
0113 02:51:30.559207 (+1171098us) ts_tablet_manager.cc:1145] Deleting tablet cd6dcfecf4324542b241f909926756ea
0113 02:51:30.559217 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:30.559315 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1170924}
I20250113 02:51:30.563586 1192 tablet_bootstrap.cc:492] T a77c38a394114d6c8a8da99d7eef7669 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
W20250113 02:51:30.563584 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet cd6dcfecf4324542b241f909926756ea because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cd6dcfecf4324542b241f909926756ea already in progress: creating tablet
I20250113 02:51:30.563711 1192 ts_tablet_manager.cc:1397] T a77c38a394114d6c8a8da99d7eef7669 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.003s
I20250113 02:51:30.564437 1192 raft_consensus.cc:357] T a77c38a394114d6c8a8da99d7eef7669 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:30.564635 1192 raft_consensus.cc:383] T a77c38a394114d6c8a8da99d7eef7669 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:30.564693 1192 raft_consensus.cc:738] T a77c38a394114d6c8a8da99d7eef7669 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:30.564888 1192 consensus_queue.cc:260] T a77c38a394114d6c8a8da99d7eef7669 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:30.565485 1192 ts_tablet_manager.cc:1428] T a77c38a394114d6c8a8da99d7eef7669 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250113 02:51:30.565953 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_bcc759b8b5ed4a4798c2127eda9b969d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:30.566161 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 712) took 1139 ms. Trace:
I20250113 02:51:30.566242 1195 rpcz_store.cc:276] 0113 02:51:29.426630 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:29.433972 (+ 7342us) service_pool.cc:225] Handling call
0113 02:51:30.564000 (+1130028us) ts_tablet_manager.cc:1145] Deleting tablet e468b652f71f44e3945f8d919ac0e1a8
0113 02:51:30.564724 (+ 724us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:30.566144 (+ 1420us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1129832}
W20250113 02:51:30.566640 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet e468b652f71f44e3945f8d919ac0e1a8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e468b652f71f44e3945f8d919ac0e1a8 already in progress: creating tablet
W20250113 02:51:30.566653 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bcc759b8b5ed4a4798c2127eda9b969d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250113 02:51:30.567307 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet fc9189a9d1e4418e9f9205386f473232 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fc9189a9d1e4418e9f9205386f473232 already in progress: creating tablet
I20250113 02:51:30.567698 1195 tablet_replica.cc:331] T d7920c0544c0466bad6bd8c09aae1a7d P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:30.567853 1195 raft_consensus.cc:2238] T d7920c0544c0466bad6bd8c09aae1a7d P 1186fa869eb14812832fa6562531f196 [term 1 LEADER]: Raft consensus shutting down.
I20250113 02:51:30.568135 1195 raft_consensus.cc:2267] T d7920c0544c0466bad6bd8c09aae1a7d P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:30.568511 448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_8d2d5a8abea04c0a993b9c0cc55d7c3a" } new_table_name: "default.Table_ab30cf183b59419689df6d79621cd053" modify_external_catalogs: true
I20250113 02:51:30.568598 1195 ts_tablet_manager.cc:1905] T d7920c0544c0466bad6bd8c09aae1a7d P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:30.569180 1040 tablet_service.cc:1514] Processing DeleteTablet for tablet 0953b2f89e35402c9e9e84f3589c23ac with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:30 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:30.569537 446 catalog_manager.cc:6071] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: default.Table_9a9b1ced3bef4e648088e0781851128e [id=ae544735623e4cfd894dc9340aca177c] alter complete (version 1)
W20250113 02:51:30.569676 439 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.60:42067: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20250113 02:51:30.572340 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet a0ca2e3a3ce94f3aa20df64c89c7cf95 with delete_type TABLET_DATA_DELETED (replaced by tablet 2470b010b571444eba3f132323038f33) from {username='slave'} at 127.0.0.1:60230
W20250113 02:51:30.572695 439 consensus_peers.cc:487] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9 -> Peer 45e9d43ecac0401fb2c18fe5eaf089b5 (127.0.104.60:42067): Couldn't send request to peer 45e9d43ecac0401fb2c18fe5eaf089b5. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.60:42067: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20250113 02:51:30.574800 736 tablet_service.cc:1514] Processing DeleteTablet for tablet 0810b9671f824d0a84741bc1904df247 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:20 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:30.578035 1191 tablet_bootstrap.cc:492] T be4b2a8a3927410eaf12e7592646e4af P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:30.578178 1191 ts_tablet_manager.cc:1397] T be4b2a8a3927410eaf12e7592646e4af P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.025s user 0.001s sys 0.001s
I20250113 02:51:30.578927 1191 raft_consensus.cc:357] T be4b2a8a3927410eaf12e7592646e4af P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:30.579125 1191 raft_consensus.cc:383] T be4b2a8a3927410eaf12e7592646e4af P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:30.579180 1191 raft_consensus.cc:738] T be4b2a8a3927410eaf12e7592646e4af P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:30.579342 1040 tablet_service.cc:1514] Processing DeleteTablet for tablet c94eedb5dce44e6d896192c93127903a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:20 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:30.579363 1191 consensus_queue.cc:260] T be4b2a8a3927410eaf12e7592646e4af P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:30.580122 1191 ts_tablet_manager.cc:1428] T be4b2a8a3927410eaf12e7592646e4af P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250113 02:51:30.581024 1191 tablet_bootstrap.cc:492] T 41fcde988f72496b824271fa36c1a226 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:30.583441 1191 tablet_bootstrap.cc:654] T 41fcde988f72496b824271fa36c1a226 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:30.585083 454 catalog_manager.cc:5618] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467) does not have the latest schema for tablet 8adfa738c17a4169a5d99f6ceca75100 (table default.Table_9a9b1ced3bef4e648088e0781851128e [id=ae544735623e4cfd894dc9340aca177c]). Expected version 1 got 0
I20250113 02:51:30.585657 1195 ts_tablet_manager.cc:1918] T d7920c0544c0466bad6bd8c09aae1a7d P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250113 02:51:30.585754 1195 log.cc:1198] T d7920c0544c0466bad6bd8c09aae1a7d P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/d7920c0544c0466bad6bd8c09aae1a7d
I20250113 02:51:30.586095 1195 ts_tablet_manager.cc:1939] T d7920c0544c0466bad6bd8c09aae1a7d P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:30.586916 1195 tablet_replica.cc:331] T c94eedb5dce44e6d896192c93127903a P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:30.587092 1195 raft_consensus.cc:2238] T c94eedb5dce44e6d896192c93127903a P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:30.587224 1195 raft_consensus.cc:2267] T c94eedb5dce44e6d896192c93127903a P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:30.587474 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet d7920c0544c0466bad6bd8c09aae1a7d (table default.Table_c9a2c2b1c62849babefee677197da844 [id=05755562cfc24370a3e6fb245a573dd4]) successfully deleted
I20250113 02:51:30.587692 1195 ts_tablet_manager.cc:1905] T c94eedb5dce44e6d896192c93127903a P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:30.597137 1195 ts_tablet_manager.cc:1918] T c94eedb5dce44e6d896192c93127903a P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:30.597256 1195 log.cc:1198] T c94eedb5dce44e6d896192c93127903a P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/c94eedb5dce44e6d896192c93127903a
I20250113 02:51:30.597532 1195 ts_tablet_manager.cc:1939] T c94eedb5dce44e6d896192c93127903a P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:30.599063 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet c94eedb5dce44e6d896192c93127903a (table default.Table_616eda5ca6ce4155b20351b6b8b000f5 [id=e0cdd94a9fc947fc8a9555150b7272f0]) successfully deleted
I20250113 02:51:30.599953 742 tablet_service.cc:1514] Processing DeleteTablet for tablet ff8ef6290698477391721fd65b9eeb28 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:20 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:30.601403 736 tablet_service.cc:1514] Processing DeleteTablet for tablet 840773ee80c7447097ea000b106fc89d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:19 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:30.605561 1192 tablet_bootstrap.cc:492] T 41fcde988f72496b824271fa36c1a226 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:30.606146 1926 raft_consensus.cc:491] T b4806b18a4194f9c95350d89ba62e13a P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:30.606284 1926 raft_consensus.cc:513] T b4806b18a4194f9c95350d89ba62e13a P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:30.606750 1195 tablet_replica.cc:331] T 572e4eed1e924381bada27f8389821a7 P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:30.606950 1195 raft_consensus.cc:2238] T 572e4eed1e924381bada27f8389821a7 P 1186fa869eb14812832fa6562531f196 [term 1 LEADER]: Raft consensus shutting down.
I20250113 02:51:30.606992 1926 leader_election.cc:290] T b4806b18a4194f9c95350d89ba62e13a P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:30.607297 1195 raft_consensus.cc:2267] T 572e4eed1e924381bada27f8389821a7 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:30.607393 1191 tablet_bootstrap.cc:492] T 41fcde988f72496b824271fa36c1a226 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:30.607496 1191 ts_tablet_manager.cc:1397] T 41fcde988f72496b824271fa36c1a226 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.027s user 0.001s sys 0.008s
I20250113 02:51:30.607517 1192 tablet_bootstrap.cc:654] T 41fcde988f72496b824271fa36c1a226 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:30.608160 1195 ts_tablet_manager.cc:1905] T 572e4eed1e924381bada27f8389821a7 P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:30.608315 1191 raft_consensus.cc:357] T 41fcde988f72496b824271fa36c1a226 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:30.608474 1191 raft_consensus.cc:383] T 41fcde988f72496b824271fa36c1a226 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:30.608527 1191 raft_consensus.cc:738] T 41fcde988f72496b824271fa36c1a226 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:30.608736 1191 consensus_queue.cc:260] T 41fcde988f72496b824271fa36c1a226 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:30.609606 1191 ts_tablet_manager.cc:1428] T 41fcde988f72496b824271fa36c1a226 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250113 02:51:30.611039 1191 tablet_bootstrap.cc:492] T a77c38a394114d6c8a8da99d7eef7669 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:30.613001 1191 tablet_bootstrap.cc:654] T a77c38a394114d6c8a8da99d7eef7669 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
W20250113 02:51:30.614281 995 leader_election.cc:343] T b4806b18a4194f9c95350d89ba62e13a P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Not found: Tablet not found: b4806b18a4194f9c95350d89ba62e13a
I20250113 02:51:30.614584 1926 raft_consensus.cc:491] T a0ca2e3a3ce94f3aa20df64c89c7cf95 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:30.614728 1926 raft_consensus.cc:513] T a0ca2e3a3ce94f3aa20df64c89c7cf95 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:30.615449 1926 leader_election.cc:290] T a0ca2e3a3ce94f3aa20df64c89c7cf95 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
I20250113 02:51:30.616726 1192 tablet_bootstrap.cc:492] T 41fcde988f72496b824271fa36c1a226 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:30.616844 1192 ts_tablet_manager.cc:1397] T 41fcde988f72496b824271fa36c1a226 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.003s
I20250113 02:51:30.617189 1198 tablet_replica.cc:331] T ddf79eec41214693859f526265a55492 P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
W20250113 02:51:30.617213 995 leader_election.cc:343] T a0ca2e3a3ce94f3aa20df64c89c7cf95 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Not found: Tablet not found: a0ca2e3a3ce94f3aa20df64c89c7cf95
I20250113 02:51:30.617533 1198 raft_consensus.cc:2238] T ddf79eec41214693859f526265a55492 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:30.617503 1192 raft_consensus.cc:357] T 41fcde988f72496b824271fa36c1a226 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:30.617683 1192 raft_consensus.cc:383] T 41fcde988f72496b824271fa36c1a226 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:30.617751 1192 raft_consensus.cc:738] T 41fcde988f72496b824271fa36c1a226 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:30.617957 1192 consensus_queue.cc:260] T 41fcde988f72496b824271fa36c1a226 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:30.618387 1192 ts_tablet_manager.cc:1428] T 41fcde988f72496b824271fa36c1a226 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20250113 02:51:30.618623 1192 tablet_bootstrap.cc:492] T 11950e996c814590bc9f3f22c0354902 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:30.620982 1192 tablet_bootstrap.cc:654] T 11950e996c814590bc9f3f22c0354902 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:30.622211 1195 ts_tablet_manager.cc:1918] T 572e4eed1e924381bada27f8389821a7 P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250113 02:51:30.622319 1195 log.cc:1198] T 572e4eed1e924381bada27f8389821a7 P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/572e4eed1e924381bada27f8389821a7
I20250113 02:51:30.622633 1195 ts_tablet_manager.cc:1939] T 572e4eed1e924381bada27f8389821a7 P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:30.617699 1198 raft_consensus.cc:2267] T ddf79eec41214693859f526265a55492 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:30.624364 1198 ts_tablet_manager.cc:1905] T ddf79eec41214693859f526265a55492 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:30.624966 1195 tablet_replica.cc:331] T de4a58bf63054890808e2e2fe46e43d2 P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:30.625224 1195 raft_consensus.cc:2238] T de4a58bf63054890808e2e2fe46e43d2 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:30.625402 1195 raft_consensus.cc:2267] T de4a58bf63054890808e2e2fe46e43d2 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:30.625977 1195 ts_tablet_manager.cc:1905] T de4a58bf63054890808e2e2fe46e43d2 P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:30.629012 1191 tablet_bootstrap.cc:492] T a77c38a394114d6c8a8da99d7eef7669 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:30.629148 1191 ts_tablet_manager.cc:1397] T a77c38a394114d6c8a8da99d7eef7669 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.000s
I20250113 02:51:30.629858 1191 raft_consensus.cc:357] T a77c38a394114d6c8a8da99d7eef7669 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:30.630019 1191 raft_consensus.cc:383] T a77c38a394114d6c8a8da99d7eef7669 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:30.630061 1191 raft_consensus.cc:738] T a77c38a394114d6c8a8da99d7eef7669 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:30.630287 1191 consensus_queue.cc:260] T a77c38a394114d6c8a8da99d7eef7669 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:30.635059 1191 ts_tablet_manager.cc:1428] T a77c38a394114d6c8a8da99d7eef7669 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20250113 02:51:30.635792 1191 tablet_bootstrap.cc:492] T efccefd6787943469131492741ee2bd5 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:30.637833 1191 tablet_bootstrap.cc:654] T efccefd6787943469131492741ee2bd5 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:30.638793 1195 ts_tablet_manager.cc:1918] T de4a58bf63054890808e2e2fe46e43d2 P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:30.638883 1195 log.cc:1198] T de4a58bf63054890808e2e2fe46e43d2 P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/de4a58bf63054890808e2e2fe46e43d2
I20250113 02:51:30.639194 1195 ts_tablet_manager.cc:1939] T de4a58bf63054890808e2e2fe46e43d2 P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:30.644244 1198 ts_tablet_manager.cc:1918] T ddf79eec41214693859f526265a55492 P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:30.644390 1198 log.cc:1198] T ddf79eec41214693859f526265a55492 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/ddf79eec41214693859f526265a55492
I20250113 02:51:30.644714 1198 ts_tablet_manager.cc:1939] T ddf79eec41214693859f526265a55492 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:30.645059 1195 tablet_replica.cc:331] T 0810b9671f824d0a84741bc1904df247 P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:30.645582 1195 raft_consensus.cc:2238] T 0810b9671f824d0a84741bc1904df247 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:30.645812 1198 tablet_replica.cc:331] T f68f9f4076d9475d938e795e72bdd468 P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:30.646008 1198 raft_consensus.cc:2238] T f68f9f4076d9475d938e795e72bdd468 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:30.646173 1198 raft_consensus.cc:2267] T f68f9f4076d9475d938e795e72bdd468 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:30.646200 1195 raft_consensus.cc:2267] T 0810b9671f824d0a84741bc1904df247 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:30.646724 1198 ts_tablet_manager.cc:1905] T f68f9f4076d9475d938e795e72bdd468 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:30.647219 1195 ts_tablet_manager.cc:1905] T 0810b9671f824d0a84741bc1904df247 P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:30.649085 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet d7920c0544c0466bad6bd8c09aae1a7d (table default.Table_c9a2c2b1c62849babefee677197da844 [id=05755562cfc24370a3e6fb245a573dd4]) successfully deleted
I20250113 02:51:30.649370 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet ddf79eec41214693859f526265a55492 (table default.Table_f5342954ee0548ec90d482ed62198240 [id=888ae0b48950407086e5e2faa7fcf506]) successfully deleted
I20250113 02:51:30.649724 454 catalog_manager.cc:5618] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095) does not have the latest schema for tablet be4b2a8a3927410eaf12e7592646e4af (table default.Table_24d9af3d0b0740d69c791fc96916ef0a [id=124669ccadaf49e19d2660b1495f9647]). Expected version 3 got 0
W20250113 02:51:30.650570 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 542df00229f645788f1b821c0ce00c46 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 542df00229f645788f1b821c0ce00c46 already in progress: creating tablet
I20250113 02:51:30.653649 1191 tablet_bootstrap.cc:492] T efccefd6787943469131492741ee2bd5 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:30.653836 1191 ts_tablet_manager.cc:1397] T efccefd6787943469131492741ee2bd5 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.001s
I20250113 02:51:30.654778 1191 raft_consensus.cc:357] T efccefd6787943469131492741ee2bd5 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:30.655071 1191 raft_consensus.cc:383] T efccefd6787943469131492741ee2bd5 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:30.655144 1191 raft_consensus.cc:738] T efccefd6787943469131492741ee2bd5 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:30.655445 1191 consensus_queue.cc:260] T efccefd6787943469131492741ee2bd5 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:30.656558 1191 ts_tablet_manager.cc:1428] T efccefd6787943469131492741ee2bd5 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20250113 02:51:30.657548 1191 tablet_bootstrap.cc:492] T 7d3562a48c6e4badbbcf37640e91b5ba P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
W20250113 02:51:30.658417 439 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.104.1:40095, user_credentials={real_user=slave}} blocked reactor thread for 45647.4us
W20250113 02:51:30.658885 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 64aa8390b1b1475c8743b6491d40674c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 64aa8390b1b1475c8743b6491d40674c already in progress: creating tablet
I20250113 02:51:30.659358 1191 tablet_bootstrap.cc:654] T 7d3562a48c6e4badbbcf37640e91b5ba P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
W20250113 02:51:30.659785 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet f4af42b9bae44e3fb009310212cec587 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f4af42b9bae44e3fb009310212cec587 already in progress: creating tablet
W20250113 02:51:30.660070 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet cd21f10d74c946a2872a84cadccf7950 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cd21f10d74c946a2872a84cadccf7950 already in progress: creating tablet
I20250113 02:51:30.660815 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet 572e4eed1e924381bada27f8389821a7 (table default.Table_1d910c3d2b244474a576fc4088301143 [id=75cb0745bbb84e1c883cb13b546ecb11]) successfully deleted
W20250113 02:51:30.661084 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 85a82ad6e93346ecb1a0cb38b62f5786 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 85a82ad6e93346ecb1a0cb38b62f5786 already in progress: creating tablet
I20250113 02:51:30.662051 1195 ts_tablet_manager.cc:1918] T 0810b9671f824d0a84741bc1904df247 P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:30.663558 1195 log.cc:1198] T 0810b9671f824d0a84741bc1904df247 P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/0810b9671f824d0a84741bc1904df247
I20250113 02:51:30.664057 1195 ts_tablet_manager.cc:1939] T 0810b9671f824d0a84741bc1904df247 P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:30.665886 1195 tablet_replica.cc:331] T ff8ef6290698477391721fd65b9eeb28 P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:30.666178 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet acae3ff634104b5d931ece75d1eba5d6
I20250113 02:51:30.666363 1195 raft_consensus.cc:2238] T ff8ef6290698477391721fd65b9eeb28 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:30.666528 1195 raft_consensus.cc:2267] T ff8ef6290698477391721fd65b9eeb28 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:30.667213 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet acae3ff634104b5d931ece75d1eba5d6 with delete_type TABLET_DATA_DELETED (replaced by tablet 95de1523f15d43fa8e9b12b75860b1dc) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:30.667685 1195 ts_tablet_manager.cc:1905] T ff8ef6290698477391721fd65b9eeb28 P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250113 02:51:30.670311 448 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 42a8e01bb71f49b09c83a8f5c905f38d consensus state has no leader
I20250113 02:51:30.674611 1195 ts_tablet_manager.cc:1918] T ff8ef6290698477391721fd65b9eeb28 P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250113 02:51:30.674588 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet cf7b8bd6801743f49e9b9473747447c5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cf7b8bd6801743f49e9b9473747447c5 already in progress: creating tablet
I20250113 02:51:30.674777 1198 ts_tablet_manager.cc:1918] T f68f9f4076d9475d938e795e72bdd468 P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:30.674909 1198 log.cc:1198] T f68f9f4076d9475d938e795e72bdd468 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/f68f9f4076d9475d938e795e72bdd468
W20250113 02:51:30.674887 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet ef99c223dcf04b11bf785703df55f877 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ef99c223dcf04b11bf785703df55f877 already in progress: creating tablet
I20250113 02:51:30.675252 1198 ts_tablet_manager.cc:1939] T f68f9f4076d9475d938e795e72bdd468 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:30.676160 1198 tablet_replica.cc:331] T b4806b18a4194f9c95350d89ba62e13a P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:30.676422 1198 raft_consensus.cc:2238] T b4806b18a4194f9c95350d89ba62e13a P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:30.676609 1198 raft_consensus.cc:2267] T b4806b18a4194f9c95350d89ba62e13a P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus is shut down!
W20250113 02:51:30.677026 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 79aaab73797c474bae06831fa308b066 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 79aaab73797c474bae06831fa308b066 already in progress: creating tablet
I20250113 02:51:30.677249 1198 ts_tablet_manager.cc:1905] T b4806b18a4194f9c95350d89ba62e13a P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250113 02:51:30.677294 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 51d9483ecacb413796d0fd9e7560eac1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 51d9483ecacb413796d0fd9e7560eac1 already in progress: creating tablet
W20250113 02:51:30.678747 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 65f42cda151744c3bd3b0ec9415253c6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 65f42cda151744c3bd3b0ec9415253c6 already in progress: creating tablet
W20250113 02:51:30.679068 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet a0385f2c6aad45748ebf86d2cfdd40d6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a0385f2c6aad45748ebf86d2cfdd40d6 already in progress: creating tablet
W20250113 02:51:30.679231 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet cfd1697a03494ad2aaa41a056e334ca7 has no consensus state
I20250113 02:51:30.681459 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet f68f9f4076d9475d938e795e72bdd468 (table default.Table_92ea541283614df981d96c5b74362f1d [id=9e58548d1a184cc0b362f148c44005ed]) successfully deleted
I20250113 02:51:30.681521 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 42fad91fe1b64a21a2c8412a2585af47
I20250113 02:51:30.674988 1191 tablet_bootstrap.cc:492] T 7d3562a48c6e4badbbcf37640e91b5ba P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:30.681819 1195 log.cc:1198] T ff8ef6290698477391721fd65b9eeb28 P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/ff8ef6290698477391721fd65b9eeb28
I20250113 02:51:30.681890 1191 ts_tablet_manager.cc:1397] T 7d3562a48c6e4badbbcf37640e91b5ba P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.025s user 0.006s sys 0.003s
I20250113 02:51:30.681962 448 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_b8e04994e6894fe4a4ff012d0050eb6c" } modify_external_catalogs: true
I20250113 02:51:30.682166 448 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_b8e04994e6894fe4a4ff012d0050eb6c" } modify_external_catalogs: true
I20250113 02:51:30.682436 1195 ts_tablet_manager.cc:1939] T ff8ef6290698477391721fd65b9eeb28 P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:30.682430 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet 42fad91fe1b64a21a2c8412a2585af47 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:30 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:30.682816 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 573cffe308cf4774a2108a7390f08e90
I20250113 02:51:30.682861 1191 raft_consensus.cc:357] T 7d3562a48c6e4badbbcf37640e91b5ba P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:30.683036 1191 raft_consensus.cc:383] T 7d3562a48c6e4badbbcf37640e91b5ba P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:30.683084 1191 raft_consensus.cc:738] T 7d3562a48c6e4badbbcf37640e91b5ba P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:30.683312 1191 consensus_queue.cc:260] T 7d3562a48c6e4badbbcf37640e91b5ba P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:30.683806 1191 ts_tablet_manager.cc:1428] T 7d3562a48c6e4badbbcf37640e91b5ba P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250113 02:51:30.684079 1191 tablet_bootstrap.cc:492] T bffec36d567544c5a1c87e64a7e9730b P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:30.685336 1195 tablet_replica.cc:331] T 840773ee80c7447097ea000b106fc89d P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:30.685739 1195 raft_consensus.cc:2238] T 840773ee80c7447097ea000b106fc89d P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:30.686107 1195 raft_consensus.cc:2267] T 840773ee80c7447097ea000b106fc89d P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:30.686676 1191 tablet_bootstrap.cc:654] T bffec36d567544c5a1c87e64a7e9730b P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:30.687026 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_8b441c8a6ec646afbe7c211d1b5d7eda"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:30.687554 1195 ts_tablet_manager.cc:1905] T 840773ee80c7447097ea000b106fc89d P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250113 02:51:30.687618 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8b441c8a6ec646afbe7c211d1b5d7eda in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:30.687950 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet 573cffe308cf4774a2108a7390f08e90 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:30 UTC) from {username='slave'} at 127.0.0.1:60230
W20250113 02:51:30.689079 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet ee289c3236834dc48d922062041be3b5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ee289c3236834dc48d922062041be3b5 already in progress: creating tablet
W20250113 02:51:30.689564 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet f21eb3fa0f414816a29e3d4560cdc0fe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f21eb3fa0f414816a29e3d4560cdc0fe already in progress: creating tablet
I20250113 02:51:30.689339 452 master_service.cc:946] ReplaceTablet: received request to replace tablet f362b8ab6c464159a7dc2164ac92d7c1 from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:30.689265 451 catalog_manager.cc:6596] ReplaceTablet: tablet acae3ff634104b5d931ece75d1eba5d6 deleted and replaced by tablet 95de1523f15d43fa8e9b12b75860b1dc
I20250113 02:51:30.691066 1198 ts_tablet_manager.cc:1918] T b4806b18a4194f9c95350d89ba62e13a P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250113 02:51:30.691092 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 4b07a985897641c3ac48021c2a4daedb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4b07a985897641c3ac48021c2a4daedb already in progress: creating tablet
I20250113 02:51:30.691296 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet de4a58bf63054890808e2e2fe46e43d2 (table default.Table_5b0e60572cb54fcca8b3b5b94326ae3a [id=9f5add219a7c48da9a8afa966e22d1bd]) successfully deleted
I20250113 02:51:30.691453 446 master_service.cc:946] ReplaceTablet: received request to replace tablet c215ffe61b9b426286cc5e808463078d from {username='slave'} at 127.0.0.1:49258
W20250113 02:51:30.693574 439 catalog_manager.cc:4839] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet ddf79eec41214693859f526265a55492 because the tablet was not found. No further retry: Not found: Tablet not found: ddf79eec41214693859f526265a55492
I20250113 02:51:30.694727 1198 log.cc:1198] T b4806b18a4194f9c95350d89ba62e13a P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/b4806b18a4194f9c95350d89ba62e13a
I20250113 02:51:30.695138 1198 ts_tablet_manager.cc:1939] T b4806b18a4194f9c95350d89ba62e13a P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:30.694607 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_65417a56ebb74c158456a735cfd14cc2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:30.696283 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet b4806b18a4194f9c95350d89ba62e13a (table default.Table_4369a143c72643b084df58d25314c6dd [id=36ff239c08a54be99ca307354810fff3]) successfully deleted
W20250113 02:51:30.696691 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_65417a56ebb74c158456a735cfd14cc2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250113 02:51:30.697455 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 542df00229f645788f1b821c0ce00c46 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 542df00229f645788f1b821c0ce00c46 already in progress: creating tablet
W20250113 02:51:30.698233 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet f4af42b9bae44e3fb009310212cec587 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f4af42b9bae44e3fb009310212cec587 already in progress: creating tablet
I20250113 02:51:30.695493 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_7c472d82db9f4c2f96b4af7e343e1d55"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:30.698976 1195 ts_tablet_manager.cc:1918] T 840773ee80c7447097ea000b106fc89d P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:30.699076 1195 log.cc:1198] T 840773ee80c7447097ea000b106fc89d P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/840773ee80c7447097ea000b106fc89d
W20250113 02:51:30.699046 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7c472d82db9f4c2f96b4af7e343e1d55 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:30.699366 1195 ts_tablet_manager.cc:1939] T 840773ee80c7447097ea000b106fc89d P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:30.696043 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_bc51f0b8366d47bcb5f9e7069597036b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:30.700474 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bc51f0b8366d47bcb5f9e7069597036b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250113 02:51:30.700767 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet cd21f10d74c946a2872a84cadccf7950 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cd21f10d74c946a2872a84cadccf7950 already in progress: creating tablet
W20250113 02:51:30.701359 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet cf7b8bd6801743f49e9b9473747447c5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cf7b8bd6801743f49e9b9473747447c5 already in progress: creating tablet
W20250113 02:51:30.701763 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 0953b2f89e35402c9e9e84f3589c23ac because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0953b2f89e35402c9e9e84f3589c23ac already in progress: creating tablet
W20250113 02:51:30.701980 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 64aa8390b1b1475c8743b6491d40674c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 64aa8390b1b1475c8743b6491d40674c already in progress: creating tablet
I20250113 02:51:30.702453 1198 tablet_replica.cc:331] T 572e4eed1e924381bada27f8389821a7 P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
W20250113 02:51:30.702580 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 85a82ad6e93346ecb1a0cb38b62f5786 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 85a82ad6e93346ecb1a0cb38b62f5786 already in progress: creating tablet
I20250113 02:51:30.702658 1198 raft_consensus.cc:2238] T 572e4eed1e924381bada27f8389821a7 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:30.702870 1198 raft_consensus.cc:2267] T 572e4eed1e924381bada27f8389821a7 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Raft consensus is shut down!
W20250113 02:51:30.702966 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 9174df876fe64adc805fba0a36fb4d38 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9174df876fe64adc805fba0a36fb4d38 already in progress: creating tablet
W20250113 02:51:30.703219 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 936b8ceaedd34a019a4fb89a80dd2708 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 936b8ceaedd34a019a4fb89a80dd2708 already in progress: creating tablet
I20250113 02:51:30.703433 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet 0810b9671f824d0a84741bc1904df247 (table default.Table_796801d17a3c458693693af6e39cf0f5 [id=df24b379d986402187cec33190501012]) successfully deleted
I20250113 02:51:30.703616 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet ff8ef6290698477391721fd65b9eeb28 (table default.Table_6c7330704479422a817543c4cbbd70d3 [id=7e176c2354964cdd8e6b724f6bb84b99]) successfully deleted
I20250113 02:51:30.703801 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet 840773ee80c7447097ea000b106fc89d (table default.Table_5aafd46e1cbf4e03a5b87797d5d97bdd [id=b6d8d0de849a4d3c988b8fd468b14d62]) successfully deleted
I20250113 02:51:30.704378 736 tablet_service.cc:1514] Processing DeleteTablet for tablet acae3ff634104b5d931ece75d1eba5d6 with delete_type TABLET_DATA_DELETED (replaced by tablet 95de1523f15d43fa8e9b12b75860b1dc) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:30.704471 1191 tablet_bootstrap.cc:492] T bffec36d567544c5a1c87e64a7e9730b P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:30.704588 1191 ts_tablet_manager.cc:1397] T bffec36d567544c5a1c87e64a7e9730b P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.021s user 0.009s sys 0.001s
I20250113 02:51:30.705102 1198 ts_tablet_manager.cc:1905] T 572e4eed1e924381bada27f8389821a7 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:30.705595 1191 raft_consensus.cc:357] T bffec36d567544c5a1c87e64a7e9730b P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:30.705866 1191 raft_consensus.cc:383] T bffec36d567544c5a1c87e64a7e9730b P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:30.705925 1191 raft_consensus.cc:738] T bffec36d567544c5a1c87e64a7e9730b P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:30.706146 1191 consensus_queue.cc:260] T bffec36d567544c5a1c87e64a7e9730b P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:30.707259 1191 ts_tablet_manager.cc:1428] T bffec36d567544c5a1c87e64a7e9730b P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20250113 02:51:30.702174 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_1a6787745a7a47f9834090244c9f5654"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:30.707934 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1a6787745a7a47f9834090244c9f5654 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:30.709060 1191 tablet_bootstrap.cc:492] T 23e04961f072440fa3e3f9485e40f79c P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:30.709877 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_f069a5be395340ee9e86f3f25eaa5197" } new_table_name: "default.Table_acdc5be1660742d7a356474b86e8b35f" modify_external_catalogs: true
I20250113 02:51:30.710536 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_5f5a6c9d71fa4b5dbd75fc758c95af38"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:30.710949 736 tablet_service.cc:1514] Processing DeleteTablet for tablet 573cffe308cf4774a2108a7390f08e90 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:30 UTC) from {username='slave'} at 127.0.0.1:55398
W20250113 02:51:30.711053 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5f5a6c9d71fa4b5dbd75fc758c95af38 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:30.711216 1191 tablet_bootstrap.cc:654] T 23e04961f072440fa3e3f9485e40f79c P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:30.711650 736 tablet_service.cc:1514] Processing DeleteTablet for tablet 42fad91fe1b64a21a2c8412a2585af47 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:30 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:30.712334 1887 raft_consensus.cc:491] T 1cfbcd682b944c89847b40fe69f0a75e P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250113 02:51:30.712352 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet acae3ff634104b5d931ece75d1eba5d6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet acae3ff634104b5d931ece75d1eba5d6 already in progress: creating tablet
I20250113 02:51:30.712467 1887 raft_consensus.cc:513] T 1cfbcd682b944c89847b40fe69f0a75e P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
W20250113 02:51:30.712610 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 573cffe308cf4774a2108a7390f08e90 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 573cffe308cf4774a2108a7390f08e90 already in progress: creating tablet
W20250113 02:51:30.712808 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 42fad91fe1b64a21a2c8412a2585af47 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 42fad91fe1b64a21a2c8412a2585af47 already in progress: creating tablet
I20250113 02:51:30.713151 1887 leader_election.cc:290] T 1cfbcd682b944c89847b40fe69f0a75e P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:30.714916 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cfbcd682b944c89847b40fe69f0a75e" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
W20250113 02:51:30.715548 709 leader_election.cc:343] T 1cfbcd682b944c89847b40fe69f0a75e P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:30.722998 1191 tablet_bootstrap.cc:492] T 23e04961f072440fa3e3f9485e40f79c P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:30.723168 1191 ts_tablet_manager.cc:1397] T 23e04961f072440fa3e3f9485e40f79c P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.000s
I20250113 02:51:30.724118 1191 raft_consensus.cc:357] T 23e04961f072440fa3e3f9485e40f79c P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:30.724373 1191 raft_consensus.cc:383] T 23e04961f072440fa3e3f9485e40f79c P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:30.724444 1191 raft_consensus.cc:738] T 23e04961f072440fa3e3f9485e40f79c P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:30.724749 1191 consensus_queue.cc:260] T 23e04961f072440fa3e3f9485e40f79c P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:30.725842 1191 ts_tablet_manager.cc:1428] T 23e04961f072440fa3e3f9485e40f79c P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20250113 02:51:30.727437 1191 tablet_bootstrap.cc:492] T b0b267b0d9c54ba496c89f6cb287dffd P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:30.728047 736 tablet_service.cc:1514] Processing DeleteTablet for tablet efccefd6787943469131492741ee2bd5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:21 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:30.728277 1195 tablet_replica.cc:331] T efccefd6787943469131492741ee2bd5 P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:30.728444 1195 raft_consensus.cc:2238] T efccefd6787943469131492741ee2bd5 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:30.728596 1195 raft_consensus.cc:2267] T efccefd6787943469131492741ee2bd5 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:30.728623 736 tablet_service.cc:1514] Processing DeleteTablet for tablet 41fcde988f72496b824271fa36c1a226 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:23 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:30.729566 736 tablet_service.cc:1514] Processing DeleteTablet for tablet a77c38a394114d6c8a8da99d7eef7669 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:20 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:30.729786 1195 ts_tablet_manager.cc:1905] T efccefd6787943469131492741ee2bd5 P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:30.730472 1191 tablet_bootstrap.cc:654] T b0b267b0d9c54ba496c89f6cb287dffd P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:30.734784 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet f362b8ab6c464159a7dc2164ac92d7c1
I20250113 02:51:30.735374 452 catalog_manager.cc:6596] ReplaceTablet: tablet f362b8ab6c464159a7dc2164ac92d7c1 deleted and replaced by tablet f748397a164d42e3a7fda3585673e633
I20250113 02:51:30.736105 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet f362b8ab6c464159a7dc2164ac92d7c1 with delete_type TABLET_DATA_DELETED (replaced by tablet f748397a164d42e3a7fda3585673e633) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:30.738111 1195 ts_tablet_manager.cc:1918] T efccefd6787943469131492741ee2bd5 P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:30.738270 1195 log.cc:1198] T efccefd6787943469131492741ee2bd5 P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/efccefd6787943469131492741ee2bd5
I20250113 02:51:30.738651 1195 ts_tablet_manager.cc:1939] T efccefd6787943469131492741ee2bd5 P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:30.739184 736 tablet_service.cc:1514] Processing DeleteTablet for tablet f362b8ab6c464159a7dc2164ac92d7c1 with delete_type TABLET_DATA_DELETED (replaced by tablet f748397a164d42e3a7fda3585673e633) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:30.739521 1195 tablet_replica.cc:331] T 41fcde988f72496b824271fa36c1a226 P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:30.739730 1195 raft_consensus.cc:2238] T 41fcde988f72496b824271fa36c1a226 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:30.739879 1195 raft_consensus.cc:2267] T 41fcde988f72496b824271fa36c1a226 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:30.740578 1195 ts_tablet_manager.cc:1905] T 41fcde988f72496b824271fa36c1a226 P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:30.742393 1198 ts_tablet_manager.cc:1918] T 572e4eed1e924381bada27f8389821a7 P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250113 02:51:30.743584 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_da161f6412074e37b23da57bd4febeee" } modify_external_catalogs: true
I20250113 02:51:30.743832 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_da161f6412074e37b23da57bd4febeee" } modify_external_catalogs: true
I20250113 02:51:30.744851 1191 tablet_bootstrap.cc:492] T b0b267b0d9c54ba496c89f6cb287dffd P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:30.745010 1191 ts_tablet_manager.cc:1397] T b0b267b0d9c54ba496c89f6cb287dffd P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.000s
I20250113 02:51:30.745513 736 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d3562a48c6e4badbbcf37640e91b5ba with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:21 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:30.746042 1191 raft_consensus.cc:357] T b0b267b0d9c54ba496c89f6cb287dffd P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:30.746268 1191 raft_consensus.cc:383] T b0b267b0d9c54ba496c89f6cb287dffd P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:30.746418 736 tablet_service.cc:1514] Processing DeleteTablet for tablet 23e04961f072440fa3e3f9485e40f79c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:20 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:30.746428 1191 raft_consensus.cc:738] T b0b267b0d9c54ba496c89f6cb287dffd P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:30.746771 1198 log.cc:1198] T 572e4eed1e924381bada27f8389821a7 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/572e4eed1e924381bada27f8389821a7
I20250113 02:51:30.746704 1191 consensus_queue.cc:260] T b0b267b0d9c54ba496c89f6cb287dffd P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:30.746979 736 tablet_service.cc:1514] Processing DeleteTablet for tablet bffec36d567544c5a1c87e64a7e9730b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:21 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:30.747787 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet efccefd6787943469131492741ee2bd5 (table default.Table_faf2fafcef874bf6a5798ee6b19ef704 [id=bcdd761e38934a2e9197b9c1b9165b27]) successfully deleted
I20250113 02:51:30.748431 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_b7013bb28efb4fd5a1bdb6b64048e437" } new_table_name: "default.Table_aac49fd87c1f4c3f9fc59bd8afe6e48f" modify_external_catalogs: true
I20250113 02:51:30.749814 1191 ts_tablet_manager.cc:1428] T b0b267b0d9c54ba496c89f6cb287dffd P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20250113 02:51:30.750120 1191 tablet_bootstrap.cc:492] T 8337f3c82d654ecdac2cbc1ffa88ef05 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:30.750453 1198 ts_tablet_manager.cc:1939] T 572e4eed1e924381bada27f8389821a7 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:30.752643 1191 tablet_bootstrap.cc:654] T 8337f3c82d654ecdac2cbc1ffa88ef05 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:30.753582 1059 raft_consensus.cc:1270] T 401c4f5dbd304950877b59a865f417b3 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Refusing update from remote peer 1186fa869eb14812832fa6562531f196: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20250113 02:51:30.754549 711 consensus_peers.cc:487] T 401c4f5dbd304950877b59a865f417b3 P 1186fa869eb14812832fa6562531f196 -> Peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Couldn't send request to peer df69f48656b44bb890f76587ee69cc0f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250113 02:51:30.755260 1195 ts_tablet_manager.cc:1918] T 41fcde988f72496b824271fa36c1a226 P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:30.755200 1904 consensus_queue.cc:1035] T 401c4f5dbd304950877b59a865f417b3 P 1186fa869eb14812832fa6562531f196 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 02:51:30.755396 1195 log.cc:1198] T 41fcde988f72496b824271fa36c1a226 P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/41fcde988f72496b824271fa36c1a226
I20250113 02:51:30.757443 1195 ts_tablet_manager.cc:1939] T 41fcde988f72496b824271fa36c1a226 P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:30.759443 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet 41fcde988f72496b824271fa36c1a226 (table default.Table_2caed919541e43c78a858513a2128f50 [id=26ba87b9bd384b99a016b29fe5b5babd]) successfully deleted
I20250113 02:51:30.760350 1195 tablet_replica.cc:331] T a77c38a394114d6c8a8da99d7eef7669 P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:30.760542 1195 raft_consensus.cc:2238] T a77c38a394114d6c8a8da99d7eef7669 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:30.761102 1195 raft_consensus.cc:2267] T a77c38a394114d6c8a8da99d7eef7669 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:30.765240 1872 tablet.cc:1721] T 401c4f5dbd304950877b59a865f417b3 P 1186fa869eb14812832fa6562531f196: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250113 02:51:30.775413 1906 tablet.cc:1721] T 401c4f5dbd304950877b59a865f417b3 P 4216a7f853d642b7bc3e5cff458f3aca: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250113 02:51:30.776582 1195 ts_tablet_manager.cc:1905] T a77c38a394114d6c8a8da99d7eef7669 P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:30.778177 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet 572e4eed1e924381bada27f8389821a7 (table default.Table_1d910c3d2b244474a576fc4088301143 [id=75cb0745bbb84e1c883cb13b546ecb11]) successfully deleted
I20250113 02:51:30.786621 455 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_0fcdb63a92324044972d1e69630f524d" } modify_external_catalogs: true
I20250113 02:51:30.787256 1198 tablet_replica.cc:331] T a0ca2e3a3ce94f3aa20df64c89c7cf95 P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:30.787479 1198 raft_consensus.cc:2238] T a0ca2e3a3ce94f3aa20df64c89c7cf95 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:30.787549 1195 ts_tablet_manager.cc:1918] T a77c38a394114d6c8a8da99d7eef7669 P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:30.787673 1195 log.cc:1198] T a77c38a394114d6c8a8da99d7eef7669 P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/a77c38a394114d6c8a8da99d7eef7669
I20250113 02:51:30.787662 1198 raft_consensus.cc:2267] T a0ca2e3a3ce94f3aa20df64c89c7cf95 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus is shut down!
W20250113 02:51:30.787748 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet ef99c223dcf04b11bf785703df55f877 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ef99c223dcf04b11bf785703df55f877 already in progress: creating tablet
W20250113 02:51:30.788189 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 79aaab73797c474bae06831fa308b066 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 79aaab73797c474bae06831fa308b066 already in progress: creating tablet
I20250113 02:51:30.788290 1198 ts_tablet_manager.cc:1905] T a0ca2e3a3ce94f3aa20df64c89c7cf95 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250113 02:51:30.788412 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 51d9483ecacb413796d0fd9e7560eac1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 51d9483ecacb413796d0fd9e7560eac1 already in progress: creating tablet
W20250113 02:51:30.788625 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 65f42cda151744c3bd3b0ec9415253c6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 65f42cda151744c3bd3b0ec9415253c6 already in progress: creating tablet
W20250113 02:51:30.788866 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet a0385f2c6aad45748ebf86d2cfdd40d6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a0385f2c6aad45748ebf86d2cfdd40d6 already in progress: creating tablet
W20250113 02:51:30.789116 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet ee289c3236834dc48d922062041be3b5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ee289c3236834dc48d922062041be3b5 already in progress: creating tablet
W20250113 02:51:30.789299 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet f21eb3fa0f414816a29e3d4560cdc0fe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f21eb3fa0f414816a29e3d4560cdc0fe already in progress: creating tablet
W20250113 02:51:30.789498 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 4b07a985897641c3ac48021c2a4daedb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4b07a985897641c3ac48021c2a4daedb already in progress: creating tablet
W20250113 02:51:30.789714 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 9174df876fe64adc805fba0a36fb4d38 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9174df876fe64adc805fba0a36fb4d38 already in progress: creating tablet
W20250113 02:51:30.789924 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 936b8ceaedd34a019a4fb89a80dd2708 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 936b8ceaedd34a019a4fb89a80dd2708 already in progress: creating tablet
W20250113 02:51:30.790112 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 0953b2f89e35402c9e9e84f3589c23ac because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0953b2f89e35402c9e9e84f3589c23ac already in progress: creating tablet
I20250113 02:51:30.790493 455 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_0fcdb63a92324044972d1e69630f524d" } modify_external_catalogs: true
I20250113 02:51:30.791677 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_ccebddd68b0448a49d408b9922a78a4d" } new_table_name: "default.Table_669ef67f4e884a939ff3b3a0bc409ce3" modify_external_catalogs: true
I20250113 02:51:30.793604 736 tablet_service.cc:1514] Processing DeleteTablet for tablet b0b267b0d9c54ba496c89f6cb287dffd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:25 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:30.796737 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_7d2663697456437195949d256c39cb4e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:30.797384 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7d2663697456437195949d256c39cb4e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:30.800021 1198 ts_tablet_manager.cc:1918] T a0ca2e3a3ce94f3aa20df64c89c7cf95 P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:30.801365 1195 ts_tablet_manager.cc:1939] T a77c38a394114d6c8a8da99d7eef7669 P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:30.802403 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet a77c38a394114d6c8a8da99d7eef7669 (table default.Table_649d3c3eaebd4d4da258735b332a4a75 [id=d1b33c3e180343f5b88dcb200e7c9dde]) successfully deleted
W20250113 02:51:30.804104 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet f362b8ab6c464159a7dc2164ac92d7c1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f362b8ab6c464159a7dc2164ac92d7c1 already in progress: creating tablet
I20250113 02:51:30.804733 1195 tablet_replica.cc:331] T 7d3562a48c6e4badbbcf37640e91b5ba P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:30.805191 1195 raft_consensus.cc:2238] T 7d3562a48c6e4badbbcf37640e91b5ba P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:30.806685 1198 log.cc:1198] T a0ca2e3a3ce94f3aa20df64c89c7cf95 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/a0ca2e3a3ce94f3aa20df64c89c7cf95
I20250113 02:51:30.807111 1198 ts_tablet_manager.cc:1939] T a0ca2e3a3ce94f3aa20df64c89c7cf95 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:30.504477 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f68f9f4076d9475d938e795e72bdd468" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:30.808092 1196 log.cc:1198] T fdbf03c89bff438a9a25115eddf60be9 P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/fdbf03c89bff438a9a25115eddf60be9
I20250113 02:51:30.506520 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0810b9671f824d0a84741bc1904df247" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:30.515154 891 tablet_service.cc:1467] Processing CreateTablet for tablet 2460e7cddcd84788a54dab342e7bb27b (DEFAULT_TABLE table=default.Table_b2598c65e9d04d8aa594a9f3d61d1e5a [id=e4f6f4ec007c454eba94263a37690f56]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:30.808781 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2460e7cddcd84788a54dab342e7bb27b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:30.808866 1196 ts_tablet_manager.cc:1939] T fdbf03c89bff438a9a25115eddf60be9 P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
W20250113 02:51:30.808970 711 leader_election.cc:343] T 0810b9671f824d0a84741bc1904df247 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:30.809084 711 leader_election.cc:304] T 0810b9671f824d0a84741bc1904df247 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
W20250113 02:51:30.809252 711 raft_consensus.cc:2681] T 0810b9671f824d0a84741bc1904df247 P 1186fa869eb14812832fa6562531f196: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250113 02:51:30.809494 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet fdbf03c89bff438a9a25115eddf60be9 (table default.Table_732e990e6f154d6486889304bbf88cef [id=48c7c3f6b4a840ee9b9e3f3844add13d]) successfully deleted
W20250113 02:51:30.808537 998 leader_election.cc:343] T f68f9f4076d9475d938e795e72bdd468 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:30.811777 998 leader_election.cc:304] T f68f9f4076d9475d938e795e72bdd468 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4216a7f853d642b7bc3e5cff458f3aca; no voters: 1186fa869eb14812832fa6562531f196, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:30.541476 899 tablet_service.cc:1514] Processing DeleteTablet for tablet 0953b2f89e35402c9e9e84f3589c23ac with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:30 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:30.543495 1873 tablet.cc:1721] T 8adfa738c17a4169a5d99f6ceca75100 P df69f48656b44bb890f76587ee69cc0f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20250113 02:51:30.812070 998 raft_consensus.cc:2681] T f68f9f4076d9475d938e795e72bdd468 P 4216a7f853d642b7bc3e5cff458f3aca: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250113 02:51:30.561182 886 tablet_service.cc:1514] Processing DeleteTablet for tablet 936b8ceaedd34a019a4fb89a80dd2708 with delete_type TABLET_DATA_DELETED (replaced by tablet eba80548f52f4c32a098d1540c9cded0) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:30.813477 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet a0ca2e3a3ce94f3aa20df64c89c7cf95 (table default.Table_f08525e2a08e48ec85ef55287d0bac49 [id=96ff1ee9327f41a5a1bb2757b188254a]) successfully deleted
I20250113 02:51:30.813808 1198 tablet_replica.cc:331] T c94eedb5dce44e6d896192c93127903a P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:30.814006 1198 raft_consensus.cc:2238] T c94eedb5dce44e6d896192c93127903a P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: Raft consensus shutting down.
I20250113 02:51:30.814227 1198 pending_rounds.cc:62] T c94eedb5dce44e6d896192c93127903a P 4216a7f853d642b7bc3e5cff458f3aca: Trying to abort 1 pending ops.
I20250113 02:51:30.814268 1198 pending_rounds.cc:69] T c94eedb5dce44e6d896192c93127903a P 4216a7f853d642b7bc3e5cff458f3aca: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113673483646603264 op_type: NO_OP noop_request { }
I20250113 02:51:30.814370 1198 raft_consensus.cc:2883] T c94eedb5dce44e6d896192c93127903a P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250113 02:51:30.814441 1198 raft_consensus.cc:2267] T c94eedb5dce44e6d896192c93127903a P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:30.817862 1198 ts_tablet_manager.cc:1905] T c94eedb5dce44e6d896192c93127903a P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:30.820053 1192 tablet_bootstrap.cc:492] T 11950e996c814590bc9f3f22c0354902 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:30.820197 1192 ts_tablet_manager.cc:1397] T 11950e996c814590bc9f3f22c0354902 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.202s user 0.024s sys 0.054s
I20250113 02:51:30.820322 1191 tablet_bootstrap.cc:492] T 8337f3c82d654ecdac2cbc1ffa88ef05 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:30.820448 1191 ts_tablet_manager.cc:1397] T 8337f3c82d654ecdac2cbc1ffa88ef05 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.070s user 0.010s sys 0.010s
I20250113 02:51:30.821036 1192 raft_consensus.cc:357] T 11950e996c814590bc9f3f22c0354902 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:30.821233 1192 raft_consensus.cc:383] T 11950e996c814590bc9f3f22c0354902 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:30.821157 1191 raft_consensus.cc:357] T 8337f3c82d654ecdac2cbc1ffa88ef05 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:30.821280 1192 raft_consensus.cc:738] T 11950e996c814590bc9f3f22c0354902 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:30.821327 1191 raft_consensus.cc:383] T 8337f3c82d654ecdac2cbc1ffa88ef05 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:30.821374 1191 raft_consensus.cc:738] T 8337f3c82d654ecdac2cbc1ffa88ef05 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:30.821517 1192 consensus_queue.cc:260] T 11950e996c814590bc9f3f22c0354902 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:30.822005 1192 ts_tablet_manager.cc:1428] T 11950e996c814590bc9f3f22c0354902 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250113 02:51:30.822285 1192 tablet_bootstrap.cc:492] T 0810b9671f824d0a84741bc1904df247 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
W20250113 02:51:30.823488 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 79aaab73797c474bae06831fa308b066 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 79aaab73797c474bae06831fa308b066 already in progress: creating tablet
I20250113 02:51:30.822296 1191 consensus_queue.cc:260] T 8337f3c82d654ecdac2cbc1ffa88ef05 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:30.571123 889 tablet_service.cc:1514] Processing DeleteTablet for tablet 9174df876fe64adc805fba0a36fb4d38 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:30 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:30.823926 1192 tablet_bootstrap.cc:654] T 0810b9671f824d0a84741bc1904df247 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:30.616263 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0ca2e3a3ce94f3aa20df64c89c7cf95" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:30.824504 998 leader_election.cc:343] T a0ca2e3a3ce94f3aa20df64c89c7cf95 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:30.824597 998 leader_election.cc:304] T a0ca2e3a3ce94f3aa20df64c89c7cf95 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4216a7f853d642b7bc3e5cff458f3aca; no voters: 1186fa869eb14812832fa6562531f196, df69f48656b44bb890f76587ee69cc0f
W20250113 02:51:30.824748 998 raft_consensus.cc:2681] T a0ca2e3a3ce94f3aa20df64c89c7cf95 P 4216a7f853d642b7bc3e5cff458f3aca: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250113 02:51:30.825623 1192 tablet_bootstrap.cc:492] T 0810b9671f824d0a84741bc1904df247 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
W20250113 02:51:30.825606 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 51d9483ecacb413796d0fd9e7560eac1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 51d9483ecacb413796d0fd9e7560eac1 already in progress: creating tablet
I20250113 02:51:30.825726 1192 ts_tablet_manager.cc:1397] T 0810b9671f824d0a84741bc1904df247 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
W20250113 02:51:30.825842 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 65f42cda151744c3bd3b0ec9415253c6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 65f42cda151744c3bd3b0ec9415253c6 already in progress: creating tablet
W20250113 02:51:30.826081 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet a0385f2c6aad45748ebf86d2cfdd40d6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a0385f2c6aad45748ebf86d2cfdd40d6 already in progress: creating tablet
I20250113 02:51:30.617543 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4806b18a4194f9c95350d89ba62e13a" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:30.677024 880 tablet_service.cc:1514] Processing DeleteTablet for tablet acae3ff634104b5d931ece75d1eba5d6 with delete_type TABLET_DATA_DELETED (replaced by tablet 95de1523f15d43fa8e9b12b75860b1dc) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:30.686540 883 tablet_service.cc:1514] Processing DeleteTablet for tablet 573cffe308cf4774a2108a7390f08e90 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:30 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:30.686689 895 tablet_service.cc:1514] Processing DeleteTablet for tablet 42fad91fe1b64a21a2c8412a2585af47 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:30 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:30.713759 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cfbcd682b944c89847b40fe69f0a75e" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:30.737526 896 tablet_service.cc:1514] Processing DeleteTablet for tablet f362b8ab6c464159a7dc2164ac92d7c1 with delete_type TABLET_DATA_DELETED (replaced by tablet f748397a164d42e3a7fda3585673e633) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:30.795845 1438 raft_consensus.cc:491] T 258fad818b7b4aa4b337f9949e20aa46 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250113 02:51:30.826789 998 leader_election.cc:343] T b4806b18a4194f9c95350d89ba62e13a P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:30.826748 1192 raft_consensus.cc:357] T 0810b9671f824d0a84741bc1904df247 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:30.826869 998 leader_election.cc:304] T b4806b18a4194f9c95350d89ba62e13a P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4216a7f853d642b7bc3e5cff458f3aca; no voters: 1186fa869eb14812832fa6562531f196, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:30.826928 1192 raft_consensus.cc:383] T 0810b9671f824d0a84741bc1904df247 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:30.826792 1438 raft_consensus.cc:513] T 258fad818b7b4aa4b337f9949e20aa46 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:30.826972 1192 raft_consensus.cc:738] T 0810b9671f824d0a84741bc1904df247 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
W20250113 02:51:30.827019 998 raft_consensus.cc:2681] T b4806b18a4194f9c95350d89ba62e13a P 4216a7f853d642b7bc3e5cff458f3aca: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250113 02:51:30.827162 1192 consensus_queue.cc:260] T 0810b9671f824d0a84741bc1904df247 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:30.827483 1438 leader_election.cc:290] T 258fad818b7b4aa4b337f9949e20aa46 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587), 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
W20250113 02:51:30.826969 711 leader_election.cc:343] T 1cfbcd682b944c89847b40fe69f0a75e P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:30.827590 1192 ts_tablet_manager.cc:1428] T 0810b9671f824d0a84741bc1904df247 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250113 02:51:30.827620 711 leader_election.cc:304] T 1cfbcd682b944c89847b40fe69f0a75e P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:30.827818 1192 tablet_bootstrap.cc:492] T 1cfbcd682b944c89847b40fe69f0a75e P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:30.827924 1191 ts_tablet_manager.cc:1428] T 8337f3c82d654ecdac2cbc1ffa88ef05 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20250113 02:51:30.828055 1904 raft_consensus.cc:2743] T 1cfbcd682b944c89847b40fe69f0a75e P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:30.828531 1191 tablet_bootstrap.cc:492] T 42fad91fe1b64a21a2c8412a2585af47 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
W20250113 02:51:30.828627 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet ee289c3236834dc48d922062041be3b5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ee289c3236834dc48d922062041be3b5 already in progress: creating tablet
I20250113 02:51:30.828971 1196 tablet_replica.cc:331] T 258fad818b7b4aa4b337f9949e20aa46 P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:30.829159 1196 raft_consensus.cc:2238] T 258fad818b7b4aa4b337f9949e20aa46 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:30.829301 1196 raft_consensus.cc:2267] T 258fad818b7b4aa4b337f9949e20aa46 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:30.829536 1192 tablet_bootstrap.cc:654] T 1cfbcd682b944c89847b40fe69f0a75e P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
W20250113 02:51:30.831727 851 leader_election.cc:343] T 258fad818b7b4aa4b337f9949e20aa46 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Not found: Tablet not found: 258fad818b7b4aa4b337f9949e20aa46
W20250113 02:51:30.832161 852 leader_election.cc:343] T 258fad818b7b4aa4b337f9949e20aa46 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Not found: Tablet not found: 258fad818b7b4aa4b337f9949e20aa46
I20250113 02:51:30.832299 852 leader_election.cc:304] T 258fad818b7b4aa4b337f9949e20aa46 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df69f48656b44bb890f76587ee69cc0f; no voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca
W20250113 02:51:30.832461 852 raft_consensus.cc:2681] T 258fad818b7b4aa4b337f9949e20aa46 P df69f48656b44bb890f76587ee69cc0f: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250113 02:51:30.835790 1196 ts_tablet_manager.cc:1905] T 258fad818b7b4aa4b337f9949e20aa46 P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:30.806576 1194 tablet_bootstrap.cc:492] T 2c399784af46426b8327d175c7a6cce8 P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:30.837857 1194 ts_tablet_manager.cc:1397] T 2c399784af46426b8327d175c7a6cce8 P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.512s user 0.014s sys 0.007s
I20250113 02:51:30.838536 1195 raft_consensus.cc:2267] T 7d3562a48c6e4badbbcf37640e91b5ba P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:30.838892 1194 raft_consensus.cc:357] T 2c399784af46426b8327d175c7a6cce8 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:30.839174 1194 raft_consensus.cc:383] T 2c399784af46426b8327d175c7a6cce8 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:30.839282 1194 raft_consensus.cc:738] T 2c399784af46426b8327d175c7a6cce8 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:30.839542 1194 consensus_queue.cc:260] T 2c399784af46426b8327d175c7a6cce8 P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:30.840445 1195 ts_tablet_manager.cc:1905] T 7d3562a48c6e4badbbcf37640e91b5ba P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:30.840970 1194 ts_tablet_manager.cc:1428] T 2c399784af46426b8327d175c7a6cce8 P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250113 02:51:30.841295 1194 tablet_bootstrap.cc:492] T f760fc1e4bed433dbb987a42c12a1ee6 P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:30.845582 1194 tablet_bootstrap.cc:654] T f760fc1e4bed433dbb987a42c12a1ee6 P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:30.846235 1191 tablet_bootstrap.cc:654] T 42fad91fe1b64a21a2c8412a2585af47 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:30.851830 1195 ts_tablet_manager.cc:1918] T 7d3562a48c6e4badbbcf37640e91b5ba P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:30.851941 1195 log.cc:1198] T 7d3562a48c6e4badbbcf37640e91b5ba P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/7d3562a48c6e4badbbcf37640e91b5ba
I20250113 02:51:30.852236 1195 ts_tablet_manager.cc:1939] T 7d3562a48c6e4badbbcf37640e91b5ba P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:30.853214 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet 7d3562a48c6e4badbbcf37640e91b5ba (table default.Table_06f81a97cb014a3bad594e88c8cf4733 [id=7c2bc6d4232f4585b84d71e3e1f85794]) successfully deleted
I20250113 02:51:30.854594 1198 ts_tablet_manager.cc:1918] T c94eedb5dce44e6d896192c93127903a P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250113 02:51:30.855865 1195 tablet_replica.cc:331] T 23e04961f072440fa3e3f9485e40f79c P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:30.856093 1195 raft_consensus.cc:2238] T 23e04961f072440fa3e3f9485e40f79c P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:30.856766 1195 raft_consensus.cc:2267] T 23e04961f072440fa3e3f9485e40f79c P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:30.857703 1195 ts_tablet_manager.cc:1905] T 23e04961f072440fa3e3f9485e40f79c P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:30.858055 1198 log.cc:1198] T c94eedb5dce44e6d896192c93127903a P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/c94eedb5dce44e6d896192c93127903a
I20250113 02:51:30.858640 1198 ts_tablet_manager.cc:1939] T c94eedb5dce44e6d896192c93127903a P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
W20250113 02:51:30.860397 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9438472464304da9a0c7658d4f0736b6 consensus state has no leader
I20250113 02:51:30.862154 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_5f7bbbd09edc47978900639818dae863"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:30.862792 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5f7bbbd09edc47978900639818dae863 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:30.866344 454 master_service.cc:946] ReplaceTablet: received request to replace tablet 55ebe9c044d944bc813396c699ba9aa1 from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:30.866642 1191 tablet_bootstrap.cc:492] T 42fad91fe1b64a21a2c8412a2585af47 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:30.866756 1191 ts_tablet_manager.cc:1397] T 42fad91fe1b64a21a2c8412a2585af47 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.038s user 0.001s sys 0.003s
I20250113 02:51:30.867483 1191 raft_consensus.cc:357] T 42fad91fe1b64a21a2c8412a2585af47 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:30.867676 1191 raft_consensus.cc:383] T 42fad91fe1b64a21a2c8412a2585af47 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:30.867722 1191 raft_consensus.cc:738] T 42fad91fe1b64a21a2c8412a2585af47 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:30.867940 1191 consensus_queue.cc:260] T 42fad91fe1b64a21a2c8412a2585af47 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:30.869058 1191 ts_tablet_manager.cc:1428] T 42fad91fe1b64a21a2c8412a2585af47 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250113 02:51:30.869438 1904 raft_consensus.cc:491] T 4c615b0b83d74fcfae0c51047dbfbbf4 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:30.869506 1191 tablet_bootstrap.cc:492] T c1fb2b36c0504cb4ba32671ae4c620d1 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:30.869565 1904 raft_consensus.cc:513] T 4c615b0b83d74fcfae0c51047dbfbbf4 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:30.870162 1904 leader_election.cc:290] T 4c615b0b83d74fcfae0c51047dbfbbf4 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:30.870641 1195 ts_tablet_manager.cc:1918] T 23e04961f072440fa3e3f9485e40f79c P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:30.870750 1195 log.cc:1198] T 23e04961f072440fa3e3f9485e40f79c P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/23e04961f072440fa3e3f9485e40f79c
I20250113 02:51:30.871064 1195 ts_tablet_manager.cc:1939] T 23e04961f072440fa3e3f9485e40f79c P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:30.871515 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 9798a2b066ea4caaa6761d62644b34e9
I20250113 02:51:30.871953 1191 tablet_bootstrap.cc:654] T c1fb2b36c0504cb4ba32671ae4c620d1 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:30.872119 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet 23e04961f072440fa3e3f9485e40f79c (table default.Table_4a0258998d4f413b80d701c74526b86d [id=6fd9215b11274a76b06813227889927b]) successfully deleted
I20250113 02:51:30.872457 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c615b0b83d74fcfae0c51047dbfbbf4" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:30.872711 736 tablet_service.cc:1514] Processing DeleteTablet for tablet 9798a2b066ea4caaa6761d62644b34e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:30 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:30.873131 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c615b0b83d74fcfae0c51047dbfbbf4" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:30.873827 891 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c399784af46426b8327d175c7a6cce8 with delete_type TABLET_DATA_DELETED (replaced by tablet 9e736211eb3c4536b9fd1d90d2d83838) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:30.873960 1192 tablet_bootstrap.cc:492] T 1cfbcd682b944c89847b40fe69f0a75e P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:30.874032 889 tablet_service.cc:1514] Processing DeleteTablet for tablet 9798a2b066ea4caaa6761d62644b34e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:30 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:30.874091 1192 ts_tablet_manager.cc:1397] T 1cfbcd682b944c89847b40fe69f0a75e P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.046s user 0.019s sys 0.002s
W20250113 02:51:30.874248 709 leader_election.cc:343] T 4c615b0b83d74fcfae0c51047dbfbbf4 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
W20250113 02:51:30.874773 711 leader_election.cc:343] T 4c615b0b83d74fcfae0c51047dbfbbf4 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:30.874886 711 leader_election.cc:304] T 4c615b0b83d74fcfae0c51047dbfbbf4 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:30.875101 1192 raft_consensus.cc:357] T 1cfbcd682b944c89847b40fe69f0a75e P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:30.875299 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet c94eedb5dce44e6d896192c93127903a (table default.Table_616eda5ca6ce4155b20351b6b8b000f5 [id=e0cdd94a9fc947fc8a9555150b7272f0]) successfully deleted
I20250113 02:51:30.875347 1192 raft_consensus.cc:383] T 1cfbcd682b944c89847b40fe69f0a75e P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:30.875396 1192 raft_consensus.cc:738] T 1cfbcd682b944c89847b40fe69f0a75e P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
W20250113 02:51:30.875564 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet acae3ff634104b5d931ece75d1eba5d6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet acae3ff634104b5d931ece75d1eba5d6 already in progress: creating tablet
I20250113 02:51:30.875608 1192 consensus_queue.cc:260] T 1cfbcd682b944c89847b40fe69f0a75e P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
W20250113 02:51:30.875849 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 42fad91fe1b64a21a2c8412a2585af47 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 42fad91fe1b64a21a2c8412a2585af47 already in progress: creating tablet
I20250113 02:51:30.876180 1192 ts_tablet_manager.cc:1428] T 1cfbcd682b944c89847b40fe69f0a75e P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
W20250113 02:51:30.876312 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 573cffe308cf4774a2108a7390f08e90 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 573cffe308cf4774a2108a7390f08e90 already in progress: creating tablet
W20250113 02:51:30.878165 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet f362b8ab6c464159a7dc2164ac92d7c1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f362b8ab6c464159a7dc2164ac92d7c1 already in progress: creating tablet
I20250113 02:51:30.878423 1904 raft_consensus.cc:2743] T 4c615b0b83d74fcfae0c51047dbfbbf4 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:30.879303 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_9ad8a6a024ee4ff7a664add48a47c662"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:30.879550 1191 tablet_bootstrap.cc:492] T c1fb2b36c0504cb4ba32671ae4c620d1 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:30.879676 1191 ts_tablet_manager.cc:1397] T c1fb2b36c0504cb4ba32671ae4c620d1 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20250113 02:51:30.880038 1192 tablet_bootstrap.cc:492] T 092892a36e0e47cb8d2184c2bbde95d7 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:30.880403 1191 raft_consensus.cc:357] T c1fb2b36c0504cb4ba32671ae4c620d1 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:30.880584 1191 raft_consensus.cc:383] T c1fb2b36c0504cb4ba32671ae4c620d1 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:30.880638 1191 raft_consensus.cc:738] T c1fb2b36c0504cb4ba32671ae4c620d1 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:30.880898 1191 consensus_queue.cc:260] T c1fb2b36c0504cb4ba32671ae4c620d1 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:30.881385 1191 ts_tablet_manager.cc:1428] T c1fb2b36c0504cb4ba32671ae4c620d1 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
W20250113 02:51:30.881148 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9ad8a6a024ee4ff7a664add48a47c662 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:30.881631 1191 tablet_bootstrap.cc:492] T c508724edea147e5bf6e20bca2f50397 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:30.883483 1192 tablet_bootstrap.cc:654] T 092892a36e0e47cb8d2184c2bbde95d7 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:30.883595 1191 tablet_bootstrap.cc:654] T c508724edea147e5bf6e20bca2f50397 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:30.883725 1195 tablet_replica.cc:331] T bffec36d567544c5a1c87e64a7e9730b P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:30.883896 1195 raft_consensus.cc:2238] T bffec36d567544c5a1c87e64a7e9730b P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:30.884075 1195 raft_consensus.cc:2267] T bffec36d567544c5a1c87e64a7e9730b P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:30.887511 1195 ts_tablet_manager.cc:1905] T bffec36d567544c5a1c87e64a7e9730b P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:30.888573 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_33d7dda5ebe341bc8ef01b7edb3cd1c6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:30.889147 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_33d7dda5ebe341bc8ef01b7edb3cd1c6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:30.890916 1031 tablet_service.cc:1514] Processing DeleteTablet for tablet 9798a2b066ea4caaa6761d62644b34e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:30 UTC) from {username='slave'} at 127.0.0.1:60230
W20250113 02:51:30.891582 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 9798a2b066ea4caaa6761d62644b34e9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9798a2b066ea4caaa6761d62644b34e9 already in progress: creating tablet
I20250113 02:51:30.892711 1191 tablet_bootstrap.cc:492] T c508724edea147e5bf6e20bca2f50397 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:30.892849 1191 ts_tablet_manager.cc:1397] T c508724edea147e5bf6e20bca2f50397 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.005s
I20250113 02:51:30.893572 1191 raft_consensus.cc:357] T c508724edea147e5bf6e20bca2f50397 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:30.893755 1191 raft_consensus.cc:383] T c508724edea147e5bf6e20bca2f50397 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:30.893803 1191 raft_consensus.cc:738] T c508724edea147e5bf6e20bca2f50397 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:30.894024 1191 consensus_queue.cc:260] T c508724edea147e5bf6e20bca2f50397 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:30.898891 1192 tablet_bootstrap.cc:492] T 092892a36e0e47cb8d2184c2bbde95d7 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:30.899034 1192 ts_tablet_manager.cc:1397] T 092892a36e0e47cb8d2184c2bbde95d7 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.001s
I20250113 02:51:30.901984 1191 ts_tablet_manager.cc:1428] T c508724edea147e5bf6e20bca2f50397 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20250113 02:51:30.902302 1191 tablet_bootstrap.cc:492] T 6d4d8b5cc8ad4c9e935d8afaf5766e66 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:30.904969 1191 tablet_bootstrap.cc:654] T 6d4d8b5cc8ad4c9e935d8afaf5766e66 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:30.914402 1192 raft_consensus.cc:357] T 092892a36e0e47cb8d2184c2bbde95d7 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:30.914705 1192 raft_consensus.cc:383] T 092892a36e0e47cb8d2184c2bbde95d7 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:30.914778 1192 raft_consensus.cc:738] T 092892a36e0e47cb8d2184c2bbde95d7 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:30.915073 1192 consensus_queue.cc:260] T 092892a36e0e47cb8d2184c2bbde95d7 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:30.915671 1192 ts_tablet_manager.cc:1428] T 092892a36e0e47cb8d2184c2bbde95d7 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.017s user 0.001s sys 0.001s
I20250113 02:51:30.916226 1192 tablet_bootstrap.cc:492] T 840773ee80c7447097ea000b106fc89d P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:30.918049 1192 tablet_bootstrap.cc:654] T 840773ee80c7447097ea000b106fc89d P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
W20250113 02:51:30.919679 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5f7c91d8c018455e847e5969f7b5e1cc consensus state has no leader
I20250113 02:51:30.919920 1195 ts_tablet_manager.cc:1918] T bffec36d567544c5a1c87e64a7e9730b P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:30.920058 1195 log.cc:1198] T bffec36d567544c5a1c87e64a7e9730b P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/bffec36d567544c5a1c87e64a7e9730b
I20250113 02:51:30.920382 1195 ts_tablet_manager.cc:1939] T bffec36d567544c5a1c87e64a7e9730b P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:30.921447 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet bffec36d567544c5a1c87e64a7e9730b (table default.Table_c79704f27bf14cba8a0afc2dd36a143f [id=f26e79c7d9164cc89d184f1ffbd21036]) successfully deleted
I20250113 02:51:30.922056 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_9b57c1f76abd4ef1b85f88572215474a" } new_table_name: "default.Table_3b46963a95a74a1d964cb3fef22375ba" modify_external_catalogs: true
I20250113 02:51:30.923183 1191 tablet_bootstrap.cc:492] T 6d4d8b5cc8ad4c9e935d8afaf5766e66 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:30.923573 1191 ts_tablet_manager.cc:1397] T 6d4d8b5cc8ad4c9e935d8afaf5766e66 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.021s user 0.008s sys 0.004s
I20250113 02:51:30.924336 1191 raft_consensus.cc:357] T 6d4d8b5cc8ad4c9e935d8afaf5766e66 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:30.924527 1191 raft_consensus.cc:383] T 6d4d8b5cc8ad4c9e935d8afaf5766e66 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:30.924589 1191 raft_consensus.cc:738] T 6d4d8b5cc8ad4c9e935d8afaf5766e66 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:30.924795 1191 consensus_queue.cc:260] T 6d4d8b5cc8ad4c9e935d8afaf5766e66 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:30.926313 1191 ts_tablet_manager.cc:1428] T 6d4d8b5cc8ad4c9e935d8afaf5766e66 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250113 02:51:30.926558 1195 tablet_replica.cc:331] T b0b267b0d9c54ba496c89f6cb287dffd P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:30.926811 1195 raft_consensus.cc:2238] T b0b267b0d9c54ba496c89f6cb287dffd P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:30.926986 1195 raft_consensus.cc:2267] T b0b267b0d9c54ba496c89f6cb287dffd P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:30.927855 1195 ts_tablet_manager.cc:1905] T b0b267b0d9c54ba496c89f6cb287dffd P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250113 02:51:30.928676 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5d2d14548b734b1fbe57d68c0ea3e2f8 consensus state has no leader
I20250113 02:51:30.929039 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_24ea41f0ca3d4b4b919da4716b6c08a2" } new_table_name: "default.Table_0820d78702cd41a3bfa9b7544842d8d4" modify_external_catalogs: true
I20250113 02:51:30.931929 1926 raft_consensus.cc:491] T 092892a36e0e47cb8d2184c2bbde95d7 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:30.932126 1926 raft_consensus.cc:513] T 092892a36e0e47cb8d2184c2bbde95d7 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:30.932756 1926 leader_election.cc:290] T 092892a36e0e47cb8d2184c2bbde95d7 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
I20250113 02:51:30.933369 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "092892a36e0e47cb8d2184c2bbde95d7" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:30.934033 998 leader_election.cc:343] T 092892a36e0e47cb8d2184c2bbde95d7 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
W20250113 02:51:30.934248 995 leader_election.cc:343] T 092892a36e0e47cb8d2184c2bbde95d7 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Not found: Tablet not found: 092892a36e0e47cb8d2184c2bbde95d7
I20250113 02:51:30.934310 995 leader_election.cc:304] T 092892a36e0e47cb8d2184c2bbde95d7 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4216a7f853d642b7bc3e5cff458f3aca; no voters: 1186fa869eb14812832fa6562531f196, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:30.934602 1926 raft_consensus.cc:2743] T 092892a36e0e47cb8d2184c2bbde95d7 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:30.940260 1191 tablet_bootstrap.cc:492] T 77cf8b0c824f401aba1880eb7fd1807b P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:30.942409 1191 tablet_bootstrap.cc:654] T 77cf8b0c824f401aba1880eb7fd1807b P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:30.945339 1904 raft_consensus.cc:491] T be4b2a8a3927410eaf12e7592646e4af P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:30.945467 1904 raft_consensus.cc:513] T be4b2a8a3927410eaf12e7592646e4af P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:30.946055 1904 leader_election.cc:290] T be4b2a8a3927410eaf12e7592646e4af P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:30.946187 1195 ts_tablet_manager.cc:1918] T b0b267b0d9c54ba496c89f6cb287dffd P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:30.946341 1195 log.cc:1198] T b0b267b0d9c54ba496c89f6cb287dffd P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/b0b267b0d9c54ba496c89f6cb287dffd
I20250113 02:51:30.946651 1195 ts_tablet_manager.cc:1939] T b0b267b0d9c54ba496c89f6cb287dffd P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:30.946866 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be4b2a8a3927410eaf12e7592646e4af" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:30.947654 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet b0b267b0d9c54ba496c89f6cb287dffd (table default.Table_eaa71690c1224509a1b527415d09f340 [id=dae9b614d621428ab168fe6a978799a0]) successfully deleted
I20250113 02:51:30.947657 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be4b2a8a3927410eaf12e7592646e4af" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
W20250113 02:51:30.948283 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 9798a2b066ea4caaa6761d62644b34e9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9798a2b066ea4caaa6761d62644b34e9 already in progress: creating tablet
I20250113 02:51:30.948596 1194 tablet_bootstrap.cc:492] T f760fc1e4bed433dbb987a42c12a1ee6 P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:30.948745 1194 ts_tablet_manager.cc:1397] T f760fc1e4bed433dbb987a42c12a1ee6 P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.108s user 0.007s sys 0.016s
W20250113 02:51:30.949024 711 leader_election.cc:343] T be4b2a8a3927410eaf12e7592646e4af P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
W20250113 02:51:30.949468 709 leader_election.cc:343] T be4b2a8a3927410eaf12e7592646e4af P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:30.949430 1194 raft_consensus.cc:357] T f760fc1e4bed433dbb987a42c12a1ee6 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:30.949560 709 leader_election.cc:304] T be4b2a8a3927410eaf12e7592646e4af P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:30.949620 1194 raft_consensus.cc:383] T f760fc1e4bed433dbb987a42c12a1ee6 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:30.949663 1194 raft_consensus.cc:738] T f760fc1e4bed433dbb987a42c12a1ee6 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:30.949841 1194 consensus_queue.cc:260] T f760fc1e4bed433dbb987a42c12a1ee6 P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:30.950018 1904 raft_consensus.cc:2743] T be4b2a8a3927410eaf12e7592646e4af P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:30.950297 1194 ts_tablet_manager.cc:1428] T f760fc1e4bed433dbb987a42c12a1ee6 P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20250113 02:51:30.950533 1194 tablet_bootstrap.cc:492] T 55ebe9c044d944bc813396c699ba9aa1 P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:30.957176 1194 tablet_bootstrap.cc:654] T 55ebe9c044d944bc813396c699ba9aa1 P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:30.969110 1194 tablet_bootstrap.cc:492] T 55ebe9c044d944bc813396c699ba9aa1 P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:30.969276 1194 ts_tablet_manager.cc:1397] T 55ebe9c044d944bc813396c699ba9aa1 P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.004s
I20250113 02:51:30.970018 1194 raft_consensus.cc:357] T 55ebe9c044d944bc813396c699ba9aa1 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:30.970222 1194 raft_consensus.cc:383] T 55ebe9c044d944bc813396c699ba9aa1 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:30.970275 1194 raft_consensus.cc:738] T 55ebe9c044d944bc813396c699ba9aa1 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:30.970508 1194 consensus_queue.cc:260] T 55ebe9c044d944bc813396c699ba9aa1 P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:30.971952 1194 ts_tablet_manager.cc:1428] T 55ebe9c044d944bc813396c699ba9aa1 P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20250113 02:51:30.973996 1194 tablet_bootstrap.cc:492] T e8ab4516789e4585a101c8766d338af3 P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:30.975616 1194 tablet_bootstrap.cc:654] T e8ab4516789e4585a101c8766d338af3 P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:30.985505 1191 tablet_bootstrap.cc:492] T 77cf8b0c824f401aba1880eb7fd1807b P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:30.985659 1191 ts_tablet_manager.cc:1397] T 77cf8b0c824f401aba1880eb7fd1807b P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.045s user 0.005s sys 0.007s
I20250113 02:51:30.986121 1196 ts_tablet_manager.cc:1918] T 258fad818b7b4aa4b337f9949e20aa46 P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:30.986248 1196 log.cc:1198] T 258fad818b7b4aa4b337f9949e20aa46 P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/258fad818b7b4aa4b337f9949e20aa46
I20250113 02:51:30.986546 1196 ts_tablet_manager.cc:1939] T 258fad818b7b4aa4b337f9949e20aa46 P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:30.986671 1191 raft_consensus.cc:357] T 77cf8b0c824f401aba1880eb7fd1807b P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:30.986932 1191 raft_consensus.cc:383] T 77cf8b0c824f401aba1880eb7fd1807b P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:30.986990 1191 raft_consensus.cc:738] T 77cf8b0c824f401aba1880eb7fd1807b P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:30.987280 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet 258fad818b7b4aa4b337f9949e20aa46 (table default.Table_e44b0cdc7f4640ec867d92338d94426b [id=9538429e42bb48c7b7fbae43644f2b2d]) successfully deleted
I20250113 02:51:30.987253 1191 consensus_queue.cc:260] T 77cf8b0c824f401aba1880eb7fd1807b P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:30.987888 1191 ts_tablet_manager.cc:1428] T 77cf8b0c824f401aba1880eb7fd1807b P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
W20250113 02:51:30.988011 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet f21eb3fa0f414816a29e3d4560cdc0fe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f21eb3fa0f414816a29e3d4560cdc0fe already in progress: creating tablet
W20250113 02:51:30.988590 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 4b07a985897641c3ac48021c2a4daedb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4b07a985897641c3ac48021c2a4daedb already in progress: creating tablet
I20250113 02:51:30.988209 1191 tablet_bootstrap.cc:492] T c826a0b8327f44f9a00e5afe3a36cd8a P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
W20250113 02:51:30.989523 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 0953b2f89e35402c9e9e84f3589c23ac because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0953b2f89e35402c9e9e84f3589c23ac already in progress: creating tablet
I20250113 02:51:30.990823 1191 tablet_bootstrap.cc:654] T c826a0b8327f44f9a00e5afe3a36cd8a P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:31.001935 1989 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.60:42067 every 8 connection(s)
I20250113 02:51:31.005292 1706 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.60:42067
I20250113 02:51:31.009836 1191 tablet_bootstrap.cc:492] T c826a0b8327f44f9a00e5afe3a36cd8a P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:31.010030 1191 ts_tablet_manager.cc:1397] T c826a0b8327f44f9a00e5afe3a36cd8a P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.022s user 0.004s sys 0.005s
I20250113 02:51:31.010912 1191 raft_consensus.cc:357] T c826a0b8327f44f9a00e5afe3a36cd8a P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:31.011181 1191 raft_consensus.cc:383] T c826a0b8327f44f9a00e5afe3a36cd8a P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:31.011263 1191 raft_consensus.cc:738] T c826a0b8327f44f9a00e5afe3a36cd8a P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:31.011508 1191 consensus_queue.cc:260] T c826a0b8327f44f9a00e5afe3a36cd8a P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
W20250113 02:51:31.011888 711 consensus_peers.cc:487] T 9fff52e99c64462a8a73fccc3c22ca63 P 1186fa869eb14812832fa6562531f196 -> Peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Couldn't send request to peer df69f48656b44bb890f76587ee69cc0f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250113 02:51:31.012544 1191 ts_tablet_manager.cc:1428] T c826a0b8327f44f9a00e5afe3a36cd8a P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250113 02:51:31.012832 1191 tablet_bootstrap.cc:492] T 0a7e7713bc8648378b668d8b769ac927 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:31.014438 1191 tablet_bootstrap.cc:654] T 0a7e7713bc8648378b668d8b769ac927 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:31.014989 446 catalog_manager.cc:6071] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: default.Table_c991f9dbd70d4c4682664b8711f05db4 [id=2c64da64ecdb4bf48e11b523533902e8] alter complete (version 1)
W20250113 02:51:31.015429 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d7193e2799a54950902c4ceac65b663b consensus state has no leader
I20250113 02:51:31.015751 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_3b76cf9fac334ebf9b1ecb9f8d07ea74" } modify_external_catalogs: true
I20250113 02:51:31.015956 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_3b76cf9fac334ebf9b1ecb9f8d07ea74" } modify_external_catalogs: true
I20250113 02:51:31.016628 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_67eab203697544008a476488a93df8a9" } modify_external_catalogs: true
I20250113 02:51:31.016821 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_67eab203697544008a476488a93df8a9" } modify_external_catalogs: true
I20250113 02:51:31.017853 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_0bba725d39ff4d44a742e905c275724b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:31.018752 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_f736da3d9cd249a989263c2481ea307c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:31.019466 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3c59ae14a6584892bf1ebf6def81d497 consensus state has no leader
I20250113 02:51:31.019806 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_9f58d5227a9841b9aacf80945f8a65a3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:31.020808 889 tablet_service.cc:1514] Processing DeleteTablet for tablet f760fc1e4bed433dbb987a42c12a1ee6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:19 UTC) from {username='slave'} at 127.0.0.1:37366
W20250113 02:51:31.020839 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 936b8ceaedd34a019a4fb89a80dd2708 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 936b8ceaedd34a019a4fb89a80dd2708 already in progress: creating tablet
W20250113 02:51:31.021087 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet acae3ff634104b5d931ece75d1eba5d6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet acae3ff634104b5d931ece75d1eba5d6 already in progress: creating tablet
W20250113 02:51:31.021309 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 573cffe308cf4774a2108a7390f08e90 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 573cffe308cf4774a2108a7390f08e90 already in progress: creating tablet
I20250113 02:51:31.021347 1191 tablet_bootstrap.cc:492] T 0a7e7713bc8648378b668d8b769ac927 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:31.021484 1191 ts_tablet_manager.cc:1397] T 0a7e7713bc8648378b668d8b769ac927 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.002s
W20250113 02:51:31.021499 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 42fad91fe1b64a21a2c8412a2585af47 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 42fad91fe1b64a21a2c8412a2585af47 already in progress: creating tablet
I20250113 02:51:31.021551 1196 tablet_replica.cc:331] T 2c399784af46426b8327d175c7a6cce8 P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
W20250113 02:51:31.021704 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet f362b8ab6c464159a7dc2164ac92d7c1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f362b8ab6c464159a7dc2164ac92d7c1 already in progress: creating tablet
I20250113 02:51:31.021855 1196 raft_consensus.cc:2238] T 2c399784af46426b8327d175c7a6cce8 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
W20250113 02:51:31.021909 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 9174df876fe64adc805fba0a36fb4d38 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9174df876fe64adc805fba0a36fb4d38 already in progress: creating tablet
I20250113 02:51:31.022130 1196 raft_consensus.cc:2267] T 2c399784af46426b8327d175c7a6cce8 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:31.022286 1191 raft_consensus.cc:357] T 0a7e7713bc8648378b668d8b769ac927 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:31.022495 1191 raft_consensus.cc:383] T 0a7e7713bc8648378b668d8b769ac927 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:31.022552 1191 raft_consensus.cc:738] T 0a7e7713bc8648378b668d8b769ac927 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:31.022826 1191 consensus_queue.cc:260] T 0a7e7713bc8648378b668d8b769ac927 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:31.023553 1191 ts_tablet_manager.cc:1428] T 0a7e7713bc8648378b668d8b769ac927 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250113 02:51:31.024879 1196 ts_tablet_manager.cc:1905] T 2c399784af46426b8327d175c7a6cce8 P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:31.027856 452 master_service.cc:946] ReplaceTablet: received request to replace tablet ef6dfe0431ef4a73a957559f60f24756 from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:31.028699 889 tablet_service.cc:1514] Processing DeleteTablet for tablet 55ebe9c044d944bc813396c699ba9aa1 with delete_type TABLET_DATA_DELETED (replaced by tablet 41fcde988f72496b824271fa36c1a226) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:31.031347 1191 tablet_bootstrap.cc:492] T 0752fde3a30a4947a2a688c56ceadab5 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:31.032578 1926 raft_consensus.cc:491] T 41fcde988f72496b824271fa36c1a226 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:31.032749 1926 raft_consensus.cc:513] T 41fcde988f72496b824271fa36c1a226 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:31.033563 1926 leader_election.cc:290] T 41fcde988f72496b824271fa36c1a226 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:31.034133 1706 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/master-2/data/info.pb
I20250113 02:51:31.034667 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41fcde988f72496b824271fa36c1a226" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:31.035014 995 leader_election.cc:343] T 41fcde988f72496b824271fa36c1a226 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Not found: Tablet not found: 41fcde988f72496b824271fa36c1a226
W20250113 02:51:31.035567 998 leader_election.cc:343] T 41fcde988f72496b824271fa36c1a226 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:31.035670 998 leader_election.cc:304] T 41fcde988f72496b824271fa36c1a226 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4216a7f853d642b7bc3e5cff458f3aca; no voters: 1186fa869eb14812832fa6562531f196, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:31.036074 1926 raft_consensus.cc:2743] T 41fcde988f72496b824271fa36c1a226 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:31.037923 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskXJrxug/build/asan/bin/kudu as pid 1706
I20250113 02:51:31.039377 1191 tablet_bootstrap.cc:654] T 0752fde3a30a4947a2a688c56ceadab5 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:31.039799 1194 tablet_bootstrap.cc:492] T e8ab4516789e4585a101c8766d338af3 P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:31.040096 1194 ts_tablet_manager.cc:1397] T e8ab4516789e4585a101c8766d338af3 P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.066s user 0.004s sys 0.002s
I20250113 02:51:31.041978 1194 raft_consensus.cc:357] T e8ab4516789e4585a101c8766d338af3 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:31.042426 1194 raft_consensus.cc:383] T e8ab4516789e4585a101c8766d338af3 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:31.042523 1194 raft_consensus.cc:738] T e8ab4516789e4585a101c8766d338af3 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:31.058897 1192 tablet_bootstrap.cc:492] T 840773ee80c7447097ea000b106fc89d P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:31.059077 1192 ts_tablet_manager.cc:1397] T 840773ee80c7447097ea000b106fc89d P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.143s user 0.011s sys 0.048s
I20250113 02:51:31.059937 1192 raft_consensus.cc:357] T 840773ee80c7447097ea000b106fc89d P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:31.060216 1192 raft_consensus.cc:383] T 840773ee80c7447097ea000b106fc89d P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:31.060276 1192 raft_consensus.cc:738] T 840773ee80c7447097ea000b106fc89d P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:31.060495 1192 consensus_queue.cc:260] T 840773ee80c7447097ea000b106fc89d P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:31.061003 1192 ts_tablet_manager.cc:1428] T 840773ee80c7447097ea000b106fc89d P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250113 02:51:31.061307 1192 tablet_bootstrap.cc:492] T ff8ef6290698477391721fd65b9eeb28 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:31.063043 1192 tablet_bootstrap.cc:654] T ff8ef6290698477391721fd65b9eeb28 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:31.068614 1191 tablet_bootstrap.cc:492] T 0752fde3a30a4947a2a688c56ceadab5 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:31.068814 1191 ts_tablet_manager.cc:1397] T 0752fde3a30a4947a2a688c56ceadab5 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.038s user 0.003s sys 0.011s
W20250113 02:51:31.070706 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f736da3d9cd249a989263c2481ea307c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:31.071662 736 tablet_service.cc:1514] Processing DeleteTablet for tablet b0b267b0d9c54ba496c89f6cb287dffd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:25 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:31.072103 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_5cfdcca1a3b0477ca840ba718e978c6d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:31.072510 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0bba725d39ff4d44a742e905c275724b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250113 02:51:31.072769 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5cfdcca1a3b0477ca840ba718e978c6d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:31.069696 1191 raft_consensus.cc:357] T 0752fde3a30a4947a2a688c56ceadab5 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:31.074555 1191 raft_consensus.cc:383] T 0752fde3a30a4947a2a688c56ceadab5 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:31.074625 1191 raft_consensus.cc:738] T 0752fde3a30a4947a2a688c56ceadab5 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:31.074815 1191 consensus_queue.cc:260] T 0752fde3a30a4947a2a688c56ceadab5 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:31.076808 1191 ts_tablet_manager.cc:1428] T 0752fde3a30a4947a2a688c56ceadab5 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.008s user 0.000s sys 0.002s
I20250113 02:51:31.077091 1191 tablet_bootstrap.cc:492] T 1a63f485caf54068abd6b50d7de3d14a P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
W20250113 02:51:31.078639 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9f58d5227a9841b9aacf80945f8a65a3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:31.079689 1872 tablet.cc:1721] T e8ab4516789e4585a101c8766d338af3 P 1186fa869eb14812832fa6562531f196: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20250113 02:51:31.080175 439 catalog_manager.cc:4839] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet b0b267b0d9c54ba496c89f6cb287dffd because the tablet was not found. No further retry: Not found: Tablet not found: b0b267b0d9c54ba496c89f6cb287dffd
I20250113 02:51:31.084439 1906 tablet.cc:1721] T e8ab4516789e4585a101c8766d338af3 P 4216a7f853d642b7bc3e5cff458f3aca: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250113 02:51:31.089325 1191 tablet_bootstrap.cc:654] T 1a63f485caf54068abd6b50d7de3d14a P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:31.090329 1192 tablet_bootstrap.cc:492] T ff8ef6290698477391721fd65b9eeb28 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:31.090205 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_2aa1258a7cff4416b0682966ee8150e9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:31.090461 1192 ts_tablet_manager.cc:1397] T ff8ef6290698477391721fd65b9eeb28 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.029s user 0.001s sys 0.002s
W20250113 02:51:31.090803 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2aa1258a7cff4416b0682966ee8150e9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:31.091372 1192 raft_consensus.cc:357] T ff8ef6290698477391721fd65b9eeb28 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:31.091610 1192 raft_consensus.cc:383] T ff8ef6290698477391721fd65b9eeb28 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:31.091670 1192 raft_consensus.cc:738] T ff8ef6290698477391721fd65b9eeb28 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:31.091930 1192 consensus_queue.cc:260] T ff8ef6290698477391721fd65b9eeb28 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:31.092553 1192 ts_tablet_manager.cc:1428] T ff8ef6290698477391721fd65b9eeb28 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250113 02:51:31.092864 1192 tablet_bootstrap.cc:492] T efccefd6787943469131492741ee2bd5 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:31.095688 1192 tablet_bootstrap.cc:654] T efccefd6787943469131492741ee2bd5 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:31.097740 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_2c0aebce064848d7ace63d6a47b5daf7" } new_table_name: "default.Table_511c88ce360749d0a2bbcab80bb64629" modify_external_catalogs: true
I20250113 02:51:31.097828 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_8d74f8c0d6b24538b60e7a61451ad264"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:31.098373 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8d74f8c0d6b24538b60e7a61451ad264 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:31.098462 1191 tablet_bootstrap.cc:492] T 1a63f485caf54068abd6b50d7de3d14a P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:31.098600 1191 ts_tablet_manager.cc:1397] T 1a63f485caf54068abd6b50d7de3d14a P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.022s user 0.004s sys 0.000s
I20250113 02:51:31.100780 1191 raft_consensus.cc:357] T 1a63f485caf54068abd6b50d7de3d14a P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:31.100968 1191 raft_consensus.cc:383] T 1a63f485caf54068abd6b50d7de3d14a P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:31.101025 1191 raft_consensus.cc:738] T 1a63f485caf54068abd6b50d7de3d14a P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:31.101761 1191 consensus_queue.cc:260] T 1a63f485caf54068abd6b50d7de3d14a P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:31.104532 1191 ts_tablet_manager.cc:1428] T 1a63f485caf54068abd6b50d7de3d14a P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250113 02:51:31.104794 1191 tablet_bootstrap.cc:492] T a97efc70946c4af487e33a85abf6b90c P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:31.107666 1194 consensus_queue.cc:260] T e8ab4516789e4585a101c8766d338af3 P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:31.109853 1191 tablet_bootstrap.cc:654] T a97efc70946c4af487e33a85abf6b90c P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:31.110469 1194 ts_tablet_manager.cc:1428] T e8ab4516789e4585a101c8766d338af3 P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.070s user 0.002s sys 0.000s
I20250113 02:51:31.110845 1194 tablet_bootstrap.cc:492] T 19f14b3db0b64847b5422bfd5ed323d0 P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:31.118858 1192 tablet_bootstrap.cc:492] T efccefd6787943469131492741ee2bd5 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:31.119024 1192 ts_tablet_manager.cc:1397] T efccefd6787943469131492741ee2bd5 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.026s user 0.001s sys 0.003s
I20250113 02:51:31.119426 1438 raft_consensus.cc:491] T 55ebe9c044d944bc813396c699ba9aa1 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:31.119563 1438 raft_consensus.cc:513] T 55ebe9c044d944bc813396c699ba9aa1 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:31.120015 1192 raft_consensus.cc:357] T efccefd6787943469131492741ee2bd5 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:31.120160 1438 leader_election.cc:290] T 55ebe9c044d944bc813396c699ba9aa1 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587), 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
I20250113 02:51:31.120213 1192 raft_consensus.cc:383] T efccefd6787943469131492741ee2bd5 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:31.120270 1192 raft_consensus.cc:738] T efccefd6787943469131492741ee2bd5 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:31.120450 1192 consensus_queue.cc:260] T efccefd6787943469131492741ee2bd5 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
W20250113 02:51:31.120810 852 leader_election.cc:343] T 55ebe9c044d944bc813396c699ba9aa1 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Not found: Tablet not found: 55ebe9c044d944bc813396c699ba9aa1
I20250113 02:51:31.120971 1192 ts_tablet_manager.cc:1428] T efccefd6787943469131492741ee2bd5 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250113 02:51:31.121363 1192 tablet_bootstrap.cc:492] T 4c615b0b83d74fcfae0c51047dbfbbf4 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:31.121924 1191 tablet_bootstrap.cc:492] T a97efc70946c4af487e33a85abf6b90c P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
W20250113 02:51:31.122000 851 leader_election.cc:343] T 55ebe9c044d944bc813396c699ba9aa1 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Not found: Tablet not found: 55ebe9c044d944bc813396c699ba9aa1
I20250113 02:51:31.122076 1191 ts_tablet_manager.cc:1397] T a97efc70946c4af487e33a85abf6b90c P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.002s
I20250113 02:51:31.122109 851 leader_election.cc:304] T 55ebe9c044d944bc813396c699ba9aa1 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df69f48656b44bb890f76587ee69cc0f; no voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca
I20250113 02:51:31.122525 1438 raft_consensus.cc:2743] T 55ebe9c044d944bc813396c699ba9aa1 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:31.123044 1192 tablet_bootstrap.cc:654] T 4c615b0b83d74fcfae0c51047dbfbbf4 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:31.124716 1191 raft_consensus.cc:357] T a97efc70946c4af487e33a85abf6b90c P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:31.124931 1191 raft_consensus.cc:383] T a97efc70946c4af487e33a85abf6b90c P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:31.124997 1191 raft_consensus.cc:738] T a97efc70946c4af487e33a85abf6b90c P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:31.125252 1191 consensus_queue.cc:260] T a97efc70946c4af487e33a85abf6b90c P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:31.129854 1191 ts_tablet_manager.cc:1428] T a97efc70946c4af487e33a85abf6b90c P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20250113 02:51:31.134143 1191 tablet_bootstrap.cc:492] T 05bf5ab0df874dd5b6559f6d82069184 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:31.136384 454 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_4484fe40b4454c5ea4943a4d38f49c5e" } modify_external_catalogs: true
I20250113 02:51:31.136610 454 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_4484fe40b4454c5ea4943a4d38f49c5e" } modify_external_catalogs: true
I20250113 02:51:31.136901 1191 tablet_bootstrap.cc:654] T 05bf5ab0df874dd5b6559f6d82069184 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
W20250113 02:51:31.137306 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 524b4bd640a04123b4abbd716e7d40f2 consensus state has no leader
I20250113 02:51:31.137602 449 master_service.cc:946] ReplaceTablet: received request to replace tablet e13267498917432ba92b6badb76f5d73 from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:31.140254 1194 tablet_bootstrap.cc:654] T 19f14b3db0b64847b5422bfd5ed323d0 P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:31.146631 1107 heartbeater.cc:346] Connected to a master server at 127.0.104.60:42067
I20250113 02:51:31.154814 1192 tablet_bootstrap.cc:492] T 4c615b0b83d74fcfae0c51047dbfbbf4 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:31.155074 1192 ts_tablet_manager.cc:1397] T 4c615b0b83d74fcfae0c51047dbfbbf4 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.034s user 0.000s sys 0.008s
I20250113 02:51:31.155959 1192 raft_consensus.cc:357] T 4c615b0b83d74fcfae0c51047dbfbbf4 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:31.156219 1192 raft_consensus.cc:383] T 4c615b0b83d74fcfae0c51047dbfbbf4 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:31.156294 1192 raft_consensus.cc:738] T 4c615b0b83d74fcfae0c51047dbfbbf4 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:31.156555 1192 consensus_queue.cc:260] T 4c615b0b83d74fcfae0c51047dbfbbf4 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:31.157265 1192 ts_tablet_manager.cc:1428] T 4c615b0b83d74fcfae0c51047dbfbbf4 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250113 02:51:31.157819 1192 tablet_bootstrap.cc:492] T bffec36d567544c5a1c87e64a7e9730b P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:31.160347 1904 raft_consensus.cc:491] T 42fad91fe1b64a21a2c8412a2585af47 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:31.160501 1904 raft_consensus.cc:513] T 42fad91fe1b64a21a2c8412a2585af47 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:31.161155 1904 leader_election.cc:290] T 42fad91fe1b64a21a2c8412a2585af47 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:31.161656 1192 tablet_bootstrap.cc:654] T bffec36d567544c5a1c87e64a7e9730b P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:31.162075 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42fad91fe1b64a21a2c8412a2585af47" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:31.162482 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42fad91fe1b64a21a2c8412a2585af47" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:31.163153 709 leader_election.cc:343] T 42fad91fe1b64a21a2c8412a2585af47 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
W20250113 02:51:31.163522 711 leader_election.cc:343] T 42fad91fe1b64a21a2c8412a2585af47 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:31.163626 711 leader_election.cc:304] T 42fad91fe1b64a21a2c8412a2585af47 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:31.164232 1904 raft_consensus.cc:2743] T 42fad91fe1b64a21a2c8412a2585af47 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:31.166837 1196 ts_tablet_manager.cc:1918] T 2c399784af46426b8327d175c7a6cce8 P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:31.167289 1196 log.cc:1198] T 2c399784af46426b8327d175c7a6cce8 P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/2c399784af46426b8327d175c7a6cce8
I20250113 02:51:31.167799 1196 ts_tablet_manager.cc:1939] T 2c399784af46426b8327d175c7a6cce8 P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:31.168432 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet 2c399784af46426b8327d175c7a6cce8 (table default.Table_f8af5d3e0dc64d3fae054f82a7b33ccf [id=4b739228564344b8b394a4718750dbb0]) successfully deleted
I20250113 02:51:31.168792 1107 heartbeater.cc:346] Connected to a master server at 127.0.104.60:42067
W20250113 02:51:31.169189 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 9798a2b066ea4caaa6761d62644b34e9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9798a2b066ea4caaa6761d62644b34e9 already in progress: creating tablet
I20250113 02:51:31.169530 1196 tablet_replica.cc:331] T f760fc1e4bed433dbb987a42c12a1ee6 P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:31.169762 1196 raft_consensus.cc:2238] T f760fc1e4bed433dbb987a42c12a1ee6 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:31.170012 1196 raft_consensus.cc:2267] T f760fc1e4bed433dbb987a42c12a1ee6 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:31.171741 1192 tablet_bootstrap.cc:492] T bffec36d567544c5a1c87e64a7e9730b P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:31.171919 1192 ts_tablet_manager.cc:1397] T bffec36d567544c5a1c87e64a7e9730b P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.005s
I20250113 02:51:31.172942 1192 raft_consensus.cc:357] T bffec36d567544c5a1c87e64a7e9730b P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:31.173202 1192 raft_consensus.cc:383] T bffec36d567544c5a1c87e64a7e9730b P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:31.173282 1192 raft_consensus.cc:738] T bffec36d567544c5a1c87e64a7e9730b P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:31.173555 1192 consensus_queue.cc:260] T bffec36d567544c5a1c87e64a7e9730b P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:31.174292 1192 ts_tablet_manager.cc:1428] T bffec36d567544c5a1c87e64a7e9730b P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250113 02:51:31.174610 1192 tablet_bootstrap.cc:492] T be4b2a8a3927410eaf12e7592646e4af P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:31.178625 1192 tablet_bootstrap.cc:654] T be4b2a8a3927410eaf12e7592646e4af P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
W20250113 02:51:31.183914 998 consensus_peers.cc:487] T 378d59d314a84ddb84cf2ab041c51cf1 P 4216a7f853d642b7bc3e5cff458f3aca -> Peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Couldn't send request to peer df69f48656b44bb890f76587ee69cc0f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250113 02:51:31.188570 447 catalog_manager.cc:5618] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587) does not have the latest schema for tablet 378d59d314a84ddb84cf2ab041c51cf1 (table default.Table_2794f9d2a3cc49f78de33a775c1619f1 [id=0c01441cec7043b687d6744e72a0e45d]). Expected version 1 got 0
I20250113 02:51:31.189082 447 catalog_manager.cc:5618] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587) does not have the latest schema for tablet 32cba8f3efc54e78b4df41c15df82eea (table default.Table_27e1ad9a94f8455f806f48768ef4ddac [id=8aff13d6d3c5483eadb596b705fe5151]). Expected version 5 got 0
I20250113 02:51:31.190801 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet e8ab4516789e4585a101c8766d338af3
I20250113 02:51:31.191597 1031 tablet_service.cc:1514] Processing DeleteTablet for tablet e8ab4516789e4585a101c8766d338af3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:31 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:31.191640 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_5182a1c5c7a2462ba930940ae04b505f" } new_table_name: "default.Table_ee56f8e59e4f40efbaf24ff31dcf0920" modify_external_catalogs: true
I20250113 02:51:31.191854 1198 tablet_replica.cc:331] T e8ab4516789e4585a101c8766d338af3 P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:31.191954 1192 tablet_bootstrap.cc:492] T be4b2a8a3927410eaf12e7592646e4af P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:31.192081 1198 raft_consensus.cc:2238] T e8ab4516789e4585a101c8766d338af3 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:31.192200 1192 ts_tablet_manager.cc:1397] T be4b2a8a3927410eaf12e7592646e4af P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.006s
I20250113 02:51:31.192252 1198 raft_consensus.cc:2267] T e8ab4516789e4585a101c8766d338af3 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:31.193240 453 master_service.cc:946] ReplaceTablet: received request to replace tablet 19f14b3db0b64847b5422bfd5ed323d0 from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:31.193161 1192 raft_consensus.cc:357] T be4b2a8a3927410eaf12e7592646e4af P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:31.193430 1192 raft_consensus.cc:383] T be4b2a8a3927410eaf12e7592646e4af P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:31.193493 1192 raft_consensus.cc:738] T be4b2a8a3927410eaf12e7592646e4af P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:31.193738 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_e3d1ee1c42934e6a92821ecf7a3d1621" } new_table_name: "default.Table_eefebb7e9073472daecb078bef21b08d" modify_external_catalogs: true
I20250113 02:51:31.193789 1192 consensus_queue.cc:260] T be4b2a8a3927410eaf12e7592646e4af P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:31.194587 1192 ts_tablet_manager.cc:1428] T be4b2a8a3927410eaf12e7592646e4af P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250113 02:51:31.194928 1192 tablet_bootstrap.cc:492] T de4a58bf63054890808e2e2fe46e43d2 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:31.195281 736 tablet_service.cc:1514] Processing DeleteTablet for tablet e8ab4516789e4585a101c8766d338af3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:31 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:31.195525 1195 tablet_replica.cc:331] T e8ab4516789e4585a101c8766d338af3 P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:31.195689 1198 ts_tablet_manager.cc:1905] T e8ab4516789e4585a101c8766d338af3 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:31.195716 1195 raft_consensus.cc:2238] T e8ab4516789e4585a101c8766d338af3 P 1186fa869eb14812832fa6562531f196 [term 1 LEADER]: Raft consensus shutting down.
I20250113 02:51:31.196234 1195 raft_consensus.cc:2267] T e8ab4516789e4585a101c8766d338af3 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:31.197059 1195 ts_tablet_manager.cc:1905] T e8ab4516789e4585a101c8766d338af3 P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:31.198639 447 master_service.cc:946] ReplaceTablet: received request to replace tablet f8a6eebb6b6d4b63903e68750e181707 from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:31.200706 1904 raft_consensus.cc:491] T a97efc70946c4af487e33a85abf6b90c P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:31.200838 1904 raft_consensus.cc:513] T a97efc70946c4af487e33a85abf6b90c P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:31.201495 1904 leader_election.cc:290] T a97efc70946c4af487e33a85abf6b90c P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:31.202297 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a97efc70946c4af487e33a85abf6b90c" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
W20250113 02:51:31.203282 709 leader_election.cc:343] T a97efc70946c4af487e33a85abf6b90c P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:31.203383 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet a77c38a394114d6c8a8da99d7eef7669 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:20 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:31.203580 1040 tablet_service.cc:1514] Processing DeleteTablet for tablet c94eedb5dce44e6d896192c93127903a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:20 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:31.203742 1031 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d63f0e5835540a78c82020b20a07dac with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:23 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:31.207190 1198 ts_tablet_manager.cc:1918] T e8ab4516789e4585a101c8766d338af3 P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250113 02:51:31.207379 1198 log.cc:1198] T e8ab4516789e4585a101c8766d338af3 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/e8ab4516789e4585a101c8766d338af3
I20250113 02:51:31.207631 1195 ts_tablet_manager.cc:1918] T e8ab4516789e4585a101c8766d338af3 P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250113 02:51:31.207864 1195 log.cc:1198] T e8ab4516789e4585a101c8766d338af3 P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/e8ab4516789e4585a101c8766d338af3
I20250113 02:51:31.207958 1198 ts_tablet_manager.cc:1939] T e8ab4516789e4585a101c8766d338af3 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:31.208473 1195 ts_tablet_manager.cc:1939] T e8ab4516789e4585a101c8766d338af3 P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:31.209072 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet e8ab4516789e4585a101c8766d338af3 (table default.Table_3b76cf9fac334ebf9b1ecb9f8d07ea74 [id=b112ac4e2e58460db083da1eb788ca07]) successfully deleted
I20250113 02:51:31.209434 1198 tablet_replica.cc:331] T a77c38a394114d6c8a8da99d7eef7669 P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:31.209688 1198 raft_consensus.cc:2238] T a77c38a394114d6c8a8da99d7eef7669 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:31.209908 1198 raft_consensus.cc:2267] T a77c38a394114d6c8a8da99d7eef7669 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:31.210451 1198 ts_tablet_manager.cc:1905] T a77c38a394114d6c8a8da99d7eef7669 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:31.210824 1192 tablet_bootstrap.cc:654] T de4a58bf63054890808e2e2fe46e43d2 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:31.212576 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet e8ab4516789e4585a101c8766d338af3 (table default.Table_3b76cf9fac334ebf9b1ecb9f8d07ea74 [id=b112ac4e2e58460db083da1eb788ca07]) successfully deleted
I20250113 02:51:31.215934 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 2e4a68bbcd684785a7bacdbabb506718
I20250113 02:51:31.217003 736 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e4a68bbcd684785a7bacdbabb506718 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:30 UTC) from {username='slave'} at 127.0.0.1:55398
W20250113 02:51:31.217748 439 catalog_manager.cc:4839] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 2e4a68bbcd684785a7bacdbabb506718 because the tablet was not found. No further retry: Not found: Tablet not found: 2e4a68bbcd684785a7bacdbabb506718
I20250113 02:51:31.217846 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 4c759831773849f2bc52e484ffa0cf49
I20250113 02:51:31.218736 736 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c759831773849f2bc52e484ffa0cf49 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:30 UTC) from {username='slave'} at 127.0.0.1:55398
W20250113 02:51:31.219442 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 4c759831773849f2bc52e484ffa0cf49 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4c759831773849f2bc52e484ffa0cf49 already in progress: creating tablet
I20250113 02:51:31.220041 1904 raft_consensus.cc:491] T 1a63f485caf54068abd6b50d7de3d14a P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:31.220222 1904 raft_consensus.cc:513] T 1a63f485caf54068abd6b50d7de3d14a P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:31.221007 1904 leader_election.cc:290] T 1a63f485caf54068abd6b50d7de3d14a P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:31.221796 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a63f485caf54068abd6b50d7de3d14a" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:31.222066 1040 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c759831773849f2bc52e484ffa0cf49 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:30 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:31.222263 1031 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e4a68bbcd684785a7bacdbabb506718 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:30 UTC) from {username='slave'} at 127.0.0.1:60230
W20250113 02:51:31.223390 709 leader_election.cc:343] T 1a63f485caf54068abd6b50d7de3d14a P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
W20250113 02:51:31.225420 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet cb897004cbfb4a0283b42d06509a9ee5 has no consensus state
W20250113 02:51:31.225883 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c368b5b0c3a940c4b4efb64c946375c3 consensus state has no leader
I20250113 02:51:31.226649 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_bceb3f766ee942aba58803219ef3b245"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:31.227229 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bceb3f766ee942aba58803219ef3b245 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:31.228942 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_df6d561e8e21480ba865be6710c85dc6" } new_table_name: "default.Table_402477cbe689499490baedf0ede9910f" modify_external_catalogs: true
I20250113 02:51:31.230509 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_9c2c7930fdf148e8af11382a5838bdb5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:31.231134 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9c2c7930fdf148e8af11382a5838bdb5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250113 02:51:31.231863 439 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet c508724edea147e5bf6e20bca2f50397 (table default.Table_810d62357c794aeaa18da712725a338f [id=08db71e435a24e36bdb20f9a5f1cc096], current schema version=3) failed: Not found: failed to reset TS proxy: No leader found: Tablet c508724edea147e5bf6e20bca2f50397 consensus state has no leader
I20250113 02:51:31.240782 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_5b0a22c0da534b649ee7a211ee6718fa" } new_table_name: "default.Table_b171295c6e384e378d1dee2a0a362e40" modify_external_catalogs: true
W20250113 02:51:31.244697 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 68c4a7e8c0fb4feb85498709b00d3258 consensus state has no leader
I20250113 02:51:31.245170 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_220afbc6ffc34d63a882ab817a21087f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:31.245692 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_220afbc6ffc34d63a882ab817a21087f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:31.247360 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_635a7ba43392414d91ff119440a6cdc7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:31.247839 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_635a7ba43392414d91ff119440a6cdc7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:31.249119 820 heartbeater.cc:346] Connected to a master server at 127.0.104.60:42067
W20250113 02:51:31.251152 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet cfd1697a03494ad2aaa41a056e334ca7 has no consensus state
I20250113 02:51:31.197034 889 tablet_service.cc:1514] Processing DeleteTablet for tablet e8ab4516789e4585a101c8766d338af3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:31 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:31.202855 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a97efc70946c4af487e33a85abf6b90c" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:31.221503 891 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e4a68bbcd684785a7bacdbabb506718 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:30 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:31.221668 896 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c759831773849f2bc52e484ffa0cf49 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:30 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:31.222894 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a63f485caf54068abd6b50d7de3d14a" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:31.254740 1438 raft_consensus.cc:668] T f760fc1e4bed433dbb987a42c12a1ee6 P df69f48656b44bb890f76587ee69cc0f: failed to trigger leader election: Illegal state: RaftConsensus is not running: State = Stopped
W20250113 02:51:31.255033 711 leader_election.cc:343] T a97efc70946c4af487e33a85abf6b90c P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:31.255226 711 leader_election.cc:304] T a97efc70946c4af487e33a85abf6b90c P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
W20250113 02:51:31.255702 711 leader_election.cc:343] T 1a63f485caf54068abd6b50d7de3d14a P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:31.255798 711 leader_election.cc:304] T 1a63f485caf54068abd6b50d7de3d14a P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:31.256213 1904 raft_consensus.cc:2743] T a97efc70946c4af487e33a85abf6b90c P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:31.256387 1904 raft_consensus.cc:2743] T 1a63f485caf54068abd6b50d7de3d14a P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:31.256392 446 catalog_manager.cc:5618] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095) does not have the latest schema for tablet c508724edea147e5bf6e20bca2f50397 (table default.Table_810d62357c794aeaa18da712725a338f [id=08db71e435a24e36bdb20f9a5f1cc096]). Expected version 3 got 0
I20250113 02:51:31.235630 1194 tablet_bootstrap.cc:492] T 19f14b3db0b64847b5422bfd5ed323d0 P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:31.251758 964 heartbeater.cc:346] Connected to a master server at 127.0.104.60:42067
I20250113 02:51:31.257028 1194 ts_tablet_manager.cc:1397] T 19f14b3db0b64847b5422bfd5ed323d0 P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.146s user 0.002s sys 0.002s
I20250113 02:51:31.258107 1194 raft_consensus.cc:357] T 19f14b3db0b64847b5422bfd5ed323d0 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:31.258445 1194 raft_consensus.cc:383] T 19f14b3db0b64847b5422bfd5ed323d0 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:31.258558 1194 raft_consensus.cc:738] T 19f14b3db0b64847b5422bfd5ed323d0 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:31.258844 1194 consensus_queue.cc:260] T 19f14b3db0b64847b5422bfd5ed323d0 P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:31.259493 745 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d4d8b5cc8ad4c9e935d8afaf5766e66 with delete_type TABLET_DATA_DELETED (replaced by tablet e57ac440265e4646bfc5d5d9ca4c35d2) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:31.259773 1195 tablet_replica.cc:331] T 6d4d8b5cc8ad4c9e935d8afaf5766e66 P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:31.259815 740 tablet_service.cc:1514] Processing DeleteTablet for tablet c826a0b8327f44f9a00e5afe3a36cd8a with delete_type TABLET_DATA_DELETED (replaced by tablet bc533518faa24dffa3306ff092e8d507) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:31.259853 742 tablet_service.cc:1514] Processing DeleteTablet for tablet 77cf8b0c824f401aba1880eb7fd1807b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:25 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:31.259997 1195 raft_consensus.cc:2238] T 6d4d8b5cc8ad4c9e935d8afaf5766e66 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:31.260156 1195 raft_consensus.cc:2267] T 6d4d8b5cc8ad4c9e935d8afaf5766e66 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:31.259745 736 tablet_service.cc:1514] Processing DeleteTablet for tablet 42fad91fe1b64a21a2c8412a2585af47 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:30 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:31.260869 1195 ts_tablet_manager.cc:1905] T 6d4d8b5cc8ad4c9e935d8afaf5766e66 P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:31.259758 746 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a7e7713bc8648378b668d8b769ac927 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:21 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:31.259997 747 tablet_service.cc:1514] Processing DeleteTablet for tablet c1fb2b36c0504cb4ba32671ae4c620d1 with delete_type TABLET_DATA_DELETED (replaced by tablet fb2458e2044d49029ed2d6851db9de25) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:31.264300 1194 ts_tablet_manager.cc:1428] T 19f14b3db0b64847b5422bfd5ed323d0 P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20250113 02:51:31.265426 1194 tablet_bootstrap.cc:492] T f31dd916336a4d8fb3ab015af7de8ccb P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:31.267477 448 catalog_manager.cc:5618] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587) does not have the latest schema for tablet 1cfbcd682b944c89847b40fe69f0a75e (table default.Table_32e88abaae96492fbcce39fb2cd7c221 [id=2b84deb07e5c4ab8b550fe42945d7b71]). Expected version 2 got 0
I20250113 02:51:31.267832 448 catalog_manager.cc:5618] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587) does not have the latest schema for tablet 4c615b0b83d74fcfae0c51047dbfbbf4 (table default.Table_bf689f8eb8884e88a72e9b50598fc389 [id=a956f95f6dd94b578f6bd3cb3b97f9da]). Expected version 2 got 0
I20250113 02:51:31.268090 448 catalog_manager.cc:5618] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587) does not have the latest schema for tablet be4b2a8a3927410eaf12e7592646e4af (table default.Table_24d9af3d0b0740d69c791fc96916ef0a [id=124669ccadaf49e19d2660b1495f9647]). Expected version 3 got 0
I20250113 02:51:31.269418 1195 ts_tablet_manager.cc:1918] T 6d4d8b5cc8ad4c9e935d8afaf5766e66 P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:31.269610 1195 log.cc:1198] T 6d4d8b5cc8ad4c9e935d8afaf5766e66 P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/6d4d8b5cc8ad4c9e935d8afaf5766e66
I20250113 02:51:31.270047 1195 ts_tablet_manager.cc:1939] T 6d4d8b5cc8ad4c9e935d8afaf5766e66 P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:31.271075 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet 6d4d8b5cc8ad4c9e935d8afaf5766e66 (table default.Table_54119752ebc147b8b34429f1c38b71d9 [id=2ab34074f7624b419c1cd36e9a09b4c6]) successfully deleted
I20250113 02:51:31.271422 1195 tablet_replica.cc:331] T c826a0b8327f44f9a00e5afe3a36cd8a P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:31.271631 1195 raft_consensus.cc:2238] T c826a0b8327f44f9a00e5afe3a36cd8a P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:31.271835 1195 raft_consensus.cc:2267] T c826a0b8327f44f9a00e5afe3a36cd8a P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:31.272578 1195 ts_tablet_manager.cc:1905] T c826a0b8327f44f9a00e5afe3a36cd8a P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:31.272969 1194 tablet_bootstrap.cc:654] T f31dd916336a4d8fb3ab015af7de8ccb P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:31.278395 1194 tablet_bootstrap.cc:492] T f31dd916336a4d8fb3ab015af7de8ccb P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:31.278550 1194 ts_tablet_manager.cc:1397] T f31dd916336a4d8fb3ab015af7de8ccb P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.005s
I20250113 02:51:31.279451 1194 raft_consensus.cc:357] T f31dd916336a4d8fb3ab015af7de8ccb P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:31.279706 1194 raft_consensus.cc:383] T f31dd916336a4d8fb3ab015af7de8ccb P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:31.279754 1194 raft_consensus.cc:738] T f31dd916336a4d8fb3ab015af7de8ccb P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:31.282150 1195 ts_tablet_manager.cc:1918] T c826a0b8327f44f9a00e5afe3a36cd8a P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:31.282228 1194 consensus_queue.cc:260] T f31dd916336a4d8fb3ab015af7de8ccb P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:31.283360 964 heartbeater.cc:346] Connected to a master server at 127.0.104.60:42067
I20250113 02:51:31.283548 1195 log.cc:1198] T c826a0b8327f44f9a00e5afe3a36cd8a P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/c826a0b8327f44f9a00e5afe3a36cd8a
I20250113 02:51:31.285753 1195 ts_tablet_manager.cc:1939] T c826a0b8327f44f9a00e5afe3a36cd8a P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:31.287364 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet c826a0b8327f44f9a00e5afe3a36cd8a (table default.Table_03babf194cc448f296512a09b2e3fe3e [id=64b42698782f41a589577af20b33501e]) successfully deleted
I20250113 02:51:31.288796 1195 tablet_replica.cc:331] T 77cf8b0c824f401aba1880eb7fd1807b P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:31.288929 1196 ts_tablet_manager.cc:1905] T f760fc1e4bed433dbb987a42c12a1ee6 P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:31.288964 653 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 2e4a68bbcd684785a7bacdbabb506718
I20250113 02:51:31.289359 1195 raft_consensus.cc:2238] T 77cf8b0c824f401aba1880eb7fd1807b P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:31.289634 896 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e4a68bbcd684785a7bacdbabb506718 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:30 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:31.289798 1195 raft_consensus.cc:2267] T 77cf8b0c824f401aba1880eb7fd1807b P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:31.291043 1195 ts_tablet_manager.cc:1905] T 77cf8b0c824f401aba1880eb7fd1807b P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:31.294713 1992 sys_catalog.cc:263] Verifying existing consensus state
I20250113 02:51:31.296104 1903 raft_consensus.cc:491] T c1fb2b36c0504cb4ba32671ae4c620d1 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:31.296245 1903 raft_consensus.cc:513] T c1fb2b36c0504cb4ba32671ae4c620d1 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:31.296854 1903 leader_election.cc:290] T c1fb2b36c0504cb4ba32671ae4c620d1 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:31.297297 1194 ts_tablet_manager.cc:1428] T f31dd916336a4d8fb3ab015af7de8ccb P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.019s user 0.002s sys 0.001s
I20250113 02:51:31.297267 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1fb2b36c0504cb4ba32671ae4c620d1" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:31.297516 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1fb2b36c0504cb4ba32671ae4c620d1" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:31.297832 709 leader_election.cc:343] T c1fb2b36c0504cb4ba32671ae4c620d1 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
W20250113 02:51:31.298127 711 leader_election.cc:343] T c1fb2b36c0504cb4ba32671ae4c620d1 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:31.298224 711 leader_election.cc:304] T c1fb2b36c0504cb4ba32671ae4c620d1 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:31.297607 1194 tablet_bootstrap.cc:492] T 63efcb6b2e194c469fb30a197bd6565b P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:31.298635 1903 raft_consensus.cc:2743] T c1fb2b36c0504cb4ba32671ae4c620d1 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:31.300164 2010 raft_consensus.cc:491] T ff8ef6290698477391721fd65b9eeb28 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:31.302875 1194 tablet_bootstrap.cc:654] T 63efcb6b2e194c469fb30a197bd6565b P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:31.304409 747 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e4a68bbcd684785a7bacdbabb506718 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:30 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:31.305402 896 tablet_service.cc:1467] Processing CreateTablet for tablet 6d659f76c66d47118edbf80c0fa28d14 (DEFAULT_TABLE table=default.Table_b95dec5993da4c6fb06409ef6cc17a5b [id=64ee5f59543f4481a2399db80d4ae784]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:31.305771 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d659f76c66d47118edbf80c0fa28d14. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:31.305814 747 tablet_service.cc:1467] Processing CreateTablet for tablet 6d659f76c66d47118edbf80c0fa28d14 (DEFAULT_TABLE table=default.Table_b95dec5993da4c6fb06409ef6cc17a5b [id=64ee5f59543f4481a2399db80d4ae784]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:31.306188 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d659f76c66d47118edbf80c0fa28d14. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:31.306819 891 tablet_service.cc:1467] Processing CreateTablet for tablet 5af53982fd3f4c0aa45cbde8a5ca2241 (DEFAULT_TABLE table=default.Table_f68a86b5448042cd898cfd456b3924a7 [id=5929aa5e0e0d416790b1d10ac4c8efdd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:31.307126 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5af53982fd3f4c0aa45cbde8a5ca2241. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:31.323499 1903 consensus_queue.cc:1035] T 9fff52e99c64462a8a73fccc3c22ca63 P 1186fa869eb14812832fa6562531f196 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 02:51:31.300843 1031 tablet_service.cc:1514] Processing DeleteTablet for tablet 0810b9671f824d0a84741bc1904df247 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:20 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:31.301107 1040 tablet_service.cc:1514] Processing DeleteTablet for tablet 41fcde988f72496b824271fa36c1a226 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:23 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:31.301378 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet e8ab4516789e4585a101c8766d338af3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:31 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:31.301652 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet efccefd6787943469131492741ee2bd5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:21 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:31.302098 2012 raft_consensus.cc:491] T 0810b9671f824d0a84741bc1904df247 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:31.302181 1033 tablet_service.cc:1514] Processing DeleteTablet for tablet bffec36d567544c5a1c87e64a7e9730b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:21 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:31.302608 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet 11950e996c814590bc9f3f22c0354902 with delete_type TABLET_DATA_DELETED (replaced by tablet 00d4381876234f16b19336523640bbc8) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:31.303051 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet 092892a36e0e47cb8d2184c2bbde95d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:22 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:31.303282 1041 tablet_service.cc:1514] Processing DeleteTablet for tablet 840773ee80c7447097ea000b106fc89d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:19 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:31.303534 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet ff8ef6290698477391721fd65b9eeb28 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:20 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:31.327397 2012 raft_consensus.cc:513] T 0810b9671f824d0a84741bc1904df247 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:31.303696 1042 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e4a68bbcd684785a7bacdbabb506718 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:30 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:31.314092 1023 tablet_service.cc:1467] Processing CreateTablet for tablet 6d659f76c66d47118edbf80c0fa28d14 (DEFAULT_TABLE table=default.Table_b95dec5993da4c6fb06409ef6cc17a5b [id=64ee5f59543f4481a2399db80d4ae784]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:31.326103 2009 raft_consensus.cc:491] T 32cba8f3efc54e78b4df41c15df82eea P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:31.327795 2009 raft_consensus.cc:513] T 32cba8f3efc54e78b4df41c15df82eea P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:31.328073 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d659f76c66d47118edbf80c0fa28d14. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:31.328554 2009 leader_election.cc:290] T 32cba8f3efc54e78b4df41c15df82eea P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
I20250113 02:51:31.329439 2012 leader_election.cc:290] T 0810b9671f824d0a84741bc1904df247 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
I20250113 02:51:31.327004 2010 raft_consensus.cc:513] T ff8ef6290698477391721fd65b9eeb28 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:31.330091 2010 leader_election.cc:290] T ff8ef6290698477391721fd65b9eeb28 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
I20250113 02:51:31.331933 1194 tablet_bootstrap.cc:492] T 63efcb6b2e194c469fb30a197bd6565b P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:31.331864 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32cba8f3efc54e78b4df41c15df82eea" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:31.332105 1194 ts_tablet_manager.cc:1397] T 63efcb6b2e194c469fb30a197bd6565b P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.035s user 0.002s sys 0.006s
I20250113 02:51:31.332551 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32cba8f3efc54e78b4df41c15df82eea" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
I20250113 02:51:31.332764 775 raft_consensus.cc:2463] T 32cba8f3efc54e78b4df41c15df82eea P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 0.
I20250113 02:51:31.332926 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0810b9671f824d0a84741bc1904df247" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:31.333549 995 leader_election.cc:343] T 0810b9671f824d0a84741bc1904df247 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Not found: Tablet not found: 0810b9671f824d0a84741bc1904df247
W20250113 02:51:31.333726 995 leader_election.cc:343] T ff8ef6290698477391721fd65b9eeb28 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Not found: Tablet not found: ff8ef6290698477391721fd65b9eeb28
I20250113 02:51:31.333729 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff8ef6290698477391721fd65b9eeb28" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:31.333513 1194 raft_consensus.cc:357] T 63efcb6b2e194c469fb30a197bd6565b P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:31.333878 1194 raft_consensus.cc:383] T 63efcb6b2e194c469fb30a197bd6565b P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:31.333885 995 leader_election.cc:304] T 32cba8f3efc54e78b4df41c15df82eea P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters:
I20250113 02:51:31.333933 1194 raft_consensus.cc:738] T 63efcb6b2e194c469fb30a197bd6565b P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:31.334206 1926 raft_consensus.cc:2798] T 32cba8f3efc54e78b4df41c15df82eea P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 02:51:31.334383 1926 raft_consensus.cc:491] T 32cba8f3efc54e78b4df41c15df82eea P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 02:51:31.334797 1926 raft_consensus.cc:3054] T 32cba8f3efc54e78b4df41c15df82eea P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:31.334107 1194 consensus_queue.cc:260] T 63efcb6b2e194c469fb30a197bd6565b P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
W20250113 02:51:31.334648 998 leader_election.cc:343] T 32cba8f3efc54e78b4df41c15df82eea P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:31.335635 1194 ts_tablet_manager.cc:1428] T 63efcb6b2e194c469fb30a197bd6565b P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
W20250113 02:51:31.335956 998 leader_election.cc:343] T 0810b9671f824d0a84741bc1904df247 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:31.337110 998 leader_election.cc:304] T 0810b9671f824d0a84741bc1904df247 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4216a7f853d642b7bc3e5cff458f3aca; no voters: 1186fa869eb14812832fa6562531f196, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:31.337581 2012 raft_consensus.cc:2743] T 0810b9671f824d0a84741bc1904df247 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:31.336139 2010 consensus_queue.cc:1035] T 378d59d314a84ddb84cf2ab041c51cf1 P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 02:51:31.337870 1194 tablet_bootstrap.cc:492] T c215ffe61b9b426286cc5e808463078d P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
W20250113 02:51:31.339098 998 leader_election.cc:343] T ff8ef6290698477391721fd65b9eeb28 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:31.339423 998 leader_election.cc:304] T ff8ef6290698477391721fd65b9eeb28 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4216a7f853d642b7bc3e5cff458f3aca; no voters: 1186fa869eb14812832fa6562531f196, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:31.341441 1194 tablet_bootstrap.cc:654] T c215ffe61b9b426286cc5e808463078d P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:31.341909 2010 raft_consensus.cc:2743] T ff8ef6290698477391721fd65b9eeb28 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:31.345188 1191 tablet_bootstrap.cc:492] T 05bf5ab0df874dd5b6559f6d82069184 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:31.345351 1191 ts_tablet_manager.cc:1397] T 05bf5ab0df874dd5b6559f6d82069184 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.211s user 0.041s sys 0.069s
I20250113 02:51:31.346212 1191 raft_consensus.cc:357] T 05bf5ab0df874dd5b6559f6d82069184 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:31.346410 1191 raft_consensus.cc:383] T 05bf5ab0df874dd5b6559f6d82069184 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:31.346489 1191 raft_consensus.cc:738] T 05bf5ab0df874dd5b6559f6d82069184 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:31.346738 1191 consensus_queue.cc:260] T 05bf5ab0df874dd5b6559f6d82069184 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:31.347277 1191 ts_tablet_manager.cc:1428] T 05bf5ab0df874dd5b6559f6d82069184 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250113 02:51:31.341159 1926 raft_consensus.cc:513] T 32cba8f3efc54e78b4df41c15df82eea P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:31.347535 1191 tablet_bootstrap.cc:492] T b8b6ffe690fc459db78b2b3bc12671dd P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:31.348789 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32cba8f3efc54e78b4df41c15df82eea" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196"
I20250113 02:51:31.348964 772 raft_consensus.cc:3054] T 32cba8f3efc54e78b4df41c15df82eea P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:31.349162 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32cba8f3efc54e78b4df41c15df82eea" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f"
W20250113 02:51:31.349774 998 leader_election.cc:343] T 32cba8f3efc54e78b4df41c15df82eea P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:31.351946 772 raft_consensus.cc:2463] T 32cba8f3efc54e78b4df41c15df82eea P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 1.
I20250113 02:51:31.352974 1191 tablet_bootstrap.cc:654] T b8b6ffe690fc459db78b2b3bc12671dd P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:31.352939 446 catalog_manager.cc:5618] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095) does not have the latest schema for tablet 1a63f485caf54068abd6b50d7de3d14a (table default.Table_f8a6812c7fb94ea69cd9bf775b914ea6 [id=71508871e2d5469a91aaf0e19b02b638]). Expected version 1 got 0
I20250113 02:51:31.354756 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_2bc942069d8446cfb28a36c5bce8c530"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:31.355379 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2bc942069d8446cfb28a36c5bce8c530 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:31.356343 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_2cbb88c393cc492ea908e11fea450a58"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:31.357043 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2cbb88c393cc492ea908e11fea450a58 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:31.356945 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_429f8ee6edea4d9287e029125ee5c861"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:31.357477 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_429f8ee6edea4d9287e029125ee5c861 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:31.358992 995 leader_election.cc:304] T 32cba8f3efc54e78b4df41c15df82eea P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters: df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:31.360335 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_183f57ddd85940d5b90ccf97c7ad0bbf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:31.360792 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_183f57ddd85940d5b90ccf97c7ad0bbf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:31.363286 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_4d8dbfe62717412eb35024d3c0d158b8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:31.367162 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet e13267498917432ba92b6badb76f5d73
I20250113 02:51:31.367610 449 catalog_manager.cc:6596] ReplaceTablet: tablet e13267498917432ba92b6badb76f5d73 deleted and replaced by tablet 166f58d45b7b443ea54087e4ab48eaba
I20250113 02:51:31.368358 747 tablet_service.cc:1514] Processing DeleteTablet for tablet a97efc70946c4af487e33a85abf6b90c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:20 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:31.368706 746 tablet_service.cc:1467] Processing CreateTablet for tablet 5af53982fd3f4c0aa45cbde8a5ca2241 (DEFAULT_TABLE table=default.Table_f68a86b5448042cd898cfd456b3924a7 [id=5929aa5e0e0d416790b1d10ac4c8efdd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:31.369216 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5af53982fd3f4c0aa45cbde8a5ca2241. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:31.370281 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_d2a96259765e46f497379cf235dc8212"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:31.371388 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d2a96259765e46f497379cf235dc8212 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:31.373157 747 tablet_service.cc:1514] Processing DeleteTablet for tablet e13267498917432ba92b6badb76f5d73 with delete_type TABLET_DATA_DELETED (replaced by tablet 166f58d45b7b443ea54087e4ab48eaba) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:31.374150 889 tablet_service.cc:1514] Processing DeleteTablet for tablet 63efcb6b2e194c469fb30a197bd6565b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:20 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:31.374421 895 tablet_service.cc:1514] Processing DeleteTablet for tablet f31dd916336a4d8fb3ab015af7de8ccb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:23 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:31.374696 1042 tablet_service.cc:1514] Processing DeleteTablet for tablet e13267498917432ba92b6badb76f5d73 with delete_type TABLET_DATA_DELETED (replaced by tablet 166f58d45b7b443ea54087e4ab48eaba) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:31.374837 883 tablet_service.cc:1467] Processing CreateTablet for tablet 019bf3b326174af59fe5f76525fe35ab (DEFAULT_TABLE table=default.Table_2abb5d2c250c4d489df2815d6ec71827 [id=23bd3e4b5654488d89d84d6449c517b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:31.375305 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 019bf3b326174af59fe5f76525fe35ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:31.376013 1042 tablet_service.cc:1467] Processing CreateTablet for tablet 5af53982fd3f4c0aa45cbde8a5ca2241 (DEFAULT_TABLE table=default.Table_f68a86b5448042cd898cfd456b3924a7 [id=5929aa5e0e0d416790b1d10ac4c8efdd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:31.376813 2010 raft_consensus.cc:491] T 1cfbcd682b944c89847b40fe69f0a75e P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:31.376935 2010 raft_consensus.cc:513] T 1cfbcd682b944c89847b40fe69f0a75e P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:31.378491 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cfbcd682b944c89847b40fe69f0a75e" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
I20250113 02:51:31.378677 772 raft_consensus.cc:2463] T 1cfbcd682b944c89847b40fe69f0a75e P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 0.
I20250113 02:51:31.379076 995 leader_election.cc:304] T 1cfbcd682b944c89847b40fe69f0a75e P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters:
I20250113 02:51:31.379103 747 tablet_service.cc:1467] Processing CreateTablet for tablet 019bf3b326174af59fe5f76525fe35ab (DEFAULT_TABLE table=default.Table_2abb5d2c250c4d489df2815d6ec71827 [id=23bd3e4b5654488d89d84d6449c517b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:31.379451 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 019bf3b326174af59fe5f76525fe35ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:31.379664 1025 tablet_service.cc:1467] Processing CreateTablet for tablet 019bf3b326174af59fe5f76525fe35ab (DEFAULT_TABLE table=default.Table_2abb5d2c250c4d489df2815d6ec71827 [id=23bd3e4b5654488d89d84d6449c517b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:31.380527 1903 raft_consensus.cc:491] T 8337f3c82d654ecdac2cbc1ffa88ef05 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:31.380643 1903 raft_consensus.cc:513] T 8337f3c82d654ecdac2cbc1ffa88ef05 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:31.381570 1903 leader_election.cc:290] T 8337f3c82d654ecdac2cbc1ffa88ef05 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:31.382037 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8337f3c82d654ecdac2cbc1ffa88ef05" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:31.382186 2010 leader_election.cc:290] T 1cfbcd682b944c89847b40fe69f0a75e P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
I20250113 02:51:31.382339 2010 raft_consensus.cc:2798] T 32cba8f3efc54e78b4df41c15df82eea P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Leader election won for term 1
I20250113 02:51:31.382455 2010 raft_consensus.cc:695] T 32cba8f3efc54e78b4df41c15df82eea P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: Becoming Leader. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Running, Role: LEADER
I20250113 02:51:31.382740 895 tablet_service.cc:1467] Processing CreateTablet for tablet cfd1697a03494ad2aaa41a056e334ca7 (DEFAULT_TABLE table=default.Table_b171295c6e384e378d1dee2a0a362e40 [id=00c70c7d77ea4c29a4abfea9dfb74c7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:31.382695 2010 consensus_queue.cc:237] T 32cba8f3efc54e78b4df41c15df82eea P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:31.382825 2009 raft_consensus.cc:2798] T 1cfbcd682b944c89847b40fe69f0a75e P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 02:51:31.382906 2009 raft_consensus.cc:491] T 1cfbcd682b944c89847b40fe69f0a75e P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 02:51:31.383000 2009 raft_consensus.cc:3054] T 1cfbcd682b944c89847b40fe69f0a75e P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:31.383149 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cfd1697a03494ad2aaa41a056e334ca7. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:31.383139 711 leader_election.cc:343] T 8337f3c82d654ecdac2cbc1ffa88ef05 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:31.384526 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cfbcd682b944c89847b40fe69f0a75e" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:31.385203 1062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8337f3c82d654ecdac2cbc1ffa88ef05" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
W20250113 02:51:31.385618 998 leader_election.cc:343] T 1cfbcd682b944c89847b40fe69f0a75e P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
W20250113 02:51:31.386096 709 leader_election.cc:343] T 8337f3c82d654ecdac2cbc1ffa88ef05 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:31.386195 709 leader_election.cc:304] T 8337f3c82d654ecdac2cbc1ffa88ef05 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
W20250113 02:51:31.386576 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4d8dbfe62717412eb35024d3c0d158b8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:31.396174 1926 leader_election.cc:290] T 32cba8f3efc54e78b4df41c15df82eea P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Requested vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
I20250113 02:51:31.398442 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet df6299f38ffa4c2f9d6c8a5e5285905f
I20250113 02:51:31.399384 747 tablet_service.cc:1514] Processing DeleteTablet for tablet df6299f38ffa4c2f9d6c8a5e5285905f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:31 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:31.400424 451 catalog_manager.cc:5618] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587) does not have the latest schema for tablet 9fff52e99c64462a8a73fccc3c22ca63 (table default.Table_9ef18ee71ad843c5921750a2f8478855 [id=bd8bd572ed524f2cae01f820f07e3c36]). Expected version 2 got 0
I20250113 02:51:31.401191 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_0ba8bb338353424c992f00c5940f3fbc" } new_table_name: "default.Table_d1c2e202f1bc4708ac8ff97a63254ea8" modify_external_catalogs: true
I20250113 02:51:31.401834 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet ef6dfe0431ef4a73a957559f60f24756
I20250113 02:51:31.402269 452 catalog_manager.cc:6596] ReplaceTablet: tablet ef6dfe0431ef4a73a957559f60f24756 deleted and replaced by tablet b86d035e503344e1a3bd85c55131274e
I20250113 02:51:31.402666 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet f8a6eebb6b6d4b63903e68750e181707
I20250113 02:51:31.403450 1903 raft_consensus.cc:2743] T 8337f3c82d654ecdac2cbc1ffa88ef05 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:31.404331 747 tablet_service.cc:1514] Processing DeleteTablet for tablet ef6dfe0431ef4a73a957559f60f24756 with delete_type TABLET_DATA_DELETED (replaced by tablet b86d035e503344e1a3bd85c55131274e) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:31.404446 736 tablet_service.cc:1514] Processing DeleteTablet for tablet f8a6eebb6b6d4b63903e68750e181707 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:31 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:31.404657 1041 tablet_service.cc:1514] Processing DeleteTablet for tablet df6299f38ffa4c2f9d6c8a5e5285905f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:31 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:31.406133 1041 tablet_service.cc:1514] Processing DeleteTablet for tablet ef6dfe0431ef4a73a957559f60f24756 with delete_type TABLET_DATA_DELETED (replaced by tablet b86d035e503344e1a3bd85c55131274e) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:31.406325 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet f8a6eebb6b6d4b63903e68750e181707 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:31 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:31.408373 880 tablet_service.cc:1514] Processing DeleteTablet for tablet f8a6eebb6b6d4b63903e68750e181707 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:31 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:31.408819 447 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_6d2c74a667ab4954a8380355f54d648f" } modify_external_catalogs: true
I20250113 02:51:31.409003 447 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_6d2c74a667ab4954a8380355f54d648f" } modify_external_catalogs: true
I20250113 02:51:31.410866 895 tablet_service.cc:1514] Processing DeleteTablet for tablet df6299f38ffa4c2f9d6c8a5e5285905f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:31 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:31.411247 2010 raft_consensus.cc:491] T 6d63f0e5835540a78c82020b20a07dac P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:31.411438 2010 raft_consensus.cc:513] T 6d63f0e5835540a78c82020b20a07dac P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:31.412197 889 tablet_service.cc:1514] Processing DeleteTablet for tablet ef6dfe0431ef4a73a957559f60f24756 with delete_type TABLET_DATA_DELETED (replaced by tablet b86d035e503344e1a3bd85c55131274e) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:31.412524 2010 leader_election.cc:290] T 6d63f0e5835540a78c82020b20a07dac P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
I20250113 02:51:31.414520 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d63f0e5835540a78c82020b20a07dac" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:31.415107 455 catalog_manager.cc:5618] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095) does not have the latest schema for tablet 378d59d314a84ddb84cf2ab041c51cf1 (table default.Table_2794f9d2a3cc49f78de33a775c1619f1 [id=0c01441cec7043b687d6744e72a0e45d]). Expected version 1 got 0
I20250113 02:51:31.416687 736 tablet_service.cc:1467] Processing CreateTablet for tablet cfd1697a03494ad2aaa41a056e334ca7 (DEFAULT_TABLE table=default.Table_b171295c6e384e378d1dee2a0a362e40 [id=00c70c7d77ea4c29a4abfea9dfb74c7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:31.417104 736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cfd1697a03494ad2aaa41a056e334ca7. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:31.413457 995 leader_election.cc:343] T 6d63f0e5835540a78c82020b20a07dac P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Not found: Tablet not found: 6d63f0e5835540a78c82020b20a07dac
W20250113 02:51:31.420118 998 leader_election.cc:343] T 6d63f0e5835540a78c82020b20a07dac P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:31.420244 998 leader_election.cc:304] T 6d63f0e5835540a78c82020b20a07dac P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4216a7f853d642b7bc3e5cff458f3aca; no voters: 1186fa869eb14812832fa6562531f196, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:31.420733 2012 raft_consensus.cc:2743] T 6d63f0e5835540a78c82020b20a07dac P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:31.423661 747 tablet_service.cc:1514] Processing DeleteTablet for tablet 05bf5ab0df874dd5b6559f6d82069184 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:21 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:31.424593 1992 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45e9d43ecac0401fb2c18fe5eaf089b5: Bootstrap starting.
I20250113 02:51:31.424680 891 tablet_service.cc:1514] Processing DeleteTablet for tablet e13267498917432ba92b6badb76f5d73 with delete_type TABLET_DATA_DELETED (replaced by tablet 166f58d45b7b443ea54087e4ab48eaba) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:31.425869 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_21648ca05a3344848aeac2ad300f7400" } modify_external_catalogs: true
I20250113 02:51:31.426085 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_21648ca05a3344848aeac2ad300f7400" } modify_external_catalogs: true
I20250113 02:51:31.428767 1029 tablet_service.cc:1467] Processing CreateTablet for tablet cfd1697a03494ad2aaa41a056e334ca7 (DEFAULT_TABLE table=default.Table_b171295c6e384e378d1dee2a0a362e40 [id=00c70c7d77ea4c29a4abfea9dfb74c7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:31.429298 454 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_aebf098beedb49a4acfdb767bf5ac454" } new_table_name: "default.Table_d778815edb8043aba80cdc3273511c51" modify_external_catalogs: true
I20250113 02:51:31.430691 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_580e03ac0ded4964948e4809ddc91945" } new_table_name: "default.Table_f04d8a874e0b4594837d8ec8cf2b5fb9" modify_external_catalogs: true
I20250113 02:51:31.435523 1438 raft_consensus.cc:491] T f31dd916336a4d8fb3ab015af7de8ccb P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:31.435710 1438 raft_consensus.cc:513] T f31dd916336a4d8fb3ab015af7de8ccb P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:31.436424 455 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_3b2ea4aaa2ee46acb69f2040e1ebdc41" } modify_external_catalogs: true
I20250113 02:51:31.436609 455 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_3b2ea4aaa2ee46acb69f2040e1ebdc41" } modify_external_catalogs: true
I20250113 02:51:31.437587 1438 leader_election.cc:290] T f31dd916336a4d8fb3ab015af7de8ccb P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
W20250113 02:51:31.438617 851 leader_election.cc:343] T f31dd916336a4d8fb3ab015af7de8ccb P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Not found: Tablet not found: f31dd916336a4d8fb3ab015af7de8ccb
W20250113 02:51:31.439962 852 leader_election.cc:343] T f31dd916336a4d8fb3ab015af7de8ccb P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Not found: Tablet not found: f31dd916336a4d8fb3ab015af7de8ccb
I20250113 02:51:31.440101 852 leader_election.cc:304] T f31dd916336a4d8fb3ab015af7de8ccb P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df69f48656b44bb890f76587ee69cc0f; no voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca
I20250113 02:51:31.441133 1438 raft_consensus.cc:2743] T f31dd916336a4d8fb3ab015af7de8ccb P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:31.444944 896 tablet_service.cc:1514] Processing DeleteTablet for tablet 19f14b3db0b64847b5422bfd5ed323d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:19 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:31.454106 1438 raft_consensus.cc:491] T 63efcb6b2e194c469fb30a197bd6565b P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:31.454279 1438 raft_consensus.cc:513] T 63efcb6b2e194c469fb30a197bd6565b P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:31.455024 1438 leader_election.cc:290] T 63efcb6b2e194c469fb30a197bd6565b P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587), 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
W20250113 02:51:31.456297 851 leader_election.cc:343] T 63efcb6b2e194c469fb30a197bd6565b P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Not found: Tablet not found: 63efcb6b2e194c469fb30a197bd6565b
W20250113 02:51:31.456447 852 leader_election.cc:343] T 63efcb6b2e194c469fb30a197bd6565b P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Not found: Tablet not found: 63efcb6b2e194c469fb30a197bd6565b
I20250113 02:51:31.456552 852 leader_election.cc:304] T 63efcb6b2e194c469fb30a197bd6565b P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df69f48656b44bb890f76587ee69cc0f; no voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca
I20250113 02:51:31.457057 1438 raft_consensus.cc:2743] T 63efcb6b2e194c469fb30a197bd6565b P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:31.469903 1196 ts_tablet_manager.cc:1918] T f760fc1e4bed433dbb987a42c12a1ee6 P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:31.470058 1196 log.cc:1198] T f760fc1e4bed433dbb987a42c12a1ee6 P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/f760fc1e4bed433dbb987a42c12a1ee6
I20250113 02:51:31.470373 1196 ts_tablet_manager.cc:1939] T f760fc1e4bed433dbb987a42c12a1ee6 P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:31.470978 1041 tablet_service.cc:1467] Processing CreateTablet for tablet 16375d658e014cc4bd42a4491c5d64ff (DEFAULT_TABLE table=default.Table_ae350a6e57174618a0dcf77428d84826 [id=1ab6145f7f88438594681721426ed645]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:31.472102 736 tablet_service.cc:1467] Processing CreateTablet for tablet 125ebde7a9f14918ba328a75e88a7746 (DEFAULT_TABLE table=default.Table_6df0545f12364f96a61aafaa9e3ff8f5 [id=ecc77ba623ab40049a2ebdce7665de8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:31.472501 736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 125ebde7a9f14918ba328a75e88a7746. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:31.473001 747 tablet_service.cc:1467] Processing CreateTablet for tablet 9fcf7a6638414f9a99403b3327eeb831 (DEFAULT_TABLE table=default.Table_bbd4259916374725a0e633d94ed73eb4 [id=3e1ef9079d8646a7ae8eba6100298cb3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:31.473335 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9fcf7a6638414f9a99403b3327eeb831. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:31.473834 889 tablet_service.cc:1467] Processing CreateTablet for tablet abc49f968ea94bdbb0443fba6864a04a (DEFAULT_TABLE table=default.Table_28444afd72804ea5bc5099ae319b1321 [id=f1ee9ae051844c85940dab96c2fc7465]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:31.473162 896 tablet_service.cc:1467] Processing CreateTablet for tablet 125ebde7a9f14918ba328a75e88a7746 (DEFAULT_TABLE table=default.Table_6df0545f12364f96a61aafaa9e3ff8f5 [id=ecc77ba623ab40049a2ebdce7665de8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:31.475004 891 tablet_service.cc:1467] Processing CreateTablet for tablet 9fcf7a6638414f9a99403b3327eeb831 (DEFAULT_TABLE table=default.Table_bbd4259916374725a0e633d94ed73eb4 [id=3e1ef9079d8646a7ae8eba6100298cb3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:31.476262 1024 tablet_service.cc:1467] Processing CreateTablet for tablet 125ebde7a9f14918ba328a75e88a7746 (DEFAULT_TABLE table=default.Table_6df0545f12364f96a61aafaa9e3ff8f5 [id=ecc77ba623ab40049a2ebdce7665de8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:31.476336 883 tablet_service.cc:1467] Processing CreateTablet for tablet 16375d658e014cc4bd42a4491c5d64ff (DEFAULT_TABLE table=default.Table_ae350a6e57174618a0dcf77428d84826 [id=1ab6145f7f88438594681721426ed645]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:31.477212 1033 tablet_service.cc:1467] Processing CreateTablet for tablet 9fcf7a6638414f9a99403b3327eeb831 (DEFAULT_TABLE table=default.Table_bbd4259916374725a0e633d94ed73eb4 [id=3e1ef9079d8646a7ae8eba6100298cb3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:31.482931 1192 tablet_bootstrap.cc:492] T de4a58bf63054890808e2e2fe46e43d2 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:31.483611 1192 ts_tablet_manager.cc:1397] T de4a58bf63054890808e2e2fe46e43d2 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.289s user 0.013s sys 0.043s
I20250113 02:51:31.490269 446 catalog_manager.cc:5618] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095) does not have the latest schema for tablet 378d59d314a84ddb84cf2ab041c51cf1 (table default.Table_2794f9d2a3cc49f78de33a775c1619f1 [id=0c01441cec7043b687d6744e72a0e45d]). Expected version 1 got 0
I20250113 02:51:31.496197 746 tablet_service.cc:1467] Processing CreateTablet for tablet 16375d658e014cc4bd42a4491c5d64ff (DEFAULT_TABLE table=default.Table_ae350a6e57174618a0dcf77428d84826 [id=1ab6145f7f88438594681721426ed645]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:31.499251 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_80b52e8cafc549b1b530660a5fd2bb1d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:31.499845 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_80b52e8cafc549b1b530660a5fd2bb1d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:31.499929 747 tablet_service.cc:1467] Processing CreateTablet for tablet abc49f968ea94bdbb0443fba6864a04a (DEFAULT_TABLE table=default.Table_28444afd72804ea5bc5099ae319b1321 [id=f1ee9ae051844c85940dab96c2fc7465]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:31.500330 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet abc49f968ea94bdbb0443fba6864a04a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:31.500422 895 tablet_service.cc:1467] Processing CreateTablet for tablet b37a7f4090cb461faff31d89d71c0261 (DEFAULT_TABLE table=default.Table_5cc38270bc7e429b8491000d00a7a782 [id=dd8205645a9b47039987128e162fdd45]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:31.501292 742 tablet_service.cc:1467] Processing CreateTablet for tablet b37a7f4090cb461faff31d89d71c0261 (DEFAULT_TABLE table=default.Table_5cc38270bc7e429b8491000d00a7a782 [id=dd8205645a9b47039987128e162fdd45]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:31.501430 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_2bc942069d8446cfb28a36c5bce8c530" } new_table_name: "default.Table_d2545982503b4f978884a98f68949dfb" modify_external_catalogs: true
I20250113 02:51:31.501582 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b37a7f4090cb461faff31d89d71c0261. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:31.502980 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_08f7721ed9f1449ba1fbce603d01b13b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:31.503443 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_08f7721ed9f1449ba1fbce603d01b13b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:31.504137 1905 raft_consensus.cc:491] T c508724edea147e5bf6e20bca2f50397 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:31.504259 1905 raft_consensus.cc:513] T c508724edea147e5bf6e20bca2f50397 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:31.504880 1905 leader_election.cc:290] T c508724edea147e5bf6e20bca2f50397 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:31.505559 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c508724edea147e5bf6e20bca2f50397" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:31.506117 711 leader_election.cc:343] T c508724edea147e5bf6e20bca2f50397 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:31.506860 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c508724edea147e5bf6e20bca2f50397" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:31.507431 898 tablet_service.cc:1467] Processing CreateTablet for tablet 62b71866acc941e8b116ec4192a8e1e7 (DEFAULT_TABLE table=default.Table_73f84e2870c44b50b38f55848ef2bbbe [id=9482c1d3706a4ab0a2e8958450b380cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:31.507501 880 tablet_service.cc:1467] Processing CreateTablet for tablet 3a438fd452ba403ca6283138913b3e03 (DEFAULT_TABLE table=default.Table_5d4d2d0f30f44ba89da01f4af3b1a4c3 [id=1bb446f0aaa84882bf815f8d87f60901]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:31.508092 886 tablet_service.cc:1467] Processing CreateTablet for tablet 1f128d9bf2924fb1bada3a8462818004 (DEFAULT_TABLE table=default.Table_fc7ae5b847754786984b5b25591def13 [id=5106be6db8684e5da6d34da81c849b50]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:31.508599 899 tablet_service.cc:1467] Processing CreateTablet for tablet e90242bf273c42beb57ef160991a89ed (DEFAULT_TABLE table=default.Table_3fe958090174491c874058061bc4c252 [id=b5ecafc6fc354091ac68da6f1ba73ca4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:31.508965 1027 tablet_service.cc:1467] Processing CreateTablet for tablet abc49f968ea94bdbb0443fba6864a04a (DEFAULT_TABLE table=default.Table_28444afd72804ea5bc5099ae319b1321 [id=f1ee9ae051844c85940dab96c2fc7465]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:31.509121 881 tablet_service.cc:1467] Processing CreateTablet for tablet f7b219de824c4bfdb097cb81ae5f16f2 (DEFAULT_TABLE table=default.Table_720678f42d2849259edbacc2f1a9c24b [id=2c1ca0ae0b64480c8b9fc4e410a73084]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:31.509481 1035 tablet_service.cc:1467] Processing CreateTablet for tablet b37a7f4090cb461faff31d89d71c0261 (DEFAULT_TABLE table=default.Table_5cc38270bc7e429b8491000d00a7a782 [id=dd8205645a9b47039987128e162fdd45]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:31.509656 897 tablet_service.cc:1467] Processing CreateTablet for tablet fff1a64765ee48729e9e91f1323bf40d (DEFAULT_TABLE table=default.Table_eb8e889492074afe8bdb891316a92caf [id=5ac2c8cce60348989124680c6156cbb6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:31.509939 1040 tablet_service.cc:1467] Processing CreateTablet for tablet 3a438fd452ba403ca6283138913b3e03 (DEFAULT_TABLE table=default.Table_5d4d2d0f30f44ba89da01f4af3b1a4c3 [id=1bb446f0aaa84882bf815f8d87f60901]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:31.510396 1031 tablet_service.cc:1467] Processing CreateTablet for tablet 1f128d9bf2924fb1bada3a8462818004 (DEFAULT_TABLE table=default.Table_fc7ae5b847754786984b5b25591def13 [id=5106be6db8684e5da6d34da81c849b50]), partition=RANGE (key) PARTITION UNBOUNDED
W20250113 02:51:31.510452 709 leader_election.cc:343] T c508724edea147e5bf6e20bca2f50397 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:31.510569 709 leader_election.cc:304] T c508724edea147e5bf6e20bca2f50397 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:31.510824 1037 tablet_service.cc:1467] Processing CreateTablet for tablet e90242bf273c42beb57ef160991a89ed (DEFAULT_TABLE table=default.Table_3fe958090174491c874058061bc4c252 [id=b5ecafc6fc354091ac68da6f1ba73ca4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:31.511071 1905 raft_consensus.cc:2743] T c508724edea147e5bf6e20bca2f50397 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:31.511827 1036 tablet_service.cc:1467] Processing CreateTablet for tablet 62b71866acc941e8b116ec4192a8e1e7 (DEFAULT_TABLE table=default.Table_73f84e2870c44b50b38f55848ef2bbbe [id=9482c1d3706a4ab0a2e8958450b380cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:31.512275 1028 tablet_service.cc:1467] Processing CreateTablet for tablet f7b219de824c4bfdb097cb81ae5f16f2 (DEFAULT_TABLE table=default.Table_720678f42d2849259edbacc2f1a9c24b [id=2c1ca0ae0b64480c8b9fc4e410a73084]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:31.512813 745 tablet_service.cc:1467] Processing CreateTablet for tablet e90242bf273c42beb57ef160991a89ed (DEFAULT_TABLE table=default.Table_3fe958090174491c874058061bc4c252 [id=b5ecafc6fc354091ac68da6f1ba73ca4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:31.512863 755 tablet_service.cc:1467] Processing CreateTablet for tablet f7b219de824c4bfdb097cb81ae5f16f2 (DEFAULT_TABLE table=default.Table_720678f42d2849259edbacc2f1a9c24b [id=2c1ca0ae0b64480c8b9fc4e410a73084]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:31.513056 739 tablet_service.cc:1467] Processing CreateTablet for tablet 62b71866acc941e8b116ec4192a8e1e7 (DEFAULT_TABLE table=default.Table_73f84e2870c44b50b38f55848ef2bbbe [id=9482c1d3706a4ab0a2e8958450b380cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:31.513166 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e90242bf273c42beb57ef160991a89ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:31.513322 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62b71866acc941e8b116ec4192a8e1e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:31.512888 740 tablet_service.cc:1467] Processing CreateTablet for tablet 1f128d9bf2924fb1bada3a8462818004 (DEFAULT_TABLE table=default.Table_fc7ae5b847754786984b5b25591def13 [id=5106be6db8684e5da6d34da81c849b50]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:31.513644 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1f128d9bf2924fb1bada3a8462818004. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:31.516224 748 tablet_service.cc:1467] Processing CreateTablet for tablet fff1a64765ee48729e9e91f1323bf40d (DEFAULT_TABLE table=default.Table_eb8e889492074afe8bdb891316a92caf [id=5ac2c8cce60348989124680c6156cbb6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:31.516537 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fff1a64765ee48729e9e91f1323bf40d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:31.518354 446 catalog_manager.cc:5526] T 32cba8f3efc54e78b4df41c15df82eea P 4216a7f853d642b7bc3e5cff458f3aca reported cstate change: term changed from 0 to 1, leader changed from <none> to 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3). New cstate: current_term: 1 leader_uuid: "4216a7f853d642b7bc3e5cff458f3aca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } health_report { overall_health: UNKNOWN } } }
I20250113 02:51:31.518711 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f7b219de824c4bfdb097cb81ae5f16f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:31.518886 446 catalog_manager.cc:5618] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587) does not have the latest schema for tablet 32cba8f3efc54e78b4df41c15df82eea (table default.Table_27e1ad9a94f8455f806f48768ef4ddac [id=8aff13d6d3c5483eadb596b705fe5151]). Expected version 5 got 0
I20250113 02:51:31.533038 747 tablet_service.cc:1467] Processing CreateTablet for tablet 3a438fd452ba403ca6283138913b3e03 (DEFAULT_TABLE table=default.Table_5d4d2d0f30f44ba89da01f4af3b1a4c3 [id=1bb446f0aaa84882bf815f8d87f60901]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:31.535820 1030 tablet_service.cc:1467] Processing CreateTablet for tablet fff1a64765ee48729e9e91f1323bf40d (DEFAULT_TABLE table=default.Table_eb8e889492074afe8bdb891316a92caf [id=5ac2c8cce60348989124680c6156cbb6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:31.541738 884 tablet_service.cc:1467] Processing CreateTablet for tablet 9fa6f11e7a86410099069389bbdec6c3 (DEFAULT_TABLE table=default.Table_ef81f67732f246589e2e2f2c8808dc00 [id=123edf937eae4e50a175e8ea641c97c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:31.546950 755 tablet_service.cc:1467] Processing CreateTablet for tablet 9fa6f11e7a86410099069389bbdec6c3 (DEFAULT_TABLE table=default.Table_ef81f67732f246589e2e2f2c8808dc00 [id=123edf937eae4e50a175e8ea641c97c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:31.555320 1039 tablet_service.cc:1467] Processing CreateTablet for tablet 9fa6f11e7a86410099069389bbdec6c3 (DEFAULT_TABLE table=default.Table_ef81f67732f246589e2e2f2c8808dc00 [id=123edf937eae4e50a175e8ea641c97c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:31.562184 2009 raft_consensus.cc:513] T 1cfbcd682b944c89847b40fe69f0a75e P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:31.562669 892 tablet_service.cc:1467] Processing CreateTablet for tablet b2a23259f2c448cc8a904a13a291d6ca (DEFAULT_TABLE table=default.Table_174390617a294f1090362caca3ab703f [id=efd834e62cb44ed190770b9763986060]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:31.562875 2009 leader_election.cc:290] T 1cfbcd682b944c89847b40fe69f0a75e P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Requested vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
I20250113 02:51:31.563589 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cfbcd682b944c89847b40fe69f0a75e" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196"
I20250113 02:51:31.563768 775 raft_consensus.cc:3054] T 1cfbcd682b944c89847b40fe69f0a75e P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:31.565373 740 tablet_service.cc:1467] Processing CreateTablet for tablet b2a23259f2c448cc8a904a13a291d6ca (DEFAULT_TABLE table=default.Table_174390617a294f1090362caca3ab703f [id=efd834e62cb44ed190770b9763986060]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:31.566346 775 raft_consensus.cc:2463] T 1cfbcd682b944c89847b40fe69f0a75e P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 1.
I20250113 02:51:31.566550 1196 tablet_replica.cc:331] T 55ebe9c044d944bc813396c699ba9aa1 P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:31.566747 1196 raft_consensus.cc:2238] T 55ebe9c044d944bc813396c699ba9aa1 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:31.566815 995 leader_election.cc:304] T 1cfbcd682b944c89847b40fe69f0a75e P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters:
I20250113 02:51:31.566916 1196 raft_consensus.cc:2267] T 55ebe9c044d944bc813396c699ba9aa1 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:31.566884 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet f760fc1e4bed433dbb987a42c12a1ee6 (table default.Table_c45864b4578e45b59a53d2d913da12f9 [id=f02cfa6472eb42de956f8591417eb7b4]) successfully deleted
I20250113 02:51:31.567394 2009 raft_consensus.cc:2798] T 1cfbcd682b944c89847b40fe69f0a75e P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Leader election won for term 1
I20250113 02:51:31.567451 1196 ts_tablet_manager.cc:1905] T 55ebe9c044d944bc813396c699ba9aa1 P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:31.567519 2009 raft_consensus.cc:695] T 1cfbcd682b944c89847b40fe69f0a75e P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: Becoming Leader. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Running, Role: LEADER
I20250113 02:51:31.567780 2009 consensus_queue.cc:237] T 1cfbcd682b944c89847b40fe69f0a75e P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:31.570809 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cfbcd682b944c89847b40fe69f0a75e" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f"
W20250113 02:51:31.571447 998 leader_election.cc:343] T 1cfbcd682b944c89847b40fe69f0a75e P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
W20250113 02:51:31.573045 439 catalog_manager.cc:4945] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): alter failed for tablet 572e4eed1e924381bada27f8389821a7 (table default.Table_1d910c3d2b244474a576fc4088301143 [id=75cb0745bbb84e1c883cb13b546ecb11]),no further retry: Not found: Tablet not found: 572e4eed1e924381bada27f8389821a7
I20250113 02:51:31.579793 1038 tablet_service.cc:1467] Processing CreateTablet for tablet b2a23259f2c448cc8a904a13a291d6ca (DEFAULT_TABLE table=default.Table_174390617a294f1090362caca3ab703f [id=efd834e62cb44ed190770b9763986060]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:31.581948 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f7b219de824c4bfdb097cb81ae5f16f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:31.582222 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9fa6f11e7a86410099069389bbdec6c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:31.582422 1040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a438fd452ba403ca6283138913b3e03. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:31.588120 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1f128d9bf2924fb1bada3a8462818004. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:31.588454 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b2a23259f2c448cc8a904a13a291d6ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:31.588673 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet abc49f968ea94bdbb0443fba6864a04a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:31.591079 745 tablet_service.cc:1467] Processing CreateTablet for tablet 8898e1e9efce453488fa4a4562a82127 (DEFAULT_TABLE table=default.Table_9417f16fb6384fde8e191ca02808aa78 [id=b49a91d3b86d44e1824a987db3de13b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:31.591493 1036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62b71866acc941e8b116ec4192a8e1e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:31.592149 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fff1a64765ee48729e9e91f1323bf40d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:31.595108 1905 raft_consensus.cc:491] T 05bf5ab0df874dd5b6559f6d82069184 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:31.595358 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b37a7f4090cb461faff31d89d71c0261. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:31.595254 1905 raft_consensus.cc:513] T 05bf5ab0df874dd5b6559f6d82069184 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:31.595937 1905 leader_election.cc:290] T 05bf5ab0df874dd5b6559f6d82069184 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:31.596472 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05bf5ab0df874dd5b6559f6d82069184" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:31.597263 711 leader_election.cc:343] T 05bf5ab0df874dd5b6559f6d82069184 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:31.598075 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05bf5ab0df874dd5b6559f6d82069184" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
W20250113 02:51:31.598747 709 leader_election.cc:343] T 05bf5ab0df874dd5b6559f6d82069184 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:31.598893 709 leader_election.cc:304] T 05bf5ab0df874dd5b6559f6d82069184 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:31.599153 1905 raft_consensus.cc:2743] T 05bf5ab0df874dd5b6559f6d82069184 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:31.600111 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 125ebde7a9f14918ba328a75e88a7746. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:31.607098 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 019bf3b326174af59fe5f76525fe35ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:31.607527 887 tablet_service.cc:1467] Processing CreateTablet for tablet 8898e1e9efce453488fa4a4562a82127 (DEFAULT_TABLE table=default.Table_9417f16fb6384fde8e191ca02808aa78 [id=b49a91d3b86d44e1824a987db3de13b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:31.615360 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cfd1697a03494ad2aaa41a056e334ca7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:31.619513 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 16375d658e014cc4bd42a4491c5d64ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:31.620424 1042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5af53982fd3f4c0aa45cbde8a5ca2241. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:31.625645 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9fcf7a6638414f9a99403b3327eeb831. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:31.629612 1034 tablet_service.cc:1467] Processing CreateTablet for tablet 8898e1e9efce453488fa4a4562a82127 (DEFAULT_TABLE table=default.Table_9417f16fb6384fde8e191ca02808aa78 [id=b49a91d3b86d44e1824a987db3de13b6]), partition=RANGE (key) PARTITION UNBOUNDED
W20250113 02:51:31.632788 451 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f86318637d5b4e378ce47e89a5d52b07 consensus state has no leader
I20250113 02:51:31.633797 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_adf2c40363104ff9a3343f5a7659917f" } new_table_name: "default.Table_6a7ae5ff14af4fa0aacbafcc8a8dfc9e" modify_external_catalogs: true
I20250113 02:51:31.635320 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 88a776aa820c44b3a3854ca5022bd353
I20250113 02:51:31.636111 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_a3a87b4d6edb4210820b2ccc7a71dbd4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:31.636701 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a3a87b4d6edb4210820b2ccc7a71dbd4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:31.636631 742 tablet_service.cc:1514] Processing DeleteTablet for tablet 88a776aa820c44b3a3854ca5022bd353 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:31 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:31.637445 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 1d15c7a01f854f0aaabcd30ec3166e31
I20250113 02:51:31.638187 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_6c9405a4b90a4010b3033c83c9c8a8c4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:31.638682 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6c9405a4b90a4010b3033c83c9c8a8c4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250113 02:51:31.639343 454 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 89b6fbf30ca44488ad7969930f1356e2 consensus state has no leader
I20250113 02:51:31.639703 454 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_a7a5a9569d064b488facac137ec35a85" } modify_external_catalogs: true
I20250113 02:51:31.639871 454 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_a7a5a9569d064b488facac137ec35a85" } modify_external_catalogs: true
I20250113 02:51:31.641834 742 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d15c7a01f854f0aaabcd30ec3166e31 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:31 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:31.642191 1032 tablet_service.cc:1514] Processing DeleteTablet for tablet 88a776aa820c44b3a3854ca5022bd353 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:31 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:31.642309 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d15c7a01f854f0aaabcd30ec3166e31 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:31 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:31.647956 885 tablet_service.cc:1514] Processing DeleteTablet for tablet 88a776aa820c44b3a3854ca5022bd353 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:31 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:31.648603 885 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d15c7a01f854f0aaabcd30ec3166e31 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:31 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:31.650462 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a438fd452ba403ca6283138913b3e03. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:31.652135 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8898e1e9efce453488fa4a4562a82127. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:31.656725 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b2a23259f2c448cc8a904a13a291d6ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:31.660094 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9fa6f11e7a86410099069389bbdec6c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:31.628312 1037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e90242bf273c42beb57ef160991a89ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:31.664453 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8898e1e9efce453488fa4a4562a82127. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:31.668344 1198 ts_tablet_manager.cc:1918] T a77c38a394114d6c8a8da99d7eef7669 P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:31.668556 1198 log.cc:1198] T a77c38a394114d6c8a8da99d7eef7669 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/a77c38a394114d6c8a8da99d7eef7669
I20250113 02:51:31.669308 1198 ts_tablet_manager.cc:1939] T a77c38a394114d6c8a8da99d7eef7669 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:31.670377 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet a77c38a394114d6c8a8da99d7eef7669 (table default.Table_649d3c3eaebd4d4da258735b332a4a75 [id=d1b33c3e180343f5b88dcb200e7c9dde]) successfully deleted
W20250113 02:51:31.671293 440 catalog_manager.cc:4839] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet c94eedb5dce44e6d896192c93127903a because the tablet was not found. No further retry: Not found: Tablet not found: c94eedb5dce44e6d896192c93127903a
I20250113 02:51:31.671695 1198 tablet_replica.cc:331] T 6d63f0e5835540a78c82020b20a07dac P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:31.672148 1198 raft_consensus.cc:2238] T 6d63f0e5835540a78c82020b20a07dac P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:31.672531 1198 raft_consensus.cc:2267] T 6d63f0e5835540a78c82020b20a07dac P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:31.676991 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet abc49f968ea94bdbb0443fba6864a04a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:31.677210 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b2a23259f2c448cc8a904a13a291d6ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:31.677764 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62b71866acc941e8b116ec4192a8e1e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:31.679757 1026 tablet_service.cc:1467] Processing CreateTablet for tablet 702e03e213b14ef5a1b64a02df1b3bf9 (DEFAULT_TABLE table=default.Table_4d5b54216098465fb88c7b9d8b605344 [id=e992f40fef2e466e9a49b8d3307e2a1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:31.680132 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 702e03e213b14ef5a1b64a02df1b3bf9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:31.684549 736 tablet_service.cc:1467] Processing CreateTablet for tablet 7d0d6f8669dd41e39d402ad1d237418d (DEFAULT_TABLE table=default.Table_22ace32627b84d33b3fe01731f8d2c07 [id=2661227c8a684d4eb08d2c46738f7aed]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:31.684960 736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d0d6f8669dd41e39d402ad1d237418d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:31.685251 1032 tablet_service.cc:1467] Processing CreateTablet for tablet 29b1444efd28483297bafc73a33e7a54 (DEFAULT_TABLE table=default.Table_1d910c3d2b244474a576fc4088301143 [id=75cb0745bbb84e1c883cb13b546ecb11]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:31.685613 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 29b1444efd28483297bafc73a33e7a54. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:31.688648 739 tablet_service.cc:1467] Processing CreateTablet for tablet 29b1444efd28483297bafc73a33e7a54 (DEFAULT_TABLE table=default.Table_1d910c3d2b244474a576fc4088301143 [id=75cb0745bbb84e1c883cb13b546ecb11]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:31.696353 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 29b1444efd28483297bafc73a33e7a54. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:31.703595 439 catalog_manager.cc:4945] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): alter failed for tablet 572e4eed1e924381bada27f8389821a7 (table default.Table_1d910c3d2b244474a576fc4088301143 [id=75cb0745bbb84e1c883cb13b546ecb11]),no further retry: Not found: Tablet not found: 572e4eed1e924381bada27f8389821a7
I20250113 02:51:31.713953 740 tablet_service.cc:1467] Processing CreateTablet for tablet 702e03e213b14ef5a1b64a02df1b3bf9 (DEFAULT_TABLE table=default.Table_4d5b54216098465fb88c7b9d8b605344 [id=e992f40fef2e466e9a49b8d3307e2a1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:31.717195 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 702e03e213b14ef5a1b64a02df1b3bf9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:31.717864 1195 ts_tablet_manager.cc:1918] T 77cf8b0c824f401aba1880eb7fd1807b P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:31.718176 1195 log.cc:1198] T 77cf8b0c824f401aba1880eb7fd1807b P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/77cf8b0c824f401aba1880eb7fd1807b
I20250113 02:51:31.718993 1195 ts_tablet_manager.cc:1939] T 77cf8b0c824f401aba1880eb7fd1807b P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:31.720655 885 tablet_service.cc:1467] Processing CreateTablet for tablet 702e03e213b14ef5a1b64a02df1b3bf9 (DEFAULT_TABLE table=default.Table_4d5b54216098465fb88c7b9d8b605344 [id=e992f40fef2e466e9a49b8d3307e2a1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:31.722159 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e90242bf273c42beb57ef160991a89ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:31.722345 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8898e1e9efce453488fa4a4562a82127. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:31.722512 888 tablet_service.cc:1467] Processing CreateTablet for tablet 7d0d6f8669dd41e39d402ad1d237418d (DEFAULT_TABLE table=default.Table_22ace32627b84d33b3fe01731f8d2c07 [id=2661227c8a684d4eb08d2c46738f7aed]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:31.722628 1198 ts_tablet_manager.cc:1905] T 6d63f0e5835540a78c82020b20a07dac P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:31.728513 745 tablet_service.cc:1467] Processing CreateTablet for tablet 448d3daf897247f8bd53ffbcdc16a22c (DEFAULT_TABLE table=default.Table_8bca38c8f95a4ea8aaec77a9e6c57f96 [id=6c256d970ac74b4d9a894b28edabdb82]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:31.729724 1192 raft_consensus.cc:357] T de4a58bf63054890808e2e2fe46e43d2 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:31.730104 1192 raft_consensus.cc:383] T de4a58bf63054890808e2e2fe46e43d2 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:31.730232 1192 raft_consensus.cc:738] T de4a58bf63054890808e2e2fe46e43d2 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:31.732167 1037 tablet_service.cc:1467] Processing CreateTablet for tablet 448d3daf897247f8bd53ffbcdc16a22c (DEFAULT_TABLE table=default.Table_8bca38c8f95a4ea8aaec77a9e6c57f96 [id=6c256d970ac74b4d9a894b28edabdb82]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:31.732213 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 448d3daf897247f8bd53ffbcdc16a22c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:31.732757 1037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 448d3daf897247f8bd53ffbcdc16a22c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:31.734274 1438 raft_consensus.cc:491] T 19f14b3db0b64847b5422bfd5ed323d0 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:31.736193 2009 raft_consensus.cc:491] T 11950e996c814590bc9f3f22c0354902 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:31.722990 893 tablet_service.cc:1467] Processing CreateTablet for tablet 29b1444efd28483297bafc73a33e7a54 (DEFAULT_TABLE table=default.Table_1d910c3d2b244474a576fc4088301143 [id=75cb0745bbb84e1c883cb13b546ecb11]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:31.736330 2009 raft_consensus.cc:513] T 11950e996c814590bc9f3f22c0354902 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:31.736992 2009 leader_election.cc:290] T 11950e996c814590bc9f3f22c0354902 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
W20250113 02:51:31.737927 995 leader_election.cc:343] T 11950e996c814590bc9f3f22c0354902 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Not found: Tablet not found: 11950e996c814590bc9f3f22c0354902
I20250113 02:51:31.737917 1438 raft_consensus.cc:513] T 19f14b3db0b64847b5422bfd5ed323d0 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:31.738541 1438 leader_election.cc:290] T 19f14b3db0b64847b5422bfd5ed323d0 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587), 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
I20250113 02:51:31.739216 882 tablet_service.cc:1467] Processing CreateTablet for tablet 448d3daf897247f8bd53ffbcdc16a22c (DEFAULT_TABLE table=default.Table_8bca38c8f95a4ea8aaec77a9e6c57f96 [id=6c256d970ac74b4d9a894b28edabdb82]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:31.739665 894 tablet_service.cc:1467] Processing CreateTablet for tablet 5a772093041b47528e3a2f18e4c25b01 (DEFAULT_TABLE table=default.Table_0236f2bec3074997a4cbfdba55c455bd [id=1bbcc54125044712ae41ca31dd211faa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:31.739847 736 tablet_service.cc:1467] Processing CreateTablet for tablet 5a772093041b47528e3a2f18e4c25b01 (DEFAULT_TABLE table=default.Table_0236f2bec3074997a4cbfdba55c455bd [id=1bbcc54125044712ae41ca31dd211faa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:31.740209 736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5a772093041b47528e3a2f18e4c25b01. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:31.740286 851 leader_election.cc:343] T 19f14b3db0b64847b5422bfd5ed323d0 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Not found: Tablet not found: 19f14b3db0b64847b5422bfd5ed323d0
I20250113 02:51:31.740406 1438 raft_consensus.cc:491] T e8ab4516789e4585a101c8766d338af3 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:31.740535 1438 raft_consensus.cc:513] T e8ab4516789e4585a101c8766d338af3 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
W20250113 02:51:31.741102 852 leader_election.cc:343] T 19f14b3db0b64847b5422bfd5ed323d0 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Not found: Tablet not found: 19f14b3db0b64847b5422bfd5ed323d0
I20250113 02:51:31.741207 852 leader_election.cc:304] T 19f14b3db0b64847b5422bfd5ed323d0 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df69f48656b44bb890f76587ee69cc0f; no voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca
I20250113 02:51:31.742094 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet 77cf8b0c824f401aba1880eb7fd1807b (table default.Table_ea18ad8d07b34ee4b3ef07b8f0431cfd [id=6069bb344e624c3bb8ca97e6c7fe23cc]) successfully deleted
I20250113 02:51:31.744004 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f7b219de824c4bfdb097cb81ae5f16f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:31.744210 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 702e03e213b14ef5a1b64a02df1b3bf9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:31.744695 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9fa6f11e7a86410099069389bbdec6c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:31.746004 1438 leader_election.cc:290] T e8ab4516789e4585a101c8766d338af3 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587), 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
I20250113 02:51:31.746132 1438 raft_consensus.cc:2743] T 19f14b3db0b64847b5422bfd5ed323d0 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:31.746290 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11950e996c814590bc9f3f22c0354902" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:31.746795 998 leader_election.cc:343] T 11950e996c814590bc9f3f22c0354902 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:31.746905 998 leader_election.cc:304] T 11950e996c814590bc9f3f22c0354902 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4216a7f853d642b7bc3e5cff458f3aca; no voters: 1186fa869eb14812832fa6562531f196, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:31.748447 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a438fd452ba403ca6283138913b3e03. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:31.748612 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 29b1444efd28483297bafc73a33e7a54. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:31.748734 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 448d3daf897247f8bd53ffbcdc16a22c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:31.748871 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5a772093041b47528e3a2f18e4c25b01. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:31.749612 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fff1a64765ee48729e9e91f1323bf40d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:31.749740 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b37a7f4090cb461faff31d89d71c0261. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:31.751223 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d0d6f8669dd41e39d402ad1d237418d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:31.752568 2009 raft_consensus.cc:2743] T 11950e996c814590bc9f3f22c0354902 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:31.753032 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 16375d658e014cc4bd42a4491c5d64ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:31.753438 1195 tablet_replica.cc:331] T 42fad91fe1b64a21a2c8412a2585af47 P 1186fa869eb14812832fa6562531f196: stopping tablet replica
W20250113 02:51:31.753780 851 leader_election.cc:343] T e8ab4516789e4585a101c8766d338af3 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Not found: Tablet not found: e8ab4516789e4585a101c8766d338af3
I20250113 02:51:31.759282 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 125ebde7a9f14918ba328a75e88a7746. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:31.760296 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9fcf7a6638414f9a99403b3327eeb831. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:31.761322 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1f128d9bf2924fb1bada3a8462818004. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:31.761534 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 16375d658e014cc4bd42a4491c5d64ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:31.761868 1195 raft_consensus.cc:2238] T 42fad91fe1b64a21a2c8412a2585af47 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:31.765153 1195 raft_consensus.cc:2267] T 42fad91fe1b64a21a2c8412a2585af47 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:31.765717 1195 ts_tablet_manager.cc:1905] T 42fad91fe1b64a21a2c8412a2585af47 P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:31.778151 1905 raft_consensus.cc:491] T 0752fde3a30a4947a2a688c56ceadab5 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:31.778301 1905 raft_consensus.cc:513] T 0752fde3a30a4947a2a688c56ceadab5 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:31.778929 1905 leader_election.cc:290] T 0752fde3a30a4947a2a688c56ceadab5 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:31.779750 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0752fde3a30a4947a2a688c56ceadab5" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
W20250113 02:51:31.780377 709 leader_election.cc:343] T 0752fde3a30a4947a2a688c56ceadab5 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:31.782693 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet f4379f7fc6b54e6785f27975d9030cce
W20250113 02:51:31.782941 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6ee2ea47a519435ebd2a5babe0b9a980 consensus state has no leader
I20250113 02:51:31.783782 755 tablet_service.cc:1514] Processing DeleteTablet for tablet f4379f7fc6b54e6785f27975d9030cce with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:31 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:31.784770 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_64856bab03754bdc9ded6e123a7d1c7e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:31.785392 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_64856bab03754bdc9ded6e123a7d1c7e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250113 02:51:31.785756 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b17cadf93237435ead176af3635baeca has no consensus state
I20250113 02:51:31.785985 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_eef7fe6c891b4fe984f8fb4a1c667a8e" } modify_external_catalogs: true
I20250113 02:51:31.786214 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_d7b990d07d59419da5ad4a9bc9dcc215"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:31.786777 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d7b990d07d59419da5ad4a9bc9dcc215 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:31.787765 742 tablet_service.cc:1467] Processing CreateTablet for tablet e8a0482683c7493eb19e7da5c78c095c (DEFAULT_TABLE table=default.Table_5893047163024e76bd537c27b9b10458 [id=4ddfcb2bed074dfcbecf0d91a339f416]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:31.787758 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_eef7fe6c891b4fe984f8fb4a1c667a8e" } modify_external_catalogs: true
I20250113 02:51:31.788476 755 tablet_service.cc:1467] Processing CreateTablet for tablet a446180f980640268170b4ab652d32cc (DEFAULT_TABLE table=default.Table_54127c30dcd54c1dbe1f089100b2a4b6 [id=518d4e16350248eba21d32d672f8eb09]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:31.790474 748 tablet_service.cc:1467] Processing CreateTablet for tablet 2e4a68bbcd684785a7bacdbabb506718 (DEFAULT_TABLE table=default.Table_b8e04994e6894fe4a4ff012d0050eb6c [id=ea173b551a6a421cbda20addabceeecb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:31.791359 1033 tablet_service.cc:1467] Processing CreateTablet for tablet e8a0482683c7493eb19e7da5c78c095c (DEFAULT_TABLE table=default.Table_5893047163024e76bd537c27b9b10458 [id=4ddfcb2bed074dfcbecf0d91a339f416]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:31.791460 1024 tablet_service.cc:1467] Processing CreateTablet for tablet 2e4a68bbcd684785a7bacdbabb506718 (DEFAULT_TABLE table=default.Table_b8e04994e6894fe4a4ff012d0050eb6c [id=ea173b551a6a421cbda20addabceeecb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:31.791731 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e8a0482683c7493eb19e7da5c78c095c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:31.791946 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e4a68bbcd684785a7bacdbabb506718. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:31.792066 1037 tablet_service.cc:1467] Processing CreateTablet for tablet 5a772093041b47528e3a2f18e4c25b01 (DEFAULT_TABLE table=default.Table_0236f2bec3074997a4cbfdba55c455bd [id=1bbcc54125044712ae41ca31dd211faa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:31.792362 1037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5a772093041b47528e3a2f18e4c25b01. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:31.793640 1191 tablet_bootstrap.cc:492] T b8b6ffe690fc459db78b2b3bc12671dd P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:31.793747 1191 ts_tablet_manager.cc:1397] T b8b6ffe690fc459db78b2b3bc12671dd P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.446s user 0.027s sys 0.048s
I20250113 02:51:31.794454 1191 raft_consensus.cc:357] T b8b6ffe690fc459db78b2b3bc12671dd P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:31.794638 1191 raft_consensus.cc:383] T b8b6ffe690fc459db78b2b3bc12671dd P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:31.794682 1191 raft_consensus.cc:738] T b8b6ffe690fc459db78b2b3bc12671dd P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:31.794909 1191 consensus_queue.cc:260] T b8b6ffe690fc459db78b2b3bc12671dd P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:31.795511 1191 ts_tablet_manager.cc:1428] T b8b6ffe690fc459db78b2b3bc12671dd P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250113 02:51:31.730829 1192 consensus_queue.cc:260] T de4a58bf63054890808e2e2fe46e43d2 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:31.797298 1192 ts_tablet_manager.cc:1428] T de4a58bf63054890808e2e2fe46e43d2 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.313s user 0.003s sys 0.000s
I20250113 02:51:31.797502 882 tablet_service.cc:1514] Processing DeleteTablet for tablet f4379f7fc6b54e6785f27975d9030cce with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:31 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:31.797572 1192 tablet_bootstrap.cc:492] T 7d3562a48c6e4badbbcf37640e91b5ba P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:31.799160 1192 tablet_bootstrap.cc:654] T 7d3562a48c6e4badbbcf37640e91b5ba P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:31.800988 1905 raft_consensus.cc:491] T b8b6ffe690fc459db78b2b3bc12671dd P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:31.801116 1905 raft_consensus.cc:513] T b8b6ffe690fc459db78b2b3bc12671dd P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:31.801708 1905 leader_election.cc:290] T b8b6ffe690fc459db78b2b3bc12671dd P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:31.801838 889 tablet_service.cc:1467] Processing CreateTablet for tablet a446180f980640268170b4ab652d32cc (DEFAULT_TABLE table=default.Table_54127c30dcd54c1dbe1f089100b2a4b6 [id=518d4e16350248eba21d32d672f8eb09]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:31.802362 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8b6ffe690fc459db78b2b3bc12671dd" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:31.802784 2009 raft_consensus.cc:491] T de4a58bf63054890808e2e2fe46e43d2 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250113 02:51:31.802872 709 leader_election.cc:343] T b8b6ffe690fc459db78b2b3bc12671dd P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:31.802909 2009 raft_consensus.cc:513] T de4a58bf63054890808e2e2fe46e43d2 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:31.803946 2009 leader_election.cc:290] T de4a58bf63054890808e2e2fe46e43d2 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
W20250113 02:51:31.805756 995 leader_election.cc:343] T de4a58bf63054890808e2e2fe46e43d2 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Not found: Tablet not found: de4a58bf63054890808e2e2fe46e43d2
I20250113 02:51:31.807482 1029 tablet_service.cc:1467] Processing CreateTablet for tablet 7d0d6f8669dd41e39d402ad1d237418d (DEFAULT_TABLE table=default.Table_22ace32627b84d33b3fe01731f8d2c07 [id=2661227c8a684d4eb08d2c46738f7aed]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:31.807838 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d0d6f8669dd41e39d402ad1d237418d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:31.808730 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a446180f980640268170b4ab652d32cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:31.809888 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_d7372f791f52447593557efc3f8226ff"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:31.810746 750 tablet_service.cc:1467] Processing CreateTablet for tablet a8aedbc8d7bb4183bb1ab9902f07bea7 (DEFAULT_TABLE table=default.Table_ed883bfc96094be9bb6e07bc510b6042 [id=3e7c139aef3847df965d0400bd2ffa45]), partition=RANGE (key) PARTITION UNBOUNDED
W20250113 02:51:31.811043 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d7372f791f52447593557efc3f8226ff in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:31.811229 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e8a0482683c7493eb19e7da5c78c095c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:31.811368 1037 tablet_service.cc:1467] Processing CreateTablet for tablet a8aedbc8d7bb4183bb1ab9902f07bea7 (DEFAULT_TABLE table=default.Table_ed883bfc96094be9bb6e07bc510b6042 [id=3e7c139aef3847df965d0400bd2ffa45]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:31.811673 1037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8aedbc8d7bb4183bb1ab9902f07bea7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:31.816236 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a446180f980640268170b4ab652d32cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:31.817584 745 tablet_service.cc:1467] Processing CreateTablet for tablet 70ffabffb8c34677bb484ae3e4fb0394 (DEFAULT_TABLE table=default.Table_3196fc458c3740f2a1af55de808edb06 [id=364160fde8ce4c87afdf1f6a36103ce3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:31.817952 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70ffabffb8c34677bb484ae3e4fb0394. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:31.822002 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet f4379f7fc6b54e6785f27975d9030cce with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:31 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:31.822094 2012 raft_consensus.cc:491] T bffec36d567544c5a1c87e64a7e9730b P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:31.822412 2009 raft_consensus.cc:491] T 4c615b0b83d74fcfae0c51047dbfbbf4 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:31.822625 1191 tablet_bootstrap.cc:492] T 51d9483ecacb413796d0fd9e7560eac1 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:31.822561 2012 raft_consensus.cc:513] T bffec36d567544c5a1c87e64a7e9730b P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:31.822540 2009 raft_consensus.cc:513] T 4c615b0b83d74fcfae0c51047dbfbbf4 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:31.823577 2009 leader_election.cc:290] T 4c615b0b83d74fcfae0c51047dbfbbf4 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
I20250113 02:51:31.824232 1191 tablet_bootstrap.cc:654] T 51d9483ecacb413796d0fd9e7560eac1 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:31.824149 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c615b0b83d74fcfae0c51047dbfbbf4" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
I20250113 02:51:31.824363 772 raft_consensus.cc:2463] T 4c615b0b83d74fcfae0c51047dbfbbf4 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 0.
W20250113 02:51:31.824402 995 leader_election.cc:343] T bffec36d567544c5a1c87e64a7e9730b P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Not found: Tablet not found: bffec36d567544c5a1c87e64a7e9730b
I20250113 02:51:31.824851 995 leader_election.cc:304] T 4c615b0b83d74fcfae0c51047dbfbbf4 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters:
I20250113 02:51:31.825165 2012 leader_election.cc:290] T bffec36d567544c5a1c87e64a7e9730b P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
I20250113 02:51:31.825295 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e4a68bbcd684785a7bacdbabb506718. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:31.825402 2012 raft_consensus.cc:2798] T 4c615b0b83d74fcfae0c51047dbfbbf4 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 02:51:31.825517 2012 raft_consensus.cc:491] T 4c615b0b83d74fcfae0c51047dbfbbf4 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 02:51:31.825639 2012 raft_consensus.cc:3054] T 4c615b0b83d74fcfae0c51047dbfbbf4 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:31.828176 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8aedbc8d7bb4183bb1ab9902f07bea7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:31.828577 2012 raft_consensus.cc:513] T 4c615b0b83d74fcfae0c51047dbfbbf4 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:31.832942 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c615b0b83d74fcfae0c51047dbfbbf4" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196"
I20250113 02:51:31.833106 772 raft_consensus.cc:3054] T 4c615b0b83d74fcfae0c51047dbfbbf4 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:31.833163 2012 leader_election.cc:290] T 4c615b0b83d74fcfae0c51047dbfbbf4 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Requested vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
I20250113 02:51:31.835496 772 raft_consensus.cc:2463] T 4c615b0b83d74fcfae0c51047dbfbbf4 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 1.
I20250113 02:51:31.836014 995 leader_election.cc:304] T 4c615b0b83d74fcfae0c51047dbfbbf4 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters:
I20250113 02:51:31.839807 2012 raft_consensus.cc:2798] T 4c615b0b83d74fcfae0c51047dbfbbf4 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Leader election won for term 1
I20250113 02:51:31.840016 2012 raft_consensus.cc:695] T 4c615b0b83d74fcfae0c51047dbfbbf4 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: Becoming Leader. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Running, Role: LEADER
I20250113 02:51:31.840468 446 catalog_manager.cc:5618] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095) does not have the latest schema for tablet 9fff52e99c64462a8a73fccc3c22ca63 (table default.Table_9ef18ee71ad843c5921750a2f8478855 [id=bd8bd572ed524f2cae01f820f07e3c36]). Expected version 2 got 0
I20250113 02:51:31.841006 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_a4b585b4849b4950ba3a372aa4190e7c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:31.841316 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_a7ed8a8a7d00492b8fc543baeb433a27"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:31.841588 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a4b585b4849b4950ba3a372aa4190e7c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250113 02:51:31.841768 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a7ed8a8a7d00492b8fc543baeb433a27 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250113 02:51:31.842934 451 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d2699bb529bf4f0f9d7f2668ffe4c9ff consensus state has no leader
I20250113 02:51:31.844084 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 883027967e0c442a91ac820210b9c6c7
I20250113 02:51:31.844162 2012 consensus_queue.cc:237] T 4c615b0b83d74fcfae0c51047dbfbbf4 P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:31.845762 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_fc6121581e0a4a8c85474adbe1bdbcf8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:31.846215 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fc6121581e0a4a8c85474adbe1bdbcf8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:31.846649 882 tablet_service.cc:1467] Processing CreateTablet for tablet e8a0482683c7493eb19e7da5c78c095c (DEFAULT_TABLE table=default.Table_5893047163024e76bd537c27b9b10458 [id=4ddfcb2bed074dfcbecf0d91a339f416]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:31.847069 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e8a0482683c7493eb19e7da5c78c095c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:31.849145 736 tablet_service.cc:1514] Processing DeleteTablet for tablet 883027967e0c442a91ac820210b9c6c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:31 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:31.850919 1030 tablet_service.cc:1467] Processing CreateTablet for tablet a446180f980640268170b4ab652d32cc (DEFAULT_TABLE table=default.Table_54127c30dcd54c1dbe1f089100b2a4b6 [id=518d4e16350248eba21d32d672f8eb09]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:31.851310 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a446180f980640268170b4ab652d32cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:31.854485 1042 tablet_service.cc:1467] Processing CreateTablet for tablet 70ffabffb8c34677bb484ae3e4fb0394 (DEFAULT_TABLE table=default.Table_3196fc458c3740f2a1af55de808edb06 [id=364160fde8ce4c87afdf1f6a36103ce3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:31.854856 1042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70ffabffb8c34677bb484ae3e4fb0394. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:31.856132 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_5f7bbbd09edc47978900639818dae863" } modify_external_catalogs: true
I20250113 02:51:31.856328 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_5f7bbbd09edc47978900639818dae863" } modify_external_catalogs: true
I20250113 02:51:31.857028 890 tablet_service.cc:1467] Processing CreateTablet for tablet 2e4a68bbcd684785a7bacdbabb506718 (DEFAULT_TABLE table=default.Table_b8e04994e6894fe4a4ff012d0050eb6c [id=ea173b551a6a421cbda20addabceeecb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:31.857484 447 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_9c2c7930fdf148e8af11382a5838bdb5" } modify_external_catalogs: true
I20250113 02:51:31.857659 447 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_9c2c7930fdf148e8af11382a5838bdb5" } modify_external_catalogs: true
I20250113 02:51:31.859231 454 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_707144e534a1463986168c22b636e208" } new_table_name: "default.Table_b95a9b1d12284c9aac701a8ef9c93482" modify_external_catalogs: true
I20250113 02:51:31.862164 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 16a132384b0e4264a73d09a3ea133893
I20250113 02:51:31.862417 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e4a68bbcd684785a7bacdbabb506718. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:31.862461 1037 tablet_service.cc:1514] Processing DeleteTablet for tablet 883027967e0c442a91ac820210b9c6c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:31 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:31.863107 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_d7fc9abb38eb4f5a9276aa825095c20b" } new_table_name: "default.Table_5255ab9ad2034d32aef64cbb9c9a59c8" modify_external_catalogs: true
I20250113 02:51:31.866504 736 tablet_service.cc:1514] Processing DeleteTablet for tablet 16a132384b0e4264a73d09a3ea133893 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:31 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:31.868705 1926 raft_consensus.cc:491] T 840773ee80c7447097ea000b106fc89d P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:31.868844 1926 raft_consensus.cc:513] T 840773ee80c7447097ea000b106fc89d P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:31.869436 1926 leader_election.cc:290] T 840773ee80c7447097ea000b106fc89d P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
I20250113 02:51:31.870898 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet 16a132384b0e4264a73d09a3ea133893 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:31 UTC) from {username='slave'} at 127.0.0.1:60230
W20250113 02:51:31.871513 995 leader_election.cc:343] T 840773ee80c7447097ea000b106fc89d P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Not found: Tablet not found: 840773ee80c7447097ea000b106fc89d
W20250113 02:51:31.872499 1194 thread.cc:635] log-alloc [worker] (thread pool) Time spent creating pthread: real 0.529s user 0.013s sys 0.064s
W20250113 02:51:31.872592 1194 thread.cc:601] log-alloc [worker] (thread pool) Time spent starting thread: real 0.529s user 0.013s sys 0.064s
I20250113 02:51:31.878666 1198 ts_tablet_manager.cc:1918] T 6d63f0e5835540a78c82020b20a07dac P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:31.879712 450 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet e51ffefaff114e73b4beb1ed2e609381
I20250113 02:51:31.879523 1198 log.cc:1198] T 6d63f0e5835540a78c82020b20a07dac P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/6d63f0e5835540a78c82020b20a07dac
I20250113 02:51:31.880043 447 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet 18ffb0c3b2a74b948dd11416aed59e49
I20250113 02:51:31.884047 772 raft_consensus.cc:1270] T 32cba8f3efc54e78b4df41c15df82eea P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Refusing update from remote peer 4216a7f853d642b7bc3e5cff458f3aca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250113 02:51:31.887197 1198 ts_tablet_manager.cc:1939] T 6d63f0e5835540a78c82020b20a07dac P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:31.894042 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet 6d63f0e5835540a78c82020b20a07dac (table default.Table_af3ebf543c9448b7ae473d721a23d47e [id=93c6f37549ce4544bc0cb4a267f44b00]) successfully deleted
I20250113 02:51:31.894358 2012 consensus_queue.cc:1035] T 32cba8f3efc54e78b4df41c15df82eea P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 02:51:31.901350 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bffec36d567544c5a1c87e64a7e9730b" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:31.902105 454 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0b119704bc774ca9b4155c9619d4765b consensus state has no leader
I20250113 02:51:31.902102 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c615b0b83d74fcfae0c51047dbfbbf4" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f"
I20250113 02:51:31.902455 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de4a58bf63054890808e2e2fe46e43d2" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:31.902907 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fe3d35d9aaf64188aaa3c1ac38d279d8 consensus state has no leader
I20250113 02:51:31.903945 1194 tablet_bootstrap.cc:492] T c215ffe61b9b426286cc5e808463078d P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:31.904186 1194 ts_tablet_manager.cc:1397] T c215ffe61b9b426286cc5e808463078d P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.566s user 0.015s sys 0.065s
I20250113 02:51:31.904886 755 tablet_service.cc:1514] Processing DeleteTablet for tablet b8b6ffe690fc459db78b2b3bc12671dd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:23 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:31.906034 1192 tablet_bootstrap.cc:492] T 7d3562a48c6e4badbbcf37640e91b5ba P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:31.906179 1192 ts_tablet_manager.cc:1397] T 7d3562a48c6e4badbbcf37640e91b5ba P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.109s user 0.000s sys 0.003s
I20250113 02:51:31.906946 1192 raft_consensus.cc:357] T 7d3562a48c6e4badbbcf37640e91b5ba P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:31.907140 1192 raft_consensus.cc:383] T 7d3562a48c6e4badbbcf37640e91b5ba P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:31.907199 1192 raft_consensus.cc:738] T 7d3562a48c6e4badbbcf37640e91b5ba P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:31.907439 1192 consensus_queue.cc:260] T 7d3562a48c6e4badbbcf37640e91b5ba P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:31.908144 1192 ts_tablet_manager.cc:1428] T 7d3562a48c6e4badbbcf37640e91b5ba P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20250113 02:51:31.908854 1192 tablet_bootstrap.cc:492] T 23e04961f072440fa3e3f9485e40f79c P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:31.910395 1192 tablet_bootstrap.cc:654] T 23e04961f072440fa3e3f9485e40f79c P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:31.914027 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "840773ee80c7447097ea000b106fc89d" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:31.914438 998 leader_election.cc:343] T bffec36d567544c5a1c87e64a7e9730b P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:31.914543 2012 raft_consensus.cc:491] T efccefd6787943469131492741ee2bd5 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:31.914539 998 leader_election.cc:304] T bffec36d567544c5a1c87e64a7e9730b P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4216a7f853d642b7bc3e5cff458f3aca; no voters: 1186fa869eb14812832fa6562531f196, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:31.914682 2012 raft_consensus.cc:513] T efccefd6787943469131492741ee2bd5 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
W20250113 02:51:31.914950 998 leader_election.cc:343] T 4c615b0b83d74fcfae0c51047dbfbbf4 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
W20250113 02:51:31.915185 998 leader_election.cc:343] T de4a58bf63054890808e2e2fe46e43d2 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:31.915259 998 leader_election.cc:304] T de4a58bf63054890808e2e2fe46e43d2 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4216a7f853d642b7bc3e5cff458f3aca; no voters: 1186fa869eb14812832fa6562531f196, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:31.915684 2010 raft_consensus.cc:2743] T bffec36d567544c5a1c87e64a7e9730b P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:31.915750 1926 raft_consensus.cc:2743] T de4a58bf63054890808e2e2fe46e43d2 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250113 02:51:31.915967 995 leader_election.cc:343] T efccefd6787943469131492741ee2bd5 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Not found: Tablet not found: efccefd6787943469131492741ee2bd5
I20250113 02:51:31.916108 2012 leader_election.cc:290] T efccefd6787943469131492741ee2bd5 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
W20250113 02:51:31.915693 998 leader_election.cc:343] T 840773ee80c7447097ea000b106fc89d P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:31.916285 998 leader_election.cc:304] T 840773ee80c7447097ea000b106fc89d P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4216a7f853d642b7bc3e5cff458f3aca; no voters: 1186fa869eb14812832fa6562531f196, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:31.916527 2012 raft_consensus.cc:2743] T 840773ee80c7447097ea000b106fc89d P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250113 02:51:31.916926 998 consensus_peers.cc:487] T 32cba8f3efc54e78b4df41c15df82eea P 4216a7f853d642b7bc3e5cff458f3aca -> Peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Couldn't send request to peer df69f48656b44bb890f76587ee69cc0f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250113 02:51:31.925515 1191 tablet_bootstrap.cc:492] T 51d9483ecacb413796d0fd9e7560eac1 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:31.925650 1191 ts_tablet_manager.cc:1397] T 51d9483ecacb413796d0fd9e7560eac1 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.103s user 0.003s sys 0.000s
I20250113 02:51:31.925905 889 tablet_service.cc:1467] Processing CreateTablet for tablet a8aedbc8d7bb4183bb1ab9902f07bea7 (DEFAULT_TABLE table=default.Table_ed883bfc96094be9bb6e07bc510b6042 [id=3e7c139aef3847df965d0400bd2ffa45]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:31.926277 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8aedbc8d7bb4183bb1ab9902f07bea7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:31.926419 1191 raft_consensus.cc:357] T 51d9483ecacb413796d0fd9e7560eac1 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:31.926616 1191 raft_consensus.cc:383] T 51d9483ecacb413796d0fd9e7560eac1 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:31.926667 1191 raft_consensus.cc:738] T 51d9483ecacb413796d0fd9e7560eac1 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:31.925905 893 tablet_service.cc:1467] Processing CreateTablet for tablet 70ffabffb8c34677bb484ae3e4fb0394 (DEFAULT_TABLE table=default.Table_3196fc458c3740f2a1af55de808edb06 [id=364160fde8ce4c87afdf1f6a36103ce3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:31.926858 1191 consensus_queue.cc:260] T 51d9483ecacb413796d0fd9e7560eac1 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:31.927282 1191 ts_tablet_manager.cc:1428] T 51d9483ecacb413796d0fd9e7560eac1 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250113 02:51:31.927511 1191 tablet_bootstrap.cc:492] T 056bf1e67be04d179eace3d93282b145 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:31.927518 896 tablet_service.cc:1514] Processing DeleteTablet for tablet 883027967e0c442a91ac820210b9c6c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:31 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:31.927646 896 tablet_service.cc:1514] Processing DeleteTablet for tablet 16a132384b0e4264a73d09a3ea133893 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:31 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:31.928977 1195 ts_tablet_manager.cc:1918] T 42fad91fe1b64a21a2c8412a2585af47 P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:31.929083 1195 log.cc:1198] T 42fad91fe1b64a21a2c8412a2585af47 P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/42fad91fe1b64a21a2c8412a2585af47
I20250113 02:51:31.929131 1191 tablet_bootstrap.cc:654] T 056bf1e67be04d179eace3d93282b145 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:31.929391 1195 ts_tablet_manager.cc:1939] T 42fad91fe1b64a21a2c8412a2585af47 P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:31.929842 1195 tablet_replica.cc:331] T 0a7e7713bc8648378b668d8b769ac927 P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:31.930001 1195 raft_consensus.cc:2238] T 0a7e7713bc8648378b668d8b769ac927 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:31.929960 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet 42fad91fe1b64a21a2c8412a2585af47 (table default.Table_639e38f37be1466e9c2ce92fa1c2d890 [id=290b815241d648b6b62b8a70d2efd1c9]) successfully deleted
I20250113 02:51:31.930119 1195 raft_consensus.cc:2267] T 0a7e7713bc8648378b668d8b769ac927 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:31.930595 1195 ts_tablet_manager.cc:1905] T 0a7e7713bc8648378b668d8b769ac927 P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:31.935014 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c615b0b83d74fcfae0c51047dbfbbf4" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:31.918797 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efccefd6787943469131492741ee2bd5" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:31.938942 998 leader_election.cc:343] T 4c615b0b83d74fcfae0c51047dbfbbf4 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
W20250113 02:51:31.939306 998 leader_election.cc:343] T efccefd6787943469131492741ee2bd5 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:31.939383 998 leader_election.cc:304] T efccefd6787943469131492741ee2bd5 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4216a7f853d642b7bc3e5cff458f3aca; no voters: 1186fa869eb14812832fa6562531f196, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:31.940371 2009 raft_consensus.cc:2743] T efccefd6787943469131492741ee2bd5 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:31.941448 1195 ts_tablet_manager.cc:1918] T 0a7e7713bc8648378b668d8b769ac927 P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:31.945503 1195 log.cc:1198] T 0a7e7713bc8648378b668d8b769ac927 P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/0a7e7713bc8648378b668d8b769ac927
I20250113 02:51:31.952566 1198 tablet_replica.cc:331] T 0810b9671f824d0a84741bc1904df247 P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:31.952771 1198 raft_consensus.cc:2238] T 0810b9671f824d0a84741bc1904df247 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:31.952931 1198 raft_consensus.cc:2267] T 0810b9671f824d0a84741bc1904df247 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:31.953460 1198 ts_tablet_manager.cc:1905] T 0810b9671f824d0a84741bc1904df247 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:31.954931 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70ffabffb8c34677bb484ae3e4fb0394. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:31.956665 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 4c759831773849f2bc52e484ffa0cf49 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4c759831773849f2bc52e484ffa0cf49 already in progress: creating tablet
W20250113 02:51:31.956954 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 2e4a68bbcd684785a7bacdbabb506718 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2e4a68bbcd684785a7bacdbabb506718 already in progress: creating tablet
I20250113 02:51:31.957103 1192 tablet_bootstrap.cc:492] T 23e04961f072440fa3e3f9485e40f79c P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:31.944674 1194 raft_consensus.cc:357] T c215ffe61b9b426286cc5e808463078d P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:31.957221 1192 ts_tablet_manager.cc:1397] T 23e04961f072440fa3e3f9485e40f79c P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.048s user 0.002s sys 0.000s
I20250113 02:51:31.957207 1194 raft_consensus.cc:383] T c215ffe61b9b426286cc5e808463078d P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:31.957275 1194 raft_consensus.cc:738] T c215ffe61b9b426286cc5e808463078d P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:31.958024 1192 raft_consensus.cc:357] T 23e04961f072440fa3e3f9485e40f79c P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:31.958240 1192 raft_consensus.cc:383] T 23e04961f072440fa3e3f9485e40f79c P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:31.958335 1192 raft_consensus.cc:738] T 23e04961f072440fa3e3f9485e40f79c P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:31.958590 1192 consensus_queue.cc:260] T 23e04961f072440fa3e3f9485e40f79c P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:31.959450 1192 ts_tablet_manager.cc:1428] T 23e04961f072440fa3e3f9485e40f79c P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20250113 02:51:31.966182 1192 tablet_bootstrap.cc:492] T 2778d7dc7d9c4843bbc9eb705cec71cd P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:31.968384 1192 tablet_bootstrap.cc:654] T 2778d7dc7d9c4843bbc9eb705cec71cd P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:31.970166 1191 tablet_bootstrap.cc:492] T 056bf1e67be04d179eace3d93282b145 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:31.970296 1191 ts_tablet_manager.cc:1397] T 056bf1e67be04d179eace3d93282b145 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.043s user 0.003s sys 0.000s
I20250113 02:51:31.971509 1191 raft_consensus.cc:357] T 056bf1e67be04d179eace3d93282b145 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:31.971709 1191 raft_consensus.cc:383] T 056bf1e67be04d179eace3d93282b145 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:31.971753 1191 raft_consensus.cc:738] T 056bf1e67be04d179eace3d93282b145 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:31.972020 1191 consensus_queue.cc:260] T 056bf1e67be04d179eace3d93282b145 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:31.972522 1191 ts_tablet_manager.cc:1428] T 056bf1e67be04d179eace3d93282b145 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250113 02:51:31.972792 1191 tablet_bootstrap.cc:492] T 6666922d677a4c678f80d7c6c47fd2a5 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:31.973567 1107 heartbeater.cc:346] Connected to a master server at 127.0.104.60:42067
I20250113 02:51:31.974135 1191 tablet_bootstrap.cc:654] T 6666922d677a4c678f80d7c6c47fd2a5 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:31.964010 1198 ts_tablet_manager.cc:1918] T 0810b9671f824d0a84741bc1904df247 P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:31.974390 1198 log.cc:1198] T 0810b9671f824d0a84741bc1904df247 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/0810b9671f824d0a84741bc1904df247
I20250113 02:51:31.975210 1198 ts_tablet_manager.cc:1939] T 0810b9671f824d0a84741bc1904df247 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:31.976104 1194 consensus_queue.cc:260] T c215ffe61b9b426286cc5e808463078d P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:31.976954 1195 ts_tablet_manager.cc:1939] T 0a7e7713bc8648378b668d8b769ac927 P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:31.982769 1195 tablet_replica.cc:331] T c1fb2b36c0504cb4ba32671ae4c620d1 P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:31.983937 1194 ts_tablet_manager.cc:1428] T c215ffe61b9b426286cc5e808463078d P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.080s user 0.003s sys 0.000s
I20250113 02:51:31.984359 1194 tablet_bootstrap.cc:492] T 401c4f5dbd304950877b59a865f417b3 P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:31.985211 1195 raft_consensus.cc:2238] T c1fb2b36c0504cb4ba32671ae4c620d1 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:31.986259 1195 raft_consensus.cc:2267] T c1fb2b36c0504cb4ba32671ae4c620d1 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:31.990736 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet 0810b9671f824d0a84741bc1904df247 (table default.Table_796801d17a3c458693693af6e39cf0f5 [id=df24b379d986402187cec33190501012]) successfully deleted
I20250113 02:51:31.992194 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_2302c6aa93864d4c98218c4b2299c03d" } new_table_name: "default.Table_b2261cee2d1c462093eefcdfe5242204" modify_external_catalogs: true
I20250113 02:51:31.996910 1194 tablet_bootstrap.cc:654] T 401c4f5dbd304950877b59a865f417b3 P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:31.996845 454 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_a8d0f4a9e13b48de8ef840a9b6d690c5" } modify_external_catalogs: true
I20250113 02:51:31.997045 454 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_a8d0f4a9e13b48de8ef840a9b6d690c5" } modify_external_catalogs: true
I20250113 02:51:31.997642 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_dd3111126f5a4f97af332dd879439153"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:31.998205 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dd3111126f5a4f97af332dd879439153 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:32.000895 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_183f57ddd85940d5b90ccf97c7ad0bbf" } new_table_name: "default.Table_ae01ec20166046f3918f5d29086e4274" modify_external_catalogs: true
I20250113 02:51:31.979537 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet 0a7e7713bc8648378b668d8b769ac927 (table default.Table_93a7b6abed0446a4baf21e5e7c8f0c1b [id=9fd4d65e7b9d443a82f06a1f660a7275]) successfully deleted
I20250113 02:51:32.002521 1904 raft_consensus.cc:491] T 51d9483ecacb413796d0fd9e7560eac1 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:32.002640 1904 raft_consensus.cc:513] T 51d9483ecacb413796d0fd9e7560eac1 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:32.003242 1904 leader_election.cc:290] T 51d9483ecacb413796d0fd9e7560eac1 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:32.003952 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51d9483ecacb413796d0fd9e7560eac1" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
W20250113 02:51:32.004650 709 leader_election.cc:343] T 51d9483ecacb413796d0fd9e7560eac1 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
W20250113 02:51:32.005398 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0411f2aab6c34f09bc4f24e0a49a744a consensus state has no leader
I20250113 02:51:32.005517 1196 ts_tablet_manager.cc:1918] T 55ebe9c044d944bc813396c699ba9aa1 P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250113 02:51:32.009563 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet cb141fb2e9894757b3ae423242ad93d2 has no consensus state
I20250113 02:51:32.009750 1196 log.cc:1198] T 55ebe9c044d944bc813396c699ba9aa1 P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/55ebe9c044d944bc813396c699ba9aa1
I20250113 02:51:32.010088 1196 ts_tablet_manager.cc:1939] T 55ebe9c044d944bc813396c699ba9aa1 P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:32.010892 1192 tablet_bootstrap.cc:492] T 2778d7dc7d9c4843bbc9eb705cec71cd P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:32.011035 1192 ts_tablet_manager.cc:1397] T 2778d7dc7d9c4843bbc9eb705cec71cd P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.045s user 0.003s sys 0.003s
I20250113 02:51:32.011325 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet 55ebe9c044d944bc813396c699ba9aa1 (table default.Table_2caed919541e43c78a858513a2128f50 [id=26ba87b9bd384b99a016b29fe5b5babd]) successfully deleted
I20250113 02:51:32.011787 1192 raft_consensus.cc:357] T 2778d7dc7d9c4843bbc9eb705cec71cd P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:32.012015 1192 raft_consensus.cc:383] T 2778d7dc7d9c4843bbc9eb705cec71cd P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:32.012089 1192 raft_consensus.cc:738] T 2778d7dc7d9c4843bbc9eb705cec71cd P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:32.012284 1192 consensus_queue.cc:260] T 2778d7dc7d9c4843bbc9eb705cec71cd P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:32.012742 1192 ts_tablet_manager.cc:1428] T 2778d7dc7d9c4843bbc9eb705cec71cd P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250113 02:51:32.012984 1196 tablet_replica.cc:331] T e8ab4516789e4585a101c8766d338af3 P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:32.013017 1192 tablet_bootstrap.cc:492] T b0b267b0d9c54ba496c89f6cb287dffd P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:32.013213 1196 raft_consensus.cc:2238] T e8ab4516789e4585a101c8766d338af3 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:32.013397 1196 raft_consensus.cc:2267] T e8ab4516789e4585a101c8766d338af3 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:32.015803 1192 tablet_bootstrap.cc:654] T b0b267b0d9c54ba496c89f6cb287dffd P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:32.019722 1198 tablet_replica.cc:331] T 41fcde988f72496b824271fa36c1a226 P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:32.020040 1198 raft_consensus.cc:2238] T 41fcde988f72496b824271fa36c1a226 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:32.020439 1195 ts_tablet_manager.cc:1905] T c1fb2b36c0504cb4ba32671ae4c620d1 P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:32.020676 1198 raft_consensus.cc:2267] T 41fcde988f72496b824271fa36c1a226 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:32.020931 1191 tablet_bootstrap.cc:492] T 6666922d677a4c678f80d7c6c47fd2a5 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:32.021019 1196 ts_tablet_manager.cc:1905] T e8ab4516789e4585a101c8766d338af3 P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:32.021093 1191 ts_tablet_manager.cc:1397] T 6666922d677a4c678f80d7c6c47fd2a5 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.048s user 0.002s sys 0.000s
I20250113 02:51:32.021911 1191 raft_consensus.cc:357] T 6666922d677a4c678f80d7c6c47fd2a5 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:32.022156 1191 raft_consensus.cc:383] T 6666922d677a4c678f80d7c6c47fd2a5 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:32.022221 1191 raft_consensus.cc:738] T 6666922d677a4c678f80d7c6c47fd2a5 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:32.022449 1191 consensus_queue.cc:260] T 6666922d677a4c678f80d7c6c47fd2a5 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:32.022620 1198 ts_tablet_manager.cc:1905] T 41fcde988f72496b824271fa36c1a226 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:32.022996 1191 ts_tablet_manager.cc:1428] T 6666922d677a4c678f80d7c6c47fd2a5 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250113 02:51:32.023303 1191 tablet_bootstrap.cc:492] T 6e1284c8aaba44b5810b0d06d36757eb P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
W20250113 02:51:32.023780 852 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.3:42587, user_credentials={real_user=slave}} blocked reactor thread for 282659us
I20250113 02:51:32.025547 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0752fde3a30a4947a2a688c56ceadab5" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:32.025682 852 leader_election.cc:343] T e8ab4516789e4585a101c8766d338af3 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Not found: Tablet not found: e8ab4516789e4585a101c8766d338af3
I20250113 02:51:32.025790 852 leader_election.cc:304] T e8ab4516789e4585a101c8766d338af3 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df69f48656b44bb890f76587ee69cc0f; no voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca
W20250113 02:51:32.025990 852 raft_consensus.cc:2681] T e8ab4516789e4585a101c8766d338af3 P df69f48656b44bb890f76587ee69cc0f: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250113 02:51:32.026057 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8b6ffe690fc459db78b2b3bc12671dd" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:32.026316 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51d9483ecacb413796d0fd9e7560eac1" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:32.026512 711 leader_election.cc:343] T 0752fde3a30a4947a2a688c56ceadab5 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:32.026610 711 leader_election.cc:304] T 0752fde3a30a4947a2a688c56ceadab5 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
W20250113 02:51:32.027011 711 leader_election.cc:343] T b8b6ffe690fc459db78b2b3bc12671dd P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:32.027097 711 leader_election.cc:304] T b8b6ffe690fc459db78b2b3bc12671dd P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:32.028085 1905 raft_consensus.cc:2743] T 0752fde3a30a4947a2a688c56ceadab5 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:32.028239 1905 raft_consensus.cc:2743] T b8b6ffe690fc459db78b2b3bc12671dd P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250113 02:51:32.028689 711 leader_election.cc:343] T 51d9483ecacb413796d0fd9e7560eac1 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:32.028779 711 leader_election.cc:304] T 51d9483ecacb413796d0fd9e7560eac1 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:32.028842 1191 tablet_bootstrap.cc:654] T 6e1284c8aaba44b5810b0d06d36757eb P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:32.029162 1904 raft_consensus.cc:2743] T 51d9483ecacb413796d0fd9e7560eac1 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:32.029942 889 tablet_service.cc:1514] Processing DeleteTablet for tablet c215ffe61b9b426286cc5e808463078d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:20 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:32.038602 1191 tablet_bootstrap.cc:492] T 6e1284c8aaba44b5810b0d06d36757eb P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:32.038616 1192 tablet_bootstrap.cc:492] T b0b267b0d9c54ba496c89f6cb287dffd P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:32.038751 1191 ts_tablet_manager.cc:1397] T 6e1284c8aaba44b5810b0d06d36757eb P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.000s
I20250113 02:51:32.038766 1192 ts_tablet_manager.cc:1397] T b0b267b0d9c54ba496c89f6cb287dffd P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.026s user 0.000s sys 0.010s
I20250113 02:51:32.039573 1191 raft_consensus.cc:357] T 6e1284c8aaba44b5810b0d06d36757eb P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:32.039609 1192 raft_consensus.cc:357] T b0b267b0d9c54ba496c89f6cb287dffd P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:32.039799 1192 raft_consensus.cc:383] T b0b267b0d9c54ba496c89f6cb287dffd P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:32.039785 1191 raft_consensus.cc:383] T 6e1284c8aaba44b5810b0d06d36757eb P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:32.039876 1192 raft_consensus.cc:738] T b0b267b0d9c54ba496c89f6cb287dffd P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:32.039906 1191 raft_consensus.cc:738] T 6e1284c8aaba44b5810b0d06d36757eb P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:32.040159 1192 consensus_queue.cc:260] T b0b267b0d9c54ba496c89f6cb287dffd P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:32.040763 1192 ts_tablet_manager.cc:1428] T b0b267b0d9c54ba496c89f6cb287dffd P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250113 02:51:32.041054 1192 tablet_bootstrap.cc:492] T 8337f3c82d654ecdac2cbc1ffa88ef05 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:32.042564 1192 tablet_bootstrap.cc:654] T 8337f3c82d654ecdac2cbc1ffa88ef05 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:32.052081 448 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_b171295c6e384e378d1dee2a0a362e40" } modify_external_catalogs: true
I20250113 02:51:32.052289 448 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_b171295c6e384e378d1dee2a0a362e40" } modify_external_catalogs: true
I20250113 02:51:32.052877 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_9e652a1c2e5f47a8bd55590289e5836f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:32.053475 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9e652a1c2e5f47a8bd55590289e5836f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:32.057627 451 catalog_manager.cc:5618] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587) does not have the latest schema for tablet 9fff52e99c64462a8a73fccc3c22ca63 (table default.Table_9ef18ee71ad843c5921750a2f8478855 [id=bd8bd572ed524f2cae01f820f07e3c36]). Expected version 2 got 0
I20250113 02:51:32.057955 451 catalog_manager.cc:5526] T 1cfbcd682b944c89847b40fe69f0a75e P 4216a7f853d642b7bc3e5cff458f3aca reported cstate change: term changed from 0 to 1, leader changed from <none> to 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3). New cstate: current_term: 1 leader_uuid: "4216a7f853d642b7bc3e5cff458f3aca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } health_report { overall_health: UNKNOWN } } }
I20250113 02:51:32.058385 451 catalog_manager.cc:5618] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587) does not have the latest schema for tablet 1cfbcd682b944c89847b40fe69f0a75e (table default.Table_32e88abaae96492fbcce39fb2cd7c221 [id=2b84deb07e5c4ab8b550fe42945d7b71]). Expected version 2 got 0
I20250113 02:51:32.058678 451 catalog_manager.cc:5618] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587) does not have the latest schema for tablet 378d59d314a84ddb84cf2ab041c51cf1 (table default.Table_2794f9d2a3cc49f78de33a775c1619f1 [id=0c01441cec7043b687d6744e72a0e45d]). Expected version 1 got 0
I20250113 02:51:32.058972 451 catalog_manager.cc:5526] T 4c615b0b83d74fcfae0c51047dbfbbf4 P 4216a7f853d642b7bc3e5cff458f3aca reported cstate change: term changed from 0 to 1, leader changed from <none> to 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3). New cstate: current_term: 1 leader_uuid: "4216a7f853d642b7bc3e5cff458f3aca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } health_report { overall_health: UNKNOWN } } }
I20250113 02:51:32.059504 451 catalog_manager.cc:5618] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587) does not have the latest schema for tablet 4c615b0b83d74fcfae0c51047dbfbbf4 (table default.Table_bf689f8eb8884e88a72e9b50598fc389 [id=a956f95f6dd94b578f6bd3cb3b97f9da]). Expected version 2 got 0
I20250113 02:51:32.061790 739 tablet_service.cc:1467] Processing CreateTablet for tablet 93c0ce9f457e431193a2ceb0ad4d4af6 (DEFAULT_TABLE table=default.Table_80b52e8cafc549b1b530660a5fd2bb1d [id=413cdddd25cb4a80b5b9a32fba699a35]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:32.062089 755 tablet_service.cc:1467] Processing CreateTablet for tablet b17cadf93237435ead176af3635baeca (DEFAULT_TABLE table=default.Table_d2545982503b4f978884a98f68949dfb [id=cc140a8884964e0bb639de412c587857]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:32.062214 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 93c0ce9f457e431193a2ceb0ad4d4af6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:32.062440 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b17cadf93237435ead176af3635baeca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:32.062713 2009 raft_consensus.cc:491] T be4b2a8a3927410eaf12e7592646e4af P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:32.062711 747 tablet_service.cc:1467] Processing CreateTablet for tablet 5beb8a3490eb406393343172d7cac2ed (DEFAULT_TABLE table=default.Table_4d8dbfe62717412eb35024d3c0d158b8 [id=2d0e3dc924df4e41b0b663da08c15a4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:32.062892 2009 raft_consensus.cc:513] T be4b2a8a3927410eaf12e7592646e4af P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:32.063005 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5beb8a3490eb406393343172d7cac2ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:32.063478 2009 leader_election.cc:290] T be4b2a8a3927410eaf12e7592646e4af P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:32.067377 746 tablet_service.cc:1467] Processing CreateTablet for tablet 65196e44818647f0921e66b34fd0fd8b (DEFAULT_TABLE table=default.Table_429f8ee6edea4d9287e029125ee5c861 [id=6ade0bd7293444fd9d0a23aeb557a8cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:32.067695 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65196e44818647f0921e66b34fd0fd8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:32.068186 745 tablet_service.cc:1467] Processing CreateTablet for tablet 39a9c90677964441acb1e59d16bd1505 (DEFAULT_TABLE table=default.Table_635a7ba43392414d91ff119440a6cdc7 [id=4097079f7f6e4baca61c1e46f3ec5784]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:32.068454 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 39a9c90677964441acb1e59d16bd1505. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:32.068840 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be4b2a8a3927410eaf12e7592646e4af" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:32.069402 998 leader_election.cc:343] T be4b2a8a3927410eaf12e7592646e4af P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:32.071004 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_3b46963a95a74a1d964cb3fef22375ba" } new_table_name: "default.Table_3292864b63454728b1a46a617240a099" modify_external_catalogs: true
I20250113 02:51:32.072890 748 tablet_service.cc:1467] Processing CreateTablet for tablet afaf16655a0b48098e26c771bc2473cb (DEFAULT_TABLE table=default.Table_8d74f8c0d6b24538b60e7a61451ad264 [id=67f3a61c644940a58efabf0378e78b2b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:32.072921 736 tablet_service.cc:1467] Processing CreateTablet for tablet ee701daae00243bba9e4e8897e2473bb (DEFAULT_TABLE table=default.Table_9f58d5227a9841b9aacf80945f8a65a3 [id=34a2506431544de8bb5870dca0483d0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:32.073236 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet afaf16655a0b48098e26c771bc2473cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:32.073431 736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee701daae00243bba9e4e8897e2473bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:32.073598 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be4b2a8a3927410eaf12e7592646e4af" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
I20250113 02:51:32.073664 750 tablet_service.cc:1467] Processing CreateTablet for tablet ef777573d17748419844d07d2be14c75 (DEFAULT_TABLE table=default.Table_f736da3d9cd249a989263c2481ea307c [id=c7e34ddd463a4bc3abf042b198569ed3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:32.073786 775 raft_consensus.cc:2463] T be4b2a8a3927410eaf12e7592646e4af P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 0.
I20250113 02:51:32.073956 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef777573d17748419844d07d2be14c75. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:32.074599 1042 tablet_service.cc:1467] Processing CreateTablet for tablet 93c0ce9f457e431193a2ceb0ad4d4af6 (DEFAULT_TABLE table=default.Table_80b52e8cafc549b1b530660a5fd2bb1d [id=413cdddd25cb4a80b5b9a32fba699a35]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:32.074954 1042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 93c0ce9f457e431193a2ceb0ad4d4af6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:32.075745 1025 tablet_service.cc:1467] Processing CreateTablet for tablet b17cadf93237435ead176af3635baeca (DEFAULT_TABLE table=default.Table_d2545982503b4f978884a98f68949dfb [id=cc140a8884964e0bb639de412c587857]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:32.076156 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b17cadf93237435ead176af3635baeca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:32.076206 742 tablet_service.cc:1467] Processing CreateTablet for tablet 3010c3efa19446cdbac041fed963cf4f (DEFAULT_TABLE table=default.Table_5cfdcca1a3b0477ca840ba718e978c6d [id=c935f1f953584fe3b9f29c92d862a287]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:32.076202 1037 tablet_service.cc:1467] Processing CreateTablet for tablet 65196e44818647f0921e66b34fd0fd8b (DEFAULT_TABLE table=default.Table_429f8ee6edea4d9287e029125ee5c861 [id=6ade0bd7293444fd9d0a23aeb557a8cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:32.076373 1023 tablet_service.cc:1467] Processing CreateTablet for tablet 8f2b4df7c6a74896ade95a490ea9b54c (DEFAULT_TABLE table=default.Table_8b441c8a6ec646afbe7c211d1b5d7eda [id=f431b8a2e5f34bdcac2a3510a2929a70]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:32.076510 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3010c3efa19446cdbac041fed963cf4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:32.076596 1037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65196e44818647f0921e66b34fd0fd8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:32.079547 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f2b4df7c6a74896ade95a490ea9b54c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:32.080199 1033 tablet_service.cc:1467] Processing CreateTablet for tablet 39a9c90677964441acb1e59d16bd1505 (DEFAULT_TABLE table=default.Table_635a7ba43392414d91ff119440a6cdc7 [id=4097079f7f6e4baca61c1e46f3ec5784]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:32.080204 741 tablet_service.cc:1467] Processing CreateTablet for tablet aef2ffd67def48eea501a98fbc9dc43b (DEFAULT_TABLE table=default.Table_33d7dda5ebe341bc8ef01b7edb3cd1c6 [id=25adf2d72c8f40e7b857bca0b8adec26]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:32.080420 1041 tablet_service.cc:1467] Processing CreateTablet for tablet ef777573d17748419844d07d2be14c75 (DEFAULT_TABLE table=default.Table_f736da3d9cd249a989263c2481ea307c [id=c7e34ddd463a4bc3abf042b198569ed3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:32.080510 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aef2ffd67def48eea501a98fbc9dc43b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:32.080554 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 39a9c90677964441acb1e59d16bd1505. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:32.080749 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef777573d17748419844d07d2be14c75. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:32.080989 1040 tablet_service.cc:1467] Processing CreateTablet for tablet b7276593af4d48b38e4f5da412fec2b6 (DEFAULT_TABLE table=default.Table_9ad8a6a024ee4ff7a664add48a47c662 [id=ea5b89bc75ec45b8aaf38dfed06af9c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:32.081281 1040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b7276593af4d48b38e4f5da412fec2b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:32.084201 1035 tablet_service.cc:1467] Processing CreateTablet for tablet cb897004cbfb4a0283b42d06509a9ee5 (DEFAULT_TABLE table=default.Table_eefebb7e9073472daecb078bef21b08d [id=5b5c93f281ee42a59d3e7b3e94b740cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:32.084230 1028 tablet_service.cc:1467] Processing CreateTablet for tablet ee701daae00243bba9e4e8897e2473bb (DEFAULT_TABLE table=default.Table_9f58d5227a9841b9aacf80945f8a65a3 [id=34a2506431544de8bb5870dca0483d0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:32.084219 754 tablet_service.cc:1467] Processing CreateTablet for tablet b7276593af4d48b38e4f5da412fec2b6 (DEFAULT_TABLE table=default.Table_9ad8a6a024ee4ff7a664add48a47c662 [id=ea5b89bc75ec45b8aaf38dfed06af9c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:32.084518 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b7276593af4d48b38e4f5da412fec2b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:32.084544 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee701daae00243bba9e4e8897e2473bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:32.076202 1029 tablet_service.cc:1467] Processing CreateTablet for tablet 5beb8a3490eb406393343172d7cac2ed (DEFAULT_TABLE table=default.Table_4d8dbfe62717412eb35024d3c0d158b8 [id=2d0e3dc924df4e41b0b663da08c15a4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:32.085028 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5beb8a3490eb406393343172d7cac2ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:32.088201 751 tablet_service.cc:1467] Processing CreateTablet for tablet cd6e48a720d549f2b5f30a61e1f5e017 (DEFAULT_TABLE table=default.Table_7c472d82db9f4c2f96b4af7e343e1d55 [id=e2ea7c849b9e4c4c98ea8b0a0fb0547a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:32.088201 1036 tablet_service.cc:1467] Processing CreateTablet for tablet 3010c3efa19446cdbac041fed963cf4f (DEFAULT_TABLE table=default.Table_5cfdcca1a3b0477ca840ba718e978c6d [id=c935f1f953584fe3b9f29c92d862a287]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:32.088498 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cd6e48a720d549f2b5f30a61e1f5e017. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:32.088534 1036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3010c3efa19446cdbac041fed963cf4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:32.092214 752 tablet_service.cc:1467] Processing CreateTablet for tablet dff74d7c093b45eba7a47d8eceb0c1d1 (DEFAULT_TABLE table=default.Table_65417a56ebb74c158456a735cfd14cc2 [id=3a1f9b9f051b43b38196f73b8d4d7011]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:32.092216 1027 tablet_service.cc:1467] Processing CreateTablet for tablet aef2ffd67def48eea501a98fbc9dc43b (DEFAULT_TABLE table=default.Table_33d7dda5ebe341bc8ef01b7edb3cd1c6 [id=25adf2d72c8f40e7b857bca0b8adec26]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:32.092527 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dff74d7c093b45eba7a47d8eceb0c1d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:32.092545 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aef2ffd67def48eea501a98fbc9dc43b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:32.096208 738 tablet_service.cc:1467] Processing CreateTablet for tablet 8f2b4df7c6a74896ade95a490ea9b54c (DEFAULT_TABLE table=default.Table_8b441c8a6ec646afbe7c211d1b5d7eda [id=f431b8a2e5f34bdcac2a3510a2929a70]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:32.096494 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f2b4df7c6a74896ade95a490ea9b54c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:32.080209 1034 tablet_service.cc:1467] Processing CreateTablet for tablet afaf16655a0b48098e26c771bc2473cb (DEFAULT_TABLE table=default.Table_8d74f8c0d6b24538b60e7a61451ad264 [id=67f3a61c644940a58efabf0378e78b2b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:32.097011 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet afaf16655a0b48098e26c771bc2473cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:32.099455 1191 consensus_queue.cc:260] T 6e1284c8aaba44b5810b0d06d36757eb P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:32.100086 1191 ts_tablet_manager.cc:1428] T 6e1284c8aaba44b5810b0d06d36757eb P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.061s user 0.003s sys 0.000s
I20250113 02:51:32.100214 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb897004cbfb4a0283b42d06509a9ee5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:32.100265 743 tablet_service.cc:1467] Processing CreateTablet for tablet cb897004cbfb4a0283b42d06509a9ee5 (DEFAULT_TABLE table=default.Table_eefebb7e9073472daecb078bef21b08d [id=5b5c93f281ee42a59d3e7b3e94b740cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:32.100342 1191 tablet_bootstrap.cc:492] T c7243d27a1c642269a3d0e67632e458c P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:32.100550 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb897004cbfb4a0283b42d06509a9ee5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:32.101670 1191 tablet_bootstrap.cc:654] T c7243d27a1c642269a3d0e67632e458c P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:32.104211 753 tablet_service.cc:1467] Processing CreateTablet for tablet 874e4392e5d946acae0bf9e3b3994735 (DEFAULT_TABLE table=default.Table_a71ef07a0c614223916ca517b610739d [id=4fcc06e72d744702b0c00df756c65e36]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:32.104499 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 874e4392e5d946acae0bf9e3b3994735. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:32.107664 1024 tablet_service.cc:1467] Processing CreateTablet for tablet 2c7d213481f24748912f5b2b23a33988 (DEFAULT_TABLE table=default.Table_bceb3f766ee942aba58803219ef3b245 [id=e15aa30840fa48f1ab19adb229ea8b4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:32.108059 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c7d213481f24748912f5b2b23a33988. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:32.108737 737 tablet_service.cc:1467] Processing CreateTablet for tablet d8e81a6fef794f26bdb017e16af8b30c (DEFAULT_TABLE table=default.Table_0bba725d39ff4d44a742e905c275724b [id=75666526ca6c449ab2a889091e35a771]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:32.109050 737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8e81a6fef794f26bdb017e16af8b30c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:32.109289 749 tablet_service.cc:1467] Processing CreateTablet for tablet 726e68108f5b4de4a51ffdf778ef825d (DEFAULT_TABLE table=default.Table_220afbc6ffc34d63a882ab817a21087f [id=1181eda7bd42481e83e21447735a25c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:32.109609 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 726e68108f5b4de4a51ffdf778ef825d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:32.109477 745 tablet_service.cc:1467] Processing CreateTablet for tablet 42eaf77d78ba445a841f77ceec94e4ea (DEFAULT_TABLE table=default.Table_7d2663697456437195949d256c39cb4e [id=f6cb4c1e89fd4feb84af74b4ab3acf7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:32.110126 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 42eaf77d78ba445a841f77ceec94e4ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:32.111385 1026 tablet_service.cc:1467] Processing CreateTablet for tablet cd6e48a720d549f2b5f30a61e1f5e017 (DEFAULT_TABLE table=default.Table_7c472d82db9f4c2f96b4af7e343e1d55 [id=e2ea7c849b9e4c4c98ea8b0a0fb0547a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:32.111688 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cd6e48a720d549f2b5f30a61e1f5e017. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:32.112224 995 leader_election.cc:304] T be4b2a8a3927410eaf12e7592646e4af P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters: df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:32.108992 744 tablet_service.cc:1467] Processing CreateTablet for tablet cb141fb2e9894757b3ae423242ad93d2 (DEFAULT_TABLE table=default.Table_ae01ec20166046f3918f5d29086e4274 [id=bd969982faa24e0c88eb2c25ae3c1c28]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:32.112790 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb141fb2e9894757b3ae423242ad93d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:32.113248 736 tablet_service.cc:1467] Processing CreateTablet for tablet 8c19bf80633a40f68f8517db04d5798b (DEFAULT_TABLE table=default.Table_5f5a6c9d71fa4b5dbd75fc758c95af38 [id=9059ca69484046048852c0cebdd6e742]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:32.113518 736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c19bf80633a40f68f8517db04d5798b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:32.114928 1032 tablet_service.cc:1467] Processing CreateTablet for tablet dff74d7c093b45eba7a47d8eceb0c1d1 (DEFAULT_TABLE table=default.Table_65417a56ebb74c158456a735cfd14cc2 [id=3a1f9b9f051b43b38196f73b8d4d7011]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:32.115245 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dff74d7c093b45eba7a47d8eceb0c1d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:32.115481 745 tablet_service.cc:1467] Processing CreateTablet for tablet f748397a164d42e3a7fda3585673e633 (DEFAULT_TABLE table=default.Table_4e3ba1ddebc74043b121c831b4abacd8 [id=088596b1d7014c399feac5aee67048e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:32.115769 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f748397a164d42e3a7fda3585673e633. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:32.116187 746 tablet_service.cc:1467] Processing CreateTablet for tablet 897e44db99534ce78b5da19e62b64087 (DEFAULT_TABLE table=default.Table_1a6787745a7a47f9834090244c9f5654 [id=5716a83720754ffcb63620821bfe3a7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:32.116604 747 tablet_service.cc:1467] Processing CreateTablet for tablet 99b78c336b7d4d26b4cb548922820fdd (DEFAULT_TABLE table=default.Table_bcc759b8b5ed4a4798c2127eda9b969d [id=33c7ff60ff1741309d582e0edb17ea4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:32.118997 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 897e44db99534ce78b5da19e62b64087. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:32.118983 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet e266a9083e2f46a0a43861b7708dcdd5
I20250113 02:51:32.120229 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet cfd1697a03494ad2aaa41a056e334ca7
I20250113 02:51:32.121814 2012 raft_consensus.cc:2798] T be4b2a8a3927410eaf12e7592646e4af P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 02:51:32.121912 2012 raft_consensus.cc:491] T be4b2a8a3927410eaf12e7592646e4af P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 02:51:32.121987 2012 raft_consensus.cc:3054] T be4b2a8a3927410eaf12e7592646e4af P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:32.124100 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 99b78c336b7d4d26b4cb548922820fdd. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:32.125176 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3c59ae14a6584892bf1ebf6def81d497 consensus state has no leader
I20250113 02:51:32.125332 889 tablet_service.cc:1467] Processing CreateTablet for tablet 5beb8a3490eb406393343172d7cac2ed (DEFAULT_TABLE table=default.Table_4d8dbfe62717412eb35024d3c0d158b8 [id=2d0e3dc924df4e41b0b663da08c15a4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:32.125703 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5beb8a3490eb406393343172d7cac2ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:32.126940 1039 tablet_service.cc:1467] Processing CreateTablet for tablet 42eaf77d78ba445a841f77ceec94e4ea (DEFAULT_TABLE table=default.Table_7d2663697456437195949d256c39cb4e [id=f6cb4c1e89fd4feb84af74b4ab3acf7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:32.127349 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 42eaf77d78ba445a841f77ceec94e4ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:32.127424 883 tablet_service.cc:1467] Processing CreateTablet for tablet 65196e44818647f0921e66b34fd0fd8b (DEFAULT_TABLE table=default.Table_429f8ee6edea4d9287e029125ee5c861 [id=6ade0bd7293444fd9d0a23aeb557a8cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:32.127737 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65196e44818647f0921e66b34fd0fd8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:32.128259 891 tablet_service.cc:1467] Processing CreateTablet for tablet ee701daae00243bba9e4e8897e2473bb (DEFAULT_TABLE table=default.Table_9f58d5227a9841b9aacf80945f8a65a3 [id=34a2506431544de8bb5870dca0483d0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:32.128548 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee701daae00243bba9e4e8897e2473bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:32.129520 1042 tablet_service.cc:1467] Processing CreateTablet for tablet 99b78c336b7d4d26b4cb548922820fdd (DEFAULT_TABLE table=default.Table_bcc759b8b5ed4a4798c2127eda9b969d [id=33c7ff60ff1741309d582e0edb17ea4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:32.130216 1042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 99b78c336b7d4d26b4cb548922820fdd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:32.130781 1038 tablet_service.cc:1467] Processing CreateTablet for tablet 897e44db99534ce78b5da19e62b64087 (DEFAULT_TABLE table=default.Table_1a6787745a7a47f9834090244c9f5654 [id=5716a83720754ffcb63620821bfe3a7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:32.131108 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 897e44db99534ce78b5da19e62b64087. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:32.132189 1031 tablet_service.cc:1467] Processing CreateTablet for tablet 8c19bf80633a40f68f8517db04d5798b (DEFAULT_TABLE table=default.Table_5f5a6c9d71fa4b5dbd75fc758c95af38 [id=9059ca69484046048852c0cebdd6e742]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:32.132211 882 tablet_service.cc:1467] Processing CreateTablet for tablet b17cadf93237435ead176af3635baeca (DEFAULT_TABLE table=default.Table_d2545982503b4f978884a98f68949dfb [id=cc140a8884964e0bb639de412c587857]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:32.132496 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b17cadf93237435ead176af3635baeca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:32.132683 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c19bf80633a40f68f8517db04d5798b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:32.132222 1023 tablet_service.cc:1467] Processing CreateTablet for tablet 4ab23a0e35134ca9a6a0efe28454e17f (DEFAULT_TABLE table=default.Table_c2171604b32e4fc4b4ff30e22c529c67 [id=fad2c8edefa342caa48fa75d71be0cd6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:32.133893 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ab23a0e35134ca9a6a0efe28454e17f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:32.135284 448 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_7c472d82db9f4c2f96b4af7e343e1d55" } modify_external_catalogs: true
I20250113 02:51:32.135501 448 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_7c472d82db9f4c2f96b4af7e343e1d55" } modify_external_catalogs: true
I20250113 02:51:32.136004 881 tablet_service.cc:1467] Processing CreateTablet for tablet cd6e48a720d549f2b5f30a61e1f5e017 (DEFAULT_TABLE table=default.Table_7c472d82db9f4c2f96b4af7e343e1d55 [id=e2ea7c849b9e4c4c98ea8b0a0fb0547a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:32.136287 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cd6e48a720d549f2b5f30a61e1f5e017. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:32.136327 885 tablet_service.cc:1467] Processing CreateTablet for tablet b7276593af4d48b38e4f5da412fec2b6 (DEFAULT_TABLE table=default.Table_9ad8a6a024ee4ff7a664add48a47c662 [id=ea5b89bc75ec45b8aaf38dfed06af9c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:32.136749 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b7276593af4d48b38e4f5da412fec2b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:32.139673 887 tablet_service.cc:1467] Processing CreateTablet for tablet ef777573d17748419844d07d2be14c75 (DEFAULT_TABLE table=default.Table_f736da3d9cd249a989263c2481ea307c [id=c7e34ddd463a4bc3abf042b198569ed3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:32.140043 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef777573d17748419844d07d2be14c75. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:32.140183 899 tablet_service.cc:1467] Processing CreateTablet for tablet dff74d7c093b45eba7a47d8eceb0c1d1 (DEFAULT_TABLE table=default.Table_65417a56ebb74c158456a735cfd14cc2 [id=3a1f9b9f051b43b38196f73b8d4d7011]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:32.140456 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dff74d7c093b45eba7a47d8eceb0c1d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:32.143339 747 tablet_service.cc:1514] Processing DeleteTablet for tablet cfd1697a03494ad2aaa41a056e334ca7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:32 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:32.143527 746 tablet_service.cc:1514] Processing DeleteTablet for tablet e266a9083e2f46a0a43861b7708dcdd5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:31 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:32.143697 884 tablet_service.cc:1467] Processing CreateTablet for tablet 2c7d213481f24748912f5b2b23a33988 (DEFAULT_TABLE table=default.Table_bceb3f766ee942aba58803219ef3b245 [id=e15aa30840fa48f1ab19adb229ea8b4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:32.143930 886 tablet_service.cc:1467] Processing CreateTablet for tablet 874e4392e5d946acae0bf9e3b3994735 (DEFAULT_TABLE table=default.Table_a71ef07a0c614223916ca517b610739d [id=4fcc06e72d744702b0c00df756c65e36]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:32.145956 892 tablet_service.cc:1467] Processing CreateTablet for tablet 3010c3efa19446cdbac041fed963cf4f (DEFAULT_TABLE table=default.Table_5cfdcca1a3b0477ca840ba718e978c6d [id=c935f1f953584fe3b9f29c92d862a287]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:32.146348 1191 tablet_bootstrap.cc:492] T c7243d27a1c642269a3d0e67632e458c P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:32.146482 1191 ts_tablet_manager.cc:1397] T c7243d27a1c642269a3d0e67632e458c P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.046s user 0.002s sys 0.000s
I20250113 02:51:32.146430 1033 tablet_service.cc:1467] Processing CreateTablet for tablet cb141fb2e9894757b3ae423242ad93d2 (DEFAULT_TABLE table=default.Table_ae01ec20166046f3918f5d29086e4274 [id=bd969982faa24e0c88eb2c25ae3c1c28]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:32.146822 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb141fb2e9894757b3ae423242ad93d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:32.146795 880 tablet_service.cc:1467] Processing CreateTablet for tablet afaf16655a0b48098e26c771bc2473cb (DEFAULT_TABLE table=default.Table_8d74f8c0d6b24538b60e7a61451ad264 [id=67f3a61c644940a58efabf0378e78b2b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:32.147269 894 tablet_service.cc:1467] Processing CreateTablet for tablet cb897004cbfb4a0283b42d06509a9ee5 (DEFAULT_TABLE table=default.Table_eefebb7e9073472daecb078bef21b08d [id=5b5c93f281ee42a59d3e7b3e94b740cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:32.147749 888 tablet_service.cc:1467] Processing CreateTablet for tablet 8f2b4df7c6a74896ade95a490ea9b54c (DEFAULT_TABLE table=default.Table_8b441c8a6ec646afbe7c211d1b5d7eda [id=f431b8a2e5f34bdcac2a3510a2929a70]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:32.147701 1191 raft_consensus.cc:357] T c7243d27a1c642269a3d0e67632e458c P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:32.147868 1191 raft_consensus.cc:383] T c7243d27a1c642269a3d0e67632e458c P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:32.147922 1191 raft_consensus.cc:738] T c7243d27a1c642269a3d0e67632e458c P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:32.148131 1191 consensus_queue.cc:260] T c7243d27a1c642269a3d0e67632e458c P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:32.148216 897 tablet_service.cc:1467] Processing CreateTablet for tablet aef2ffd67def48eea501a98fbc9dc43b (DEFAULT_TABLE table=default.Table_33d7dda5ebe341bc8ef01b7edb3cd1c6 [id=25adf2d72c8f40e7b857bca0b8adec26]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:32.148629 1191 ts_tablet_manager.cc:1428] T c7243d27a1c642269a3d0e67632e458c P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250113 02:51:32.148900 1191 tablet_bootstrap.cc:492] T f129bb3fbbfd492d98822be0a6626e44 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:32.148983 1041 tablet_service.cc:1467] Processing CreateTablet for tablet d8e81a6fef794f26bdb017e16af8b30c (DEFAULT_TABLE table=default.Table_0bba725d39ff4d44a742e905c275724b [id=75666526ca6c449ab2a889091e35a771]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:32.149325 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8e81a6fef794f26bdb017e16af8b30c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:32.150091 1040 tablet_service.cc:1467] Processing CreateTablet for tablet 726e68108f5b4de4a51ffdf778ef825d (DEFAULT_TABLE table=default.Table_220afbc6ffc34d63a882ab817a21087f [id=1181eda7bd42481e83e21447735a25c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:32.150384 1040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 726e68108f5b4de4a51ffdf778ef825d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:32.152473 1029 tablet_service.cc:1467] Processing CreateTablet for tablet 874e4392e5d946acae0bf9e3b3994735 (DEFAULT_TABLE table=default.Table_a71ef07a0c614223916ca517b610739d [id=4fcc06e72d744702b0c00df756c65e36]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:32.144016 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c7d213481f24748912f5b2b23a33988. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:32.152765 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 874e4392e5d946acae0bf9e3b3994735. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:32.155144 1036 tablet_service.cc:1467] Processing CreateTablet for tablet f748397a164d42e3a7fda3585673e633 (DEFAULT_TABLE table=default.Table_4e3ba1ddebc74043b121c831b4abacd8 [id=088596b1d7014c399feac5aee67048e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:32.155474 1036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f748397a164d42e3a7fda3585673e633. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:32.156121 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 874e4392e5d946acae0bf9e3b3994735. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:32.156826 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3010c3efa19446cdbac041fed963cf4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:32.158267 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet e266a9083e2f46a0a43861b7708dcdd5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:31 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:32.158448 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet cfd1697a03494ad2aaa41a056e334ca7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:32 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:32.163535 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aef2ffd67def48eea501a98fbc9dc43b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:32.168259 895 tablet_service.cc:1467] Processing CreateTablet for tablet 4ab23a0e35134ca9a6a0efe28454e17f (DEFAULT_TABLE table=default.Table_c2171604b32e4fc4b4ff30e22c529c67 [id=fad2c8edefa342caa48fa75d71be0cd6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:32.168601 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ab23a0e35134ca9a6a0efe28454e17f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:32.169106 1191 tablet_bootstrap.cc:654] T f129bb3fbbfd492d98822be0a6626e44 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:32.171329 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet afaf16655a0b48098e26c771bc2473cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:32.171380 889 tablet_service.cc:1467] Processing CreateTablet for tablet 8c19bf80633a40f68f8517db04d5798b (DEFAULT_TABLE table=default.Table_5f5a6c9d71fa4b5dbd75fc758c95af38 [id=9059ca69484046048852c0cebdd6e742]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:32.171692 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c19bf80633a40f68f8517db04d5798b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:32.174600 891 tablet_service.cc:1467] Processing CreateTablet for tablet 42eaf77d78ba445a841f77ceec94e4ea (DEFAULT_TABLE table=default.Table_7d2663697456437195949d256c39cb4e [id=f6cb4c1e89fd4feb84af74b4ab3acf7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:32.176416 896 tablet_service.cc:1467] Processing CreateTablet for tablet 99b78c336b7d4d26b4cb548922820fdd (DEFAULT_TABLE table=default.Table_bcc759b8b5ed4a4798c2127eda9b969d [id=33c7ff60ff1741309d582e0edb17ea4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:32.176749 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 99b78c336b7d4d26b4cb548922820fdd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:32.176929 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_678cc4306399435ca453eb3ce382f5fc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:32.177525 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_678cc4306399435ca453eb3ce382f5fc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:32.179711 899 tablet_service.cc:1467] Processing CreateTablet for tablet 897e44db99534ce78b5da19e62b64087 (DEFAULT_TABLE table=default.Table_1a6787745a7a47f9834090244c9f5654 [id=5716a83720754ffcb63620821bfe3a7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:32.180173 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb897004cbfb4a0283b42d06509a9ee5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:32.180224 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_345d3448247f4d818397c10f9e3c96de"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:32.180397 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 897e44db99534ce78b5da19e62b64087. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:32.180574 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f2b4df7c6a74896ade95a490ea9b54c. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:32.180685 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_345d3448247f4d818397c10f9e3c96de in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:32.180941 1191 tablet_bootstrap.cc:492] T f129bb3fbbfd492d98822be0a6626e44 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:32.181073 1191 ts_tablet_manager.cc:1397] T f129bb3fbbfd492d98822be0a6626e44 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.032s user 0.003s sys 0.000s
I20250113 02:51:32.181986 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_ef9c0cf86b0943aab6bbb6a3fe74ba85"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:32.182431 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ef9c0cf86b0943aab6bbb6a3fe74ba85 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:32.184599 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_a85aae70bdd1465786bae1f4505e333c" } modify_external_catalogs: true
I20250113 02:51:32.184772 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_a85aae70bdd1465786bae1f4505e333c" } modify_external_catalogs: true
I20250113 02:51:32.184988 886 tablet_service.cc:1467] Processing CreateTablet for tablet cb141fb2e9894757b3ae423242ad93d2 (DEFAULT_TABLE table=default.Table_ae01ec20166046f3918f5d29086e4274 [id=bd969982faa24e0c88eb2c25ae3c1c28]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:32.185277 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb141fb2e9894757b3ae423242ad93d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:32.186076 453 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_1d910c3d2b244474a576fc4088301143" } modify_external_catalogs: true
I20250113 02:51:32.186225 453 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_1d910c3d2b244474a576fc4088301143" } modify_external_catalogs: true
I20250113 02:51:32.187690 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_41b2cae1cce14b0ab6b6de6d9c946ff5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:32.188176 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_41b2cae1cce14b0ab6b6de6d9c946ff5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:32.188292 1191 raft_consensus.cc:357] T f129bb3fbbfd492d98822be0a6626e44 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:32.188478 1191 raft_consensus.cc:383] T f129bb3fbbfd492d98822be0a6626e44 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:32.188530 1191 raft_consensus.cc:738] T f129bb3fbbfd492d98822be0a6626e44 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:32.189401 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_50cc6a4316724447bdeefc5aefe989a6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:32.189801 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_50cc6a4316724447bdeefc5aefe989a6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:32.190950 740 tablet_service.cc:1467] Processing CreateTablet for tablet 2c7d213481f24748912f5b2b23a33988 (DEFAULT_TABLE table=default.Table_bceb3f766ee942aba58803219ef3b245 [id=e15aa30840fa48f1ab19adb229ea8b4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:32.191329 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c7d213481f24748912f5b2b23a33988. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:32.194388 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_44413b1d27e14bff98f1a50e0d1a4dd4" } new_table_name: "default.Table_40dae4519f04406d97443c3871b03dd9" modify_external_catalogs: true
I20250113 02:51:32.195809 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_60f9889bafe84241a748d885f45df89c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:32.196306 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_60f9889bafe84241a748d885f45df89c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:32.199318 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 42eaf77d78ba445a841f77ceec94e4ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:32.200493 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_ed2ecde421f24146b87838f8e9009663" } new_table_name: "default.Table_a35695d3f16144fdb1aa00394d72eb14" modify_external_catalogs: true
I20250113 02:51:32.202093 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_b17de55bafab4c41b91c30d96a90b75c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:32.202297 1191 consensus_queue.cc:260] T f129bb3fbbfd492d98822be0a6626e44 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:32.202862 1191 ts_tablet_manager.cc:1428] T f129bb3fbbfd492d98822be0a6626e44 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.022s user 0.002s sys 0.000s
I20250113 02:51:32.202911 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet de4a58bf63054890808e2e2fe46e43d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:20 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:32.203135 1191 tablet_bootstrap.cc:492] T 0fe5421dc3ea4be684cab084f13d8df0 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:32.204497 1191 tablet_bootstrap.cc:654] T 0fe5421dc3ea4be684cab084f13d8df0 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
W20250113 02:51:32.202530 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b17de55bafab4c41b91c30d96a90b75c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:32.207928 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet cd6e48a720d549f2b5f30a61e1f5e017
I20250113 02:51:32.209420 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet cd6e48a720d549f2b5f30a61e1f5e017 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:32 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:32.211088 882 tablet_service.cc:1467] Processing CreateTablet for tablet d8e81a6fef794f26bdb017e16af8b30c (DEFAULT_TABLE table=default.Table_0bba725d39ff4d44a742e905c275724b [id=75666526ca6c449ab2a889091e35a771]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:32.211431 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8e81a6fef794f26bdb017e16af8b30c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:32.213639 883 tablet_service.cc:1467] Processing CreateTablet for tablet 726e68108f5b4de4a51ffdf778ef825d (DEFAULT_TABLE table=default.Table_220afbc6ffc34d63a882ab817a21087f [id=1181eda7bd42481e83e21447735a25c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:32.213986 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 726e68108f5b4de4a51ffdf778ef825d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:32.215612 2012 raft_consensus.cc:513] T be4b2a8a3927410eaf12e7592646e4af P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:32.216409 2012 leader_election.cc:290] T be4b2a8a3927410eaf12e7592646e4af P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Requested vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:32.217859 1904 raft_consensus.cc:491] T f129bb3fbbfd492d98822be0a6626e44 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:32.218007 1904 raft_consensus.cc:513] T f129bb3fbbfd492d98822be0a6626e44 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:32.218775 1904 leader_election.cc:290] T f129bb3fbbfd492d98822be0a6626e44 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:32.219628 885 tablet_service.cc:1467] Processing CreateTablet for tablet f748397a164d42e3a7fda3585673e633 (DEFAULT_TABLE table=default.Table_4e3ba1ddebc74043b121c831b4abacd8 [id=088596b1d7014c399feac5aee67048e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:32.220695 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f129bb3fbbfd492d98822be0a6626e44" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:32.221148 711 leader_election.cc:343] T f129bb3fbbfd492d98822be0a6626e44 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:32.221540 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f129bb3fbbfd492d98822be0a6626e44" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
W20250113 02:51:32.222128 709 leader_election.cc:343] T f129bb3fbbfd492d98822be0a6626e44 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:32.222229 709 leader_election.cc:304] T f129bb3fbbfd492d98822be0a6626e44 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:32.222536 892 tablet_service.cc:1514] Processing DeleteTablet for tablet e266a9083e2f46a0a43861b7708dcdd5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:31 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:32.222700 892 tablet_service.cc:1514] Processing DeleteTablet for tablet cfd1697a03494ad2aaa41a056e334ca7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:32 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:32.223368 1904 raft_consensus.cc:2743] T f129bb3fbbfd492d98822be0a6626e44 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:32.224282 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 87e6a4e4a8434bbcbbe70ddf634accba
I20250113 02:51:32.224988 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_292625e9d891410287fe0647a7c6b109"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:32.225560 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_292625e9d891410287fe0647a7c6b109 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:32.226414 1191 tablet_bootstrap.cc:492] T 0fe5421dc3ea4be684cab084f13d8df0 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:32.226742 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 29b1444efd28483297bafc73a33e7a54
I20250113 02:51:32.226881 890 tablet_service.cc:1467] Processing CreateTablet for tablet 39a9c90677964441acb1e59d16bd1505 (DEFAULT_TABLE table=default.Table_635a7ba43392414d91ff119440a6cdc7 [id=4097079f7f6e4baca61c1e46f3ec5784]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:32.227262 1191 ts_tablet_manager.cc:1397] T 0fe5421dc3ea4be684cab084f13d8df0 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.024s user 0.003s sys 0.000s
I20250113 02:51:32.227629 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet 29b1444efd28483297bafc73a33e7a54 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:32 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:32.228082 1191 raft_consensus.cc:357] T 0fe5421dc3ea4be684cab084f13d8df0 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:32.228261 1191 raft_consensus.cc:383] T 0fe5421dc3ea4be684cab084f13d8df0 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:32.228327 1191 raft_consensus.cc:738] T 0fe5421dc3ea4be684cab084f13d8df0 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:32.228518 1191 consensus_queue.cc:260] T 0fe5421dc3ea4be684cab084f13d8df0 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:32.229065 1191 ts_tablet_manager.cc:1428] T 0fe5421dc3ea4be684cab084f13d8df0 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250113 02:51:32.229327 1191 tablet_bootstrap.cc:492] T 0539cbbe47e64c039ad39c4b52edcc4b P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:32.230680 1191 tablet_bootstrap.cc:654] T 0539cbbe47e64c039ad39c4b52edcc4b P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:32.232283 1023 tablet_service.cc:1467] Processing CreateTablet for tablet ccc24d92091a4643a66959d1f8ddb865 (DEFAULT_TABLE table=default.Table_2cbb88c393cc492ea908e11fea450a58 [id=fdaa2293fd5f44539ce4f447af10396a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:32.232576 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f748397a164d42e3a7fda3585673e633. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:32.232689 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ccc24d92091a4643a66959d1f8ddb865. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:32.236435 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 39a9c90677964441acb1e59d16bd1505. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:32.240069 1042 tablet_service.cc:1514] Processing DeleteTablet for tablet 87e6a4e4a8434bbcbbe70ddf634accba with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:32 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:32.244434 1191 tablet_bootstrap.cc:492] T 0539cbbe47e64c039ad39c4b52edcc4b P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
W20250113 02:51:32.246634 439 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet fe3d35d9aaf64188aaa3c1ac38d279d8 (table default.Table_5255ab9ad2034d32aef64cbb9c9a59c8 [id=c8a8725a3ac44d4b8960e2ceccb0ccc6], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet fe3d35d9aaf64188aaa3c1ac38d279d8 consensus state has no leader
I20250113 02:51:32.247121 1191 ts_tablet_manager.cc:1397] T 0539cbbe47e64c039ad39c4b52edcc4b P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.000s
I20250113 02:51:32.247922 1191 raft_consensus.cc:357] T 0539cbbe47e64c039ad39c4b52edcc4b P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:32.248129 1191 raft_consensus.cc:383] T 0539cbbe47e64c039ad39c4b52edcc4b P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:32.248195 1191 raft_consensus.cc:738] T 0539cbbe47e64c039ad39c4b52edcc4b P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:32.248373 1191 consensus_queue.cc:260] T 0539cbbe47e64c039ad39c4b52edcc4b P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:32.248924 1191 ts_tablet_manager.cc:1428] T 0539cbbe47e64c039ad39c4b52edcc4b P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250113 02:51:32.249182 1191 tablet_bootstrap.cc:492] T 3b1910496e0849b193e390a5ef1a6b2b P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:32.249244 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be4b2a8a3927410eaf12e7592646e4af" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196"
I20250113 02:51:32.249408 775 raft_consensus.cc:3054] T be4b2a8a3927410eaf12e7592646e4af P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:32.252126 1029 tablet_service.cc:1467] Processing CreateTablet for tablet 55762ab5317846f7af014461c09e8d81 (DEFAULT_TABLE table=default.Table_d2a96259765e46f497379cf235dc8212 [id=4a9658232cf1489e8c491675646b9d48]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:32.252569 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 55762ab5317846f7af014461c09e8d81. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:32.254549 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d29899b4cccb4e37bd25978d385916f9 consensus state has no leader
I20250113 02:51:32.254585 741 tablet_service.cc:1467] Processing CreateTablet for tablet ccc24d92091a4643a66959d1f8ddb865 (DEFAULT_TABLE table=default.Table_2cbb88c393cc492ea908e11fea450a58 [id=fdaa2293fd5f44539ce4f447af10396a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:32.254951 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ccc24d92091a4643a66959d1f8ddb865. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:32.251943 775 raft_consensus.cc:2463] T be4b2a8a3927410eaf12e7592646e4af P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 1.
I20250113 02:51:32.256126 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be4b2a8a3927410eaf12e7592646e4af" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f"
W20250113 02:51:32.257056 998 leader_election.cc:343] T be4b2a8a3927410eaf12e7592646e4af P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:32.258505 995 leader_election.cc:304] T be4b2a8a3927410eaf12e7592646e4af P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters: df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:32.259034 2012 raft_consensus.cc:2798] T be4b2a8a3927410eaf12e7592646e4af P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Leader election won for term 1
I20250113 02:51:32.259167 2012 raft_consensus.cc:695] T be4b2a8a3927410eaf12e7592646e4af P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: Becoming Leader. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Running, Role: LEADER
I20250113 02:51:32.259488 2012 consensus_queue.cc:237] T be4b2a8a3927410eaf12e7592646e4af P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:32.259964 751 tablet_service.cc:1514] Processing DeleteTablet for tablet 87e6a4e4a8434bbcbbe70ddf634accba with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:32 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:32.261055 742 tablet_service.cc:1514] Processing DeleteTablet for tablet 29b1444efd28483297bafc73a33e7a54 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:32 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:32.261912 738 tablet_service.cc:1514] Processing DeleteTablet for tablet cd6e48a720d549f2b5f30a61e1f5e017 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:32 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:32.262188 1024 tablet_service.cc:1467] Processing CreateTablet for tablet 3178924445d64fea89a9721be7d241e9 (DEFAULT_TABLE table=default.Table_bc51f0b8366d47bcb5f9e7069597036b [id=2cbe5301ebd640048cbb7cba8a1f3cdf]), partition=RANGE (key) PARTITION UNBOUNDED
W20250113 02:51:32.262355 451 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f3a8c7a25aca4496919386396626de00 consensus state has no leader
I20250113 02:51:32.262969 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3178924445d64fea89a9721be7d241e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:32.263217 742 tablet_service.cc:1467] Processing CreateTablet for tablet 166f58d45b7b443ea54087e4ab48eaba (DEFAULT_TABLE table=default.Table_85ede9aae0dd4e3abefcc25b836e9d0b [id=a63ccead990445fdac16fda1312bb66c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:32.263547 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 166f58d45b7b443ea54087e4ab48eaba. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:32.264216 743 tablet_service.cc:1467] Processing CreateTablet for tablet 3178924445d64fea89a9721be7d241e9 (DEFAULT_TABLE table=default.Table_bc51f0b8366d47bcb5f9e7069597036b [id=2cbe5301ebd640048cbb7cba8a1f3cdf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:32.264535 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3178924445d64fea89a9721be7d241e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:32.265879 1191 tablet_bootstrap.cc:654] T 3b1910496e0849b193e390a5ef1a6b2b P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:32.268062 752 tablet_service.cc:1467] Processing CreateTablet for tablet b6293a3b355b4c409b1158b6ba51f727 (DEFAULT_TABLE table=default.Table_2aa1258a7cff4416b0682966ee8150e9 [id=2a4acf3cff234c99991c3c440aac9258]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:32.268409 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b6293a3b355b4c409b1158b6ba51f727. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:32.268483 738 tablet_service.cc:1467] Processing CreateTablet for tablet 55762ab5317846f7af014461c09e8d81 (DEFAULT_TABLE table=default.Table_d2a96259765e46f497379cf235dc8212 [id=4a9658232cf1489e8c491675646b9d48]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:32.268795 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 55762ab5317846f7af014461c09e8d81. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:32.268795 1029 tablet_service.cc:1467] Processing CreateTablet for tablet 166f58d45b7b443ea54087e4ab48eaba (DEFAULT_TABLE table=default.Table_85ede9aae0dd4e3abefcc25b836e9d0b [id=a63ccead990445fdac16fda1312bb66c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:32.269304 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 166f58d45b7b443ea54087e4ab48eaba. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:32.273418 1191 tablet_bootstrap.cc:492] T 3b1910496e0849b193e390a5ef1a6b2b P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:32.273984 1191 ts_tablet_manager.cc:1397] T 3b1910496e0849b193e390a5ef1a6b2b P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.025s user 0.003s sys 0.000s
I20250113 02:51:32.274852 1191 raft_consensus.cc:357] T 3b1910496e0849b193e390a5ef1a6b2b P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:32.276345 1035 tablet_service.cc:1467] Processing CreateTablet for tablet b6293a3b355b4c409b1158b6ba51f727 (DEFAULT_TABLE table=default.Table_2aa1258a7cff4416b0682966ee8150e9 [id=2a4acf3cff234c99991c3c440aac9258]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:32.276736 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b6293a3b355b4c409b1158b6ba51f727. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:32.277642 893 tablet_service.cc:1467] Processing CreateTablet for tablet 93c0ce9f457e431193a2ceb0ad4d4af6 (DEFAULT_TABLE table=default.Table_80b52e8cafc549b1b530660a5fd2bb1d [id=413cdddd25cb4a80b5b9a32fba699a35]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:32.278038 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 93c0ce9f457e431193a2ceb0ad4d4af6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:32.279819 1191 raft_consensus.cc:383] T 3b1910496e0849b193e390a5ef1a6b2b P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:32.280082 1191 raft_consensus.cc:738] T 3b1910496e0849b193e390a5ef1a6b2b P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:32.280414 1191 consensus_queue.cc:260] T 3b1910496e0849b193e390a5ef1a6b2b P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:32.281315 1191 ts_tablet_manager.cc:1428] T 3b1910496e0849b193e390a5ef1a6b2b P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20250113 02:51:32.282068 1191 tablet_bootstrap.cc:492] T 99c8046f5d124ca49899df365114c69f P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:32.282862 1031 tablet_service.cc:1467] Processing CreateTablet for tablet eba80548f52f4c32a098d1540c9cded0 (DEFAULT_TABLE table=default.Table_bb7ff740ee724b3999eeec918a03b1ae [id=1291674e6af743d08c5fff97bbb34c8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:32.283380 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eba80548f52f4c32a098d1540c9cded0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:32.284747 1191 tablet_bootstrap.cc:654] T 99c8046f5d124ca49899df365114c69f P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:32.285770 1992 log.cc:826] T 00000000000000000000000000000000 P 45e9d43ecac0401fb2c18fe5eaf089b5: Log is configured to *not* fsync() on all Append() calls
I20250113 02:51:32.287329 739 tablet_service.cc:1467] Processing CreateTablet for tablet 4ab23a0e35134ca9a6a0efe28454e17f (DEFAULT_TABLE table=default.Table_c2171604b32e4fc4b4ff30e22c529c67 [id=fad2c8edefa342caa48fa75d71be0cd6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:32.287650 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ab23a0e35134ca9a6a0efe28454e17f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:32.288268 891 tablet_service.cc:1467] Processing CreateTablet for tablet ccc24d92091a4643a66959d1f8ddb865 (DEFAULT_TABLE table=default.Table_2cbb88c393cc492ea908e11fea450a58 [id=fdaa2293fd5f44539ce4f447af10396a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:32.288630 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ccc24d92091a4643a66959d1f8ddb865. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:32.275828 755 tablet_service.cc:1467] Processing CreateTablet for tablet eba80548f52f4c32a098d1540c9cded0 (DEFAULT_TABLE table=default.Table_bb7ff740ee724b3999eeec918a03b1ae [id=1291674e6af743d08c5fff97bbb34c8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:32.291021 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eba80548f52f4c32a098d1540c9cded0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:32.295648 751 tablet_service.cc:1467] Processing CreateTablet for tablet b86d035e503344e1a3bd85c55131274e (DEFAULT_TABLE table=default.Table_f4ebf22f164942e581f60ba75f899497 [id=bc667764370048d0a6211fa11ac9c55b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:32.296068 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b86d035e503344e1a3bd85c55131274e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:32.298103 894 tablet_service.cc:1514] Processing DeleteTablet for tablet 87e6a4e4a8434bbcbbe70ddf634accba with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:32 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:32.299490 886 tablet_service.cc:1467] Processing CreateTablet for tablet b6293a3b355b4c409b1158b6ba51f727 (DEFAULT_TABLE table=default.Table_2aa1258a7cff4416b0682966ee8150e9 [id=2a4acf3cff234c99991c3c440aac9258]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:32.300161 889 tablet_service.cc:1467] Processing CreateTablet for tablet eba80548f52f4c32a098d1540c9cded0 (DEFAULT_TABLE table=default.Table_bb7ff740ee724b3999eeec918a03b1ae [id=1291674e6af743d08c5fff97bbb34c8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:32.300391 897 tablet_service.cc:1467] Processing CreateTablet for tablet 166f58d45b7b443ea54087e4ab48eaba (DEFAULT_TABLE table=default.Table_85ede9aae0dd4e3abefcc25b836e9d0b [id=a63ccead990445fdac16fda1312bb66c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:32.299870 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b6293a3b355b4c409b1158b6ba51f727. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:32.300827 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 166f58d45b7b443ea54087e4ab48eaba. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:32.301071 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eba80548f52f4c32a098d1540c9cded0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:32.299855 884 tablet_service.cc:1467] Processing CreateTablet for tablet 3178924445d64fea89a9721be7d241e9 (DEFAULT_TABLE table=default.Table_bc51f0b8366d47bcb5f9e7069597036b [id=2cbe5301ebd640048cbb7cba8a1f3cdf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:32.301503 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3178924445d64fea89a9721be7d241e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:32.301638 1904 raft_consensus.cc:491] T 056bf1e67be04d179eace3d93282b145 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:32.301774 1904 raft_consensus.cc:513] T 056bf1e67be04d179eace3d93282b145 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:32.302006 896 tablet_service.cc:1514] Processing DeleteTablet for tablet cd6e48a720d549f2b5f30a61e1f5e017 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:32 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:32.302202 894 tablet_service.cc:1514] Processing DeleteTablet for tablet 29b1444efd28483297bafc73a33e7a54 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:32 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:32.302534 448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_0bf61ef424c949cfaff504365b82c4cf" } new_table_name: "default.Table_d1d988a18e0b47b1baef18f31d0cb356" modify_external_catalogs: true
W20250113 02:51:32.304049 421 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.104.2:38467, user_credentials={real_user=slave}} blocked reactor thread for 133647us
I20250113 02:51:32.305518 1904 leader_election.cc:290] T 056bf1e67be04d179eace3d93282b145 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:32.305732 1040 tablet_service.cc:1514] Processing DeleteTablet for tablet 2778d7dc7d9c4843bbc9eb705cec71cd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:20 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:32.307356 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "056bf1e67be04d179eace3d93282b145" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:32.307926 711 leader_election.cc:343] T 056bf1e67be04d179eace3d93282b145 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:32.309660 1191 tablet_bootstrap.cc:492] T 99c8046f5d124ca49899df365114c69f P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:32.313035 1191 ts_tablet_manager.cc:1397] T 99c8046f5d124ca49899df365114c69f P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.031s user 0.003s sys 0.000s
I20250113 02:51:32.314224 1191 raft_consensus.cc:357] T 99c8046f5d124ca49899df365114c69f P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:32.316362 1191 raft_consensus.cc:383] T 99c8046f5d124ca49899df365114c69f P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:32.316602 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_d1a29205746d47a6bcf4719cdaeaf285" } new_table_name: "default.Table_1f2bc7d62d9c47fe82ab1108f7197080" modify_external_catalogs: true
I20250113 02:51:32.316843 2009 raft_consensus.cc:491] T 2778d7dc7d9c4843bbc9eb705cec71cd P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:32.317006 2009 raft_consensus.cc:513] T 2778d7dc7d9c4843bbc9eb705cec71cd P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:32.317571 2009 leader_election.cc:290] T 2778d7dc7d9c4843bbc9eb705cec71cd P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
I20250113 02:51:32.317898 1062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "056bf1e67be04d179eace3d93282b145" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:32.319562 1191 raft_consensus.cc:738] T 99c8046f5d124ca49899df365114c69f P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
W20250113 02:51:32.319895 709 leader_election.cc:343] T 056bf1e67be04d179eace3d93282b145 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:32.320043 709 leader_election.cc:304] T 056bf1e67be04d179eace3d93282b145 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:32.321383 1038 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d3562a48c6e4badbbcf37640e91b5ba with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:21 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:32.322052 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet b0b267b0d9c54ba496c89f6cb287dffd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:25 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:32.322544 1032 tablet_service.cc:1514] Processing DeleteTablet for tablet 23e04961f072440fa3e3f9485e40f79c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:20 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:32.322757 451 catalog_manager.cc:5526] T be4b2a8a3927410eaf12e7592646e4af P 4216a7f853d642b7bc3e5cff458f3aca reported cstate change: term changed from 0 to 1, leader changed from <none> to 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3). New cstate: current_term: 1 leader_uuid: "4216a7f853d642b7bc3e5cff458f3aca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } health_report { overall_health: HEALTHY } } }
I20250113 02:51:32.323303 451 catalog_manager.cc:5618] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587) does not have the latest schema for tablet be4b2a8a3927410eaf12e7592646e4af (table default.Table_24d9af3d0b0740d69c791fc96916ef0a [id=124669ccadaf49e19d2660b1495f9647]). Expected version 3 got 0
I20250113 02:51:32.323333 1026 tablet_service.cc:1467] Processing CreateTablet for tablet 95de1523f15d43fa8e9b12b75860b1dc (DEFAULT_TABLE table=default.Table_eaece5fe552b47fb9c0f3d73031f40f4 [id=27d19697fb0f42d09faee637ba85cefd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:32.320050 1191 consensus_queue.cc:260] T 99c8046f5d124ca49899df365114c69f P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:32.325474 1191 ts_tablet_manager.cc:1428] T 99c8046f5d124ca49899df365114c69f P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.012s user 0.001s sys 0.000s
I20250113 02:51:32.327579 1191 tablet_bootstrap.cc:492] T e468b652f71f44e3945f8d919ac0e1a8 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:32.328166 1904 raft_consensus.cc:2743] T 056bf1e67be04d179eace3d93282b145 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250113 02:51:32.329134 995 leader_election.cc:343] T 2778d7dc7d9c4843bbc9eb705cec71cd P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Not found: Tablet not found: 2778d7dc7d9c4843bbc9eb705cec71cd
I20250113 02:51:32.330502 736 tablet_service.cc:1467] Processing CreateTablet for tablet 95de1523f15d43fa8e9b12b75860b1dc (DEFAULT_TABLE table=default.Table_eaece5fe552b47fb9c0f3d73031f40f4 [id=27d19697fb0f42d09faee637ba85cefd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:32.330718 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_3cf8774e45444063a6da1fec1b7ed5e2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:32.331012 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2778d7dc7d9c4843bbc9eb705cec71cd" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:32.331282 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3cf8774e45444063a6da1fec1b7ed5e2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:32.331779 450 master_service.cc:946] ReplaceTablet: received request to replace tablet 0f428c555442401285fb4eab8280b10b from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:32.332517 449 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_8a3a3b6c452a437dae09bdd5ee43c8a0" } modify_external_catalogs: true
W20250113 02:51:32.333168 998 leader_election.cc:343] T 2778d7dc7d9c4843bbc9eb705cec71cd P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:32.333382 998 leader_election.cc:304] T 2778d7dc7d9c4843bbc9eb705cec71cd P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4216a7f853d642b7bc3e5cff458f3aca; no voters: 1186fa869eb14812832fa6562531f196, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:32.333794 2009 raft_consensus.cc:2743] T 2778d7dc7d9c4843bbc9eb705cec71cd P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:32.337584 880 tablet_service.cc:1467] Processing CreateTablet for tablet 55762ab5317846f7af014461c09e8d81 (DEFAULT_TABLE table=default.Table_d2a96259765e46f497379cf235dc8212 [id=4a9658232cf1489e8c491675646b9d48]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:32.337893 447 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_60f9889bafe84241a748d885f45df89c" } modify_external_catalogs: true
I20250113 02:51:32.338652 736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95de1523f15d43fa8e9b12b75860b1dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:32.340735 894 tablet_service.cc:1467] Processing CreateTablet for tablet b86d035e503344e1a3bd85c55131274e (DEFAULT_TABLE table=default.Table_f4ebf22f164942e581f60ba75f899497 [id=bc667764370048d0a6211fa11ac9c55b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:32.342948 896 tablet_service.cc:1467] Processing CreateTablet for tablet 95de1523f15d43fa8e9b12b75860b1dc (DEFAULT_TABLE table=default.Table_eaece5fe552b47fb9c0f3d73031f40f4 [id=27d19697fb0f42d09faee637ba85cefd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:32.343797 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 55762ab5317846f7af014461c09e8d81. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:32.344110 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95de1523f15d43fa8e9b12b75860b1dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:32.347334 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b86d035e503344e1a3bd85c55131274e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:32.367657 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_a4b585b4849b4950ba3a372aa4190e7c" } new_table_name: "default.Table_aafae7a2874249e7ac84dcde865d33cf" modify_external_catalogs: true
I20250113 02:51:32.367873 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_c2aba85708b64110bc3a4254a09352d7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:32.369692 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c2aba85708b64110bc3a4254a09352d7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:32.374624 1191 tablet_bootstrap.cc:654] T e468b652f71f44e3945f8d919ac0e1a8 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:32.379699 1194 tablet_bootstrap.cc:492] T 401c4f5dbd304950877b59a865f417b3 P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:32.379894 1194 ts_tablet_manager.cc:1397] T 401c4f5dbd304950877b59a865f417b3 P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.396s user 0.005s sys 0.004s
I20250113 02:51:32.381311 1194 raft_consensus.cc:357] T 401c4f5dbd304950877b59a865f417b3 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:32.381587 1194 raft_consensus.cc:383] T 401c4f5dbd304950877b59a865f417b3 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:32.381682 1194 raft_consensus.cc:738] T 401c4f5dbd304950877b59a865f417b3 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:32.381978 1194 consensus_queue.cc:260] T 401c4f5dbd304950877b59a865f417b3 P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:32.382747 1194 ts_tablet_manager.cc:1428] T 401c4f5dbd304950877b59a865f417b3 P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
W20250113 02:51:32.385872 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d7542beb84c94b258d2f35a4dc74eec5 consensus state has no leader
I20250113 02:51:32.386205 449 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_8a3a3b6c452a437dae09bdd5ee43c8a0" } modify_external_catalogs: true
I20250113 02:51:32.386576 1195 ts_tablet_manager.cc:1918] T c1fb2b36c0504cb4ba32671ae4c620d1 P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:32.386690 1195 log.cc:1198] T c1fb2b36c0504cb4ba32671ae4c620d1 P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/c1fb2b36c0504cb4ba32671ae4c620d1
I20250113 02:51:32.386998 1195 ts_tablet_manager.cc:1939] T c1fb2b36c0504cb4ba32671ae4c620d1 P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:32.387292 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 794) took 1127 ms. Trace:
I20250113 02:51:32.387370 1195 rpcz_store.cc:276] 0113 02:51:31.259618 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:31.259926 (+ 308us) service_pool.cc:225] Handling call
0113 02:51:31.980531 (+720605us) ts_tablet_manager.cc:1145] Deleting tablet c1fb2b36c0504cb4ba32671ae4c620d1
0113 02:51:31.982700 (+ 2169us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:32.383622 (+400922us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:32.386479 (+ 2857us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:32.387274 (+ 795us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":717890}
I20250113 02:51:32.387854 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet c1fb2b36c0504cb4ba32671ae4c620d1 (table default.Table_53a85211f49241c4af531e479ecdc0cb [id=ff8b884b04954ff48a7c8a72df232e8a]) successfully deleted
W20250113 02:51:32.391696 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6fd59b6773fd4bac863b1714f13c9184 has no consensus state
I20250113 02:51:32.392040 447 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_60f9889bafe84241a748d885f45df89c" } modify_external_catalogs: true
W20250113 02:51:32.394315 448 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet df87e25053a74990959f85288a85217b consensus state has no leader
I20250113 02:51:32.397716 1191 tablet_bootstrap.cc:492] T e468b652f71f44e3945f8d919ac0e1a8 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:32.397914 1191 ts_tablet_manager.cc:1397] T e468b652f71f44e3945f8d919ac0e1a8 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.070s user 0.005s sys 0.000s
I20250113 02:51:32.398764 1191 raft_consensus.cc:357] T e468b652f71f44e3945f8d919ac0e1a8 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
W20250113 02:51:32.399557 772 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.504s user 0.009s sys 0.002s
I20250113 02:51:32.399643 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 4cbc27d8ac2a46c58bffc33d8af17be2
W20250113 02:51:32.399683 772 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.504s user 0.009s sys 0.002s
I20250113 02:51:32.399021 1191 raft_consensus.cc:383] T e468b652f71f44e3945f8d919ac0e1a8 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:32.400184 1191 raft_consensus.cc:738] T e468b652f71f44e3945f8d919ac0e1a8 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:32.400461 1191 consensus_queue.cc:260] T e468b652f71f44e3945f8d919ac0e1a8 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:32.401033 1191 ts_tablet_manager.cc:1428] T e468b652f71f44e3945f8d919ac0e1a8 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250113 02:51:32.401913 1191 tablet_bootstrap.cc:492] T f56230a2c75d4763a54671c79358f4d9 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:32.403987 1191 tablet_bootstrap.cc:654] T f56230a2c75d4763a54671c79358f4d9 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:32.412776 741 tablet_service.cc:1514] Processing DeleteTablet for tablet 4cbc27d8ac2a46c58bffc33d8af17be2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:32 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:32.421185 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 0f428c555442401285fb4eab8280b10b
I20250113 02:51:32.421830 450 catalog_manager.cc:6596] ReplaceTablet: tablet 0f428c555442401285fb4eab8280b10b deleted and replaced by tablet 582aef4539b240dc8aa82d29c4261153
I20250113 02:51:32.422650 741 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f428c555442401285fb4eab8280b10b with delete_type TABLET_DATA_DELETED (replaced by tablet 582aef4539b240dc8aa82d29c4261153) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:32.374426 2009 consensus_queue.cc:1035] T 1cfbcd682b944c89847b40fe69f0a75e P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.049s
I20250113 02:51:32.393092 1196 ts_tablet_manager.cc:1918] T e8ab4516789e4585a101c8766d338af3 P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:32.400846 886 tablet_service.cc:1514] Processing DeleteTablet for tablet 4cbc27d8ac2a46c58bffc33d8af17be2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:32 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:32.428072 1196 log.cc:1198] T e8ab4516789e4585a101c8766d338af3 P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/e8ab4516789e4585a101c8766d338af3
I20250113 02:51:32.428489 1196 ts_tablet_manager.cc:1939] T e8ab4516789e4585a101c8766d338af3 P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:32.428828 1196 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37366 (request call id 747) took 1231 ms. Trace:
I20250113 02:51:32.422504 890 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f428c555442401285fb4eab8280b10b with delete_type TABLET_DATA_DELETED (replaced by tablet 582aef4539b240dc8aa82d29c4261153) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:32.425626 876 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:37356 (request call id 223) took 1019 ms. Trace:
I20250113 02:51:32.429554 876 rpcz_store.cc:276] 0113 02:51:31.406171 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:31.406236 (+ 65us) service_pool.cc:225] Handling call
0113 02:51:32.425583 (+1019347us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 02:51:32.432019 447 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet f61b635c9a6840269a98ae2765cc9123
I20250113 02:51:32.434474 449 catalog_manager.cc:5618] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095) does not have the latest schema for tablet 32cba8f3efc54e78b4df41c15df82eea (table default.Table_27e1ad9a94f8455f806f48768ef4ddac [id=8aff13d6d3c5483eadb596b705fe5151]). Expected version 5 got 0
I20250113 02:51:32.450573 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_2bd58703f99f4f8f9945ac5026a63f2b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:32.452060 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2bd58703f99f4f8f9945ac5026a63f2b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:32.453284 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 795) took 1149 ms. Trace:
I20250113 02:51:32.453397 1195 rpcz_store.cc:276] 0113 02:51:31.304292 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:31.304352 (+ 60us) service_pool.cc:225] Handling call
0113 02:51:32.453155 (+1148803us) ts_tablet_manager.cc:1145] Deleting tablet 2e4a68bbcd684785a7bacdbabb506718
0113 02:51:32.453167 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:32.453265 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1148616}
I20250113 02:51:32.453687 1195 tablet_replica.cc:331] T a97efc70946c4af487e33a85abf6b90c P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:32.453855 1195 raft_consensus.cc:2238] T a97efc70946c4af487e33a85abf6b90c P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus shutting down.
W20250113 02:51:32.453944 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 2e4a68bbcd684785a7bacdbabb506718 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2e4a68bbcd684785a7bacdbabb506718 already in progress: creating tablet
I20250113 02:51:32.454008 1195 raft_consensus.cc:2267] T a97efc70946c4af487e33a85abf6b90c P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:32.454561 1195 ts_tablet_manager.cc:1905] T a97efc70946c4af487e33a85abf6b90c P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:32.457069 451 master_service.cc:946] ReplaceTablet: received request to replace tablet fd0b8592997d4a61b8ca854ecbb67c55 from {username='slave'} at 127.0.0.1:49258
W20250113 02:51:32.475180 1030 consensus_meta.cc:287] T a446180f980640268170b4ab652d32cc P 4216a7f853d642b7bc3e5cff458f3aca: Time spent flushing consensus metadata: real 0.585s user 0.002s sys 0.001s
I20250113 02:51:32.475412 1032 tablet_service.cc:1514] Processing DeleteTablet for tablet 4cbc27d8ac2a46c58bffc33d8af17be2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:32 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:32.477123 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet fd0b8592997d4a61b8ca854ecbb67c55
I20250113 02:51:32.477334 1035 tablet_service.cc:1467] Processing CreateTablet for tablet b86d035e503344e1a3bd85c55131274e (DEFAULT_TABLE table=default.Table_f4ebf22f164942e581f60ba75f899497 [id=bc667764370048d0a6211fa11ac9c55b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:32.478060 741 tablet_service.cc:1514] Processing DeleteTablet for tablet fd0b8592997d4a61b8ca854ecbb67c55 with delete_type TABLET_DATA_DELETED (replaced by tablet e312e70334e04ecd843159ac72f4a2a6) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:32.479660 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f428c555442401285fb4eab8280b10b with delete_type TABLET_DATA_DELETED (replaced by tablet 582aef4539b240dc8aa82d29c4261153) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:32.479786 1195 ts_tablet_manager.cc:1918] T a97efc70946c4af487e33a85abf6b90c P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:32.479899 1195 log.cc:1198] T a97efc70946c4af487e33a85abf6b90c P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/a97efc70946c4af487e33a85abf6b90c
I20250113 02:51:32.480280 1195 ts_tablet_manager.cc:1939] T a97efc70946c4af487e33a85abf6b90c P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:32.480767 451 catalog_manager.cc:6596] ReplaceTablet: tablet fd0b8592997d4a61b8ca854ecbb67c55 deleted and replaced by tablet e312e70334e04ecd843159ac72f4a2a6
I20250113 02:51:32.482862 449 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_a55eab4604df4ca5b767708b6533e6d3" } modify_external_catalogs: true
I20250113 02:51:32.483050 449 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_a55eab4604df4ca5b767708b6533e6d3" } modify_external_catalogs: true
I20250113 02:51:32.483639 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_f547905e49744f99810317090e10ba97"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:32.484231 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f547905e49744f99810317090e10ba97 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:32.487409 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_517c2399a90d4b36925a3b7fb36d5963" } modify_external_catalogs: true
I20250113 02:51:32.487646 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_517c2399a90d4b36925a3b7fb36d5963" } modify_external_catalogs: true
I20250113 02:51:32.490545 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 797) took 1122 ms. Trace:
I20250113 02:51:32.480643 1032 tablet_service.cc:1514] Processing DeleteTablet for tablet fd0b8592997d4a61b8ca854ecbb67c55 with delete_type TABLET_DATA_DELETED (replaced by tablet e312e70334e04ecd843159ac72f4a2a6) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:32.490866 1195 rpcz_store.cc:276] 0113 02:51:31.368231 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:31.368291 (+ 60us) service_pool.cc:225] Handling call
0113 02:51:32.453642 (+1085351us) ts_tablet_manager.cc:1145] Deleting tablet a97efc70946c4af487e33a85abf6b90c
0113 02:51:32.453652 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:32.470346 (+ 16694us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:32.479698 (+ 9352us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:32.490528 (+ 10830us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1085143}
I20250113 02:51:32.491294 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet a97efc70946c4af487e33a85abf6b90c (table default.Table_a5e21e7ff75b4d70a6944712fb3d1392 [id=d29a78ceccaa42808de9c2736d3eb4ac]) successfully deleted
I20250113 02:51:32.491952 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 799) took 1118 ms. Trace:
I20250113 02:51:32.492686 1195 rpcz_store.cc:276] 0113 02:51:31.373054 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:31.373103 (+ 49us) service_pool.cc:225] Handling call
0113 02:51:32.491832 (+1118729us) ts_tablet_manager.cc:1145] Deleting tablet e13267498917432ba92b6badb76f5d73
0113 02:51:32.491842 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:32.491935 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1118541}
I20250113 02:51:32.493664 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b86d035e503344e1a3bd85c55131274e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:32.494210 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95de1523f15d43fa8e9b12b75860b1dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:32.428907 1196 rpcz_store.cc:276] 0113 02:51:31.196926 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:31.196983 (+ 57us) service_pool.cc:225] Handling call
0113 02:51:32.012776 (+815793us) ts_tablet_manager.cc:1145] Deleting tablet e8ab4516789e4585a101c8766d338af3
0113 02:51:32.012786 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:32.294447 (+281661us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:32.392993 (+ 98546us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:32.428812 (+ 35819us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":758554}
I20250113 02:51:32.494982 1196 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37366 (request call id 748) took 1274 ms. Trace:
I20250113 02:51:32.495069 1196 rpcz_store.cc:276] 0113 02:51:31.220722 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:31.221446 (+ 724us) service_pool.cc:225] Handling call
0113 02:51:32.494844 (+1273398us) ts_tablet_manager.cc:1145] Deleting tablet 2e4a68bbcd684785a7bacdbabb506718
0113 02:51:32.494855 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:32.494961 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1240496}
I20250113 02:51:32.495373 1196 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37366 (request call id 749) took 1274 ms. Trace:
I20250113 02:51:32.495434 1196 rpcz_store.cc:276] 0113 02:51:31.220879 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:31.221614 (+ 735us) service_pool.cc:225] Handling call
0113 02:51:32.495277 (+1273663us) ts_tablet_manager.cc:1145] Deleting tablet 4c759831773849f2bc52e484ffa0cf49
0113 02:51:32.495291 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:32.495360 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1240861}
I20250113 02:51:32.495682 1196 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37366 (request call id 750) took 1206 ms. Trace:
I20250113 02:51:32.495743 1196 rpcz_store.cc:276] 0113 02:51:31.289529 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:31.289585 (+ 56us) service_pool.cc:225] Handling call
0113 02:51:32.495606 (+1206021us) ts_tablet_manager.cc:1145] Deleting tablet 2e4a68bbcd684785a7bacdbabb506718
0113 02:51:32.495613 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:32.495670 (+ 57us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1205858}
I20250113 02:51:32.496049 1196 tablet_replica.cc:331] T 63efcb6b2e194c469fb30a197bd6565b P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:32.496215 1196 raft_consensus.cc:2238] T 63efcb6b2e194c469fb30a197bd6565b P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:32.496349 1196 raft_consensus.cc:2267] T 63efcb6b2e194c469fb30a197bd6565b P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:32.497509 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet e8ab4516789e4585a101c8766d338af3 (table default.Table_3b76cf9fac334ebf9b1ecb9f8d07ea74 [id=b112ac4e2e58460db083da1eb788ca07]) successfully deleted
W20250113 02:51:32.497740 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 2e4a68bbcd684785a7bacdbabb506718 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2e4a68bbcd684785a7bacdbabb506718 already in progress: creating tablet
W20250113 02:51:32.497958 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 4c759831773849f2bc52e484ffa0cf49 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4c759831773849f2bc52e484ffa0cf49 already in progress: creating tablet
W20250113 02:51:32.498211 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 2e4a68bbcd684785a7bacdbabb506718 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2e4a68bbcd684785a7bacdbabb506718 already in progress: creating tablet
I20250113 02:51:32.498525 890 tablet_service.cc:1514] Processing DeleteTablet for tablet fd0b8592997d4a61b8ca854ecbb67c55 with delete_type TABLET_DATA_DELETED (replaced by tablet e312e70334e04ecd843159ac72f4a2a6) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:32.501487 1196 ts_tablet_manager.cc:1905] T 63efcb6b2e194c469fb30a197bd6565b P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:32.504472 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_f86a1e2be2ba44b8bda94f4ad150c655"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:32.504896 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_c217d95911fd4ac18cb2764c65408b0b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:32.505054 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f86a1e2be2ba44b8bda94f4ad150c655 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250113 02:51:32.505371 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c217d95911fd4ac18cb2764c65408b0b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:32.504437 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_3935379456cf4f08b2e36f4ba95af912"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:32.506585 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_5f73c3e74d984bb69330b294fbc610de" } new_table_name: "default.Table_a373073bdcf740e389ce848e2e1ce7d4" modify_external_catalogs: true
W20250113 02:51:32.506585 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3935379456cf4f08b2e36f4ba95af912 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:32.506321 2053 tablet.cc:1721] T 32cba8f3efc54e78b4df41c15df82eea P 1186fa869eb14812832fa6562531f196: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
W20250113 02:51:32.508596 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet e13267498917432ba92b6badb76f5d73 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e13267498917432ba92b6badb76f5d73 already in progress: creating tablet
I20250113 02:51:32.509093 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_17633452a13c4859971518af3e51e69e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:32.509271 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 801) took 1110 ms. Trace:
W20250113 02:51:32.509631 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_17633452a13c4859971518af3e51e69e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:32.509758 1195 rpcz_store.cc:276] 0113 02:51:31.399270 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:31.399329 (+ 59us) service_pool.cc:225] Handling call
0113 02:51:32.509131 (+1109802us) ts_tablet_manager.cc:1145] Deleting tablet df6299f38ffa4c2f9d6c8a5e5285905f
0113 02:51:32.509143 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:32.509251 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1109577}
I20250113 02:51:32.512003 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_bbde1d1575c74204b7304a1cbe946c76"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:32.512593 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bbde1d1575c74204b7304a1cbe946c76 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250113 02:51:32.513567 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet df6299f38ffa4c2f9d6c8a5e5285905f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet df6299f38ffa4c2f9d6c8a5e5285905f already in progress: creating tablet
I20250113 02:51:32.514043 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 802) took 1109 ms. Trace:
I20250113 02:51:32.514199 1195 rpcz_store.cc:276] 0113 02:51:31.404200 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:31.404273 (+ 73us) service_pool.cc:225] Handling call
0113 02:51:32.513922 (+1109649us) ts_tablet_manager.cc:1145] Deleting tablet ef6dfe0431ef4a73a957559f60f24756
0113 02:51:32.513933 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:32.514027 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1109477}
I20250113 02:51:32.514894 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 803) took 1110 ms. Trace:
I20250113 02:51:32.515019 1195 rpcz_store.cc:276] 0113 02:51:31.404351 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:31.404401 (+ 50us) service_pool.cc:225] Handling call
0113 02:51:32.514773 (+1110372us) ts_tablet_manager.cc:1145] Deleting tablet f8a6eebb6b6d4b63903e68750e181707
0113 02:51:32.514783 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:32.514877 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1110212}
W20250113 02:51:32.515275 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet ef6dfe0431ef4a73a957559f60f24756 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ef6dfe0431ef4a73a957559f60f24756 already in progress: creating tablet
W20250113 02:51:32.516275 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet f8a6eebb6b6d4b63903e68750e181707 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f8a6eebb6b6d4b63903e68750e181707 already in progress: creating tablet
I20250113 02:51:32.516702 1195 tablet_replica.cc:331] T 05bf5ab0df874dd5b6559f6d82069184 P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:32.516928 1195 raft_consensus.cc:2238] T 05bf5ab0df874dd5b6559f6d82069184 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:32.517150 1195 raft_consensus.cc:2267] T 05bf5ab0df874dd5b6559f6d82069184 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:32.517237 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_518b2b6cde4441c88c49806de09f0aea" } new_table_name: "default.Table_d382199622cd467494ab3cab7c491b25" modify_external_catalogs: true
I20250113 02:51:32.519179 1195 ts_tablet_manager.cc:1905] T 05bf5ab0df874dd5b6559f6d82069184 P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250113 02:51:32.520905 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7b16286181684916ac84823f5fec563e consensus state has no leader
I20250113 02:51:32.528296 1194 tablet_bootstrap.cc:492] T 6d63f0e5835540a78c82020b20a07dac P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:32.529614 1194 tablet_bootstrap.cc:654] T 6d63f0e5835540a78c82020b20a07dac P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
W20250113 02:51:32.532059 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet cfa963a6df094ce0b9c54408f75ca6b2 consensus state has no leader
I20250113 02:51:32.537897 454 catalog_manager.cc:5618] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467) does not have the latest schema for tablet 401c4f5dbd304950877b59a865f417b3 (table default.Table_c991f9dbd70d4c4682664b8711f05db4 [id=2c64da64ecdb4bf48e11b523533902e8]). Expected version 1 got 0
I20250113 02:51:32.540946 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_0e516bc3e1f14f3ca9867668dd60d73a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:32.541663 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0e516bc3e1f14f3ca9867668dd60d73a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:32.531824 1192 tablet_bootstrap.cc:492] T 8337f3c82d654ecdac2cbc1ffa88ef05 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:32.552695 1192 ts_tablet_manager.cc:1397] T 8337f3c82d654ecdac2cbc1ffa88ef05 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.512s user 0.020s sys 0.048s
I20250113 02:51:32.552716 2051 tablet.cc:1721] T 32cba8f3efc54e78b4df41c15df82eea P 4216a7f853d642b7bc3e5cff458f3aca: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20250113 02:51:32.554188 1192 raft_consensus.cc:357] T 8337f3c82d654ecdac2cbc1ffa88ef05 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:32.554548 1192 raft_consensus.cc:383] T 8337f3c82d654ecdac2cbc1ffa88ef05 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:32.554664 1192 raft_consensus.cc:738] T 8337f3c82d654ecdac2cbc1ffa88ef05 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:32.554981 1192 consensus_queue.cc:260] T 8337f3c82d654ecdac2cbc1ffa88ef05 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:32.555199 1196 ts_tablet_manager.cc:1918] T 63efcb6b2e194c469fb30a197bd6565b P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:32.555801 1192 ts_tablet_manager.cc:1428] T 8337f3c82d654ecdac2cbc1ffa88ef05 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250113 02:51:32.552387 1198 ts_tablet_manager.cc:1918] T 41fcde988f72496b824271fa36c1a226 P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:32.555996 1198 log.cc:1198] T 41fcde988f72496b824271fa36c1a226 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/41fcde988f72496b824271fa36c1a226
I20250113 02:51:32.556214 1196 log.cc:1198] T 63efcb6b2e194c469fb30a197bd6565b P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/63efcb6b2e194c469fb30a197bd6565b
I20250113 02:51:32.556313 1198 ts_tablet_manager.cc:1939] T 41fcde988f72496b824271fa36c1a226 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:32.556664 1196 ts_tablet_manager.cc:1939] T 63efcb6b2e194c469fb30a197bd6565b P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
W20250113 02:51:32.557994 998 consensus_peers.cc:487] T 1cfbcd682b944c89847b40fe69f0a75e P 4216a7f853d642b7bc3e5cff458f3aca -> Peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Couldn't send request to peer df69f48656b44bb890f76587ee69cc0f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250113 02:51:32.559593 2053 tablet.cc:1721] T 9fff52e99c64462a8a73fccc3c22ca63 P 1186fa869eb14812832fa6562531f196: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250113 02:51:32.565032 1194 tablet_bootstrap.cc:492] T 6d63f0e5835540a78c82020b20a07dac P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:32.565183 1194 ts_tablet_manager.cc:1397] T 6d63f0e5835540a78c82020b20a07dac P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.037s user 0.002s sys 0.006s
I20250113 02:51:32.565703 1192 tablet_bootstrap.cc:492] T 0a7e7713bc8648378b668d8b769ac927 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:32.573974 1192 tablet_bootstrap.cc:654] T 0a7e7713bc8648378b668d8b769ac927 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:32.576249 448 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_4eff233920ec4a0cb1c20513a510ce8b" } modify_external_catalogs: true
I20250113 02:51:32.576962 1194 raft_consensus.cc:357] T 6d63f0e5835540a78c82020b20a07dac P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:32.577154 1194 raft_consensus.cc:383] T 6d63f0e5835540a78c82020b20a07dac P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:32.577205 1194 raft_consensus.cc:738] T 6d63f0e5835540a78c82020b20a07dac P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:32.577386 1194 consensus_queue.cc:260] T 6d63f0e5835540a78c82020b20a07dac P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:32.579111 455 catalog_manager.cc:5618] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467) does not have the latest schema for tablet 401c4f5dbd304950877b59a865f417b3 (table default.Table_c991f9dbd70d4c4682664b8711f05db4 [id=2c64da64ecdb4bf48e11b523533902e8]). Expected version 1 got 0
I20250113 02:51:32.579107 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 774) took 1278 ms. Trace:
I20250113 02:51:32.581056 1198 rpcz_store.cc:276] 0113 02:51:31.301028 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:31.301074 (+ 46us) service_pool.cc:225] Handling call
0113 02:51:32.019665 (+718591us) ts_tablet_manager.cc:1145] Deleting tablet 41fcde988f72496b824271fa36c1a226
0113 02:51:32.019678 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:32.479445 (+459767us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:32.545289 (+ 65844us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:32.579074 (+ 33785us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":665255}
I20250113 02:51:32.581543 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet 41fcde988f72496b824271fa36c1a226 (table default.Table_2caed919541e43c78a858513a2128f50 [id=26ba87b9bd384b99a016b29fe5b5babd]) successfully deleted
I20250113 02:51:32.582052 1194 ts_tablet_manager.cc:1428] T 6d63f0e5835540a78c82020b20a07dac P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.017s user 0.001s sys 0.001s
I20250113 02:51:32.585338 1192 tablet_bootstrap.cc:492] T 0a7e7713bc8648378b668d8b769ac927 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:32.585474 1192 ts_tablet_manager.cc:1397] T 0a7e7713bc8648378b668d8b769ac927 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.020s user 0.001s sys 0.001s
I20250113 02:51:32.588037 890 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d63f0e5835540a78c82020b20a07dac with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:23 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:32.588196 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 775) took 1286 ms. Trace:
I20250113 02:51:32.588217 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_d41198149c094b9cb0f1fc8c7cc6a799" } new_table_name: "default.Table_a30fe05769f5429cb4d5f680984dae16" modify_external_catalogs: true
I20250113 02:51:32.588292 1198 rpcz_store.cc:276] 0113 02:51:31.301257 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:31.301302 (+ 45us) service_pool.cc:225] Handling call
0113 02:51:32.581771 (+1280469us) ts_tablet_manager.cc:1145] Deleting tablet e8ab4516789e4585a101c8766d338af3
0113 02:51:32.588104 (+ 6333us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:32.588174 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1255147}
I20250113 02:51:32.588577 1198 tablet_replica.cc:331] T efccefd6787943469131492741ee2bd5 P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
W20250113 02:51:32.588754 440 catalog_manager.cc:4839] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet e8ab4516789e4585a101c8766d338af3 because the tablet was not found. No further retry: Not found: Tablet not found: e8ab4516789e4585a101c8766d338af3
I20250113 02:51:32.588783 1198 raft_consensus.cc:2238] T efccefd6787943469131492741ee2bd5 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:32.588932 1198 raft_consensus.cc:2267] T efccefd6787943469131492741ee2bd5 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:32.589584 1192 raft_consensus.cc:357] T 0a7e7713bc8648378b668d8b769ac927 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:32.589753 1192 raft_consensus.cc:383] T 0a7e7713bc8648378b668d8b769ac927 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:32.589818 1192 raft_consensus.cc:738] T 0a7e7713bc8648378b668d8b769ac927 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:32.590026 2051 tablet.cc:1721] T 9fff52e99c64462a8a73fccc3c22ca63 P 4216a7f853d642b7bc3e5cff458f3aca: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250113 02:51:32.590349 878 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:37356 (request call id 224) took 1012 ms. Trace:
I20250113 02:51:32.590477 878 rpcz_store.cc:276] 0113 02:51:31.577987 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:31.578059 (+ 72us) service_pool.cc:225] Handling call
0113 02:51:32.590314 (+1012255us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 02:51:32.590004 1192 consensus_queue.cc:260] T 0a7e7713bc8648378b668d8b769ac927 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:32.590325 1198 ts_tablet_manager.cc:1905] T efccefd6787943469131492741ee2bd5 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250113 02:51:32.595753 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0741955d74f64f0393444fa64cfe1c90 consensus state has no leader
I20250113 02:51:32.597069 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_8b441c8a6ec646afbe7c211d1b5d7eda" } new_table_name: "default.Table_f8a9bb5154f746afa795d4996851b7b5" modify_external_catalogs: true
I20250113 02:51:32.597262 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_bceb3f766ee942aba58803219ef3b245" } new_table_name: "default.Table_ed12a6de5e614eb39a3cc670df0f5671" modify_external_catalogs: true
I20250113 02:51:32.601114 1192 ts_tablet_manager.cc:1428] T 0a7e7713bc8648378b668d8b769ac927 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.013s user 0.001s sys 0.001s
I20250113 02:51:32.601725 453 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_fc7ae5b847754786984b5b25591def13" } modify_external_catalogs: true
I20250113 02:51:32.601922 453 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_fc7ae5b847754786984b5b25591def13" } modify_external_catalogs: true
W20250113 02:51:32.604893 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8f2b4df7c6a74896ade95a490ea9b54c consensus state has no leader
I20250113 02:51:32.606048 1194 tablet_bootstrap.cc:492] T 32cba8f3efc54e78b4df41c15df82eea P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:32.607630 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_777e191d02f54883afa306010890a105"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:32.608278 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_777e191d02f54883afa306010890a105 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250113 02:51:32.610970 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2c7d213481f24748912f5b2b23a33988 consensus state has no leader
I20250113 02:51:32.612323 1192 tablet_bootstrap.cc:492] T 77cf8b0c824f401aba1880eb7fd1807b P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:32.615887 1192 tablet_bootstrap.cc:654] T 77cf8b0c824f401aba1880eb7fd1807b P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:32.617712 1198 ts_tablet_manager.cc:1918] T efccefd6787943469131492741ee2bd5 P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:32.617800 1198 log.cc:1198] T efccefd6787943469131492741ee2bd5 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/efccefd6787943469131492741ee2bd5
I20250113 02:51:32.618064 1198 ts_tablet_manager.cc:1939] T efccefd6787943469131492741ee2bd5 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:32.618068 2045 raft_consensus.cc:491] T 0fe5421dc3ea4be684cab084f13d8df0 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:32.618207 2045 raft_consensus.cc:513] T 0fe5421dc3ea4be684cab084f13d8df0 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:32.619336 2045 leader_election.cc:290] T 0fe5421dc3ea4be684cab084f13d8df0 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:32.619833 1062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fe5421dc3ea4be684cab084f13d8df0" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
W20250113 02:51:32.620357 709 leader_election.cc:343] T 0fe5421dc3ea4be684cab084f13d8df0 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:32.620596 446 master_service.cc:946] ReplaceTablet: received request to replace tablet 7213e3284b4d4cb581d9fb946a9f87e9 from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:32.626019 447 catalog_manager.cc:5618] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095) does not have the latest schema for tablet 6e1284c8aaba44b5810b0d06d36757eb (table default.Table_bfc5031f68f64006aceefe2ee986223a [id=aa4679fc0d7148e3bef21342b3a0ea22]). Expected version 2 got 0
I20250113 02:51:32.632519 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 776) took 1331 ms. Trace:
I20250113 02:51:32.635761 1026 tablet_service.cc:1467] Processing CreateTablet for tablet f29d7a9504d740528914fe6a66fd4028 (DEFAULT_TABLE table=default.Table_292625e9d891410287fe0647a7c6b109 [id=49b50c3b70394b2a8dfba6b65c80135f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:32.636237 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f29d7a9504d740528914fe6a66fd4028. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:32.637027 897 tablet_service.cc:1467] Processing CreateTablet for tablet f29d7a9504d740528914fe6a66fd4028 (DEFAULT_TABLE table=default.Table_292625e9d891410287fe0647a7c6b109 [id=49b50c3b70394b2a8dfba6b65c80135f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:32.643244 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_f4ebf22f164942e581f60ba75f899497" } modify_external_catalogs: true
I20250113 02:51:32.646498 775 raft_consensus.cc:1270] T 4c615b0b83d74fcfae0c51047dbfbbf4 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Refusing update from remote peer 4216a7f853d642b7bc3e5cff458f3aca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250113 02:51:32.647092 2009 consensus_queue.cc:1035] T 4c615b0b83d74fcfae0c51047dbfbbf4 P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 02:51:32.648093 2045 raft_consensus.cc:491] T e468b652f71f44e3945f8d919ac0e1a8 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:32.648236 1195 ts_tablet_manager.cc:1918] T 05bf5ab0df874dd5b6559f6d82069184 P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:32.648313 1195 log.cc:1198] T 05bf5ab0df874dd5b6559f6d82069184 P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/05bf5ab0df874dd5b6559f6d82069184
I20250113 02:51:32.648221 2045 raft_consensus.cc:513] T e468b652f71f44e3945f8d919ac0e1a8 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:32.648624 1195 ts_tablet_manager.cc:1939] T 05bf5ab0df874dd5b6559f6d82069184 P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:32.648896 2045 leader_election.cc:290] T e468b652f71f44e3945f8d919ac0e1a8 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:32.649401 1062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e468b652f71f44e3945f8d919ac0e1a8" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
W20250113 02:51:32.650462 709 leader_election.cc:343] T e468b652f71f44e3945f8d919ac0e1a8 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:32.652616 1192 tablet_bootstrap.cc:492] T 77cf8b0c824f401aba1880eb7fd1807b P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:32.652761 1192 ts_tablet_manager.cc:1397] T 77cf8b0c824f401aba1880eb7fd1807b P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.041s user 0.003s sys 0.000s
I20250113 02:51:32.652940 452 catalog_manager.cc:5618] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587) does not have the latest schema for tablet 1cfbcd682b944c89847b40fe69f0a75e (table default.Table_32e88abaae96492fbcce39fb2cd7c221 [id=2b84deb07e5c4ab8b550fe42945d7b71]). Expected version 2 got 0
I20250113 02:51:32.653577 1192 raft_consensus.cc:357] T 77cf8b0c824f401aba1880eb7fd1807b P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:32.653793 1192 raft_consensus.cc:383] T 77cf8b0c824f401aba1880eb7fd1807b P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:32.653842 1192 raft_consensus.cc:738] T 77cf8b0c824f401aba1880eb7fd1807b P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:32.632640 1198 rpcz_store.cc:276] 0113 02:51:31.301532 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:31.301577 (+ 45us) service_pool.cc:225] Handling call
0113 02:51:32.588537 (+1286960us) ts_tablet_manager.cc:1145] Deleting tablet efccefd6787943469131492741ee2bd5
0113 02:51:32.588545 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:32.598123 (+ 9578us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:32.617620 (+ 19497us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:32.632496 (+ 14876us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1261854}
I20250113 02:51:32.654486 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet efccefd6787943469131492741ee2bd5 (table default.Table_faf2fafcef874bf6a5798ee6b19ef704 [id=bcdd761e38934a2e9197b9c1b9165b27]) successfully deleted
I20250113 02:51:32.654824 1198 tablet_replica.cc:331] T bffec36d567544c5a1c87e64a7e9730b P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:32.655043 1198 raft_consensus.cc:2238] T bffec36d567544c5a1c87e64a7e9730b P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:32.655210 1198 raft_consensus.cc:2267] T bffec36d567544c5a1c87e64a7e9730b P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:32.655757 1198 ts_tablet_manager.cc:1905] T bffec36d567544c5a1c87e64a7e9730b P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:32.651168 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 805) took 1227 ms. Trace:
I20250113 02:51:32.656064 1195 rpcz_store.cc:276] 0113 02:51:31.423550 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:31.423611 (+ 61us) service_pool.cc:225] Handling call
0113 02:51:32.516654 (+1093043us) ts_tablet_manager.cc:1145] Deleting tablet 05bf5ab0df874dd5b6559f6d82069184
0113 02:51:32.516663 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:32.626967 (+110304us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:32.648146 (+ 21179us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:32.651148 (+ 3002us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1092859}
I20250113 02:51:32.656468 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet 05bf5ab0df874dd5b6559f6d82069184 (table default.Table_5175e117e3f34778a5ef456bed6b5e28 [id=5c9d31a6ab2445ad949f45abffdf4cc7]) successfully deleted
I20250113 02:51:32.656863 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 821) took 1020 ms. Trace:
I20250113 02:51:32.656853 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_f4ebf22f164942e581f60ba75f899497" } modify_external_catalogs: true
I20250113 02:51:32.656947 1195 rpcz_store.cc:276] 0113 02:51:31.635938 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:31.636485 (+ 547us) service_pool.cc:225] Handling call
0113 02:51:32.656734 (+1020249us) ts_tablet_manager.cc:1145] Deleting tablet 88a776aa820c44b3a3854ca5022bd353
0113 02:51:32.656748 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:32.656847 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1019744}
W20250113 02:51:32.657421 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 88a776aa820c44b3a3854ca5022bd353 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 88a776aa820c44b3a3854ca5022bd353 already in progress: creating tablet
I20250113 02:51:32.659924 755 tablet_service.cc:1514] Processing DeleteTablet for tablet 51d9483ecacb413796d0fd9e7560eac1 with delete_type TABLET_DATA_DELETED (replaced by tablet 8898e1e9efce453488fa4a4562a82127) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:32.660583 736 tablet_service.cc:1514] Processing DeleteTablet for tablet f129bb3fbbfd492d98822be0a6626e44 with delete_type TABLET_DATA_DELETED (replaced by tablet 67baaab1e36e415381579d7385d4ab5a) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:32.660643 741 tablet_service.cc:1514] Processing DeleteTablet for tablet c7243d27a1c642269a3d0e67632e458c with delete_type TABLET_DATA_DELETED (replaced by tablet ed68a2b7ebcf4841bef54798dd3ed80d) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:32.660825 752 tablet_service.cc:1514] Processing DeleteTablet for tablet e468b652f71f44e3945f8d919ac0e1a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:29 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:32.660959 740 tablet_service.cc:1514] Processing DeleteTablet for tablet 99c8046f5d124ca49899df365114c69f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:23 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:32.661088 739 tablet_service.cc:1514] Processing DeleteTablet for tablet 0539cbbe47e64c039ad39c4b52edcc4b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:27 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:32.661244 742 tablet_service.cc:1514] Processing DeleteTablet for tablet 0fe5421dc3ea4be684cab084f13d8df0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:24 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:32.661590 738 tablet_service.cc:1467] Processing CreateTablet for tablet f29d7a9504d740528914fe6a66fd4028 (DEFAULT_TABLE table=default.Table_292625e9d891410287fe0647a7c6b109 [id=49b50c3b70394b2a8dfba6b65c80135f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:32.661784 755 tablet_service.cc:1514] Processing DeleteTablet for tablet 056bf1e67be04d179eace3d93282b145 with delete_type TABLET_DATA_DELETED (replaced by tablet dfea918c04c1408a8b86a44bb0911a7c) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:32.661993 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f29d7a9504d740528914fe6a66fd4028. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:32.662405 1026 tablet_service.cc:1467] Processing CreateTablet for tablet e4998b86d32040ec959ff6fd94b1e9eb (DEFAULT_TABLE table=default.Table_b17de55bafab4c41b91c30d96a90b75c [id=7cde3519999c4890b2e8f1fa0d64549b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:32.662737 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e4998b86d32040ec959ff6fd94b1e9eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:32.668361 2009 raft_consensus.cc:491] T 8337f3c82d654ecdac2cbc1ffa88ef05 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:32.668470 2009 raft_consensus.cc:513] T 8337f3c82d654ecdac2cbc1ffa88ef05 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:32.668584 751 tablet_service.cc:1514] Processing DeleteTablet for tablet 6666922d677a4c678f80d7c6c47fd2a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:28 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:32.669240 2009 leader_election.cc:290] T 8337f3c82d654ecdac2cbc1ffa88ef05 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
I20250113 02:51:32.670684 755 tablet_service.cc:1467] Processing CreateTablet for tablet e4998b86d32040ec959ff6fd94b1e9eb (DEFAULT_TABLE table=default.Table_b17de55bafab4c41b91c30d96a90b75c [id=7cde3519999c4890b2e8f1fa0d64549b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:32.671113 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e4998b86d32040ec959ff6fd94b1e9eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:32.671940 1031 tablet_service.cc:1467] Processing CreateTablet for tablet fc28641fbd834d768aea272e0f988688 (DEFAULT_TABLE table=default.Table_9e652a1c2e5f47a8bd55590289e5836f [id=44cb457fbe004c7d8b26741487970529]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:32.672317 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc28641fbd834d768aea272e0f988688. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:32.671417 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 822) took 1029 ms. Trace:
I20250113 02:51:32.672540 2009 raft_consensus.cc:491] T 7d3562a48c6e4badbbcf37640e91b5ba P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:32.672559 1195 rpcz_store.cc:276] 0113 02:51:31.641677 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:31.641763 (+ 86us) service_pool.cc:225] Handling call
0113 02:51:32.671298 (+1029535us) ts_tablet_manager.cc:1145] Deleting tablet 1d15c7a01f854f0aaabcd30ec3166e31
0113 02:51:32.671307 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:32.671400 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1029266}
I20250113 02:51:32.672653 2009 raft_consensus.cc:513] T 7d3562a48c6e4badbbcf37640e91b5ba P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:32.673207 1195 tablet_replica.cc:331] T b8b6ffe690fc459db78b2b3bc12671dd P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:32.673242 2009 leader_election.cc:290] T 7d3562a48c6e4badbbcf37640e91b5ba P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:32.654064 1192 consensus_queue.cc:260] T 77cf8b0c824f401aba1880eb7fd1807b P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
W20250113 02:51:32.673538 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 1d15c7a01f854f0aaabcd30ec3166e31 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1d15c7a01f854f0aaabcd30ec3166e31 already in progress: creating tablet
W20250113 02:51:32.673820 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet f4379f7fc6b54e6785f27975d9030cce because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f4379f7fc6b54e6785f27975d9030cce already in progress: creating tablet
W20250113 02:51:32.674026 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 883027967e0c442a91ac820210b9c6c7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 883027967e0c442a91ac820210b9c6c7 already in progress: creating tablet
W20250113 02:51:32.674268 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 16a132384b0e4264a73d09a3ea133893 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 16a132384b0e4264a73d09a3ea133893 already in progress: creating tablet
W20250113 02:51:32.674590 995 leader_election.cc:343] T 7d3562a48c6e4badbbcf37640e91b5ba P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Not found: Tablet not found: 7d3562a48c6e4badbbcf37640e91b5ba
I20250113 02:51:32.658456 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_b05aa043ae0e44b5a2a881764be3dbf0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:32.675354 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b05aa043ae0e44b5a2a881764be3dbf0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:32.675612 1035 tablet_service.cc:1467] Processing CreateTablet for tablet b45fc730c445456a8a5cacac2cf47efe (DEFAULT_TABLE table=default.Table_dd3111126f5a4f97af332dd879439153 [id=4a030a6a2ced4643847eba258a002751]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:32.675925 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b45fc730c445456a8a5cacac2cf47efe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:32.678674 1192 ts_tablet_manager.cc:1428] T 77cf8b0c824f401aba1880eb7fd1807b P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.026s user 0.002s sys 0.000s
I20250113 02:51:32.678953 1192 tablet_bootstrap.cc:492] T 42fad91fe1b64a21a2c8412a2585af47 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:32.680256 1192 tablet_bootstrap.cc:654] T 42fad91fe1b64a21a2c8412a2585af47 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:32.680281 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8337f3c82d654ecdac2cbc1ffa88ef05" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
I20250113 02:51:32.680490 773 raft_consensus.cc:2463] T 8337f3c82d654ecdac2cbc1ffa88ef05 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 0.
I20250113 02:51:32.673388 1195 raft_consensus.cc:2238] T b8b6ffe690fc459db78b2b3bc12671dd P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:32.681653 1195 raft_consensus.cc:2267] T b8b6ffe690fc459db78b2b3bc12671dd P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:32.682196 1195 ts_tablet_manager.cc:1905] T b8b6ffe690fc459db78b2b3bc12671dd P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:32.682350 1029 tablet_service.cc:1467] Processing CreateTablet for tablet 6fd59b6773fd4bac863b1714f13c9184 (DEFAULT_TABLE table=default.Table_aafae7a2874249e7ac84dcde865d33cf [id=e8fc03b511e94e2c9883eb31073f17e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:32.682667 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6fd59b6773fd4bac863b1714f13c9184. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:32.686836 995 leader_election.cc:304] T 8337f3c82d654ecdac2cbc1ffa88ef05 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters:
I20250113 02:51:32.686937 751 tablet_service.cc:1467] Processing CreateTablet for tablet b45fc730c445456a8a5cacac2cf47efe (DEFAULT_TABLE table=default.Table_dd3111126f5a4f97af332dd879439153 [id=4a030a6a2ced4643847eba258a002751]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:32.687086 2009 raft_consensus.cc:2798] T 8337f3c82d654ecdac2cbc1ffa88ef05 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 02:51:32.687194 2009 raft_consensus.cc:491] T 8337f3c82d654ecdac2cbc1ffa88ef05 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 02:51:32.687273 2009 raft_consensus.cc:3054] T 8337f3c82d654ecdac2cbc1ffa88ef05 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:32.687508 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b45fc730c445456a8a5cacac2cf47efe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:32.688275 738 tablet_service.cc:1467] Processing CreateTablet for tablet fc28641fbd834d768aea272e0f988688 (DEFAULT_TABLE table=default.Table_9e652a1c2e5f47a8bd55590289e5836f [id=44cb457fbe004c7d8b26741487970529]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:32.688606 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc28641fbd834d768aea272e0f988688. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:32.689477 2009 raft_consensus.cc:513] T 8337f3c82d654ecdac2cbc1ffa88ef05 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:32.690259 2009 leader_election.cc:290] T 8337f3c82d654ecdac2cbc1ffa88ef05 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Requested vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
I20250113 02:51:32.690757 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8337f3c82d654ecdac2cbc1ffa88ef05" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196"
I20250113 02:51:32.690955 773 raft_consensus.cc:3054] T 8337f3c82d654ecdac2cbc1ffa88ef05 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:32.693560 773 raft_consensus.cc:2463] T 8337f3c82d654ecdac2cbc1ffa88ef05 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 1.
I20250113 02:51:32.693950 995 leader_election.cc:304] T 8337f3c82d654ecdac2cbc1ffa88ef05 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters:
I20250113 02:51:32.694657 742 tablet_service.cc:1467] Processing CreateTablet for tablet 6fd59b6773fd4bac863b1714f13c9184 (DEFAULT_TABLE table=default.Table_aafae7a2874249e7ac84dcde865d33cf [id=e8fc03b511e94e2c9883eb31073f17e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:32.694965 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6fd59b6773fd4bac863b1714f13c9184. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:32.702291 2009 raft_consensus.cc:2798] T 8337f3c82d654ecdac2cbc1ffa88ef05 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Leader election won for term 1
I20250113 02:51:32.702425 2009 raft_consensus.cc:695] T 8337f3c82d654ecdac2cbc1ffa88ef05 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: Becoming Leader. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Running, Role: LEADER
I20250113 02:51:32.702666 2009 consensus_queue.cc:237] T 8337f3c82d654ecdac2cbc1ffa88ef05 P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:32.709720 1191 tablet_bootstrap.cc:492] T f56230a2c75d4763a54671c79358f4d9 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:32.709877 1191 ts_tablet_manager.cc:1397] T f56230a2c75d4763a54671c79358f4d9 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.309s user 0.034s sys 0.027s
I20250113 02:51:32.710777 1191 raft_consensus.cc:357] T f56230a2c75d4763a54671c79358f4d9 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:32.710985 1191 raft_consensus.cc:383] T f56230a2c75d4763a54671c79358f4d9 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:32.711037 1191 raft_consensus.cc:738] T f56230a2c75d4763a54671c79358f4d9 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:32.711262 1191 consensus_queue.cc:260] T f56230a2c75d4763a54671c79358f4d9 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:32.711745 1191 ts_tablet_manager.cc:1428] T f56230a2c75d4763a54671c79358f4d9 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250113 02:51:32.712038 1191 tablet_bootstrap.cc:492] T 377ebf66bbef40988043f6200ae0893c P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:32.712553 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_21eff11db1644c02bfa6ceb7d46e7eac"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:32.713166 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_21eff11db1644c02bfa6ceb7d46e7eac in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:32.713721 1191 tablet_bootstrap.cc:654] T 377ebf66bbef40988043f6200ae0893c P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:32.713754 743 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b1910496e0849b193e390a5ef1a6b2b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:26 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:32.719866 1031 tablet_service.cc:1467] Processing CreateTablet for tablet be5d6907a02f41be8eab71de051a6d3b (DEFAULT_TABLE table=default.Table_a7ed8a8a7d00492b8fc543baeb433a27 [id=3e6a7094710a444aac74e1919ba52ac8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:32.720283 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be5d6907a02f41be8eab71de051a6d3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:32.726958 1198 ts_tablet_manager.cc:1918] T bffec36d567544c5a1c87e64a7e9730b P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:32.729957 739 tablet_service.cc:1467] Processing CreateTablet for tablet fb715f9fbf0e4e3d935b0510e9872257 (DEFAULT_TABLE table=default.Table_d7372f791f52447593557efc3f8226ff [id=b241c454a1cf47419255537799b14eb6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:32.730360 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb715f9fbf0e4e3d935b0510e9872257. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:32.732721 740 tablet_service.cc:1467] Processing CreateTablet for tablet 9b9e7deda7334ff4a3429f70d9b3a7fc (DEFAULT_TABLE table=default.Table_64856bab03754bdc9ded6e123a7d1c7e [id=215d6a7ecc6643bea61986756ad67eca]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:32.727138 1035 tablet_service.cc:1467] Processing CreateTablet for tablet f29e16fea2534c7dbe57639c7c6b37d6 (DEFAULT_TABLE table=default.Table_d7b990d07d59419da5ad4a9bc9dcc215 [id=5f903ae1c50e4e7487a3b70555f4bb1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:32.733117 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b9e7deda7334ff4a3429f70d9b3a7fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:32.728363 1026 tablet_service.cc:1467] Processing CreateTablet for tablet 119baabad10a488082f86f2a52ac3779 (DEFAULT_TABLE table=default.Table_50cc6a4316724447bdeefc5aefe989a6 [id=608f9e63e64845fbbb0efb119c9fa091]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:32.729343 1029 tablet_service.cc:1467] Processing CreateTablet for tablet fb715f9fbf0e4e3d935b0510e9872257 (DEFAULT_TABLE table=default.Table_d7372f791f52447593557efc3f8226ff [id=b241c454a1cf47419255537799b14eb6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:32.730255 1036 tablet_service.cc:1467] Processing CreateTablet for tablet 9b9e7deda7334ff4a3429f70d9b3a7fc (DEFAULT_TABLE table=default.Table_64856bab03754bdc9ded6e123a7d1c7e [id=215d6a7ecc6643bea61986756ad67eca]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:32.730568 1192 tablet_bootstrap.cc:492] T 42fad91fe1b64a21a2c8412a2585af47 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:32.733645 1192 ts_tablet_manager.cc:1397] T 42fad91fe1b64a21a2c8412a2585af47 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.055s user 0.003s sys 0.000s
I20250113 02:51:32.733552 448 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_4eff233920ec4a0cb1c20513a510ce8b" } modify_external_catalogs: true
I20250113 02:51:32.732167 1032 tablet_service.cc:1467] Processing CreateTablet for tablet 4cc15487be474354b995c4e415276a18 (DEFAULT_TABLE table=default.Table_6c9405a4b90a4010b3033c83c9c8a8c4 [id=8146ea507a264878898f61c3d034f302]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:32.731448 1030 tablet_service.cc:1467] Processing CreateTablet for tablet 9e25a99e40ae41c39289178d897e0b99 (DEFAULT_TABLE table=default.Table_a3a87b4d6edb4210820b2ccc7a71dbd4 [id=0c14541780ef458e925fd6e6a4fdb663]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:32.734266 1024 tablet_service.cc:1467] Processing CreateTablet for tablet ae20dd1bc47d4495a0d88297ff934df6 (DEFAULT_TABLE table=default.Table_678cc4306399435ca453eb3ce382f5fc [id=d44ec6d8110f4507afa786b9d82d0463]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:32.734705 1192 raft_consensus.cc:357] T 42fad91fe1b64a21a2c8412a2585af47 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:32.736064 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 7213e3284b4d4cb581d9fb946a9f87e9
I20250113 02:51:32.736519 446 catalog_manager.cc:6596] ReplaceTablet: tablet 7213e3284b4d4cb581d9fb946a9f87e9 deleted and replaced by tablet 7c443454994a47f2bfa7e2ec7ac576e0
I20250113 02:51:32.736584 752 tablet_service.cc:1467] Processing CreateTablet for tablet 9e25a99e40ae41c39289178d897e0b99 (DEFAULT_TABLE table=default.Table_a3a87b4d6edb4210820b2ccc7a71dbd4 [id=0c14541780ef458e925fd6e6a4fdb663]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:32.734881 1192 raft_consensus.cc:383] T 42fad91fe1b64a21a2c8412a2585af47 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:32.736899 1192 raft_consensus.cc:738] T 42fad91fe1b64a21a2c8412a2585af47 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:32.736938 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e25a99e40ae41c39289178d897e0b99. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:32.736954 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_1f3f594aac1e4cd2bcec522b01f7e746"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:32.737192 1192 consensus_queue.cc:260] T 42fad91fe1b64a21a2c8412a2585af47 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:32.737480 1198 log.cc:1198] T bffec36d567544c5a1c87e64a7e9730b P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/bffec36d567544c5a1c87e64a7e9730b
I20250113 02:51:32.737797 1198 ts_tablet_manager.cc:1939] T bffec36d567544c5a1c87e64a7e9730b P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:32.739635 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_3e6f08e022664e2db587347cf97cf0a9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:32.740182 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3e6f08e022664e2db587347cf97cf0a9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:32.740311 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 1f128d9bf2924fb1bada3a8462818004
I20250113 02:51:32.740535 1192 ts_tablet_manager.cc:1428] T 42fad91fe1b64a21a2c8412a2585af47 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20250113 02:51:32.741143 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_b4f66c4bb40c4135ac5516b28550f0d5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:32.741674 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b4f66c4bb40c4135ac5516b28550f0d5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:32.742699 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet 7213e3284b4d4cb581d9fb946a9f87e9 with delete_type TABLET_DATA_DELETED (replaced by tablet 7c443454994a47f2bfa7e2ec7ac576e0) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:32.742888 1041 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f128d9bf2924fb1bada3a8462818004 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:32 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:32.742931 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet e6e8966da70746d7a10c7243ba2f6f87
I20250113 02:51:32.619482 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fe5421dc3ea4be684cab084f13d8df0" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:32.743651 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_1e7d9a88187442b7bf4ae616cfd746d0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:32.744143 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1e7d9a88187442b7bf4ae616cfd746d0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:32.745048 1041 tablet_service.cc:1514] Processing DeleteTablet for tablet e6e8966da70746d7a10c7243ba2f6f87 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:32 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:32.745165 1191 tablet_bootstrap.cc:492] T 377ebf66bbef40988043f6200ae0893c P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:32.745286 1191 ts_tablet_manager.cc:1397] T 377ebf66bbef40988043f6200ae0893c P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.033s user 0.003s sys 0.000s
I20250113 02:51:32.745321 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e468b652f71f44e3945f8d919ac0e1a8" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:32.745811 711 leader_election.cc:343] T 0fe5421dc3ea4be684cab084f13d8df0 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:32.745934 711 leader_election.cc:304] T 0fe5421dc3ea4be684cab084f13d8df0 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:32.746302 2045 raft_consensus.cc:2743] T 0fe5421dc3ea4be684cab084f13d8df0 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250113 02:51:32.748172 711 leader_election.cc:343] T e468b652f71f44e3945f8d919ac0e1a8 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:32.748279 711 leader_election.cc:304] T e468b652f71f44e3945f8d919ac0e1a8 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
W20250113 02:51:32.748557 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1f3f594aac1e4cd2bcec522b01f7e746 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:32.748739 2045 raft_consensus.cc:2743] T e468b652f71f44e3945f8d919ac0e1a8 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:32.750692 754 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f128d9bf2924fb1bada3a8462818004 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:32 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:32.750924 748 tablet_service.cc:1514] Processing DeleteTablet for tablet e6e8966da70746d7a10c7243ba2f6f87 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:32 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:32.751073 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 8cc78bf7d6324b7ba10a1b1045923f2b
I20250113 02:51:32.750988 736 tablet_service.cc:1514] Processing DeleteTablet for tablet 7213e3284b4d4cb581d9fb946a9f87e9 with delete_type TABLET_DATA_DELETED (replaced by tablet 7c443454994a47f2bfa7e2ec7ac576e0) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:32.750931 741 tablet_service.cc:1467] Processing CreateTablet for tablet 4cc15487be474354b995c4e415276a18 (DEFAULT_TABLE table=default.Table_6c9405a4b90a4010b3033c83c9c8a8c4 [id=8146ea507a264878898f61c3d034f302]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:32.751721 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4cc15487be474354b995c4e415276a18. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:32.751993 1041 tablet_service.cc:1514] Processing DeleteTablet for tablet 8cc78bf7d6324b7ba10a1b1045923f2b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:32 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:32.753687 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_4385ce4af1044706afef0ab60f2710a7" } new_table_name: "default.Table_da3e49fa1e654145a05858466ec2260f" modify_external_catalogs: true
I20250113 02:51:32.756423 1191 raft_consensus.cc:357] T 377ebf66bbef40988043f6200ae0893c P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:32.756649 1191 raft_consensus.cc:383] T 377ebf66bbef40988043f6200ae0893c P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:32.756707 1191 raft_consensus.cc:738] T 377ebf66bbef40988043f6200ae0893c P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:32.756995 1191 consensus_queue.cc:260] T 377ebf66bbef40988043f6200ae0893c P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:32.757391 2051 tablet.cc:1716] T 32cba8f3efc54e78b4df41c15df82eea P 4216a7f853d642b7bc3e5cff458f3aca: Skipping requested alter to schema version 5, tablet already version 5
I20250113 02:51:32.757781 1191 ts_tablet_manager.cc:1428] T 377ebf66bbef40988043f6200ae0893c P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20250113 02:51:32.758148 1191 tablet_bootstrap.cc:492] T aa5c172a69bf4ca79999f4b58df08de8 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:32.759011 736 tablet_service.cc:1514] Processing DeleteTablet for tablet 8cc78bf7d6324b7ba10a1b1045923f2b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:32 UTC) from {username='slave'} at 127.0.0.1:55398
W20250113 02:51:32.760210 898 consensus_meta.cc:287] T 62b71866acc941e8b116ec4192a8e1e7 P df69f48656b44bb890f76587ee69cc0f: Time spent flushing consensus metadata: real 0.837s user 0.017s sys 0.000s
I20250113 02:51:32.760293 748 tablet_service.cc:1467] Processing CreateTablet for tablet ae20dd1bc47d4495a0d88297ff934df6 (DEFAULT_TABLE table=default.Table_678cc4306399435ca453eb3ce382f5fc [id=d44ec6d8110f4507afa786b9d82d0463]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:32.760603 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae20dd1bc47d4495a0d88297ff934df6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:32.761373 1041 tablet_service.cc:1467] Processing CreateTablet for tablet 99cf22e0253c4bd6a2d22a0e540e8b68 (DEFAULT_TABLE table=default.Table_41b2cae1cce14b0ab6b6de6d9c946ff5 [id=2b3bc17b229a4a7baaa34966e5d0fe67]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:32.762348 1023 tablet_service.cc:1467] Processing CreateTablet for tablet b1cd18ee2e2145ba9273f500d468343d (DEFAULT_TABLE table=default.Table_ef9c0cf86b0943aab6bbb6a3fe74ba85 [id=330eff1816254196a8930116c238aad7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:32.763442 452 catalog_manager.cc:6071] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: default.Table_27e1ad9a94f8455f806f48768ef4ddac [id=8aff13d6d3c5483eadb596b705fe5151] alter complete (version 5)
I20250113 02:51:32.766549 880 tablet_service.cc:1467] Processing CreateTablet for tablet e4998b86d32040ec959ff6fd94b1e9eb (DEFAULT_TABLE table=default.Table_b17de55bafab4c41b91c30d96a90b75c [id=7cde3519999c4890b2e8f1fa0d64549b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:32.766847 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d3562a48c6e4badbbcf37640e91b5ba" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:32.767314 896 tablet_service.cc:1467] Processing CreateTablet for tablet fc28641fbd834d768aea272e0f988688 (DEFAULT_TABLE table=default.Table_9e652a1c2e5f47a8bd55590289e5836f [id=44cb457fbe004c7d8b26741487970529]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:32.767661 890 tablet_service.cc:1467] Processing CreateTablet for tablet b45fc730c445456a8a5cacac2cf47efe (DEFAULT_TABLE table=default.Table_dd3111126f5a4f97af332dd879439153 [id=4a030a6a2ced4643847eba258a002751]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:32.768020 894 tablet_service.cc:1467] Processing CreateTablet for tablet 6fd59b6773fd4bac863b1714f13c9184 (DEFAULT_TABLE table=default.Table_aafae7a2874249e7ac84dcde865d33cf [id=e8fc03b511e94e2c9883eb31073f17e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:32.768209 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8337f3c82d654ecdac2cbc1ffa88ef05" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:32.768357 886 tablet_service.cc:1467] Processing CreateTablet for tablet be5d6907a02f41be8eab71de051a6d3b (DEFAULT_TABLE table=default.Table_a7ed8a8a7d00492b8fc543baeb433a27 [id=3e6a7094710a444aac74e1919ba52ac8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:32.768429 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8337f3c82d654ecdac2cbc1ffa88ef05" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f"
I20250113 02:51:32.768493 881 tablet_service.cc:1514] Processing DeleteTablet for tablet 8cc78bf7d6324b7ba10a1b1045923f2b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:32 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:32.768664 882 tablet_service.cc:1467] Processing CreateTablet for tablet f29e16fea2534c7dbe57639c7c6b37d6 (DEFAULT_TABLE table=default.Table_d7b990d07d59419da5ad4a9bc9dcc215 [id=5f903ae1c50e4e7487a3b70555f4bb1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:32.768949 892 tablet_service.cc:1467] Processing CreateTablet for tablet 99cf22e0253c4bd6a2d22a0e540e8b68 (DEFAULT_TABLE table=default.Table_41b2cae1cce14b0ab6b6de6d9c946ff5 [id=2b3bc17b229a4a7baaa34966e5d0fe67]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:32.769085 884 tablet_service.cc:1467] Processing CreateTablet for tablet 119baabad10a488082f86f2a52ac3779 (DEFAULT_TABLE table=default.Table_50cc6a4316724447bdeefc5aefe989a6 [id=608f9e63e64845fbbb0efb119c9fa091]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:32.769433 883 tablet_service.cc:1467] Processing CreateTablet for tablet fb715f9fbf0e4e3d935b0510e9872257 (DEFAULT_TABLE table=default.Table_d7372f791f52447593557efc3f8226ff [id=b241c454a1cf47419255537799b14eb6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:32.769477 895 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f128d9bf2924fb1bada3a8462818004 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:32 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:32.769750 889 tablet_service.cc:1467] Processing CreateTablet for tablet 9b9e7deda7334ff4a3429f70d9b3a7fc (DEFAULT_TABLE table=default.Table_64856bab03754bdc9ded6e123a7d1c7e [id=215d6a7ecc6643bea61986756ad67eca]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:32.769945 891 tablet_service.cc:1467] Processing CreateTablet for tablet 4cc15487be474354b995c4e415276a18 (DEFAULT_TABLE table=default.Table_6c9405a4b90a4010b3033c83c9c8a8c4 [id=8146ea507a264878898f61c3d034f302]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:32.770143 1038 tablet_service.cc:1467] Processing CreateTablet for tablet 9938f5866a6149e3a1456b03389064b2 (DEFAULT_TABLE table=default.Table_fc6121581e0a4a8c85474adbe1bdbcf8 [id=ec5a69fb4ec74a04b841ee375e8018d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:32.770269 899 tablet_service.cc:1467] Processing CreateTablet for tablet ae20dd1bc47d4495a0d88297ff934df6 (DEFAULT_TABLE table=default.Table_678cc4306399435ca453eb3ce382f5fc [id=d44ec6d8110f4507afa786b9d82d0463]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:32.769732 895 tablet_service.cc:1467] Processing CreateTablet for tablet 9938f5866a6149e3a1456b03389064b2 (DEFAULT_TABLE table=default.Table_fc6121581e0a4a8c85474adbe1bdbcf8 [id=ec5a69fb4ec74a04b841ee375e8018d5]), partition=RANGE (key) PARTITION UNBOUNDED
W20250113 02:51:32.770553 998 consensus_peers.cc:487] T 4c615b0b83d74fcfae0c51047dbfbbf4 P 4216a7f853d642b7bc3e5cff458f3aca -> Peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Couldn't send request to peer df69f48656b44bb890f76587ee69cc0f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250113 02:51:32.770712 754 tablet_service.cc:1467] Processing CreateTablet for tablet b1cd18ee2e2145ba9273f500d468343d (DEFAULT_TABLE table=default.Table_ef9c0cf86b0943aab6bbb6a3fe74ba85 [id=330eff1816254196a8930116c238aad7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:32.770800 744 tablet_service.cc:1467] Processing CreateTablet for tablet 9938f5866a6149e3a1456b03389064b2 (DEFAULT_TABLE table=default.Table_fc6121581e0a4a8c85474adbe1bdbcf8 [id=ec5a69fb4ec74a04b841ee375e8018d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:32.771075 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b1cd18ee2e2145ba9273f500d468343d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:32.771116 736 tablet_service.cc:1467] Processing CreateTablet for tablet 99cf22e0253c4bd6a2d22a0e540e8b68 (DEFAULT_TABLE table=default.Table_41b2cae1cce14b0ab6b6de6d9c946ff5 [id=2b3bc17b229a4a7baaa34966e5d0fe67]), partition=RANGE (key) PARTITION UNBOUNDED
W20250113 02:51:32.771342 998 leader_election.cc:343] T 7d3562a48c6e4badbbcf37640e91b5ba P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:32.771440 998 leader_election.cc:304] T 7d3562a48c6e4badbbcf37640e91b5ba P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4216a7f853d642b7bc3e5cff458f3aca; no voters: 1186fa869eb14812832fa6562531f196, df69f48656b44bb890f76587ee69cc0f
W20250113 02:51:32.771775 998 leader_election.cc:343] T 8337f3c82d654ecdac2cbc1ffa88ef05 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
W20250113 02:51:32.772025 998 leader_election.cc:343] T 8337f3c82d654ecdac2cbc1ffa88ef05 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:32.772199 742 tablet_service.cc:1467] Processing CreateTablet for tablet 119baabad10a488082f86f2a52ac3779 (DEFAULT_TABLE table=default.Table_50cc6a4316724447bdeefc5aefe989a6 [id=608f9e63e64845fbbb0efb119c9fa091]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:32.772307 2012 raft_consensus.cc:2743] T 7d3562a48c6e4badbbcf37640e91b5ba P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:32.772650 736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 99cf22e0253c4bd6a2d22a0e540e8b68. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:32.773571 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet b86d035e503344e1a3bd85c55131274e
I20250113 02:51:32.777276 1191 tablet_bootstrap.cc:654] T aa5c172a69bf4ca79999f4b58df08de8 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:32.778091 1042 tablet_service.cc:1514] Processing DeleteTablet for tablet b86d035e503344e1a3bd85c55131274e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:32 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:32.778167 1040 tablet_service.cc:1467] Processing CreateTablet for tablet cd5589a82daa4624b3e8f94a01ae70d9 (DEFAULT_TABLE table=default.Table_08f7721ed9f1449ba1fbce603d01b13b [id=74d4555532054cc6ba5fa87376e0a74f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:32.778548 1039 tablet_service.cc:1467] Processing CreateTablet for tablet 7aa4a8fbf5204ee89aa8bdc3047bc7e5 (DEFAULT_TABLE table=default.Table_345d3448247f4d818397c10f9e3c96de [id=6483905dcab54c308d444885c9debeae]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:32.784737 885 tablet_service.cc:1467] Processing CreateTablet for tablet 9e25a99e40ae41c39289178d897e0b99 (DEFAULT_TABLE table=default.Table_a3a87b4d6edb4210820b2ccc7a71dbd4 [id=0c14541780ef458e925fd6e6a4fdb663]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:32.784914 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9938f5866a6149e3a1456b03389064b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:32.785339 1191 tablet_bootstrap.cc:492] T aa5c172a69bf4ca79999f4b58df08de8 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:32.785462 1191 ts_tablet_manager.cc:1397] T aa5c172a69bf4ca79999f4b58df08de8 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.027s user 0.004s sys 0.000s
I20250113 02:51:32.785450 881 tablet_service.cc:1467] Processing CreateTablet for tablet b1cd18ee2e2145ba9273f500d468343d (DEFAULT_TABLE table=default.Table_ef9c0cf86b0943aab6bbb6a3fe74ba85 [id=330eff1816254196a8930116c238aad7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:32.787024 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae20dd1bc47d4495a0d88297ff934df6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:32.787225 2051 tablet.cc:1721] T 4c615b0b83d74fcfae0c51047dbfbbf4 P 4216a7f853d642b7bc3e5cff458f3aca: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250113 02:51:32.792110 887 tablet_service.cc:1514] Processing DeleteTablet for tablet 7213e3284b4d4cb581d9fb946a9f87e9 with delete_type TABLET_DATA_DELETED (replaced by tablet 7c443454994a47f2bfa7e2ec7ac576e0) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:32.792577 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 119baabad10a488082f86f2a52ac3779. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:32.792778 888 tablet_service.cc:1514] Processing DeleteTablet for tablet e6e8966da70746d7a10c7243ba2f6f87 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:32 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:32.794844 751 tablet_service.cc:1467] Processing CreateTablet for tablet f29e16fea2534c7dbe57639c7c6b37d6 (DEFAULT_TABLE table=default.Table_d7b990d07d59419da5ad4a9bc9dcc215 [id=5f903ae1c50e4e7487a3b70555f4bb1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:32.800681 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f29e16fea2534c7dbe57639c7c6b37d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:32.800678 1192 tablet_bootstrap.cc:492] T 056bf1e67be04d179eace3d93282b145 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:32.802717 1192 tablet_bootstrap.cc:654] T 056bf1e67be04d179eace3d93282b145 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:32.804170 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 119baabad10a488082f86f2a52ac3779. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:32.809336 1036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b9e7deda7334ff4a3429f70d9b3a7fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:32.809657 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4cc15487be474354b995c4e415276a18. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:32.812652 887 tablet_service.cc:1467] Processing CreateTablet for tablet cd5589a82daa4624b3e8f94a01ae70d9 (DEFAULT_TABLE table=default.Table_08f7721ed9f1449ba1fbce603d01b13b [id=74d4555532054cc6ba5fa87376e0a74f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:32.812918 888 tablet_service.cc:1514] Processing DeleteTablet for tablet b86d035e503344e1a3bd85c55131274e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:32 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:32.813191 888 tablet_service.cc:1467] Processing CreateTablet for tablet 7aa4a8fbf5204ee89aa8bdc3047bc7e5 (DEFAULT_TABLE table=default.Table_345d3448247f4d818397c10f9e3c96de [id=6483905dcab54c308d444885c9debeae]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:32.817747 1192 tablet_bootstrap.cc:492] T 056bf1e67be04d179eace3d93282b145 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:32.818042 1192 ts_tablet_manager.cc:1397] T 056bf1e67be04d179eace3d93282b145 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.017s user 0.001s sys 0.002s
I20250113 02:51:32.820243 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f29d7a9504d740528914fe6a66fd4028. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:32.820331 1192 raft_consensus.cc:357] T 056bf1e67be04d179eace3d93282b145 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:32.820969 1192 raft_consensus.cc:383] T 056bf1e67be04d179eace3d93282b145 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:32.821113 1192 raft_consensus.cc:738] T 056bf1e67be04d179eace3d93282b145 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:32.821388 1192 consensus_queue.cc:260] T 056bf1e67be04d179eace3d93282b145 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:32.821458 1196 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37366 (request call id 755) took 1447 ms. Trace:
I20250113 02:51:32.821895 877 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:37356 (request call id 226) took 1201 ms. Trace:
I20250113 02:51:32.822037 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b1cd18ee2e2145ba9273f500d468343d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:32.821982 877 rpcz_store.cc:276] 0113 02:51:31.614585 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:31.614635 (+ 50us) service_pool.cc:225] Handling call
0113 02:51:32.539386 (+924751us) spinlock_profiling.cc:243] Waited 50.2 ms on lock 0x6180000a388c. stack: 00007f0b1ea7cfd6 00007f0b1ea7cd90 00007f0b1da5c48d 000000000096d544 000000000096d50d 000000000096ceda 00007f0b2b81f8ee 00007f0b2b61e1df 00007f0b2e02c948 00007f0b289878e6 00007f0b28987615 00007f0b257e430e 00007f0b257e31ee 00007f0b257e9425 00007f0b257e9d4f 00007f0b257e9b78
0113 02:51:32.816181 (+276795us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":115371904}
I20250113 02:51:32.821532 1196 rpcz_store.cc:276] 0113 02:51:31.373567 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:31.374086 (+ 519us) service_pool.cc:225] Handling call
0113 02:51:32.495968 (+1121882us) ts_tablet_manager.cc:1145] Deleting tablet 63efcb6b2e194c469fb30a197bd6565b
0113 02:51:32.496015 (+ 47us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:32.548950 (+ 52935us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:32.555102 (+ 6152us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:32.821442 (+266340us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4604,"tablet-delete.queue_time_us":1121642}
I20250113 02:51:32.823685 1196 tablet_replica.cc:331] T f31dd916336a4d8fb3ab015af7de8ccb P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:32.823846 1196 raft_consensus.cc:2238] T f31dd916336a4d8fb3ab015af7de8ccb P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:32.824048 1196 raft_consensus.cc:2267] T f31dd916336a4d8fb3ab015af7de8ccb P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:32.824568 1196 ts_tablet_manager.cc:1905] T f31dd916336a4d8fb3ab015af7de8ccb P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:32.828218 1192 ts_tablet_manager.cc:1428] T 056bf1e67be04d179eace3d93282b145 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20250113 02:51:32.830054 2045 raft_consensus.cc:491] T c7243d27a1c642269a3d0e67632e458c P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:32.830194 2045 raft_consensus.cc:513] T c7243d27a1c642269a3d0e67632e458c P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:32.831396 2009 raft_consensus.cc:491] T 42fad91fe1b64a21a2c8412a2585af47 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:32.831522 2009 raft_consensus.cc:513] T 42fad91fe1b64a21a2c8412a2585af47 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:32.832170 2009 leader_election.cc:290] T 42fad91fe1b64a21a2c8412a2585af47 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:32.832648 1192 tablet_bootstrap.cc:492] T c508724edea147e5bf6e20bca2f50397 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:32.834204 1192 tablet_bootstrap.cc:654] T c508724edea147e5bf6e20bca2f50397 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:32.836179 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42fad91fe1b64a21a2c8412a2585af47" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:32.836689 998 leader_election.cc:343] T 42fad91fe1b64a21a2c8412a2585af47 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:32.837138 1194 tablet_bootstrap.cc:654] T 32cba8f3efc54e78b4df41c15df82eea P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:32.837962 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 99cf22e0253c4bd6a2d22a0e540e8b68. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:32.840345 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet 63efcb6b2e194c469fb30a197bd6565b (table default.Table_a6447976a51a4e2f93d22bd1c657c49d [id=0485ff4fda424e97bb71b77d657ab5e6]) successfully deleted
I20250113 02:51:32.840857 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7aa4a8fbf5204ee89aa8bdc3047bc7e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:32.843530 743 tablet_service.cc:1467] Processing CreateTablet for tablet be5d6907a02f41be8eab71de051a6d3b (DEFAULT_TABLE table=default.Table_a7ed8a8a7d00492b8fc543baeb433a27 [id=3e6a7094710a444aac74e1919ba52ac8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:32.848230 1040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cd5589a82daa4624b3e8f94a01ae70d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:32.853561 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb715f9fbf0e4e3d935b0510e9872257. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:32.856199 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e25a99e40ae41c39289178d897e0b99. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:32.860371 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9938f5866a6149e3a1456b03389064b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:32.864909 746 tablet_service.cc:1467] Processing CreateTablet for tablet cd5589a82daa4624b3e8f94a01ae70d9 (DEFAULT_TABLE table=default.Table_08f7721ed9f1449ba1fbce603d01b13b [id=74d4555532054cc6ba5fa87376e0a74f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:32.866519 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e4998b86d32040ec959ff6fd94b1e9eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:32.866911 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc28641fbd834d768aea272e0f988688. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:32.870544 898 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:37366 (request call id 770) took 1363 ms. Trace:
I20250113 02:51:32.870622 898 rpcz_store.cc:276] 0113 02:51:31.507039 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:31.507206 (+ 167us) service_pool.cc:225] Handling call
0113 02:51:31.507608 (+ 402us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0113 02:51:31.507633 (+ 25us) ts_tablet_manager.cc:676] Creating new metadata...
0113 02:51:31.922956 (+415323us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:32.760370 (+837414us) tablet_replica.cc:206] Creating consensus instance
0113 02:51:32.870524 (+110154us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 02:51:32.871344 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7243d27a1c642269a3d0e67632e458c" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:32.876231 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6fd59b6773fd4bac863b1714f13c9184. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:32.878334 454 catalog_manager.cc:5618] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095) does not have the latest schema for tablet 1cfbcd682b944c89847b40fe69f0a75e (table default.Table_32e88abaae96492fbcce39fb2cd7c221 [id=2b84deb07e5c4ab8b550fe42945d7b71]). Expected version 2 got 0
I20250113 02:51:32.880236 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be5d6907a02f41be8eab71de051a6d3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:32.880539 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f29e16fea2534c7dbe57639c7c6b37d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:32.881114 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 119baabad10a488082f86f2a52ac3779. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:32.881412 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b45fc730c445456a8a5cacac2cf47efe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:32.881603 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4cc15487be474354b995c4e415276a18. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:32.882230 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae20dd1bc47d4495a0d88297ff934df6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:32.882390 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b9e7deda7334ff4a3429f70d9b3a7fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:32.883993 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9938f5866a6149e3a1456b03389064b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:32.884945 1196 ts_tablet_manager.cc:1918] T f31dd916336a4d8fb3ab015af7de8ccb P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:32.885934 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e25a99e40ae41c39289178d897e0b99. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:32.886291 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b1cd18ee2e2145ba9273f500d468343d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:32.887282 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cd5589a82daa4624b3e8f94a01ae70d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:32.887528 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7aa4a8fbf5204ee89aa8bdc3047bc7e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:32.889231 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 99cf22e0253c4bd6a2d22a0e540e8b68. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:32.890399 1196 log.cc:1198] T f31dd916336a4d8fb3ab015af7de8ccb P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/f31dd916336a4d8fb3ab015af7de8ccb
I20250113 02:51:32.892141 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb715f9fbf0e4e3d935b0510e9872257. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:32.900435 1196 ts_tablet_manager.cc:1939] T f31dd916336a4d8fb3ab015af7de8ccb P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:32.900717 1196 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37366 (request call id 756) took 1527 ms. Trace:
I20250113 02:51:32.900786 1196 rpcz_store.cc:276] 0113 02:51:31.373660 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:31.374367 (+ 707us) service_pool.cc:225] Handling call
0113 02:51:32.823645 (+1449278us) ts_tablet_manager.cc:1145] Deleting tablet f31dd916336a4d8fb3ab015af7de8ccb
0113 02:51:32.823654 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:32.827683 (+ 4029us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:32.884863 (+ 57180us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:32.900703 (+ 15840us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1449123}
I20250113 02:51:32.901780 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet f31dd916336a4d8fb3ab015af7de8ccb (table default.Table_37b1d9d091924f8d9a8a2cbf1d06af1a [id=4d85a53a49d642b798af44eef60b3f78]) successfully deleted
I20250113 02:51:32.902300 2024 raft_consensus.cc:491] T c215ffe61b9b426286cc5e808463078d P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:32.902427 2024 raft_consensus.cc:513] T c215ffe61b9b426286cc5e808463078d P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:32.903908 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_003cb5a672084be4b29d515fa4485747" } new_table_name: "default.Table_325c103d1e1349e4896118881852e462" modify_external_catalogs: true
I20250113 02:51:32.905755 454 catalog_manager.cc:5618] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095) does not have the latest schema for tablet 4c615b0b83d74fcfae0c51047dbfbbf4 (table default.Table_bf689f8eb8884e88a72e9b50598fc389 [id=a956f95f6dd94b578f6bd3cb3b97f9da]). Expected version 2 got 0
I20250113 02:51:32.907665 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_bf1d5374f8a54f8abe3314c2c9615fdf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:32.908058 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0752fde3a30a4947a2a688c56ceadab5 consensus state has no leader
I20250113 02:51:32.908129 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_93d51ec3a124475ea63b71ae5744e3c0" } new_table_name: "default.Table_b22d94712c2746ac9a3f7f71248fce80" modify_external_catalogs: true
I20250113 02:51:32.908349 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_e3586ada32644653a92140e6ce9deed9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:32.908569 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_40dae4519f04406d97443c3871b03dd9" } new_table_name: "default.Table_5870bef96fda4722aa621855265bbbab" modify_external_catalogs: true
W20250113 02:51:32.908883 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e3586ada32644653a92140e6ce9deed9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:32.909757 452 catalog_manager.cc:6071] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: default.Table_9ef18ee71ad843c5921750a2f8478855 [id=bd8bd572ed524f2cae01f820f07e3c36] alter complete (version 2)
I20250113 02:51:32.911563 1192 tablet_bootstrap.cc:492] T c508724edea147e5bf6e20bca2f50397 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:32.911736 1030 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a7e7713bc8648378b668d8b769ac927 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:21 UTC) from {username='slave'} at 127.0.0.1:60230
W20250113 02:51:32.912231 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bf1d5374f8a54f8abe3314c2c9615fdf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250113 02:51:32.912986 852 leader_election.cc:343] T c215ffe61b9b426286cc5e808463078d P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Not found: Tablet not found: c215ffe61b9b426286cc5e808463078d
I20250113 02:51:32.911739 1192 ts_tablet_manager.cc:1397] T c508724edea147e5bf6e20bca2f50397 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.079s user 0.007s sys 0.022s
I20250113 02:51:32.913982 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_d1d988a18e0b47b1baef18f31d0cb356" } new_table_name: "default.Table_2f1a7bb2c4cd4ead95b175e1d3545f20" modify_external_catalogs: true
I20250113 02:51:32.915722 1192 raft_consensus.cc:357] T c508724edea147e5bf6e20bca2f50397 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:32.915900 1192 raft_consensus.cc:383] T c508724edea147e5bf6e20bca2f50397 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:32.915954 1192 raft_consensus.cc:738] T c508724edea147e5bf6e20bca2f50397 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:32.916167 1192 consensus_queue.cc:260] T c508724edea147e5bf6e20bca2f50397 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:32.916262 2009 raft_consensus.cc:491] T 056bf1e67be04d179eace3d93282b145 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:32.916385 2009 raft_consensus.cc:513] T 056bf1e67be04d179eace3d93282b145 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:32.916232 454 catalog_manager.cc:5526] T 8337f3c82d654ecdac2cbc1ffa88ef05 P 4216a7f853d642b7bc3e5cff458f3aca reported cstate change: term changed from 0 to 1, leader changed from <none> to 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3). New cstate: current_term: 1 leader_uuid: "4216a7f853d642b7bc3e5cff458f3aca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } health_report { overall_health: UNKNOWN } } }
I20250113 02:51:32.916666 1192 ts_tablet_manager.cc:1428] T c508724edea147e5bf6e20bca2f50397 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250113 02:51:32.917032 2009 leader_election.cc:290] T 056bf1e67be04d179eace3d93282b145 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:32.918125 1192 tablet_bootstrap.cc:492] T 6d4d8b5cc8ad4c9e935d8afaf5766e66 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:32.919667 1192 tablet_bootstrap.cc:654] T 6d4d8b5cc8ad4c9e935d8afaf5766e66 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:32.920030 1196 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37366 (request call id 761) took 1511 ms. Trace:
I20250113 02:51:32.920105 1196 rpcz_store.cc:276] 0113 02:51:31.408250 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:31.408303 (+ 53us) service_pool.cc:225] Handling call
0113 02:51:32.901053 (+1492750us) ts_tablet_manager.cc:1145] Deleting tablet f8a6eebb6b6d4b63903e68750e181707
0113 02:51:32.901061 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:32.920003 (+ 18942us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1492543}
I20250113 02:51:32.920496 1196 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37366 (request call id 759) took 1512 ms. Trace:
I20250113 02:51:32.920591 1196 rpcz_store.cc:276] 0113 02:51:31.407909 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:31.410816 (+ 2907us) service_pool.cc:225] Handling call
0113 02:51:32.920368 (+1509552us) ts_tablet_manager.cc:1145] Deleting tablet df6299f38ffa4c2f9d6c8a5e5285905f
0113 02:51:32.920381 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:32.920476 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1509277}
I20250113 02:51:32.920950 1196 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37366 (request call id 760) took 1512 ms. Trace:
I20250113 02:51:32.921016 1196 rpcz_store.cc:276] 0113 02:51:31.408082 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:31.412148 (+ 4066us) service_pool.cc:225] Handling call
0113 02:51:32.920853 (+1508705us) ts_tablet_manager.cc:1145] Deleting tablet ef6dfe0431ef4a73a957559f60f24756
0113 02:51:32.920866 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:32.920936 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1508462}
I20250113 02:51:32.921283 1196 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37366 (request call id 754) took 1547 ms. Trace:
I20250113 02:51:32.921347 1196 rpcz_store.cc:276] 0113 02:51:31.373436 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:31.424580 (+ 51144us) service_pool.cc:225] Handling call
0113 02:51:32.921197 (+1496617us) ts_tablet_manager.cc:1145] Deleting tablet e13267498917432ba92b6badb76f5d73
0113 02:51:32.921204 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:32.921267 (+ 63us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1496108}
I20250113 02:51:32.921643 1196 tablet_replica.cc:331] T 19f14b3db0b64847b5422bfd5ed323d0 P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:32.921865 1196 raft_consensus.cc:2238] T 19f14b3db0b64847b5422bfd5ed323d0 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:32.922019 1196 raft_consensus.cc:2267] T 19f14b3db0b64847b5422bfd5ed323d0 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:32.922570 1196 ts_tablet_manager.cc:1905] T 19f14b3db0b64847b5422bfd5ed323d0 P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:32.923633 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_9ad6dc8c7c204c7ea01374d6d8c5844d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:32.924201 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9ad6dc8c7c204c7ea01374d6d8c5844d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:32.924501 2024 leader_election.cc:290] T c215ffe61b9b426286cc5e808463078d P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587), 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
W20250113 02:51:32.930028 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet f8a6eebb6b6d4b63903e68750e181707 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f8a6eebb6b6d4b63903e68750e181707 already in progress: creating tablet
W20250113 02:51:32.930294 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet df6299f38ffa4c2f9d6c8a5e5285905f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet df6299f38ffa4c2f9d6c8a5e5285905f already in progress: creating tablet
W20250113 02:51:32.930588 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet e13267498917432ba92b6badb76f5d73 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e13267498917432ba92b6badb76f5d73 already in progress: creating tablet
I20250113 02:51:32.932732 2009 raft_consensus.cc:491] T 23e04961f072440fa3e3f9485e40f79c P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:32.932770 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "056bf1e67be04d179eace3d93282b145" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:32.932861 2009 raft_consensus.cc:513] T 23e04961f072440fa3e3f9485e40f79c P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
W20250113 02:51:32.933290 998 leader_election.cc:343] T 056bf1e67be04d179eace3d93282b145 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:32.933472 2009 leader_election.cc:290] T 23e04961f072440fa3e3f9485e40f79c P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:32.934098 1107 heartbeater.cc:346] Connected to a master server at 127.0.104.60:42067
I20250113 02:51:32.939731 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23e04961f072440fa3e3f9485e40f79c" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:32.940184 998 leader_election.cc:343] T 23e04961f072440fa3e3f9485e40f79c P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:32.941248 1192 tablet_bootstrap.cc:492] T 6d4d8b5cc8ad4c9e935d8afaf5766e66 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:32.941519 1192 ts_tablet_manager.cc:1397] T 6d4d8b5cc8ad4c9e935d8afaf5766e66 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.023s user 0.005s sys 0.002s
I20250113 02:51:32.942893 1192 raft_consensus.cc:357] T 6d4d8b5cc8ad4c9e935d8afaf5766e66 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:32.943229 1192 raft_consensus.cc:383] T 6d4d8b5cc8ad4c9e935d8afaf5766e66 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:32.943385 1192 raft_consensus.cc:738] T 6d4d8b5cc8ad4c9e935d8afaf5766e66 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:32.943812 1192 consensus_queue.cc:260] T 6d4d8b5cc8ad4c9e935d8afaf5766e66 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:32.944876 1192 ts_tablet_manager.cc:1428] T 6d4d8b5cc8ad4c9e935d8afaf5766e66 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250113 02:51:32.945442 1192 tablet_bootstrap.cc:492] T 0752fde3a30a4947a2a688c56ceadab5 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:32.954229 1192 tablet_bootstrap.cc:654] T 0752fde3a30a4947a2a688c56ceadab5 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:32.957094 1194 tablet_bootstrap.cc:492] T 32cba8f3efc54e78b4df41c15df82eea P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:32.957238 1194 ts_tablet_manager.cc:1397] T 32cba8f3efc54e78b4df41c15df82eea P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.351s user 0.001s sys 0.002s
I20250113 02:51:32.957762 2009 raft_consensus.cc:491] T b0b267b0d9c54ba496c89f6cb287dffd P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:32.957880 2009 raft_consensus.cc:513] T b0b267b0d9c54ba496c89f6cb287dffd P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:32.958446 2009 leader_election.cc:290] T b0b267b0d9c54ba496c89f6cb287dffd P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
I20250113 02:51:32.961432 879 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:37356 (request call id 225) took 1347 ms. Trace:
I20250113 02:51:32.961520 879 rpcz_store.cc:276] 0113 02:51:31.614288 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:31.614359 (+ 71us) service_pool.cc:225] Handling call
0113 02:51:32.572033 (+957674us) spinlock_profiling.cc:243] Waited 135 ms on lock 0x6180000a388c. stack: 00007f0b1ea7cfd6 00007f0b1ea7cd90 00007f0b1da5c48d 000000000096d544 000000000096d50d 000000000096ceda 00007f0b2b81f8ee 00007f0b2b61e1df 00007f0b2e02c948 00007f0b289878e6 00007f0b28987615 00007f0b257e430e 00007f0b257e31ee 00007f0b257e9425 00007f0b257e9d4f 00007f0b257e9b78
0113 02:51:32.961410 (+389377us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":309910784}
I20250113 02:51:32.969720 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0b267b0d9c54ba496c89f6cb287dffd" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:32.971463 998 leader_election.cc:343] T b0b267b0d9c54ba496c89f6cb287dffd P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
W20250113 02:51:32.977806 998 consensus_peers.cc:487] T be4b2a8a3927410eaf12e7592646e4af P 4216a7f853d642b7bc3e5cff458f3aca -> Peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Couldn't send request to peer df69f48656b44bb890f76587ee69cc0f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250113 02:51:32.979138 447 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_9a9b1ced3bef4e648088e0781851128e" } modify_external_catalogs: true
I20250113 02:51:32.979283 1192 tablet_bootstrap.cc:492] T 0752fde3a30a4947a2a688c56ceadab5 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:32.979450 447 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_9a9b1ced3bef4e648088e0781851128e" } modify_external_catalogs: true
I20250113 02:51:32.979481 1192 ts_tablet_manager.cc:1397] T 0752fde3a30a4947a2a688c56ceadab5 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.034s user 0.004s sys 0.001s
I20250113 02:51:32.980502 2024 raft_consensus.cc:491] T 401c4f5dbd304950877b59a865f417b3 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:32.980675 1192 raft_consensus.cc:357] T 0752fde3a30a4947a2a688c56ceadab5 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:32.980991 1192 raft_consensus.cc:383] T 0752fde3a30a4947a2a688c56ceadab5 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:32.980937 2024 raft_consensus.cc:513] T 401c4f5dbd304950877b59a865f417b3 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:32.981099 1192 raft_consensus.cc:738] T 0752fde3a30a4947a2a688c56ceadab5 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:32.981447 1192 consensus_queue.cc:260] T 0752fde3a30a4947a2a688c56ceadab5 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:32.982959 1192 ts_tablet_manager.cc:1428] T 0752fde3a30a4947a2a688c56ceadab5 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250113 02:51:32.984395 1192 tablet_bootstrap.cc:492] T c1fb2b36c0504cb4ba32671ae4c620d1 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:32.988121 2024 leader_election.cc:290] T 401c4f5dbd304950877b59a865f417b3 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:32.989255 1062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "401c4f5dbd304950877b59a865f417b3" candidate_uuid: "df69f48656b44bb890f76587ee69cc0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:32.997325 1192 tablet_bootstrap.cc:654] T c1fb2b36c0504cb4ba32671ae4c620d1 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:32.997469 1905 raft_consensus.cc:491] T 6e1284c8aaba44b5810b0d06d36757eb P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:32.997650 1905 raft_consensus.cc:513] T 6e1284c8aaba44b5810b0d06d36757eb P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:32.998437 1905 leader_election.cc:290] T 6e1284c8aaba44b5810b0d06d36757eb P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:32.999019 750 tablet_service.cc:1514] Processing DeleteTablet for tablet b86d035e503344e1a3bd85c55131274e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:32 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:32.999269 2045 leader_election.cc:290] T c7243d27a1c642269a3d0e67632e458c P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
W20250113 02:51:32.997452 711 leader_election.cc:343] T c7243d27a1c642269a3d0e67632e458c P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:32.997469 747 tablet_service.cc:1467] Processing CreateTablet for tablet 7aa4a8fbf5204ee89aa8bdc3047bc7e5 (DEFAULT_TABLE table=default.Table_345d3448247f4d818397c10f9e3c96de [id=6483905dcab54c308d444885c9debeae]), partition=RANGE (key) PARTITION UNBOUNDED
W20250113 02:51:33.000134 711 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:38467, user_credentials={real_user=slave}} blocked reactor thread for 128346us
I20250113 02:51:33.002543 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 777) took 1700 ms. Trace:
I20250113 02:51:33.002800 1198 rpcz_store.cc:276] 0113 02:51:31.302043 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:31.302096 (+ 53us) service_pool.cc:225] Handling call
0113 02:51:32.654771 (+1352675us) ts_tablet_manager.cc:1145] Deleting tablet bffec36d567544c5a1c87e64a7e9730b
0113 02:51:32.654783 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:32.724314 (+ 69531us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:32.726885 (+ 2571us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:33.002486 (+275601us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1327394}
I20250113 02:51:33.003528 1198 tablet_replica.cc:331] T 11950e996c814590bc9f3f22c0354902 P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:33.003957 1198 raft_consensus.cc:2238] T 11950e996c814590bc9f3f22c0354902 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:33.004315 1198 raft_consensus.cc:2267] T 11950e996c814590bc9f3f22c0354902 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:33.004873 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet bffec36d567544c5a1c87e64a7e9730b (table default.Table_c79704f27bf14cba8a0afc2dd36a143f [id=f26e79c7d9164cc89d184f1ffbd21036]) successfully deleted
I20250113 02:51:33.005640 1198 ts_tablet_manager.cc:1905] T 11950e996c814590bc9f3f22c0354902 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:33.007445 2053 tablet.cc:1716] T 32cba8f3efc54e78b4df41c15df82eea P 1186fa869eb14812832fa6562531f196: Skipping requested alter to schema version 5, tablet already version 5
I20250113 02:51:33.008113 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e1284c8aaba44b5810b0d06d36757eb" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:33.008562 711 leader_election.cc:343] T 6e1284c8aaba44b5810b0d06d36757eb P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:32.978757 1194 raft_consensus.cc:357] T 32cba8f3efc54e78b4df41c15df82eea P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:33.008756 1194 raft_consensus.cc:383] T 32cba8f3efc54e78b4df41c15df82eea P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:33.008832 1194 raft_consensus.cc:738] T 32cba8f3efc54e78b4df41c15df82eea P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:33.009151 1194 consensus_queue.cc:260] T 32cba8f3efc54e78b4df41c15df82eea P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
W20250113 02:51:33.009289 851 leader_election.cc:343] T c215ffe61b9b426286cc5e808463078d P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Not found: Tablet not found: c215ffe61b9b426286cc5e808463078d
I20250113 02:51:33.009385 851 leader_election.cc:304] T c215ffe61b9b426286cc5e808463078d P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df69f48656b44bb890f76587ee69cc0f; no voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca
I20250113 02:51:33.009816 2024 raft_consensus.cc:2743] T c215ffe61b9b426286cc5e808463078d P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:33.010756 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "401c4f5dbd304950877b59a865f417b3" candidate_uuid: "df69f48656b44bb890f76587ee69cc0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
I20250113 02:51:33.012517 1195 ts_tablet_manager.cc:1918] T b8b6ffe690fc459db78b2b3bc12671dd P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:33.012615 1195 log.cc:1198] T b8b6ffe690fc459db78b2b3bc12671dd P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/b8b6ffe690fc459db78b2b3bc12671dd
I20250113 02:51:33.014199 851 leader_election.cc:304] T 401c4f5dbd304950877b59a865f417b3 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df69f48656b44bb890f76587ee69cc0f; no voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca
I20250113 02:51:33.015090 2024 raft_consensus.cc:2743] T 401c4f5dbd304950877b59a865f417b3 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:33.015110 2045 raft_consensus.cc:491] T 6666922d677a4c678f80d7c6c47fd2a5 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:33.015206 2045 raft_consensus.cc:513] T 6666922d677a4c678f80d7c6c47fd2a5 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:33.010586 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 36726b40ac5145bfb371871b3fff079a
I20250113 02:51:33.016453 888 tablet_service.cc:1514] Processing DeleteTablet for tablet 36726b40ac5145bfb371871b3fff079a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:32 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:33.016428 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6666922d677a4c678f80d7c6c47fd2a5" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:33.017113 711 leader_election.cc:343] T 6666922d677a4c678f80d7c6c47fd2a5 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
W20250113 02:51:33.017092 995 leader_election.cc:343] T 42fad91fe1b64a21a2c8412a2585af47 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Not found: Tablet not found: 42fad91fe1b64a21a2c8412a2585af47
I20250113 02:51:33.017232 995 leader_election.cc:304] T 42fad91fe1b64a21a2c8412a2585af47 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4216a7f853d642b7bc3e5cff458f3aca; no voters: 1186fa869eb14812832fa6562531f196, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:33.017483 2045 leader_election.cc:290] T 6666922d677a4c678f80d7c6c47fd2a5 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
W20250113 02:51:33.017621 995 leader_election.cc:343] T 23e04961f072440fa3e3f9485e40f79c P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Not found: Tablet not found: 23e04961f072440fa3e3f9485e40f79c
I20250113 02:51:33.017709 995 leader_election.cc:304] T 23e04961f072440fa3e3f9485e40f79c P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4216a7f853d642b7bc3e5cff458f3aca; no voters: 1186fa869eb14812832fa6562531f196, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:33.018021 2009 raft_consensus.cc:2743] T 42fad91fe1b64a21a2c8412a2585af47 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:33.017999 1191 raft_consensus.cc:357] T aa5c172a69bf4ca79999f4b58df08de8 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:33.018148 1191 raft_consensus.cc:383] T aa5c172a69bf4ca79999f4b58df08de8 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:33.018188 1191 raft_consensus.cc:738] T aa5c172a69bf4ca79999f4b58df08de8 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:33.018180 2009 raft_consensus.cc:2743] T 23e04961f072440fa3e3f9485e40f79c P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:33.018337 1191 consensus_queue.cc:260] T aa5c172a69bf4ca79999f4b58df08de8 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:33.018734 1191 ts_tablet_manager.cc:1428] T aa5c172a69bf4ca79999f4b58df08de8 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.233s user 0.001s sys 0.001s
I20250113 02:51:33.020753 1191 tablet_bootstrap.cc:492] T 4b07a985897641c3ac48021c2a4daedb P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:33.022567 739 tablet_service.cc:1514] Processing DeleteTablet for tablet 36726b40ac5145bfb371871b3fff079a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:32 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:33.023034 1195 ts_tablet_manager.cc:1939] T b8b6ffe690fc459db78b2b3bc12671dd P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
W20250113 02:51:33.023316 995 leader_election.cc:343] T b0b267b0d9c54ba496c89f6cb287dffd P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Not found: Tablet not found: b0b267b0d9c54ba496c89f6cb287dffd
I20250113 02:51:33.023428 995 leader_election.cc:304] T b0b267b0d9c54ba496c89f6cb287dffd P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4216a7f853d642b7bc3e5cff458f3aca; no voters: 1186fa869eb14812832fa6562531f196, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:33.023839 1926 raft_consensus.cc:2743] T b0b267b0d9c54ba496c89f6cb287dffd P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:33.017086 1041 tablet_service.cc:1514] Processing DeleteTablet for tablet 36726b40ac5145bfb371871b3fff079a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:32 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:33.024565 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 837) took 1154 ms. Trace:
I20250113 02:51:33.024644 1195 rpcz_store.cc:276] 0113 02:51:31.869802 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:31.904794 (+ 34992us) service_pool.cc:225] Handling call
0113 02:51:32.673171 (+768377us) ts_tablet_manager.cc:1145] Deleting tablet b8b6ffe690fc459db78b2b3bc12671dd
0113 02:51:32.673177 (+ 6us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:32.685447 (+ 12270us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:33.012425 (+326978us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:33.024549 (+ 12124us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":768147}
I20250113 02:51:33.025933 1195 tablet_replica.cc:331] T 51d9483ecacb413796d0fd9e7560eac1 P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:33.026038 820 heartbeater.cc:346] Connected to a master server at 127.0.104.60:42067
I20250113 02:51:33.026121 1195 raft_consensus.cc:2238] T 51d9483ecacb413796d0fd9e7560eac1 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:33.026252 1195 raft_consensus.cc:2267] T 51d9483ecacb413796d0fd9e7560eac1 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:33.026733 1195 ts_tablet_manager.cc:1905] T 51d9483ecacb413796d0fd9e7560eac1 P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:33.027540 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet b8b6ffe690fc459db78b2b3bc12671dd (table default.Table_b77c072e465c474db9349f1a3b31c323 [id=d7fe00879d3143c1bf0ff603f9284750]) successfully deleted
W20250113 02:51:33.027839 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet cfd1697a03494ad2aaa41a056e334ca7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cfd1697a03494ad2aaa41a056e334ca7 already in progress: creating tablet
W20250113 02:51:33.028092 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet e266a9083e2f46a0a43861b7708dcdd5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e266a9083e2f46a0a43861b7708dcdd5 already in progress: creating tablet
W20250113 02:51:33.028302 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 87e6a4e4a8434bbcbbe70ddf634accba because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 87e6a4e4a8434bbcbbe70ddf634accba already in progress: creating tablet
W20250113 02:51:33.028481 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 29b1444efd28483297bafc73a33e7a54 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 29b1444efd28483297bafc73a33e7a54 already in progress: creating tablet
W20250113 02:51:33.028671 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet cd6e48a720d549f2b5f30a61e1f5e017 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cd6e48a720d549f2b5f30a61e1f5e017 already in progress: creating tablet
W20250113 02:51:33.028888 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 4cbc27d8ac2a46c58bffc33d8af17be2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4cbc27d8ac2a46c58bffc33d8af17be2 already in progress: creating tablet
W20250113 02:51:33.029093 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 0f428c555442401285fb4eab8280b10b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0f428c555442401285fb4eab8280b10b already in progress: creating tablet
W20250113 02:51:33.029291 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet fd0b8592997d4a61b8ca854ecbb67c55 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fd0b8592997d4a61b8ca854ecbb67c55 already in progress: creating tablet
I20250113 02:51:33.032171 2053 tablet.cc:1721] T 4c615b0b83d74fcfae0c51047dbfbbf4 P 1186fa869eb14812832fa6562531f196: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250113 02:51:33.033057 769 raft_consensus.cc:1270] T be4b2a8a3927410eaf12e7592646e4af P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Refusing update from remote peer 4216a7f853d642b7bc3e5cff458f3aca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250113 02:51:33.033413 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "056bf1e67be04d179eace3d93282b145" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
I20250113 02:51:33.033596 771 raft_consensus.cc:2463] T 056bf1e67be04d179eace3d93282b145 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 0.
I20250113 02:51:33.034317 995 leader_election.cc:304] T 056bf1e67be04d179eace3d93282b145 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters: df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:33.034754 2009 consensus_queue.cc:1035] T be4b2a8a3927410eaf12e7592646e4af P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 02:51:33.035279 2009 raft_consensus.cc:2798] T 056bf1e67be04d179eace3d93282b145 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 02:51:33.035346 2009 raft_consensus.cc:491] T 056bf1e67be04d179eace3d93282b145 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 02:51:33.035414 2009 raft_consensus.cc:3054] T 056bf1e67be04d179eace3d93282b145 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:33.038033 2009 raft_consensus.cc:513] T 056bf1e67be04d179eace3d93282b145 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:33.039574 1194 ts_tablet_manager.cc:1428] T 32cba8f3efc54e78b4df41c15df82eea P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.082s user 0.001s sys 0.001s
I20250113 02:51:33.040026 2009 leader_election.cc:290] T 056bf1e67be04d179eace3d93282b145 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Requested vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:33.039934 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "056bf1e67be04d179eace3d93282b145" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f"
W20250113 02:51:33.040714 998 leader_election.cc:343] T 056bf1e67be04d179eace3d93282b145 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:33.041306 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "056bf1e67be04d179eace3d93282b145" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196"
I20250113 02:51:33.041487 771 raft_consensus.cc:3054] T 056bf1e67be04d179eace3d93282b145 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:33.044452 771 raft_consensus.cc:2463] T 056bf1e67be04d179eace3d93282b145 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 1.
I20250113 02:51:33.044873 995 leader_election.cc:304] T 056bf1e67be04d179eace3d93282b145 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters: df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:33.045266 1926 raft_consensus.cc:2798] T 056bf1e67be04d179eace3d93282b145 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Leader election won for term 1
I20250113 02:51:33.045382 1926 raft_consensus.cc:695] T 056bf1e67be04d179eace3d93282b145 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: Becoming Leader. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Running, Role: LEADER
I20250113 02:51:33.045646 1926 consensus_queue.cc:237] T 056bf1e67be04d179eace3d93282b145 P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:33.052938 2045 raft_consensus.cc:491] T 0539cbbe47e64c039ad39c4b52edcc4b P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:33.053066 2045 raft_consensus.cc:513] T 0539cbbe47e64c039ad39c4b52edcc4b P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:33.053587 2045 leader_election.cc:290] T 0539cbbe47e64c039ad39c4b52edcc4b P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:33.054152 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0539cbbe47e64c039ad39c4b52edcc4b" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:33.054735 711 leader_election.cc:343] T 0539cbbe47e64c039ad39c4b52edcc4b P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:33.059407 1062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e1284c8aaba44b5810b0d06d36757eb" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:33.059839 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7243d27a1c642269a3d0e67632e458c" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:33.060285 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6666922d677a4c678f80d7c6c47fd2a5" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:33.061571 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0539cbbe47e64c039ad39c4b52edcc4b" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
W20250113 02:51:33.063653 709 leader_election.cc:343] T 6e1284c8aaba44b5810b0d06d36757eb P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:33.063922 709 leader_election.cc:304] T 6e1284c8aaba44b5810b0d06d36757eb P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
W20250113 02:51:33.064747 709 leader_election.cc:343] T c7243d27a1c642269a3d0e67632e458c P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:33.065027 2072 raft_consensus.cc:2743] T 6e1284c8aaba44b5810b0d06d36757eb P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:33.065101 709 leader_election.cc:304] T c7243d27a1c642269a3d0e67632e458c P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:33.065707 2072 raft_consensus.cc:2743] T c7243d27a1c642269a3d0e67632e458c P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250113 02:51:33.066322 709 leader_election.cc:343] T 6666922d677a4c678f80d7c6c47fd2a5 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:33.066536 709 leader_election.cc:304] T 6666922d677a4c678f80d7c6c47fd2a5 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:33.066922 2072 raft_consensus.cc:2743] T 6666922d677a4c678f80d7c6c47fd2a5 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:33.068616 1192 tablet_bootstrap.cc:492] T c1fb2b36c0504cb4ba32671ae4c620d1 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
W20250113 02:51:33.069753 709 leader_election.cc:343] T 0539cbbe47e64c039ad39c4b52edcc4b P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:33.070050 709 leader_election.cc:304] T 0539cbbe47e64c039ad39c4b52edcc4b P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:33.070596 2072 raft_consensus.cc:2743] T 0539cbbe47e64c039ad39c4b52edcc4b P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:33.069195 1192 ts_tablet_manager.cc:1397] T c1fb2b36c0504cb4ba32671ae4c620d1 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.085s user 0.005s sys 0.000s
I20250113 02:51:33.075609 1192 raft_consensus.cc:357] T c1fb2b36c0504cb4ba32671ae4c620d1 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:33.076563 1192 raft_consensus.cc:383] T c1fb2b36c0504cb4ba32671ae4c620d1 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:33.076822 1192 raft_consensus.cc:738] T c1fb2b36c0504cb4ba32671ae4c620d1 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:33.077602 1192 consensus_queue.cc:260] T c1fb2b36c0504cb4ba32671ae4c620d1 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:33.078678 2010 raft_consensus.cc:491] T c508724edea147e5bf6e20bca2f50397 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:33.078789 2010 raft_consensus.cc:513] T c508724edea147e5bf6e20bca2f50397 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:33.079300 2010 leader_election.cc:290] T c508724edea147e5bf6e20bca2f50397 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:33.079890 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c508724edea147e5bf6e20bca2f50397" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:33.080509 1192 ts_tablet_manager.cc:1428] T c1fb2b36c0504cb4ba32671ae4c620d1 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.008s user 0.002s sys 0.001s
W20250113 02:51:33.081571 998 leader_election.cc:343] T c508724edea147e5bf6e20bca2f50397 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:33.081919 1194 tablet_bootstrap.cc:492] T ddf79eec41214693859f526265a55492 P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:33.082850 964 heartbeater.cc:346] Connected to a master server at 127.0.104.60:42067
I20250113 02:51:33.081601 1192 tablet_bootstrap.cc:492] T c826a0b8327f44f9a00e5afe3a36cd8a P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:33.084131 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c508724edea147e5bf6e20bca2f50397" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
I20250113 02:51:33.084309 771 raft_consensus.cc:2463] T c508724edea147e5bf6e20bca2f50397 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 0.
I20250113 02:51:33.084719 995 leader_election.cc:304] T c508724edea147e5bf6e20bca2f50397 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters: df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:33.085250 2009 raft_consensus.cc:2798] T c508724edea147e5bf6e20bca2f50397 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 02:51:33.085333 2009 raft_consensus.cc:491] T c508724edea147e5bf6e20bca2f50397 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 02:51:33.085389 2009 raft_consensus.cc:3054] T c508724edea147e5bf6e20bca2f50397 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:33.085431 1194 tablet_bootstrap.cc:654] T ddf79eec41214693859f526265a55492 P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:33.087874 2009 raft_consensus.cc:513] T c508724edea147e5bf6e20bca2f50397 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:33.089118 2009 leader_election.cc:290] T c508724edea147e5bf6e20bca2f50397 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Requested vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:33.089183 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c508724edea147e5bf6e20bca2f50397" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f"
W20250113 02:51:33.089859 998 leader_election.cc:343] T c508724edea147e5bf6e20bca2f50397 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:33.090159 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c508724edea147e5bf6e20bca2f50397" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196"
I20250113 02:51:33.090319 771 raft_consensus.cc:3054] T c508724edea147e5bf6e20bca2f50397 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:33.092229 1192 tablet_bootstrap.cc:654] T c826a0b8327f44f9a00e5afe3a36cd8a P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:33.095577 771 raft_consensus.cc:2463] T c508724edea147e5bf6e20bca2f50397 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 1.
I20250113 02:51:33.096060 995 leader_election.cc:304] T c508724edea147e5bf6e20bca2f50397 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters: df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:33.096475 2009 raft_consensus.cc:2798] T c508724edea147e5bf6e20bca2f50397 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Leader election won for term 1
I20250113 02:51:33.096695 2009 raft_consensus.cc:695] T c508724edea147e5bf6e20bca2f50397 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: Becoming Leader. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Running, Role: LEADER
I20250113 02:51:33.096958 2009 consensus_queue.cc:237] T c508724edea147e5bf6e20bca2f50397 P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
W20250113 02:51:33.104588 421 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.104.2:38467, user_credentials={real_user=slave}} blocked reactor thread for 116498us
I20250113 02:51:33.115599 1192 tablet_bootstrap.cc:492] T c826a0b8327f44f9a00e5afe3a36cd8a P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:33.115953 1192 ts_tablet_manager.cc:1397] T c826a0b8327f44f9a00e5afe3a36cd8a P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.035s user 0.004s sys 0.004s
I20250113 02:51:33.117944 1192 raft_consensus.cc:357] T c826a0b8327f44f9a00e5afe3a36cd8a P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:33.118467 1192 raft_consensus.cc:383] T c826a0b8327f44f9a00e5afe3a36cd8a P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:33.118619 1192 raft_consensus.cc:738] T c826a0b8327f44f9a00e5afe3a36cd8a P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:33.119118 1192 consensus_queue.cc:260] T c826a0b8327f44f9a00e5afe3a36cd8a P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:33.120625 1192 ts_tablet_manager.cc:1428] T c826a0b8327f44f9a00e5afe3a36cd8a P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20250113 02:51:33.125730 1192 tablet_bootstrap.cc:492] T 1a63f485caf54068abd6b50d7de3d14a P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:33.126590 2051 tablet.cc:1721] T be4b2a8a3927410eaf12e7592646e4af P 4216a7f853d642b7bc3e5cff458f3aca: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250113 02:51:33.129961 1192 tablet_bootstrap.cc:654] T 1a63f485caf54068abd6b50d7de3d14a P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:33.135689 2080 tablet.cc:1721] T be4b2a8a3927410eaf12e7592646e4af P 1186fa869eb14812832fa6562531f196: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250113 02:51:33.139719 1926 raft_consensus.cc:491] T 0752fde3a30a4947a2a688c56ceadab5 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:33.139863 1926 raft_consensus.cc:513] T 0752fde3a30a4947a2a688c56ceadab5 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:33.140857 1194 tablet_bootstrap.cc:492] T ddf79eec41214693859f526265a55492 P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:33.140817 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0752fde3a30a4947a2a688c56ceadab5" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
I20250113 02:51:33.141002 771 raft_consensus.cc:2463] T 0752fde3a30a4947a2a688c56ceadab5 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 0.
I20250113 02:51:33.141085 1194 ts_tablet_manager.cc:1397] T ddf79eec41214693859f526265a55492 P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.059s user 0.001s sys 0.002s
I20250113 02:51:33.141393 995 leader_election.cc:304] T 0752fde3a30a4947a2a688c56ceadab5 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters:
I20250113 02:51:33.141748 2009 raft_consensus.cc:2798] T 0752fde3a30a4947a2a688c56ceadab5 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 02:51:33.141821 2009 raft_consensus.cc:491] T 0752fde3a30a4947a2a688c56ceadab5 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 02:51:33.141877 2009 raft_consensus.cc:3054] T 0752fde3a30a4947a2a688c56ceadab5 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:33.141916 1194 raft_consensus.cc:357] T ddf79eec41214693859f526265a55492 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:33.142114 1194 raft_consensus.cc:383] T ddf79eec41214693859f526265a55492 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:33.142181 1194 raft_consensus.cc:738] T ddf79eec41214693859f526265a55492 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:33.142371 1194 consensus_queue.cc:260] T ddf79eec41214693859f526265a55492 P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:33.144811 2009 raft_consensus.cc:513] T 0752fde3a30a4947a2a688c56ceadab5 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:33.145897 2009 leader_election.cc:290] T 0752fde3a30a4947a2a688c56ceadab5 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Requested vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:33.146304 1926 leader_election.cc:290] T 0752fde3a30a4947a2a688c56ceadab5 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:33.146574 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0752fde3a30a4947a2a688c56ceadab5" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196"
I20250113 02:51:33.146821 771 raft_consensus.cc:3054] T 0752fde3a30a4947a2a688c56ceadab5 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:33.148248 1194 ts_tablet_manager.cc:1428] T ddf79eec41214693859f526265a55492 P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20250113 02:51:33.149981 771 raft_consensus.cc:2463] T 0752fde3a30a4947a2a688c56ceadab5 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 1.
I20250113 02:51:33.150480 995 leader_election.cc:304] T 0752fde3a30a4947a2a688c56ceadab5 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters:
I20250113 02:51:33.150565 1192 tablet_bootstrap.cc:492] T 1a63f485caf54068abd6b50d7de3d14a P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:33.150768 1926 raft_consensus.cc:2798] T 0752fde3a30a4947a2a688c56ceadab5 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Leader election won for term 1
I20250113 02:51:33.150940 1926 raft_consensus.cc:695] T 0752fde3a30a4947a2a688c56ceadab5 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: Becoming Leader. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Running, Role: LEADER
I20250113 02:51:33.150846 1192 ts_tablet_manager.cc:1397] T 1a63f485caf54068abd6b50d7de3d14a P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.025s user 0.002s sys 0.007s
I20250113 02:51:33.151269 1926 consensus_queue.cc:237] T 0752fde3a30a4947a2a688c56ceadab5 P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:33.152858 1192 raft_consensus.cc:357] T 1a63f485caf54068abd6b50d7de3d14a P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:33.153893 1194 tablet_bootstrap.cc:492] T 2778d7dc7d9c4843bbc9eb705cec71cd P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:33.153357 1192 raft_consensus.cc:383] T 1a63f485caf54068abd6b50d7de3d14a P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:33.155167 1192 raft_consensus.cc:738] T 1a63f485caf54068abd6b50d7de3d14a P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:33.155624 1192 consensus_queue.cc:260] T 1a63f485caf54068abd6b50d7de3d14a P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:33.157702 1192 ts_tablet_manager.cc:1428] T 1a63f485caf54068abd6b50d7de3d14a P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20250113 02:51:33.159145 1192 tablet_bootstrap.cc:492] T 05bf5ab0df874dd5b6559f6d82069184 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
W20250113 02:51:33.167330 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet df87e25053a74990959f85288a85217b consensus state has no leader
W20250113 02:51:33.167443 451 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a04cec5ed9f34a16a80e112e5c9671e6 consensus state has no leader
I20250113 02:51:33.167838 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_d778815edb8043aba80cdc3273511c51" } modify_external_catalogs: true
I20250113 02:51:33.168056 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_d778815edb8043aba80cdc3273511c51" } modify_external_catalogs: true
I20250113 02:51:33.168438 446 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_a373073bdcf740e389ce848e2e1ce7d4" } modify_external_catalogs: true
I20250113 02:51:33.168607 446 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_a373073bdcf740e389ce848e2e1ce7d4" } modify_external_catalogs: true
W20250113 02:51:33.169559 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d29899b4cccb4e37bd25978d385916f9 consensus state has no leader
I20250113 02:51:33.169979 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_d2d35cb51870448bbf935e20885ec960"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:33.170974 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_9ef18ee71ad843c5921750a2f8478855" } modify_external_catalogs: true
I20250113 02:51:33.171159 455 catalog_manager.cc:5618] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467) does not have the latest schema for tablet 32cba8f3efc54e78b4df41c15df82eea (table default.Table_27e1ad9a94f8455f806f48768ef4ddac [id=8aff13d6d3c5483eadb596b705fe5151]). Expected version 5 got 0
I20250113 02:51:33.172046 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_c9adaef9de094549a9cbd3340c1b33b8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:33.171139 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_9ef18ee71ad843c5921750a2f8478855" } modify_external_catalogs: true
W20250113 02:51:33.172531 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c9adaef9de094549a9cbd3340c1b33b8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250113 02:51:33.174999 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d2d35cb51870448bbf935e20885ec960 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:33.177289 452 master_service.cc:946] ReplaceTablet: received request to replace tablet ef6dfe0431ef4a73a957559f60f24756 from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:33.177028 1192 tablet_bootstrap.cc:654] T 05bf5ab0df874dd5b6559f6d82069184 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:33.178761 452 catalog_manager.cc:5618] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467) does not have the latest schema for tablet 32cba8f3efc54e78b4df41c15df82eea (table default.Table_27e1ad9a94f8455f806f48768ef4ddac [id=8aff13d6d3c5483eadb596b705fe5151]). Expected version 5 got 0
I20250113 02:51:33.180367 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_d0642c34ba7d40c9888ec7521d3cb1e3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:33.180781 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d0642c34ba7d40c9888ec7521d3cb1e3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:33.194366 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cd5589a82daa4624b3e8f94a01ae70d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:33.197321 1191 tablet_bootstrap.cc:654] T 4b07a985897641c3ac48021c2a4daedb P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:33.198110 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be5d6907a02f41be8eab71de051a6d3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:33.200080 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7aa4a8fbf5204ee89aa8bdc3047bc7e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:33.204093 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f29e16fea2534c7dbe57639c7c6b37d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:33.204916 1195 ts_tablet_manager.cc:1918] T 51d9483ecacb413796d0fd9e7560eac1 P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:33.205085 1195 log.cc:1198] T 51d9483ecacb413796d0fd9e7560eac1 P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/51d9483ecacb413796d0fd9e7560eac1
I20250113 02:51:33.206763 2072 raft_consensus.cc:491] T 3b1910496e0849b193e390a5ef1a6b2b P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:33.212131 2072 raft_consensus.cc:513] T 3b1910496e0849b193e390a5ef1a6b2b P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:33.212826 2072 leader_election.cc:290] T 3b1910496e0849b193e390a5ef1a6b2b P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:33.210172 2045 raft_consensus.cc:491] T 99c8046f5d124ca49899df365114c69f P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:33.213364 2045 raft_consensus.cc:513] T 99c8046f5d124ca49899df365114c69f P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:33.213419 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b1910496e0849b193e390a5ef1a6b2b" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:33.213876 2045 leader_election.cc:290] T 99c8046f5d124ca49899df365114c69f P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
W20250113 02:51:33.214452 711 leader_election.cc:343] T 3b1910496e0849b193e390a5ef1a6b2b P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:33.214450 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99c8046f5d124ca49899df365114c69f" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:33.214998 711 leader_election.cc:343] T 99c8046f5d124ca49899df365114c69f P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:33.215461 1195 ts_tablet_manager.cc:1939] T 51d9483ecacb413796d0fd9e7560eac1 P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:33.216125 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet 51d9483ecacb413796d0fd9e7560eac1 (table default.Table_9417f16fb6384fde8e191ca02808aa78 [id=b49a91d3b86d44e1824a987db3de13b6]) successfully deleted
I20250113 02:51:33.216499 1195 tablet_replica.cc:331] T f129bb3fbbfd492d98822be0a6626e44 P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:33.216743 1195 raft_consensus.cc:2238] T f129bb3fbbfd492d98822be0a6626e44 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:33.216949 1195 raft_consensus.cc:2267] T f129bb3fbbfd492d98822be0a6626e44 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:33.217000 1191 tablet_bootstrap.cc:492] T 4b07a985897641c3ac48021c2a4daedb P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:33.217237 1191 ts_tablet_manager.cc:1397] T 4b07a985897641c3ac48021c2a4daedb P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.197s user 0.011s sys 0.012s
I20250113 02:51:33.217517 1195 ts_tablet_manager.cc:1905] T f129bb3fbbfd492d98822be0a6626e44 P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:33.217959 1191 raft_consensus.cc:357] T 4b07a985897641c3ac48021c2a4daedb P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:33.220136 1191 raft_consensus.cc:383] T 4b07a985897641c3ac48021c2a4daedb P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:33.220239 1191 raft_consensus.cc:738] T 4b07a985897641c3ac48021c2a4daedb P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:33.220506 1191 consensus_queue.cc:260] T 4b07a985897641c3ac48021c2a4daedb P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:33.221184 1191 ts_tablet_manager.cc:1428] T 4b07a985897641c3ac48021c2a4daedb P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250113 02:51:33.223554 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b1910496e0849b193e390a5ef1a6b2b" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:33.223913 1192 tablet_bootstrap.cc:492] T 05bf5ab0df874dd5b6559f6d82069184 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:33.224050 1192 ts_tablet_manager.cc:1397] T 05bf5ab0df874dd5b6559f6d82069184 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.065s user 0.001s sys 0.004s
I20250113 02:51:33.223955 1062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99c8046f5d124ca49899df365114c69f" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
W20250113 02:51:33.224613 709 leader_election.cc:343] T 3b1910496e0849b193e390a5ef1a6b2b P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:33.224722 709 leader_election.cc:304] T 3b1910496e0849b193e390a5ef1a6b2b P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
W20250113 02:51:33.225073 709 leader_election.cc:343] T 99c8046f5d124ca49899df365114c69f P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:33.225168 709 leader_election.cc:304] T 99c8046f5d124ca49899df365114c69f P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:33.225562 2045 raft_consensus.cc:2743] T 3b1910496e0849b193e390a5ef1a6b2b P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:33.225720 2045 raft_consensus.cc:2743] T 99c8046f5d124ca49899df365114c69f P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:33.227279 1194 tablet_bootstrap.cc:654] T 2778d7dc7d9c4843bbc9eb705cec71cd P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:33.227488 1192 raft_consensus.cc:357] T 05bf5ab0df874dd5b6559f6d82069184 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:33.227658 1192 raft_consensus.cc:383] T 05bf5ab0df874dd5b6559f6d82069184 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:33.227700 1192 raft_consensus.cc:738] T 05bf5ab0df874dd5b6559f6d82069184 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:33.227859 1192 consensus_queue.cc:260] T 05bf5ab0df874dd5b6559f6d82069184 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:33.228400 1192 ts_tablet_manager.cc:1428] T 05bf5ab0df874dd5b6559f6d82069184 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250113 02:51:33.228663 1192 tablet_bootstrap.cc:492] T a97efc70946c4af487e33a85abf6b90c P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:33.230583 1192 tablet_bootstrap.cc:654] T a97efc70946c4af487e33a85abf6b90c P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
W20250113 02:51:33.231493 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 00d4381876234f16b19336523640bbc8 consensus state has no leader
I20250113 02:51:33.231952 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_deef0377723a44e4a4ed576829e0e763"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:33.238008 2072 raft_consensus.cc:491] T f56230a2c75d4763a54671c79358f4d9 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:33.238153 2072 raft_consensus.cc:513] T f56230a2c75d4763a54671c79358f4d9 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:33.238291 1191 tablet_bootstrap.cc:492] T 42cf1e708644427f9d4dcd41ae2bcb37 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:33.238724 2072 leader_election.cc:290] T f56230a2c75d4763a54671c79358f4d9 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:33.239872 1191 tablet_bootstrap.cc:654] T 42cf1e708644427f9d4dcd41ae2bcb37 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:33.241746 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 9fff52e99c64462a8a73fccc3c22ca63
I20250113 02:51:33.242789 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_3cfdcce7afbf48ee8cf67f1a4c839612"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:33.244642 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_64b83cf4ac3548bd89fefb3779bd0e1a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:33.245304 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_64b83cf4ac3548bd89fefb3779bd0e1a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:33.247130 746 tablet_service.cc:1514] Processing DeleteTablet for tablet 9fff52e99c64462a8a73fccc3c22ca63 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:33 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:33.248709 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 89b6fbf30ca44488ad7969930f1356e2
I20250113 02:51:33.249513 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_e98730a6e1c74c4fbd2b39ac115701f6" } new_table_name: "default.Table_294d733d092b4d9ca0ddd79b52024853" modify_external_catalogs: true
I20250113 02:51:33.250698 1192 tablet_bootstrap.cc:492] T a97efc70946c4af487e33a85abf6b90c P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:33.250828 1192 ts_tablet_manager.cc:1397] T a97efc70946c4af487e33a85abf6b90c P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.002s
I20250113 02:51:33.251149 746 tablet_service.cc:1514] Processing DeleteTablet for tablet 89b6fbf30ca44488ad7969930f1356e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:33 UTC) from {username='slave'} at 127.0.0.1:55398
W20250113 02:51:33.251561 440 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 3c59ae14a6584892bf1ebf6def81d497 (table default.Table_3292864b63454728b1a46a617240a099 [id=5a72ce8b06c54138817637abfe92be99], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 3c59ae14a6584892bf1ebf6def81d497 consensus state has no leader
I20250113 02:51:33.251570 1192 raft_consensus.cc:357] T a97efc70946c4af487e33a85abf6b90c P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:33.251762 1192 raft_consensus.cc:383] T a97efc70946c4af487e33a85abf6b90c P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:33.251816 1192 raft_consensus.cc:738] T a97efc70946c4af487e33a85abf6b90c P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:33.252027 1192 consensus_queue.cc:260] T a97efc70946c4af487e33a85abf6b90c P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:33.252208 1039 tablet_service.cc:1514] Processing DeleteTablet for tablet 89b6fbf30ca44488ad7969930f1356e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:33 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:33.252573 1192 ts_tablet_manager.cc:1428] T a97efc70946c4af487e33a85abf6b90c P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250113 02:51:33.252591 1062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f56230a2c75d4763a54671c79358f4d9" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
W20250113 02:51:33.253831 709 leader_election.cc:343] T f56230a2c75d4763a54671c79358f4d9 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:33.256157 454 catalog_manager.cc:6071] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: default.Table_bf689f8eb8884e88a72e9b50598fc389 [id=a956f95f6dd94b578f6bd3cb3b97f9da] alter complete (version 2)
I20250113 02:51:33.256927 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_fd78ad35561d4253aac07bd501c6aa30"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:33.257481 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fd78ad35561d4253aac07bd501c6aa30 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:33.259604 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet 056bf1e67be04d179eace3d93282b145 with delete_type TABLET_DATA_DELETED (replaced by tablet dfea918c04c1408a8b86a44bb0911a7c) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:33.259776 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet 42fad91fe1b64a21a2c8412a2585af47 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:30 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:33.261922 1039 tablet_service.cc:1514] Processing DeleteTablet for tablet 77cf8b0c824f401aba1880eb7fd1807b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:25 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:33.264230 1192 tablet_bootstrap.cc:492] T 6e1284c8aaba44b5810b0d06d36757eb P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:33.265923 1192 tablet_bootstrap.cc:654] T 6e1284c8aaba44b5810b0d06d36757eb P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
W20250113 02:51:33.267443 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3cfdcce7afbf48ee8cf67f1a4c839612 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:33.268244 1198 ts_tablet_manager.cc:1918] T 11950e996c814590bc9f3f22c0354902 P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:33.268674 1041 tablet_service.cc:1514] Processing DeleteTablet for tablet 9fff52e99c64462a8a73fccc3c22ca63 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:33 UTC) from {username='slave'} at 127.0.0.1:60230
W20250113 02:51:33.269791 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_deef0377723a44e4a4ed576829e0e763 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:33.271243 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 8adfa738c17a4169a5d99f6ceca75100
I20250113 02:51:33.271868 1041 tablet_service.cc:1514] Processing DeleteTablet for tablet 8adfa738c17a4169a5d99f6ceca75100 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:32 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:33.272210 1195 ts_tablet_manager.cc:1918] T f129bb3fbbfd492d98822be0a6626e44 P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:33.273234 744 tablet_service.cc:1514] Processing DeleteTablet for tablet 8adfa738c17a4169a5d99f6ceca75100 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:32 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:33.268328 1198 log.cc:1198] T 11950e996c814590bc9f3f22c0354902 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/11950e996c814590bc9f3f22c0354902
I20250113 02:51:33.273567 1195 log.cc:1198] T f129bb3fbbfd492d98822be0a6626e44 P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/f129bb3fbbfd492d98822be0a6626e44
I20250113 02:51:33.274044 1195 ts_tablet_manager.cc:1939] T f129bb3fbbfd492d98822be0a6626e44 P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:33.274374 1198 ts_tablet_manager.cc:1939] T 11950e996c814590bc9f3f22c0354902 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:33.275352 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet f129bb3fbbfd492d98822be0a6626e44 (table default.Table_ee56f8e59e4f40efbaf24ff31dcf0920 [id=6666a614063942c99a78b95d35add037]) successfully deleted
I20250113 02:51:33.241667 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f56230a2c75d4763a54671c79358f4d9" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:33.277179 711 leader_election.cc:343] T f56230a2c75d4763a54671c79358f4d9 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:33.277294 711 leader_election.cc:304] T f56230a2c75d4763a54671c79358f4d9 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:33.277518 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 778) took 1975 ms. Trace:
I20250113 02:51:33.277693 2072 raft_consensus.cc:2743] T f56230a2c75d4763a54671c79358f4d9 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:33.277648 1198 rpcz_store.cc:276] 0113 02:51:31.302370 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:31.302534 (+ 164us) service_pool.cc:225] Handling call
0113 02:51:33.003456 (+1700922us) ts_tablet_manager.cc:1145] Deleting tablet 11950e996c814590bc9f3f22c0354902
0113 02:51:33.003464 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:33.239606 (+236142us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:33.268148 (+ 28542us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:33.277502 (+ 9354us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1676008}
I20250113 02:51:33.278038 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet 11950e996c814590bc9f3f22c0354902 (table default.Table_325c103d1e1349e4896118881852e462 [id=2e275f7f92ac440ca3ea6e29c5566f00]) successfully deleted
I20250113 02:51:33.278430 1195 tablet_replica.cc:331] T c7243d27a1c642269a3d0e67632e458c P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:33.278692 1198 tablet_replica.cc:331] T 092892a36e0e47cb8d2184c2bbde95d7 P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:33.278925 1195 raft_consensus.cc:2238] T c7243d27a1c642269a3d0e67632e458c P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:33.279299 1198 raft_consensus.cc:2238] T 092892a36e0e47cb8d2184c2bbde95d7 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:33.279546 1195 raft_consensus.cc:2267] T c7243d27a1c642269a3d0e67632e458c P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:33.262528 1196 ts_tablet_manager.cc:1918] T 19f14b3db0b64847b5422bfd5ed323d0 P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:33.280128 1198 raft_consensus.cc:2267] T 092892a36e0e47cb8d2184c2bbde95d7 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:33.280160 1196 log.cc:1198] T 19f14b3db0b64847b5422bfd5ed323d0 P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/19f14b3db0b64847b5422bfd5ed323d0
I20250113 02:51:33.280589 1196 ts_tablet_manager.cc:1939] T 19f14b3db0b64847b5422bfd5ed323d0 P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:33.281190 1196 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37366 (request call id 753) took 1907 ms. Trace:
I20250113 02:51:33.281478 1198 ts_tablet_manager.cc:1905] T 092892a36e0e47cb8d2184c2bbde95d7 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:33.281944 1195 ts_tablet_manager.cc:1905] T c7243d27a1c642269a3d0e67632e458c P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:33.284212 1196 rpcz_store.cc:276] 0113 02:51:31.373293 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:31.444862 (+ 71569us) service_pool.cc:225] Handling call
0113 02:51:32.921597 (+1476735us) ts_tablet_manager.cc:1145] Deleting tablet 19f14b3db0b64847b5422bfd5ed323d0
0113 02:51:32.921608 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:33.094831 (+173223us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:33.159806 (+ 64975us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:33.281171 (+121365us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1476388}
I20250113 02:51:33.284744 1196 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37366 (request call id 776) took 1637 ms. Trace:
I20250113 02:51:33.285094 1196 rpcz_store.cc:276] 0113 02:51:31.647787 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:31.647867 (+ 80us) service_pool.cc:225] Handling call
0113 02:51:33.284603 (+1636736us) ts_tablet_manager.cc:1145] Deleting tablet 88a776aa820c44b3a3854ca5022bd353
0113 02:51:33.284616 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:33.284724 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1636475}
I20250113 02:51:33.285557 1196 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37366 (request call id 777) took 1637 ms. Trace:
I20250113 02:51:33.285676 1196 rpcz_store.cc:276] 0113 02:51:31.648480 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:31.648533 (+ 53us) service_pool.cc:225] Handling call
0113 02:51:33.285453 (+1636920us) ts_tablet_manager.cc:1145] Deleting tablet 1d15c7a01f854f0aaabcd30ec3166e31
0113 02:51:33.285461 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:33.285543 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1636698}
I20250113 02:51:33.285826 1191 tablet_bootstrap.cc:492] T 42cf1e708644427f9d4dcd41ae2bcb37 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:33.285951 1191 ts_tablet_manager.cc:1397] T 42cf1e708644427f9d4dcd41ae2bcb37 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.048s user 0.000s sys 0.004s
I20250113 02:51:33.286125 1196 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37366 (request call id 783) took 1489 ms. Trace:
I20250113 02:51:33.286242 1196 rpcz_store.cc:276] 0113 02:51:31.797007 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:31.797448 (+ 441us) service_pool.cc:225] Handling call
0113 02:51:33.285981 (+1488533us) ts_tablet_manager.cc:1145] Deleting tablet f4379f7fc6b54e6785f27975d9030cce
0113 02:51:33.285989 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:33.286111 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1488359}
I20250113 02:51:33.282562 1192 tablet_bootstrap.cc:492] T 6e1284c8aaba44b5810b0d06d36757eb P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:33.286445 1192 ts_tablet_manager.cc:1397] T 6e1284c8aaba44b5810b0d06d36757eb P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.001s
I20250113 02:51:33.286639 1196 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37366 (request call id 789) took 1413 ms. Trace:
I20250113 02:51:33.286747 1196 rpcz_store.cc:276] 0113 02:51:31.873005 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:31.927479 (+ 54474us) service_pool.cc:225] Handling call
0113 02:51:33.286500 (+1359021us) ts_tablet_manager.cc:1145] Deleting tablet 883027967e0c442a91ac820210b9c6c7
0113 02:51:33.286554 (+ 54us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:33.286624 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1358879}
I20250113 02:51:33.286835 1191 raft_consensus.cc:357] T 42cf1e708644427f9d4dcd41ae2bcb37 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:33.287052 1191 raft_consensus.cc:383] T 42cf1e708644427f9d4dcd41ae2bcb37 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:33.287163 1196 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37366 (request call id 790) took 1414 ms. Trace:
I20250113 02:51:33.287204 1191 raft_consensus.cc:738] T 42cf1e708644427f9d4dcd41ae2bcb37 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:33.287331 1196 rpcz_store.cc:276] 0113 02:51:31.873094 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:31.927627 (+ 54533us) service_pool.cc:225] Handling call
0113 02:51:33.287069 (+1359442us) ts_tablet_manager.cc:1145] Deleting tablet 16a132384b0e4264a73d09a3ea133893
0113 02:51:33.287079 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:33.287149 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1359323}
I20250113 02:51:33.287432 1191 consensus_queue.cc:260] T 42cf1e708644427f9d4dcd41ae2bcb37 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:33.287793 1196 tablet_replica.cc:331] T c215ffe61b9b426286cc5e808463078d P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:33.288030 1191 ts_tablet_manager.cc:1428] T 42cf1e708644427f9d4dcd41ae2bcb37 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250113 02:51:33.288165 1196 raft_consensus.cc:2238] T c215ffe61b9b426286cc5e808463078d P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:33.288288 1191 tablet_bootstrap.cc:492] T 0916b41c171a419791797559fbc2807d P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:33.288198 1192 raft_consensus.cc:357] T 6e1284c8aaba44b5810b0d06d36757eb P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:33.288357 1192 raft_consensus.cc:383] T 6e1284c8aaba44b5810b0d06d36757eb P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:33.288415 1192 raft_consensus.cc:738] T 6e1284c8aaba44b5810b0d06d36757eb P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:33.288628 1192 consensus_queue.cc:260] T 6e1284c8aaba44b5810b0d06d36757eb P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:33.289208 1192 ts_tablet_manager.cc:1428] T 6e1284c8aaba44b5810b0d06d36757eb P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250113 02:51:33.289819 1191 tablet_bootstrap.cc:654] T 0916b41c171a419791797559fbc2807d P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:33.295363 1194 tablet_bootstrap.cc:492] T 2778d7dc7d9c4843bbc9eb705cec71cd P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:33.295360 1198 ts_tablet_manager.cc:1918] T 092892a36e0e47cb8d2184c2bbde95d7 P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:33.295500 1194 ts_tablet_manager.cc:1397] T 2778d7dc7d9c4843bbc9eb705cec71cd P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.142s user 0.004s sys 0.001s
I20250113 02:51:33.297679 1192 tablet_bootstrap.cc:492] T c7243d27a1c642269a3d0e67632e458c P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:33.299804 1192 tablet_bootstrap.cc:654] T c7243d27a1c642269a3d0e67632e458c P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
W20250113 02:51:33.301249 451 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fe8f075d4c9a4f9abe372bfdb81a829c consensus state has no leader
I20250113 02:51:33.307691 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_09819d1f01c94668aba0478d44f251f8" } modify_external_catalogs: true
I20250113 02:51:33.307868 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_09819d1f01c94668aba0478d44f251f8" } modify_external_catalogs: true
I20250113 02:51:33.321820 1192 tablet_bootstrap.cc:492] T c7243d27a1c642269a3d0e67632e458c P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:33.321975 1192 ts_tablet_manager.cc:1397] T c7243d27a1c642269a3d0e67632e458c P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.024s user 0.003s sys 0.000s
I20250113 02:51:33.322695 1192 raft_consensus.cc:357] T c7243d27a1c642269a3d0e67632e458c P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:33.322893 1192 raft_consensus.cc:383] T c7243d27a1c642269a3d0e67632e458c P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:33.322953 1192 raft_consensus.cc:738] T c7243d27a1c642269a3d0e67632e458c P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:33.323191 1192 consensus_queue.cc:260] T c7243d27a1c642269a3d0e67632e458c P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:33.326097 453 master_service.cc:946] ReplaceTablet: received request to replace tablet 19f14b3db0b64847b5422bfd5ed323d0 from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:33.328222 1192 ts_tablet_manager.cc:1428] T c7243d27a1c642269a3d0e67632e458c P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250113 02:51:33.330361 1192 tablet_bootstrap.cc:492] T 99c8046f5d124ca49899df365114c69f P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:33.333760 1192 tablet_bootstrap.cc:654] T 99c8046f5d124ca49899df365114c69f P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:33.335186 1195 ts_tablet_manager.cc:1918] T c7243d27a1c642269a3d0e67632e458c P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:33.336143 883 tablet_service.cc:1514] Processing DeleteTablet for tablet ddf79eec41214693859f526265a55492 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:23 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:33.336728 883 tablet_service.cc:1514] Processing DeleteTablet for tablet 9fff52e99c64462a8a73fccc3c22ca63 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:33 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:33.337149 883 tablet_service.cc:1514] Processing DeleteTablet for tablet 89b6fbf30ca44488ad7969930f1356e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:33 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:33.337419 1198 log.cc:1198] T 092892a36e0e47cb8d2184c2bbde95d7 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/092892a36e0e47cb8d2184c2bbde95d7
I20250113 02:51:33.337549 883 tablet_service.cc:1514] Processing DeleteTablet for tablet 8adfa738c17a4169a5d99f6ceca75100 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:32 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:33.338196 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0752fde3a30a4947a2a688c56ceadab5" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:33.338492 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet 19f14b3db0b64847b5422bfd5ed323d0 (table default.Table_c7093afcc0ae47c78e7c5cf85219481c [id=141489c57ce44e96a33a1140a8c2b077]) successfully deleted
I20250113 02:51:33.338722 1198 ts_tablet_manager.cc:1939] T 092892a36e0e47cb8d2184c2bbde95d7 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
W20250113 02:51:33.339141 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 88a776aa820c44b3a3854ca5022bd353 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 88a776aa820c44b3a3854ca5022bd353 already in progress: creating tablet
I20250113 02:51:33.338163 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0752fde3a30a4947a2a688c56ceadab5" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f"
I20250113 02:51:33.339452 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 779) took 2036 ms. Trace:
I20250113 02:51:33.339620 1195 log.cc:1198] T c7243d27a1c642269a3d0e67632e458c P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/c7243d27a1c642269a3d0e67632e458c
W20250113 02:51:33.339732 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 1d15c7a01f854f0aaabcd30ec3166e31 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1d15c7a01f854f0aaabcd30ec3166e31 already in progress: creating tablet
I20250113 02:51:33.339872 914 raft_consensus.cc:3054] T 401c4f5dbd304950877b59a865f417b3 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:33.340008 1195 ts_tablet_manager.cc:1939] T c7243d27a1c642269a3d0e67632e458c P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
W20250113 02:51:33.340323 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet f4379f7fc6b54e6785f27975d9030cce because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f4379f7fc6b54e6785f27975d9030cce already in progress: creating tablet
W20250113 02:51:33.340809 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 883027967e0c442a91ac820210b9c6c7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 883027967e0c442a91ac820210b9c6c7 already in progress: creating tablet
W20250113 02:51:33.341249 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 16a132384b0e4264a73d09a3ea133893 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 16a132384b0e4264a73d09a3ea133893 already in progress: creating tablet
I20250113 02:51:33.342113 1196 raft_consensus.cc:2267] T c215ffe61b9b426286cc5e808463078d P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:33.342172 2024 raft_consensus.cc:491] T 6d63f0e5835540a78c82020b20a07dac P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:33.342299 2024 raft_consensus.cc:513] T 6d63f0e5835540a78c82020b20a07dac P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:33.342693 1196 ts_tablet_manager.cc:1905] T c215ffe61b9b426286cc5e808463078d P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:33.342870 2024 leader_election.cc:290] T 6d63f0e5835540a78c82020b20a07dac P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587), 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
I20250113 02:51:33.343343 1194 raft_consensus.cc:357] T 2778d7dc7d9c4843bbc9eb705cec71cd P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:33.343489 1194 raft_consensus.cc:383] T 2778d7dc7d9c4843bbc9eb705cec71cd P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:33.343529 1194 raft_consensus.cc:738] T 2778d7dc7d9c4843bbc9eb705cec71cd P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:33.343699 1194 consensus_queue.cc:260] T 2778d7dc7d9c4843bbc9eb705cec71cd P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:33.344166 1194 ts_tablet_manager.cc:1428] T 2778d7dc7d9c4843bbc9eb705cec71cd P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.049s user 0.000s sys 0.002s
I20250113 02:51:33.344401 1194 tablet_bootstrap.cc:492] T a77c38a394114d6c8a8da99d7eef7669 P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:33.345739 1194 tablet_bootstrap.cc:654] T a77c38a394114d6c8a8da99d7eef7669 P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
W20250113 02:51:33.341734 998 leader_election.cc:343] T 0752fde3a30a4947a2a688c56ceadab5 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:33.346745 1198 rpcz_store.cc:276] 0113 02:51:31.302909 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:31.303016 (+ 107us) service_pool.cc:225] Handling call
0113 02:51:33.278633 (+1975617us) ts_tablet_manager.cc:1145] Deleting tablet 092892a36e0e47cb8d2184c2bbde95d7
0113 02:51:33.278643 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:33.292543 (+ 13900us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:33.295282 (+ 2739us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:33.339432 (+ 44150us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1951109}
W20250113 02:51:33.347214 998 leader_election.cc:343] T 0752fde3a30a4947a2a688c56ceadab5 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:33.347303 1198 tablet_replica.cc:331] T 840773ee80c7447097ea000b106fc89d P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:33.347492 1198 raft_consensus.cc:2238] T 840773ee80c7447097ea000b106fc89d P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:33.347627 1198 raft_consensus.cc:2267] T 840773ee80c7447097ea000b106fc89d P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:33.347589 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet 092892a36e0e47cb8d2184c2bbde95d7 (table default.Table_03fafac4f68e4eb3be2ea41493d9e006 [id=d4bf205fb0464358af61bb3fb889c687]) successfully deleted
I20250113 02:51:33.348160 1198 ts_tablet_manager.cc:1905] T 840773ee80c7447097ea000b106fc89d P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:33.340905 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet c7243d27a1c642269a3d0e67632e458c (table default.Table_7157800e639b45d8bb947e53d4319cfc [id=39997cddb4684b6c984eeeb48d715002]) successfully deleted
I20250113 02:51:33.349893 1195 tablet_replica.cc:331] T e468b652f71f44e3945f8d919ac0e1a8 P 1186fa869eb14812832fa6562531f196: stopping tablet replica
W20250113 02:51:33.350066 851 leader_election.cc:343] T 6d63f0e5835540a78c82020b20a07dac P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Not found: Tablet not found: 6d63f0e5835540a78c82020b20a07dac
I20250113 02:51:33.350158 1195 raft_consensus.cc:2238] T e468b652f71f44e3945f8d919ac0e1a8 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:33.350306 1195 raft_consensus.cc:2267] T e468b652f71f44e3945f8d919ac0e1a8 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250113 02:51:33.350476 852 leader_election.cc:343] T 6d63f0e5835540a78c82020b20a07dac P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Not found: Tablet not found: 6d63f0e5835540a78c82020b20a07dac
I20250113 02:51:33.350574 852 leader_election.cc:304] T 6d63f0e5835540a78c82020b20a07dac P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df69f48656b44bb890f76587ee69cc0f; no voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca
I20250113 02:51:33.351053 2024 raft_consensus.cc:2743] T 6d63f0e5835540a78c82020b20a07dac P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:33.352631 1196 ts_tablet_manager.cc:1918] T c215ffe61b9b426286cc5e808463078d P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:33.360409 1195 ts_tablet_manager.cc:1905] T e468b652f71f44e3945f8d919ac0e1a8 P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:33.362138 1191 tablet_bootstrap.cc:492] T 0916b41c171a419791797559fbc2807d P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:33.362308 1191 ts_tablet_manager.cc:1397] T 0916b41c171a419791797559fbc2807d P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.074s user 0.004s sys 0.021s
I20250113 02:51:33.363270 1191 raft_consensus.cc:357] T 0916b41c171a419791797559fbc2807d P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:33.363523 1191 raft_consensus.cc:383] T 0916b41c171a419791797559fbc2807d P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:33.363593 1191 raft_consensus.cc:738] T 0916b41c171a419791797559fbc2807d P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:33.365478 1192 tablet_bootstrap.cc:492] T 99c8046f5d124ca49899df365114c69f P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:33.365607 1192 ts_tablet_manager.cc:1397] T 99c8046f5d124ca49899df365114c69f P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.035s user 0.003s sys 0.000s
I20250113 02:51:33.366281 1192 raft_consensus.cc:357] T 99c8046f5d124ca49899df365114c69f P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:33.366477 1192 raft_consensus.cc:383] T 99c8046f5d124ca49899df365114c69f P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:33.366536 1192 raft_consensus.cc:738] T 99c8046f5d124ca49899df365114c69f P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:33.366720 1192 consensus_queue.cc:260] T 99c8046f5d124ca49899df365114c69f P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:33.367187 1192 ts_tablet_manager.cc:1428] T 99c8046f5d124ca49899df365114c69f P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250113 02:51:33.371639 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_14ab7cebe87e40b4bf54f68ee6a1ab80" } new_table_name: "default.Table_f2c94bca6ad3435e9d28d82c98214064" modify_external_catalogs: true
I20250113 02:51:33.371663 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_65417a56ebb74c158456a735cfd14cc2" } new_table_name: "default.Table_8174140b535e44008667efc3800186da" modify_external_catalogs: true
I20250113 02:51:33.374166 448 catalog_manager.cc:5618] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095) does not have the latest schema for tablet 377ebf66bbef40988043f6200ae0893c (table default.Table_5ddfdff2cc8c4d12a1c4552e9a302027 [id=8b35e389de9f4917904e87b43618737e]). Expected version 1 got 0
W20250113 02:51:33.374601 421 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.104.2:38467, user_credentials={real_user=slave}} blocked reactor thread for 61285.7us
I20250113 02:51:33.375144 1198 ts_tablet_manager.cc:1918] T 840773ee80c7447097ea000b106fc89d P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:33.375921 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_14aabaa1901741fe8c6f84fd8a397d10"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:33.376447 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_14aabaa1901741fe8c6f84fd8a397d10 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:33.377804 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_a969335ba8de47a49dc4629865ee4a1a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:33.378240 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a969335ba8de47a49dc4629865ee4a1a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:33.379431 751 tablet_service.cc:1514] Processing DeleteTablet for tablet f56230a2c75d4763a54671c79358f4d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:22 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:33.379652 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 12a28557b0cc42fdad927c86fcfa7b96
I20250113 02:51:33.380688 751 tablet_service.cc:1514] Processing DeleteTablet for tablet 12a28557b0cc42fdad927c86fcfa7b96 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:33 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:33.381861 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_aafae7a2874249e7ac84dcde865d33cf" } new_table_name: "default.Table_f5951a8a804b4962bab277859f323298" modify_external_catalogs: true
W20250113 02:51:33.384239 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f1237f7b04af4ebbb39b03aa3c78a499 consensus state has no leader
W20250113 02:51:33.384682 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet dff74d7c093b45eba7a47d8eceb0c1d1 consensus state has no leader
I20250113 02:51:33.385007 1195 ts_tablet_manager.cc:1918] T e468b652f71f44e3945f8d919ac0e1a8 P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:33.385730 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_787937b2aa714be5b37053648c4240fe"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:33.386184 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_787937b2aa714be5b37053648c4240fe in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:33.395650 1195 log.cc:1198] T e468b652f71f44e3945f8d919ac0e1a8 P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/e468b652f71f44e3945f8d919ac0e1a8
I20250113 02:51:33.380137 1192 tablet_bootstrap.cc:492] T 51d9483ecacb413796d0fd9e7560eac1 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:33.381003 1041 tablet_service.cc:1514] Processing DeleteTablet for tablet 12a28557b0cc42fdad927c86fcfa7b96 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:33 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:33.395951 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 7b16286181684916ac84823f5fec563e
I20250113 02:51:33.397678 1041 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b16286181684916ac84823f5fec563e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:33 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:33.398185 1192 tablet_bootstrap.cc:654] T 51d9483ecacb413796d0fd9e7560eac1 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:33.398679 455 catalog_manager.cc:5618] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587) does not have the latest schema for tablet 1a63f485caf54068abd6b50d7de3d14a (table default.Table_f8a6812c7fb94ea69cd9bf775b914ea6 [id=71508871e2d5469a91aaf0e19b02b638]). Expected version 1 got 0
I20250113 02:51:33.399078 1198 log.cc:1198] T 840773ee80c7447097ea000b106fc89d P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/840773ee80c7447097ea000b106fc89d
I20250113 02:51:33.398964 455 catalog_manager.cc:5526] T c508724edea147e5bf6e20bca2f50397 P 4216a7f853d642b7bc3e5cff458f3aca reported cstate change: term changed from 0 to 1, leader changed from <none> to 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3). New cstate: current_term: 1 leader_uuid: "4216a7f853d642b7bc3e5cff458f3aca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } health_report { overall_health: UNKNOWN } } }
I20250113 02:51:33.399395 1198 ts_tablet_manager.cc:1939] T 840773ee80c7447097ea000b106fc89d P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:33.399385 455 catalog_manager.cc:5618] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587) does not have the latest schema for tablet c508724edea147e5bf6e20bca2f50397 (table default.Table_810d62357c794aeaa18da712725a338f [id=08db71e435a24e36bdb20f9a5f1cc096]). Expected version 3 got 0
I20250113 02:51:33.397919 751 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b16286181684916ac84823f5fec563e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:33 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:33.361219 1194 tablet_bootstrap.cc:492] T a77c38a394114d6c8a8da99d7eef7669 P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:33.364653 916 raft_consensus.cc:3054] T 32cba8f3efc54e78b4df41c15df82eea P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:33.381047 883 tablet_service.cc:1514] Processing DeleteTablet for tablet 12a28557b0cc42fdad927c86fcfa7b96 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:33 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:33.400120 1195 ts_tablet_manager.cc:1939] T e468b652f71f44e3945f8d919ac0e1a8 P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:33.400148 1194 ts_tablet_manager.cc:1397] T a77c38a394114d6c8a8da99d7eef7669 P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.056s user 0.002s sys 0.006s
I20250113 02:51:33.399772 455 catalog_manager.cc:5526] T 0752fde3a30a4947a2a688c56ceadab5 P 4216a7f853d642b7bc3e5cff458f3aca reported cstate change: term changed from 0 to 1, leader changed from <none> to 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3). New cstate: current_term: 1 leader_uuid: "4216a7f853d642b7bc3e5cff458f3aca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } health_report { overall_health: UNKNOWN } } }
I20250113 02:51:33.400786 455 catalog_manager.cc:5618] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587) does not have the latest schema for tablet 0752fde3a30a4947a2a688c56ceadab5 (table default.Table_da3e49fa1e654145a05858466ec2260f [id=94704429b49b46e38a48258a450d2705]). Expected version 1 got 0
I20250113 02:51:33.400941 1195 tablet_replica.cc:331] T 99c8046f5d124ca49899df365114c69f P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:33.400957 1194 raft_consensus.cc:357] T a77c38a394114d6c8a8da99d7eef7669 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:33.401099 1195 raft_consensus.cc:2238] T 99c8046f5d124ca49899df365114c69f P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:33.401147 1194 raft_consensus.cc:383] T a77c38a394114d6c8a8da99d7eef7669 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:33.401202 1194 raft_consensus.cc:738] T a77c38a394114d6c8a8da99d7eef7669 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:33.401224 1195 raft_consensus.cc:2267] T 99c8046f5d124ca49899df365114c69f P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:33.401405 1194 consensus_queue.cc:260] T a77c38a394114d6c8a8da99d7eef7669 P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:33.393010 2092 raft_consensus.cc:491] T ddf79eec41214693859f526265a55492 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:33.399129 885 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b16286181684916ac84823f5fec563e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:33 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:33.401917 2092 raft_consensus.cc:513] T ddf79eec41214693859f526265a55492 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:33.402168 1194 ts_tablet_manager.cc:1428] T a77c38a394114d6c8a8da99d7eef7669 P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250113 02:51:33.402422 1194 tablet_bootstrap.cc:492] T 378d59d314a84ddb84cf2ab041c51cf1 P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:33.403216 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 780) took 2100 ms. Trace:
I20250113 02:51:33.403321 1198 rpcz_store.cc:276] 0113 02:51:31.303206 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:31.303253 (+ 47us) service_pool.cc:225] Handling call
0113 02:51:33.347254 (+2044001us) ts_tablet_manager.cc:1145] Deleting tablet 840773ee80c7447097ea000b106fc89d
0113 02:51:33.347265 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:33.372275 (+ 25010us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:33.375055 (+ 2780us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:33.403173 (+ 28118us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2019686}
W20250113 02:51:33.403681 852 leader_election.cc:343] T ddf79eec41214693859f526265a55492 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Not found: Tablet not found: ddf79eec41214693859f526265a55492
I20250113 02:51:33.404078 1194 tablet_bootstrap.cc:654] T 378d59d314a84ddb84cf2ab041c51cf1 P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:33.404011 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet e468b652f71f44e3945f8d919ac0e1a8 (table default.Table_c52b68d0d3d74003bf10bc8ff205d25b [id=caf9972c69f14619be22fb3a9e563b05]) successfully deleted
I20250113 02:51:33.404301 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet 840773ee80c7447097ea000b106fc89d (table default.Table_5aafd46e1cbf4e03a5b87797d5d97bdd [id=b6d8d0de849a4d3c988b8fd468b14d62]) successfully deleted
W20250113 02:51:33.404808 851 leader_election.cc:343] T ddf79eec41214693859f526265a55492 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Not found: Tablet not found: ddf79eec41214693859f526265a55492
I20250113 02:51:33.404942 1196 log.cc:1198] T c215ffe61b9b426286cc5e808463078d P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/c215ffe61b9b426286cc5e808463078d
I20250113 02:51:33.404910 851 leader_election.cc:304] T ddf79eec41214693859f526265a55492 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df69f48656b44bb890f76587ee69cc0f; no voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca
I20250113 02:51:33.405153 2092 leader_election.cc:290] T ddf79eec41214693859f526265a55492 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587), 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
I20250113 02:51:33.405256 1196 ts_tablet_manager.cc:1939] T c215ffe61b9b426286cc5e808463078d P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:33.405225 2024 raft_consensus.cc:2743] T ddf79eec41214693859f526265a55492 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:33.405597 1196 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37366 (request call id 791) took 1375 ms. Trace:
I20250113 02:51:33.405654 1198 tablet_replica.cc:331] T ff8ef6290698477391721fd65b9eeb28 P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:33.405663 1196 rpcz_store.cc:276] 0113 02:51:32.029847 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:32.029895 (+ 48us) service_pool.cc:225] Handling call
0113 02:51:33.287736 (+1257841us) ts_tablet_manager.cc:1145] Deleting tablet c215ffe61b9b426286cc5e808463078d
0113 02:51:33.287745 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:33.346230 (+ 58485us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:33.352546 (+ 6316us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:33.405580 (+ 53034us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1257667}
I20250113 02:51:33.405877 1198 raft_consensus.cc:2238] T ff8ef6290698477391721fd65b9eeb28 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:33.406057 1198 raft_consensus.cc:2267] T ff8ef6290698477391721fd65b9eeb28 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:33.406059 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet c215ffe61b9b426286cc5e808463078d (table default.Table_d4146353239a45b281b482d6e9537502 [id=20b69dfe61d841829e1804153c0d0184]) successfully deleted
I20250113 02:51:33.406630 1198 ts_tablet_manager.cc:1905] T ff8ef6290698477391721fd65b9eeb28 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:33.407264 885 tablet_service.cc:1467] Processing CreateTablet for tablet 746abec26dac48eb8831f62477ad3a2c (DEFAULT_TABLE table=default.Table_3e6f08e022664e2db587347cf97cf0a9 [id=11b2bac0ce65435b8074bd5031ce259c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:33.407517 1041 tablet_service.cc:1467] Processing CreateTablet for tablet 746abec26dac48eb8831f62477ad3a2c (DEFAULT_TABLE table=default.Table_3e6f08e022664e2db587347cf97cf0a9 [id=11b2bac0ce65435b8074bd5031ce259c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:33.407611 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 746abec26dac48eb8831f62477ad3a2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:33.407840 1195 ts_tablet_manager.cc:1905] T 99c8046f5d124ca49899df365114c69f P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:33.363829 1191 consensus_queue.cc:260] T 0916b41c171a419791797559fbc2807d P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:33.408531 751 tablet_service.cc:1467] Processing CreateTablet for tablet 746abec26dac48eb8831f62477ad3a2c (DEFAULT_TABLE table=default.Table_3e6f08e022664e2db587347cf97cf0a9 [id=11b2bac0ce65435b8074bd5031ce259c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:33.408877 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 746abec26dac48eb8831f62477ad3a2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:33.408991 1191 ts_tablet_manager.cc:1428] T 0916b41c171a419791797559fbc2807d P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.047s user 0.003s sys 0.000s
I20250113 02:51:33.412261 744 tablet_service.cc:1467] Processing CreateTablet for tablet cee1d2c50a6145fda7f258bbd080a3c8 (DEFAULT_TABLE table=default.Table_0e516bc3e1f14f3ca9867668dd60d73a [id=2cd331302a6d45eb82d654231685e2ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:33.412959 1039 tablet_service.cc:1467] Processing CreateTablet for tablet cee1d2c50a6145fda7f258bbd080a3c8 (DEFAULT_TABLE table=default.Table_0e516bc3e1f14f3ca9867668dd60d73a [id=2cd331302a6d45eb82d654231685e2ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:33.415403 883 tablet_service.cc:1467] Processing CreateTablet for tablet ac47ba88e3d54c5c8d3298777af4c537 (DEFAULT_TABLE table=default.Table_bbde1d1575c74204b7304a1cbe946c76 [id=8cc99e7c81954c77b0b15179243eb4a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:33.415701 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac47ba88e3d54c5c8d3298777af4c537. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:33.416078 1191 tablet_bootstrap.cc:492] T 56b022c94847411785a7e9db3757a2d9 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:33.417515 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cee1d2c50a6145fda7f258bbd080a3c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:33.418037 1191 tablet_bootstrap.cc:654] T 56b022c94847411785a7e9db3757a2d9 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:33.420133 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 746abec26dac48eb8831f62477ad3a2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:33.420198 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cee1d2c50a6145fda7f258bbd080a3c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:33.427573 885 tablet_service.cc:1467] Processing CreateTablet for tablet cee1d2c50a6145fda7f258bbd080a3c8 (DEFAULT_TABLE table=default.Table_0e516bc3e1f14f3ca9867668dd60d73a [id=2cd331302a6d45eb82d654231685e2ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:33.428303 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cee1d2c50a6145fda7f258bbd080a3c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:33.429167 1192 tablet_bootstrap.cc:492] T 51d9483ecacb413796d0fd9e7560eac1 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:33.429328 1192 ts_tablet_manager.cc:1397] T 51d9483ecacb413796d0fd9e7560eac1 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.049s user 0.006s sys 0.000s
I20250113 02:51:33.430123 1192 raft_consensus.cc:357] T 51d9483ecacb413796d0fd9e7560eac1 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:33.430168 883 tablet_service.cc:1467] Processing CreateTablet for tablet ec06c9f6a3bf46a9bcb90a85a82b6ff8 (DEFAULT_TABLE table=default.Table_17633452a13c4859971518af3e51e69e [id=22b348b6392f4e2cbee3a33f5c671ca0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:33.430320 1192 raft_consensus.cc:383] T 51d9483ecacb413796d0fd9e7560eac1 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:33.430405 1192 raft_consensus.cc:738] T 51d9483ecacb413796d0fd9e7560eac1 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:33.430442 1026 tablet_service.cc:1467] Processing CreateTablet for tablet ec06c9f6a3bf46a9bcb90a85a82b6ff8 (DEFAULT_TABLE table=default.Table_17633452a13c4859971518af3e51e69e [id=22b348b6392f4e2cbee3a33f5c671ca0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:33.430682 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ec06c9f6a3bf46a9bcb90a85a82b6ff8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:33.430655 1192 consensus_queue.cc:260] T 51d9483ecacb413796d0fd9e7560eac1 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:33.431136 1192 ts_tablet_manager.cc:1428] T 51d9483ecacb413796d0fd9e7560eac1 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250113 02:51:33.432256 744 tablet_service.cc:1467] Processing CreateTablet for tablet ac47ba88e3d54c5c8d3298777af4c537 (DEFAULT_TABLE table=default.Table_bbde1d1575c74204b7304a1cbe946c76 [id=8cc99e7c81954c77b0b15179243eb4a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:33.432718 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac47ba88e3d54c5c8d3298777af4c537. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:33.433197 1192 tablet_bootstrap.cc:492] T 0539cbbe47e64c039ad39c4b52edcc4b P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:33.435488 1192 tablet_bootstrap.cc:654] T 0539cbbe47e64c039ad39c4b52edcc4b P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:33.443415 1192 tablet_bootstrap.cc:492] T 0539cbbe47e64c039ad39c4b52edcc4b P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:33.443562 1192 ts_tablet_manager.cc:1397] T 0539cbbe47e64c039ad39c4b52edcc4b P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.000s
I20250113 02:51:33.443492 454 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_bbd4259916374725a0e633d94ed73eb4" } new_table_name: "default.Table_ebe471b611d9404a922f0e404a6ea4ee" modify_external_catalogs: true
I20250113 02:51:33.443924 892 tablet_service.cc:1467] Processing CreateTablet for tablet 797d91d4aae240bd82b00021717772ae (DEFAULT_TABLE table=default.Table_3935379456cf4f08b2e36f4ba95af912 [id=5d873b4a961942a49d21be7729114c0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:33.444332 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 797d91d4aae240bd82b00021717772ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:33.444375 1192 raft_consensus.cc:357] T 0539cbbe47e64c039ad39c4b52edcc4b P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:33.444553 1192 raft_consensus.cc:383] T 0539cbbe47e64c039ad39c4b52edcc4b P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:33.444613 1192 raft_consensus.cc:738] T 0539cbbe47e64c039ad39c4b52edcc4b P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:33.444605 747 tablet_service.cc:1467] Processing CreateTablet for tablet ec06c9f6a3bf46a9bcb90a85a82b6ff8 (DEFAULT_TABLE table=default.Table_17633452a13c4859971518af3e51e69e [id=22b348b6392f4e2cbee3a33f5c671ca0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:33.444798 1192 consensus_queue.cc:260] T 0539cbbe47e64c039ad39c4b52edcc4b P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:33.444949 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ec06c9f6a3bf46a9bcb90a85a82b6ff8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:33.445292 1192 ts_tablet_manager.cc:1428] T 0539cbbe47e64c039ad39c4b52edcc4b P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250113 02:51:33.446532 1196 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37366 (request call id 818) took 1319 ms. Trace:
I20250113 02:51:33.446646 1196 rpcz_store.cc:276] 0113 02:51:32.127116 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:32.222492 (+ 95376us) service_pool.cc:225] Handling call
0113 02:51:33.446417 (+1223925us) ts_tablet_manager.cc:1145] Deleting tablet e266a9083e2f46a0a43861b7708dcdd5
0113 02:51:33.446426 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:33.446517 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1223763}
W20250113 02:51:33.447144 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet e266a9083e2f46a0a43861b7708dcdd5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e266a9083e2f46a0a43861b7708dcdd5 already in progress: creating tablet
I20250113 02:51:33.449112 1196 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37366 (request call id 819) took 1321 ms. Trace:
I20250113 02:51:33.449774 1196 rpcz_store.cc:276] 0113 02:51:32.127178 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:32.222670 (+ 95492us) service_pool.cc:225] Handling call
0113 02:51:33.449010 (+1226340us) ts_tablet_manager.cc:1145] Deleting tablet cfd1697a03494ad2aaa41a056e334ca7
0113 02:51:33.449019 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:33.449097 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1226235}
W20250113 02:51:33.450237 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet cfd1697a03494ad2aaa41a056e334ca7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cfd1697a03494ad2aaa41a056e334ca7 already in progress: creating tablet
I20250113 02:51:33.450626 1196 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37366 (request call id 822) took 1163 ms. Trace:
I20250113 02:51:33.451714 1926 raft_consensus.cc:491] T 99c8046f5d124ca49899df365114c69f P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:33.451834 1926 raft_consensus.cc:513] T 99c8046f5d124ca49899df365114c69f P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:33.451678 1196 rpcz_store.cc:276] 0113 02:51:32.286965 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:32.298043 (+ 11078us) service_pool.cc:225] Handling call
0113 02:51:33.450540 (+1152497us) ts_tablet_manager.cc:1145] Deleting tablet 87e6a4e4a8434bbcbbe70ddf634accba
0113 02:51:33.450548 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:33.450613 (+ 65us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1152299}
I20250113 02:51:33.452497 1926 leader_election.cc:290] T 99c8046f5d124ca49899df365114c69f P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:33.452759 1195 ts_tablet_manager.cc:1918] T 99c8046f5d124ca49899df365114c69f P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:33.452726 1041 tablet_service.cc:1467] Processing CreateTablet for tablet 797d91d4aae240bd82b00021717772ae (DEFAULT_TABLE table=default.Table_3935379456cf4f08b2e36f4ba95af912 [id=5d873b4a961942a49d21be7729114c0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:33.453039 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 797d91d4aae240bd82b00021717772ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:33.452944 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99c8046f5d124ca49899df365114c69f" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:33.453733 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 87e6a4e4a8434bbcbbe70ddf634accba because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 87e6a4e4a8434bbcbbe70ddf634accba already in progress: creating tablet
I20250113 02:51:33.454154 1195 log.cc:1198] T 99c8046f5d124ca49899df365114c69f P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/99c8046f5d124ca49899df365114c69f
W20250113 02:51:33.454516 998 leader_election.cc:343] T 99c8046f5d124ca49899df365114c69f P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:33.454551 1195 ts_tablet_manager.cc:1939] T 99c8046f5d124ca49899df365114c69f P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:33.454950 1196 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37366 (request call id 820) took 1174 ms. Trace:
I20250113 02:51:33.455309 1195 tablet_replica.cc:331] T 0539cbbe47e64c039ad39c4b52edcc4b P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:33.456018 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet 99c8046f5d124ca49899df365114c69f (table default.Table_7805d72c367b4ef8aa585de05f3d9b90 [id=d4ce999a9a7b47ada946f259228771bb]) successfully deleted
W20250113 02:51:33.456530 995 leader_election.cc:343] T 99c8046f5d124ca49899df365114c69f P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Not found: Tablet not found: 99c8046f5d124ca49899df365114c69f
I20250113 02:51:33.456635 995 leader_election.cc:304] T 99c8046f5d124ca49899df365114c69f P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4216a7f853d642b7bc3e5cff458f3aca; no voters: 1186fa869eb14812832fa6562531f196, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:33.457221 1196 rpcz_store.cc:276] 0113 02:51:32.280956 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:32.301967 (+ 21011us) service_pool.cc:225] Handling call
0113 02:51:33.454852 (+1152885us) ts_tablet_manager.cc:1145] Deleting tablet cd6e48a720d549f2b5f30a61e1f5e017
0113 02:51:33.454861 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:33.454937 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1152729}
W20250113 02:51:33.457576 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet cd6e48a720d549f2b5f30a61e1f5e017 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cd6e48a720d549f2b5f30a61e1f5e017 already in progress: creating tablet
I20250113 02:51:33.457789 1926 raft_consensus.cc:2743] T 99c8046f5d124ca49899df365114c69f P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:33.458045 1196 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37366 (request call id 823) took 1159 ms. Trace:
I20250113 02:51:33.460808 1198 ts_tablet_manager.cc:1918] T ff8ef6290698477391721fd65b9eeb28 P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:33.461136 1023 tablet_service.cc:1467] Processing CreateTablet for tablet fb3fecc661b843b681616d78f6c2ab00 (DEFAULT_TABLE table=default.Table_c217d95911fd4ac18cb2764c65408b0b [id=0dc56f09d97e438aaffb24dec61dfd97]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:33.462579 1196 rpcz_store.cc:276] 0113 02:51:32.299084 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:32.302163 (+ 3079us) service_pool.cc:225] Handling call
0113 02:51:33.457955 (+1155792us) ts_tablet_manager.cc:1145] Deleting tablet 29b1444efd28483297bafc73a33e7a54
0113 02:51:33.457963 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:33.458030 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1155655}
I20250113 02:51:33.462883 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb3fecc661b843b681616d78f6c2ab00. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:33.463305 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 29b1444efd28483297bafc73a33e7a54 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 29b1444efd28483297bafc73a33e7a54 already in progress: creating tablet
I20250113 02:51:33.462519 1194 tablet_bootstrap.cc:492] T 378d59d314a84ddb84cf2ab041c51cf1 P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:33.462829 892 tablet_service.cc:1467] Processing CreateTablet for tablet 67fe2f52d98048dab9f37dd820b7f712 (DEFAULT_TABLE table=default.Table_f547905e49744f99810317090e10ba97 [id=91d2bcd49bd441c0bda23004e3e14ee9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:33.463579 1194 ts_tablet_manager.cc:1397] T 378d59d314a84ddb84cf2ab041c51cf1 P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.061s user 0.001s sys 0.003s
I20250113 02:51:33.463729 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 67fe2f52d98048dab9f37dd820b7f712. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:33.464381 1194 raft_consensus.cc:357] T 378d59d314a84ddb84cf2ab041c51cf1 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:33.464540 1194 raft_consensus.cc:383] T 378d59d314a84ddb84cf2ab041c51cf1 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:33.464581 1194 raft_consensus.cc:738] T 378d59d314a84ddb84cf2ab041c51cf1 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:33.464782 1926 raft_consensus.cc:491] T 0a7e7713bc8648378b668d8b769ac927 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:33.464891 1926 raft_consensus.cc:513] T 0a7e7713bc8648378b668d8b769ac927 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:33.465405 1926 leader_election.cc:290] T 0a7e7713bc8648378b668d8b769ac927 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:33.461361 883 tablet_service.cc:1467] Processing CreateTablet for tablet fb3fecc661b843b681616d78f6c2ab00 (DEFAULT_TABLE table=default.Table_c217d95911fd4ac18cb2764c65408b0b [id=0dc56f09d97e438aaffb24dec61dfd97]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:33.465947 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a7e7713bc8648378b668d8b769ac927" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:33.466332 998 leader_election.cc:343] T 0a7e7713bc8648378b668d8b769ac927 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
W20250113 02:51:33.466435 995 leader_election.cc:343] T 0a7e7713bc8648378b668d8b769ac927 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Not found: Tablet not found: 0a7e7713bc8648378b668d8b769ac927
I20250113 02:51:33.466526 995 leader_election.cc:304] T 0a7e7713bc8648378b668d8b769ac927 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4216a7f853d642b7bc3e5cff458f3aca; no voters: 1186fa869eb14812832fa6562531f196, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:33.466890 1926 raft_consensus.cc:2743] T 0a7e7713bc8648378b668d8b769ac927 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:33.466167 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb3fecc661b843b681616d78f6c2ab00. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:33.468437 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_ddf6ee80ea4048b19e7ac7de9934b274"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:33.468667 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_b8ded36063364816988685dfc77391d2" } modify_external_catalogs: true
I20250113 02:51:33.468891 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_b8ded36063364816988685dfc77391d2" } modify_external_catalogs: true
W20250113 02:51:33.469000 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ddf6ee80ea4048b19e7ac7de9934b274 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:33.462049 1107 heartbeater.cc:346] Connected to a master server at 127.0.104.60:42067
I20250113 02:51:33.470283 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_c2fca6e58b3947bcacaeec643890f1f9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:33.462332 1192 tablet_bootstrap.cc:492] T f129bb3fbbfd492d98822be0a6626e44 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:33.470651 446 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_95e76bedf8924501b57f7a26d66a9d33" } modify_external_catalogs: true
W20250113 02:51:33.470815 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c2fca6e58b3947bcacaeec643890f1f9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:33.470836 446 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_95e76bedf8924501b57f7a26d66a9d33" } modify_external_catalogs: true
I20250113 02:51:33.471305 1198 log.cc:1198] T ff8ef6290698477391721fd65b9eeb28 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/ff8ef6290698477391721fd65b9eeb28
I20250113 02:51:33.465961 1194 consensus_queue.cc:260] T 378d59d314a84ddb84cf2ab041c51cf1 P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:33.472340 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_84475daaf7544a4b83c6846dffec23e0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:33.472762 1195 raft_consensus.cc:2238] T 0539cbbe47e64c039ad39c4b52edcc4b P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus shutting down.
W20250113 02:51:33.472784 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_84475daaf7544a4b83c6846dffec23e0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:33.473016 1194 ts_tablet_manager.cc:1428] T 378d59d314a84ddb84cf2ab041c51cf1 P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.009s user 0.001s sys 0.000s
I20250113 02:51:33.473219 1198 ts_tablet_manager.cc:1939] T ff8ef6290698477391721fd65b9eeb28 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:33.473330 1194 tablet_bootstrap.cc:492] T a0ca2e3a3ce94f3aa20df64c89c7cf95 P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:33.474923 1194 tablet_bootstrap.cc:654] T a0ca2e3a3ce94f3aa20df64c89c7cf95 P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:33.466565 742 tablet_service.cc:1467] Processing CreateTablet for tablet 67fe2f52d98048dab9f37dd820b7f712 (DEFAULT_TABLE table=default.Table_f547905e49744f99810317090e10ba97 [id=91d2bcd49bd441c0bda23004e3e14ee9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:33.476965 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 67fe2f52d98048dab9f37dd820b7f712. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:33.477962 747 tablet_service.cc:1467] Processing CreateTablet for tablet fb3fecc661b843b681616d78f6c2ab00 (DEFAULT_TABLE table=default.Table_c217d95911fd4ac18cb2764c65408b0b [id=0dc56f09d97e438aaffb24dec61dfd97]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:33.478255 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb3fecc661b843b681616d78f6c2ab00. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:33.461213 744 tablet_service.cc:1467] Processing CreateTablet for tablet 797d91d4aae240bd82b00021717772ae (DEFAULT_TABLE table=default.Table_3935379456cf4f08b2e36f4ba95af912 [id=5d873b4a961942a49d21be7729114c0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:33.479421 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 797d91d4aae240bd82b00021717772ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:33.480439 746 tablet_service.cc:1467] Processing CreateTablet for tablet 106a80fd24724020bff63de662fdabb4 (DEFAULT_TABLE table=default.Table_c2aba85708b64110bc3a4254a09352d7 [id=c5aff2851977475cbc44c925ee329c8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:33.480715 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 106a80fd24724020bff63de662fdabb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:33.484678 1191 tablet_bootstrap.cc:492] T 56b022c94847411785a7e9db3757a2d9 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:33.484853 1191 ts_tablet_manager.cc:1397] T 56b022c94847411785a7e9db3757a2d9 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.069s user 0.000s sys 0.003s
I20250113 02:51:33.485639 1191 raft_consensus.cc:357] T 56b022c94847411785a7e9db3757a2d9 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:33.485888 1191 raft_consensus.cc:383] T 56b022c94847411785a7e9db3757a2d9 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:33.485986 1191 raft_consensus.cc:738] T 56b022c94847411785a7e9db3757a2d9 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:33.486212 1191 consensus_queue.cc:260] T 56b022c94847411785a7e9db3757a2d9 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:33.487476 1191 ts_tablet_manager.cc:1428] T 56b022c94847411785a7e9db3757a2d9 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250113 02:51:33.487763 1191 tablet_bootstrap.cc:492] T 999fecb3fc65466cad21cc1ff5ef2d44 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:33.489012 746 tablet_service.cc:1467] Processing CreateTablet for tablet a1d6cbfb61564e929efab798d4681409 (DEFAULT_TABLE table=default.Table_3cf8774e45444063a6da1fec1b7ed5e2 [id=2b7cd9368495489da417f617bd3be7c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:33.489310 1191 tablet_bootstrap.cc:654] T 999fecb3fc65466cad21cc1ff5ef2d44 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:33.489442 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a1d6cbfb61564e929efab798d4681409. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:33.475208 1196 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37366 (request call id 831) took 1074 ms. Trace:
I20250113 02:51:33.490938 1196 rpcz_store.cc:276] 0113 02:51:32.400689 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:32.400766 (+ 77us) service_pool.cc:225] Handling call
0113 02:51:33.475099 (+1074333us) ts_tablet_manager.cc:1145] Deleting tablet 4cbc27d8ac2a46c58bffc33d8af17be2
0113 02:51:33.475108 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:33.475191 (+ 83us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1046956}
W20250113 02:51:33.491410 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 4cbc27d8ac2a46c58bffc33d8af17be2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4cbc27d8ac2a46c58bffc33d8af17be2 already in progress: creating tablet
I20250113 02:51:33.476141 885 tablet_service.cc:1467] Processing CreateTablet for tablet 86d59e3aa9804a2cbab167d25ec86c90 (DEFAULT_TABLE table=default.Table_2bd58703f99f4f8f9945ac5026a63f2b [id=b380affa85424b9591c0d9d57da992d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:33.492014 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 86d59e3aa9804a2cbab167d25ec86c90. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:33.492288 899 tablet_service.cc:1467] Processing CreateTablet for tablet 7c443454994a47f2bfa7e2ec7ac576e0 (DEFAULT_TABLE table=default.Table_ff6df4afcf4d4f9a964d079ee24f02b5 [id=8a5901fc685345579a254a0f3af32051]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:33.492623 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c443454994a47f2bfa7e2ec7ac576e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:33.492880 1195 raft_consensus.cc:2267] T 0539cbbe47e64c039ad39c4b52edcc4b P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:33.479563 892 tablet_service.cc:1467] Processing CreateTablet for tablet 106a80fd24724020bff63de662fdabb4 (DEFAULT_TABLE table=default.Table_c2aba85708b64110bc3a4254a09352d7 [id=c5aff2851977475cbc44c925ee329c8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:33.487089 890 tablet_service.cc:1467] Processing CreateTablet for tablet a1d6cbfb61564e929efab798d4681409 (DEFAULT_TABLE table=default.Table_3cf8774e45444063a6da1fec1b7ed5e2 [id=2b7cd9368495489da417f617bd3be7c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:33.496364 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a1d6cbfb61564e929efab798d4681409. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:33.496850 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 781) took 2193 ms. Trace:
I20250113 02:51:33.496949 1198 rpcz_store.cc:276] 0113 02:51:31.303458 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:31.303503 (+ 45us) service_pool.cc:225] Handling call
0113 02:51:33.405585 (+2102082us) ts_tablet_manager.cc:1145] Deleting tablet ff8ef6290698477391721fd65b9eeb28
0113 02:51:33.405596 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:33.419262 (+ 13666us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:33.460700 (+ 41438us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:33.496830 (+ 36130us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2077955}
I20250113 02:51:33.497413 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet ff8ef6290698477391721fd65b9eeb28 (table default.Table_6c7330704479422a817543c4cbbd70d3 [id=7e176c2354964cdd8e6b724f6bb84b99]) successfully deleted
I20250113 02:51:33.493311 742 tablet_service.cc:1467] Processing CreateTablet for tablet 7c443454994a47f2bfa7e2ec7ac576e0 (DEFAULT_TABLE table=default.Table_ff6df4afcf4d4f9a964d079ee24f02b5 [id=8a5901fc685345579a254a0f3af32051]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:33.497833 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c443454994a47f2bfa7e2ec7ac576e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:33.497931 1039 tablet_service.cc:1467] Processing CreateTablet for tablet ac47ba88e3d54c5c8d3298777af4c537 (DEFAULT_TABLE table=default.Table_bbde1d1575c74204b7304a1cbe946c76 [id=8cc99e7c81954c77b0b15179243eb4a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:33.498452 751 tablet_service.cc:1467] Processing CreateTablet for tablet 86d59e3aa9804a2cbab167d25ec86c90 (DEFAULT_TABLE table=default.Table_2bd58703f99f4f8f9945ac5026a63f2b [id=b380affa85424b9591c0d9d57da992d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:33.498725 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 86d59e3aa9804a2cbab167d25ec86c90. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:33.502342 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 106a80fd24724020bff63de662fdabb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:33.503321 1036 tablet_service.cc:1467] Processing CreateTablet for tablet 7c443454994a47f2bfa7e2ec7ac576e0 (DEFAULT_TABLE table=default.Table_ff6df4afcf4d4f9a964d079ee24f02b5 [id=8a5901fc685345579a254a0f3af32051]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:33.503707 1195 ts_tablet_manager.cc:1905] T 0539cbbe47e64c039ad39c4b52edcc4b P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250113 02:51:33.504585 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6fd59b6773fd4bac863b1714f13c9184 consensus state has no leader
I20250113 02:51:33.474908 1192 tablet_bootstrap.cc:654] T f129bb3fbbfd492d98822be0a6626e44 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:33.475301 1023 tablet_service.cc:1467] Processing CreateTablet for tablet 67fe2f52d98048dab9f37dd820b7f712 (DEFAULT_TABLE table=default.Table_f547905e49744f99810317090e10ba97 [id=91d2bcd49bd441c0bda23004e3e14ee9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:33.505781 449 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_2abef959c5c34459a8dfa3034e207995" } modify_external_catalogs: true
I20250113 02:51:33.476662 1035 tablet_service.cc:1467] Processing CreateTablet for tablet 86d59e3aa9804a2cbab167d25ec86c90 (DEFAULT_TABLE table=default.Table_2bd58703f99f4f8f9945ac5026a63f2b [id=b380affa85424b9591c0d9d57da992d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:33.505982 449 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_2abef959c5c34459a8dfa3034e207995" } modify_external_catalogs: true
I20250113 02:51:33.480530 1040 tablet_service.cc:1467] Processing CreateTablet for tablet 106a80fd24724020bff63de662fdabb4 (DEFAULT_TABLE table=default.Table_c2aba85708b64110bc3a4254a09352d7 [id=c5aff2851977475cbc44c925ee329c8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:33.487753 1041 tablet_service.cc:1467] Processing CreateTablet for tablet a1d6cbfb61564e929efab798d4681409 (DEFAULT_TABLE table=default.Table_3cf8774e45444063a6da1fec1b7ed5e2 [id=2b7cd9368495489da417f617bd3be7c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:33.488361 1926 raft_consensus.cc:491] T 0539cbbe47e64c039ad39c4b52edcc4b P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:33.506460 1926 raft_consensus.cc:513] T 0539cbbe47e64c039ad39c4b52edcc4b P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:33.507237 1926 leader_election.cc:290] T 0539cbbe47e64c039ad39c4b52edcc4b P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:33.507589 1196 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37366 (request call id 832) took 1085 ms. Trace:
I20250113 02:51:33.507669 1196 rpcz_store.cc:276] 0113 02:51:32.421838 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:32.422438 (+ 600us) service_pool.cc:225] Handling call
0113 02:51:33.507459 (+1085021us) ts_tablet_manager.cc:1145] Deleting tablet 0f428c555442401285fb4eab8280b10b
0113 02:51:33.507470 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:33.507571 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1078362}
I20250113 02:51:33.508070 1196 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37366 (request call id 833) took 1010 ms. Trace:
I20250113 02:51:33.508137 1196 rpcz_store.cc:276] 0113 02:51:32.497125 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:32.498449 (+ 1324us) service_pool.cc:225] Handling call
0113 02:51:33.507943 (+1009494us) ts_tablet_manager.cc:1145] Deleting tablet fd0b8592997d4a61b8ca854ecbb67c55
0113 02:51:33.507951 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:33.508054 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1009312}
I20250113 02:51:33.508354 1196 tablet_replica.cc:331] T 6d63f0e5835540a78c82020b20a07dac P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:33.508503 1196 raft_consensus.cc:2238] T 6d63f0e5835540a78c82020b20a07dac P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:33.508633 1196 raft_consensus.cc:2267] T 6d63f0e5835540a78c82020b20a07dac P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:33.509181 1196 ts_tablet_manager.cc:1905] T 6d63f0e5835540a78c82020b20a07dac P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:33.510200 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_3fe958090174491c874058061bc4c252" } new_table_name: "default.Table_44cadaad4c454a31b14e50db0301c838" modify_external_catalogs: true
I20250113 02:51:33.510732 451 catalog_manager.cc:5618] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095) does not have the latest schema for tablet aa5c172a69bf4ca79999f4b58df08de8 (table default.Table_3f1e7146fed34d53896f138e43a4d26e [id=5dceb76e1dbe422ab98d4da5bf6c45a0]). Expected version 1 got 0
I20250113 02:51:33.510905 1195 ts_tablet_manager.cc:1918] T 0539cbbe47e64c039ad39c4b52edcc4b P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:33.511000 1195 log.cc:1198] T 0539cbbe47e64c039ad39c4b52edcc4b P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/0539cbbe47e64c039ad39c4b52edcc4b
I20250113 02:51:33.511304 1195 ts_tablet_manager.cc:1939] T 0539cbbe47e64c039ad39c4b52edcc4b P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
W20250113 02:51:33.512562 995 leader_election.cc:343] T 0539cbbe47e64c039ad39c4b52edcc4b P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Not found: Tablet not found: 0539cbbe47e64c039ad39c4b52edcc4b
W20250113 02:51:33.513517 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 0f428c555442401285fb4eab8280b10b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0f428c555442401285fb4eab8280b10b already in progress: creating tablet
W20250113 02:51:33.516086 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet fd0b8592997d4a61b8ca854ecbb67c55 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fd0b8592997d4a61b8ca854ecbb67c55 already in progress: creating tablet
I20250113 02:51:33.516295 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0539cbbe47e64c039ad39c4b52edcc4b" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:33.516793 998 leader_election.cc:343] T 0539cbbe47e64c039ad39c4b52edcc4b P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:33.516903 998 leader_election.cc:304] T 0539cbbe47e64c039ad39c4b52edcc4b P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4216a7f853d642b7bc3e5cff458f3aca; no voters: 1186fa869eb14812832fa6562531f196, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:33.518035 1926 raft_consensus.cc:2743] T 0539cbbe47e64c039ad39c4b52edcc4b P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:33.518682 742 tablet_service.cc:1467] Processing CreateTablet for tablet e312e70334e04ecd843159ac72f4a2a6 (DEFAULT_TABLE table=default.Table_71699c9b0b1a46cda2db76dee82b90b4 [id=d1b0bb18a7b84473803cd1294b0829ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:33.519104 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e312e70334e04ecd843159ac72f4a2a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:33.526245 1032 tablet_service.cc:1467] Processing CreateTablet for tablet e312e70334e04ecd843159ac72f4a2a6 (DEFAULT_TABLE table=default.Table_71699c9b0b1a46cda2db76dee82b90b4 [id=d1b0bb18a7b84473803cd1294b0829ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:33.526739 1926 raft_consensus.cc:491] T 77cf8b0c824f401aba1880eb7fd1807b P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:33.526888 1926 raft_consensus.cc:513] T 77cf8b0c824f401aba1880eb7fd1807b P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:33.527451 1194 tablet_bootstrap.cc:492] T a0ca2e3a3ce94f3aa20df64c89c7cf95 P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:33.527571 1194 ts_tablet_manager.cc:1397] T a0ca2e3a3ce94f3aa20df64c89c7cf95 P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.054s user 0.002s sys 0.002s
I20250113 02:51:33.527649 1926 leader_election.cc:290] T 77cf8b0c824f401aba1880eb7fd1807b P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:33.528363 1194 raft_consensus.cc:357] T a0ca2e3a3ce94f3aa20df64c89c7cf95 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:33.528612 1194 raft_consensus.cc:383] T a0ca2e3a3ce94f3aa20df64c89c7cf95 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:33.528739 1194 raft_consensus.cc:738] T a0ca2e3a3ce94f3aa20df64c89c7cf95 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:33.528986 1194 consensus_queue.cc:260] T a0ca2e3a3ce94f3aa20df64c89c7cf95 P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:33.529649 1194 ts_tablet_manager.cc:1428] T a0ca2e3a3ce94f3aa20df64c89c7cf95 P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250113 02:51:33.529618 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77cf8b0c824f401aba1880eb7fd1807b" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:33.529974 1194 tablet_bootstrap.cc:492] T 4c615b0b83d74fcfae0c51047dbfbbf4 P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
W20250113 02:51:33.532107 995 leader_election.cc:343] T 77cf8b0c824f401aba1880eb7fd1807b P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Not found: Tablet not found: 77cf8b0c824f401aba1880eb7fd1807b
I20250113 02:51:33.532238 1194 tablet_bootstrap.cc:654] T 4c615b0b83d74fcfae0c51047dbfbbf4 P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
W20250113 02:51:33.532232 454 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9fcf7a6638414f9a99403b3327eeb831 consensus state has no leader
I20250113 02:51:33.532761 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_1d41ead6635b49f380159a2312885b50"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:33.533123 998 leader_election.cc:343] T 77cf8b0c824f401aba1880eb7fd1807b P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:33.533217 998 leader_election.cc:304] T 77cf8b0c824f401aba1880eb7fd1807b P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4216a7f853d642b7bc3e5cff458f3aca; no voters: 1186fa869eb14812832fa6562531f196, df69f48656b44bb890f76587ee69cc0f
W20250113 02:51:33.533452 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1d41ead6635b49f380159a2312885b50 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:33.533847 1926 raft_consensus.cc:2743] T 77cf8b0c824f401aba1880eb7fd1807b P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:33.535811 892 tablet_service.cc:1467] Processing CreateTablet for tablet e312e70334e04ecd843159ac72f4a2a6 (DEFAULT_TABLE table=default.Table_71699c9b0b1a46cda2db76dee82b90b4 [id=d1b0bb18a7b84473803cd1294b0829ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:33.491187 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ec06c9f6a3bf46a9bcb90a85a82b6ff8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:33.538236 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e312e70334e04ecd843159ac72f4a2a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:33.538828 1036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c443454994a47f2bfa7e2ec7ac576e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:33.539070 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac47ba88e3d54c5c8d3298777af4c537. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:33.539229 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 86d59e3aa9804a2cbab167d25ec86c90. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:33.543280 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 67fe2f52d98048dab9f37dd820b7f712. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:33.544108 1040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 106a80fd24724020bff63de662fdabb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:33.546259 1196 ts_tablet_manager.cc:1918] T 6d63f0e5835540a78c82020b20a07dac P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:33.546355 1196 log.cc:1198] T 6d63f0e5835540a78c82020b20a07dac P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/6d63f0e5835540a78c82020b20a07dac
I20250113 02:51:33.546636 1196 ts_tablet_manager.cc:1939] T 6d63f0e5835540a78c82020b20a07dac P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:33.548089 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e312e70334e04ecd843159ac72f4a2a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:33.555011 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_759338d187b64a61aea6fc11366d3877"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:33.555545 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_759338d187b64a61aea6fc11366d3877 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:33.565307 2045 raft_consensus.cc:491] T 377ebf66bbef40988043f6200ae0893c P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:33.565462 2045 raft_consensus.cc:513] T 377ebf66bbef40988043f6200ae0893c P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:33.565549 2109 raft_consensus.cc:491] T 42cf1e708644427f9d4dcd41ae2bcb37 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:33.565744 2109 raft_consensus.cc:513] T 42cf1e708644427f9d4dcd41ae2bcb37 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:33.566089 2045 leader_election.cc:290] T 377ebf66bbef40988043f6200ae0893c P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:33.566586 1062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "377ebf66bbef40988043f6200ae0893c" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:33.566843 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "377ebf66bbef40988043f6200ae0893c" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:33.567237 709 leader_election.cc:343] T 377ebf66bbef40988043f6200ae0893c P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
W20250113 02:51:33.567543 711 leader_election.cc:343] T 377ebf66bbef40988043f6200ae0893c P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:33.567637 711 leader_election.cc:304] T 377ebf66bbef40988043f6200ae0893c P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:33.567961 2072 raft_consensus.cc:2743] T 377ebf66bbef40988043f6200ae0893c P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:33.569669 1195 tablet_replica.cc:331] T 0fe5421dc3ea4be684cab084f13d8df0 P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:33.569929 1195 raft_consensus.cc:2238] T 0fe5421dc3ea4be684cab084f13d8df0 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:33.570140 1195 raft_consensus.cc:2267] T 0fe5421dc3ea4be684cab084f13d8df0 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:33.570091 2102 tablet.cc:1721] T 401c4f5dbd304950877b59a865f417b3 P df69f48656b44bb890f76587ee69cc0f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250113 02:51:33.570760 1195 ts_tablet_manager.cc:1905] T 0fe5421dc3ea4be684cab084f13d8df0 P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:33.572431 885 tablet_service.cc:1467] Processing CreateTablet for tablet 6265e9a6cfde4161b0cd5dd1b76be8fc (DEFAULT_TABLE table=default.Table_777e191d02f54883afa306010890a105 [id=f15e7b9c370f406183e970ec4095cf38]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:33.573518 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet 0539cbbe47e64c039ad39c4b52edcc4b (table default.Table_d1b93410e6504e71a32397040f50b464 [id=817479ce0ac04442a0f89e6115d24ba4]) successfully deleted
I20250113 02:51:33.574339 742 tablet_service.cc:1467] Processing CreateTablet for tablet 6265e9a6cfde4161b0cd5dd1b76be8fc (DEFAULT_TABLE table=default.Table_777e191d02f54883afa306010890a105 [id=f15e7b9c370f406183e970ec4095cf38]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:33.574704 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6265e9a6cfde4161b0cd5dd1b76be8fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:33.574784 883 tablet_service.cc:1467] Processing CreateTablet for tablet 18a124d212b94ff3b545c4c7f3f4fcd7 (DEFAULT_TABLE table=default.Table_f86a1e2be2ba44b8bda94f4ad150c655 [id=4c4bee64e7794cd9b1152cc717e1f4e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:33.575134 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 18a124d212b94ff3b545c4c7f3f4fcd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:33.575721 1035 tablet_service.cc:1467] Processing CreateTablet for tablet 6265e9a6cfde4161b0cd5dd1b76be8fc (DEFAULT_TABLE table=default.Table_777e191d02f54883afa306010890a105 [id=f15e7b9c370f406183e970ec4095cf38]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:33.576435 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6265e9a6cfde4161b0cd5dd1b76be8fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:33.576622 746 tablet_service.cc:1467] Processing CreateTablet for tablet 18a124d212b94ff3b545c4c7f3f4fcd7 (DEFAULT_TABLE table=default.Table_f86a1e2be2ba44b8bda94f4ad150c655 [id=4c4bee64e7794cd9b1152cc717e1f4e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:33.576987 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a1d6cbfb61564e929efab798d4681409. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:33.577001 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 18a124d212b94ff3b545c4c7f3f4fcd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:33.512574 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 782) took 2209 ms. Trace:
I20250113 02:51:33.577668 1023 tablet_service.cc:1467] Processing CreateTablet for tablet 18a124d212b94ff3b545c4c7f3f4fcd7 (DEFAULT_TABLE table=default.Table_f86a1e2be2ba44b8bda94f4ad150c655 [id=4c4bee64e7794cd9b1152cc717e1f4e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:33.579203 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 18a124d212b94ff3b545c4c7f3f4fcd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:33.579665 1926 raft_consensus.cc:491] T 6e1284c8aaba44b5810b0d06d36757eb P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:33.579816 1926 raft_consensus.cc:513] T 6e1284c8aaba44b5810b0d06d36757eb P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:33.579879 1062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42cf1e708644427f9d4dcd41ae2bcb37" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
W20250113 02:51:33.580464 709 leader_election.cc:343] T 42cf1e708644427f9d4dcd41ae2bcb37 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:33.580674 2109 leader_election.cc:290] T 42cf1e708644427f9d4dcd41ae2bcb37 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:33.581312 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42cf1e708644427f9d4dcd41ae2bcb37" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:33.581528 890 tablet_service.cc:1467] Processing CreateTablet for tablet 582aef4539b240dc8aa82d29c4261153 (DEFAULT_TABLE table=default.Table_0e1d06e34184490d8c82c4c47f22fef4 [id=f512fbac07fb44cd8b231019d979c9fc]), partition=RANGE (key) PARTITION UNBOUNDED
W20250113 02:51:33.581809 711 leader_election.cc:343] T 42cf1e708644427f9d4dcd41ae2bcb37 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:33.581945 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6265e9a6cfde4161b0cd5dd1b76be8fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:33.581912 711 leader_election.cc:304] T 42cf1e708644427f9d4dcd41ae2bcb37 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:33.582247 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 582aef4539b240dc8aa82d29c4261153. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:33.582333 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e1284c8aaba44b5810b0d06d36757eb" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:33.583458 998 leader_election.cc:343] T 6e1284c8aaba44b5810b0d06d36757eb P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:33.584615 1926 leader_election.cc:290] T 6e1284c8aaba44b5810b0d06d36757eb P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:33.585247 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e1284c8aaba44b5810b0d06d36757eb" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
I20250113 02:51:33.585433 771 raft_consensus.cc:2463] T 6e1284c8aaba44b5810b0d06d36757eb P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 0.
I20250113 02:51:33.585815 995 leader_election.cc:304] T 6e1284c8aaba44b5810b0d06d36757eb P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters: df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:33.586344 2009 raft_consensus.cc:2798] T 6e1284c8aaba44b5810b0d06d36757eb P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 02:51:33.586418 2009 raft_consensus.cc:491] T 6e1284c8aaba44b5810b0d06d36757eb P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 02:51:33.586475 2009 raft_consensus.cc:3054] T 6e1284c8aaba44b5810b0d06d36757eb P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:33.588796 2009 raft_consensus.cc:513] T 6e1284c8aaba44b5810b0d06d36757eb P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:33.589397 2009 leader_election.cc:290] T 6e1284c8aaba44b5810b0d06d36757eb P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Requested vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:33.589861 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e1284c8aaba44b5810b0d06d36757eb" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196"
I20250113 02:51:33.590024 771 raft_consensus.cc:3054] T 6e1284c8aaba44b5810b0d06d36757eb P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:33.592471 771 raft_consensus.cc:2463] T 6e1284c8aaba44b5810b0d06d36757eb P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 1.
I20250113 02:51:33.592891 995 leader_election.cc:304] T 6e1284c8aaba44b5810b0d06d36757eb P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters:
I20250113 02:51:33.593010 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet b16ff51101e0452ebc05374f0d1f4917
I20250113 02:51:33.593180 2009 raft_consensus.cc:2798] T 6e1284c8aaba44b5810b0d06d36757eb P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Leader election won for term 1
I20250113 02:51:33.593291 2009 raft_consensus.cc:695] T 6e1284c8aaba44b5810b0d06d36757eb P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: Becoming Leader. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Running, Role: LEADER
I20250113 02:51:33.593518 2009 consensus_queue.cc:237] T 6e1284c8aaba44b5810b0d06d36757eb P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:33.594416 899 tablet_service.cc:1514] Processing DeleteTablet for tablet b16ff51101e0452ebc05374f0d1f4917 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:33 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:33.594931 1040 tablet_service.cc:1514] Processing DeleteTablet for tablet b16ff51101e0452ebc05374f0d1f4917 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:33 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:33.600462 2024 raft_consensus.cc:491] T 2778d7dc7d9c4843bbc9eb705cec71cd P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:33.600618 2024 raft_consensus.cc:513] T 2778d7dc7d9c4843bbc9eb705cec71cd P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:33.601316 2024 leader_election.cc:290] T 2778d7dc7d9c4843bbc9eb705cec71cd P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587), 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
I20250113 02:51:33.602197 1062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2778d7dc7d9c4843bbc9eb705cec71cd" candidate_uuid: "df69f48656b44bb890f76587ee69cc0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:33.602406 1062 raft_consensus.cc:2463] T 2778d7dc7d9c4843bbc9eb705cec71cd P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df69f48656b44bb890f76587ee69cc0f in term 0.
I20250113 02:51:33.602763 852 leader_election.cc:304] T 2778d7dc7d9c4843bbc9eb705cec71cd P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f; no voters:
I20250113 02:51:33.603013 2024 raft_consensus.cc:2798] T 2778d7dc7d9c4843bbc9eb705cec71cd P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 02:51:33.603072 2024 raft_consensus.cc:491] T 2778d7dc7d9c4843bbc9eb705cec71cd P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 02:51:33.603135 2024 raft_consensus.cc:3054] T 2778d7dc7d9c4843bbc9eb705cec71cd P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:33.607112 1198 rpcz_store.cc:276] 0113 02:51:31.303558 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:31.303600 (+ 42us) service_pool.cc:225] Handling call
0113 02:51:33.512450 (+2208850us) ts_tablet_manager.cc:1145] Deleting tablet 2e4a68bbcd684785a7bacdbabb506718
0113 02:51:33.512462 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:33.512554 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2184758}
W20250113 02:51:33.608554 851 leader_election.cc:343] T 2778d7dc7d9c4843bbc9eb705cec71cd P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Not found: Tablet not found: 2778d7dc7d9c4843bbc9eb705cec71cd
W20250113 02:51:33.608798 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 2e4a68bbcd684785a7bacdbabb506718 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2e4a68bbcd684785a7bacdbabb506718 already in progress: creating tablet
I20250113 02:51:33.609658 771 raft_consensus.cc:1270] T c508724edea147e5bf6e20bca2f50397 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Refusing update from remote peer 4216a7f853d642b7bc3e5cff458f3aca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250113 02:51:33.610277 1926 consensus_queue.cc:1035] T c508724edea147e5bf6e20bca2f50397 P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 02:51:33.611476 2072 raft_consensus.cc:2743] T 42cf1e708644427f9d4dcd41ae2bcb37 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:33.612874 2072 raft_consensus.cc:491] T aa5c172a69bf4ca79999f4b58df08de8 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:33.612985 2072 raft_consensus.cc:513] T aa5c172a69bf4ca79999f4b58df08de8 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:33.613397 450 catalog_manager.cc:5618] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467) does not have the latest schema for tablet 401c4f5dbd304950877b59a865f417b3 (table default.Table_c991f9dbd70d4c4682664b8711f05db4 [id=2c64da64ecdb4bf48e11b523533902e8]). Expected version 1 got 0
I20250113 02:51:33.613430 1926 raft_consensus.cc:491] T 1a63f485caf54068abd6b50d7de3d14a P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:33.613593 2072 leader_election.cc:290] T aa5c172a69bf4ca79999f4b58df08de8 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:33.613539 1926 raft_consensus.cc:513] T 1a63f485caf54068abd6b50d7de3d14a P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:33.614070 2051 tablet.cc:1721] T c508724edea147e5bf6e20bca2f50397 P 4216a7f853d642b7bc3e5cff458f3aca: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250113 02:51:33.614770 1926 leader_election.cc:290] T 1a63f485caf54068abd6b50d7de3d14a P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:33.614849 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_8a9caf2b686f4017948ebdd7f3fe8f58"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:33.615453 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8a9caf2b686f4017948ebdd7f3fe8f58 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:33.616267 1062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5c172a69bf4ca79999f4b58df08de8" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:33.618305 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a63f485caf54068abd6b50d7de3d14a" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
I20250113 02:51:33.618528 771 raft_consensus.cc:2463] T 1a63f485caf54068abd6b50d7de3d14a P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 0.
I20250113 02:51:33.618876 995 leader_election.cc:304] T 1a63f485caf54068abd6b50d7de3d14a P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters:
I20250113 02:51:33.619297 1926 raft_consensus.cc:2798] T 1a63f485caf54068abd6b50d7de3d14a P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 02:51:33.619428 1926 raft_consensus.cc:491] T 1a63f485caf54068abd6b50d7de3d14a P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 02:51:33.619565 1926 raft_consensus.cc:3054] T 1a63f485caf54068abd6b50d7de3d14a P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:33.619890 2053 tablet.cc:1721] T c508724edea147e5bf6e20bca2f50397 P 1186fa869eb14812832fa6562531f196: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
W20250113 02:51:33.621263 709 leader_election.cc:343] T aa5c172a69bf4ca79999f4b58df08de8 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:33.622725 1926 raft_consensus.cc:513] T 1a63f485caf54068abd6b50d7de3d14a P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:33.623780 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a63f485caf54068abd6b50d7de3d14a" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196"
I20250113 02:51:33.624009 772 raft_consensus.cc:3054] T 1a63f485caf54068abd6b50d7de3d14a P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:33.625061 1191 tablet_bootstrap.cc:492] T 999fecb3fc65466cad21cc1ff5ef2d44 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:33.625177 1191 ts_tablet_manager.cc:1397] T 999fecb3fc65466cad21cc1ff5ef2d44 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.137s user 0.010s sys 0.002s
I20250113 02:51:33.625918 1191 raft_consensus.cc:357] T 999fecb3fc65466cad21cc1ff5ef2d44 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:33.626363 1191 raft_consensus.cc:383] T 999fecb3fc65466cad21cc1ff5ef2d44 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:33.626423 1191 raft_consensus.cc:738] T 999fecb3fc65466cad21cc1ff5ef2d44 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:33.626611 1191 consensus_queue.cc:260] T 999fecb3fc65466cad21cc1ff5ef2d44 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:33.627092 1191 ts_tablet_manager.cc:1428] T 999fecb3fc65466cad21cc1ff5ef2d44 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250113 02:51:33.627121 772 raft_consensus.cc:2463] T 1a63f485caf54068abd6b50d7de3d14a P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 1.
I20250113 02:51:33.627308 1191 tablet_bootstrap.cc:492] T 1d15c7a01f854f0aaabcd30ec3166e31 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:33.627701 995 leader_election.cc:304] T 1a63f485caf54068abd6b50d7de3d14a P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters:
I20250113 02:51:33.628197 2009 raft_consensus.cc:491] T c826a0b8327f44f9a00e5afe3a36cd8a P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:33.628320 2113 raft_consensus.cc:2798] T 1a63f485caf54068abd6b50d7de3d14a P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Leader election won for term 1
I20250113 02:51:33.628345 2009 raft_consensus.cc:513] T c826a0b8327f44f9a00e5afe3a36cd8a P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:33.628458 2113 raft_consensus.cc:695] T 1a63f485caf54068abd6b50d7de3d14a P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: Becoming Leader. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Running, Role: LEADER
I20250113 02:51:33.628782 2113 consensus_queue.cc:237] T 1a63f485caf54068abd6b50d7de3d14a P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:33.628945 1191 tablet_bootstrap.cc:654] T 1d15c7a01f854f0aaabcd30ec3166e31 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:33.629179 2009 leader_election.cc:290] T c826a0b8327f44f9a00e5afe3a36cd8a P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
W20250113 02:51:33.629912 995 leader_election.cc:343] T c826a0b8327f44f9a00e5afe3a36cd8a P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Not found: Tablet not found: c826a0b8327f44f9a00e5afe3a36cd8a
I20250113 02:51:33.630235 899 tablet_service.cc:1514] Processing DeleteTablet for tablet 2778d7dc7d9c4843bbc9eb705cec71cd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:20 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:33.633972 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c826a0b8327f44f9a00e5afe3a36cd8a" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:33.634256 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5c172a69bf4ca79999f4b58df08de8" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:33.634277 751 tablet_service.cc:1514] Processing DeleteTablet for tablet b16ff51101e0452ebc05374f0d1f4917 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:33 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:33.634492 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a63f485caf54068abd6b50d7de3d14a" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:33.635040 998 consensus_peers.cc:487] T 8337f3c82d654ecdac2cbc1ffa88ef05 P 4216a7f853d642b7bc3e5cff458f3aca -> Peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Couldn't send request to peer df69f48656b44bb890f76587ee69cc0f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250113 02:51:33.635195 998 consensus_peers.cc:487] T c508724edea147e5bf6e20bca2f50397 P 4216a7f853d642b7bc3e5cff458f3aca -> Peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Couldn't send request to peer df69f48656b44bb890f76587ee69cc0f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250113 02:51:33.635349 998 leader_election.cc:343] T c826a0b8327f44f9a00e5afe3a36cd8a P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:33.635464 998 leader_election.cc:304] T c826a0b8327f44f9a00e5afe3a36cd8a P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4216a7f853d642b7bc3e5cff458f3aca; no voters: 1186fa869eb14812832fa6562531f196, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:33.635561 2024 raft_consensus.cc:513] T 2778d7dc7d9c4843bbc9eb705cec71cd P df69f48656b44bb890f76587ee69cc0f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
W20250113 02:51:33.635855 998 leader_election.cc:343] T 1a63f485caf54068abd6b50d7de3d14a P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:33.636184 2024 leader_election.cc:290] T 2778d7dc7d9c4843bbc9eb705cec71cd P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 election: Requested vote from peers 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587), 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
I20250113 02:51:33.636269 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e1284c8aaba44b5810b0d06d36757eb" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f"
I20250113 02:51:33.636500 2009 raft_consensus.cc:2743] T c826a0b8327f44f9a00e5afe3a36cd8a P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250113 02:51:33.636505 711 leader_election.cc:343] T aa5c172a69bf4ca79999f4b58df08de8 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:33.636533 1062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2778d7dc7d9c4843bbc9eb705cec71cd" candidate_uuid: "df69f48656b44bb890f76587ee69cc0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca"
I20250113 02:51:33.636602 711 leader_election.cc:304] T aa5c172a69bf4ca79999f4b58df08de8 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:33.636674 1062 raft_consensus.cc:3054] T 2778d7dc7d9c4843bbc9eb705cec71cd P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:33.639098 1062 raft_consensus.cc:2463] T 2778d7dc7d9c4843bbc9eb705cec71cd P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df69f48656b44bb890f76587ee69cc0f in term 1.
W20250113 02:51:33.639542 851 leader_election.cc:343] T 2778d7dc7d9c4843bbc9eb705cec71cd P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Not found: Tablet not found: 2778d7dc7d9c4843bbc9eb705cec71cd
W20250113 02:51:33.639542 998 leader_election.cc:343] T 6e1284c8aaba44b5810b0d06d36757eb P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:33.640542 852 leader_election.cc:304] T 2778d7dc7d9c4843bbc9eb705cec71cd P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f; no voters: 1186fa869eb14812832fa6562531f196
I20250113 02:51:33.641057 2024 raft_consensus.cc:2798] T 2778d7dc7d9c4843bbc9eb705cec71cd P df69f48656b44bb890f76587ee69cc0f [term 1 FOLLOWER]: Leader election won for term 1
I20250113 02:51:33.641191 2024 raft_consensus.cc:695] T 2778d7dc7d9c4843bbc9eb705cec71cd P df69f48656b44bb890f76587ee69cc0f [term 1 LEADER]: Becoming Leader. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Running, Role: LEADER
I20250113 02:51:33.641391 2024 consensus_queue.cc:237] T 2778d7dc7d9c4843bbc9eb705cec71cd P df69f48656b44bb890f76587ee69cc0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:33.641623 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_9b26cd6ed9944430a1228f552a7c3d95"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:33.642239 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9b26cd6ed9944430a1228f552a7c3d95 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:33.643831 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 784) took 2269 ms. Trace:
I20250113 02:51:33.643934 1198 rpcz_store.cc:276] 0113 02:51:31.374542 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:31.374601 (+ 59us) service_pool.cc:225] Handling call
0113 02:51:33.643692 (+2269091us) ts_tablet_manager.cc:1145] Deleting tablet e13267498917432ba92b6badb76f5d73
0113 02:51:33.643705 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:33.643808 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2268850}
I20250113 02:51:33.644464 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 787) took 2245 ms. Trace:
I20250113 02:51:33.644546 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 35a13d9c065348e08de66044c04d221d
I20250113 02:51:33.644577 452 master_service.cc:946] ReplaceTablet: received request to replace tablet 8cc78bf7d6324b7ba10a1b1045923f2b from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:33.644543 1198 rpcz_store.cc:276] 0113 02:51:31.399291 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:31.404607 (+ 5316us) service_pool.cc:225] Handling call
0113 02:51:33.644361 (+2239754us) ts_tablet_manager.cc:1145] Deleting tablet df6299f38ffa4c2f9d6c8a5e5285905f
0113 02:51:33.644370 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:33.644449 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2239594}
W20250113 02:51:33.644838 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet e13267498917432ba92b6badb76f5d73 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e13267498917432ba92b6badb76f5d73 already in progress: creating tablet
I20250113 02:51:33.645021 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 788) took 2239 ms. Trace:
I20250113 02:51:33.645088 1198 rpcz_store.cc:276] 0113 02:51:31.405956 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:31.406090 (+ 134us) service_pool.cc:225] Handling call
0113 02:51:33.644938 (+2238848us) ts_tablet_manager.cc:1145] Deleting tablet ef6dfe0431ef4a73a957559f60f24756
0113 02:51:33.644945 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:33.645008 (+ 63us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2238681}
I20250113 02:51:33.645215 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_4ef3edc5b07c4e0881456bfe23741c90" } modify_external_catalogs: true
I20250113 02:51:33.645270 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_677784013a07442b9963695510a0a055" } new_table_name: "default.Table_35bfb3623c774fc39e15a926d9af88b6" modify_external_catalogs: true
W20250113 02:51:33.645375 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet df6299f38ffa4c2f9d6c8a5e5285905f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet df6299f38ffa4c2f9d6c8a5e5285905f already in progress: creating tablet
I20250113 02:51:33.645431 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_4ef3edc5b07c4e0881456bfe23741c90" } modify_external_catalogs: true
I20250113 02:51:33.646816 1041 tablet_service.cc:1514] Processing DeleteTablet for tablet 35a13d9c065348e08de66044c04d221d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:33 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:33.647362 899 tablet_service.cc:1514] Processing DeleteTablet for tablet 35a13d9c065348e08de66044c04d221d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:33 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:33.648509 1196 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37366 (request call id 834) took 1060 ms. Trace:
I20250113 02:51:33.650847 1041 tablet_service.cc:1467] Processing CreateTablet for tablet 582aef4539b240dc8aa82d29c4261153 (DEFAULT_TABLE table=default.Table_0e1d06e34184490d8c82c4c47f22fef4 [id=f512fbac07fb44cd8b231019d979c9fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:33.651230 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 582aef4539b240dc8aa82d29c4261153. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:33.651739 747 tablet_service.cc:1467] Processing CreateTablet for tablet 582aef4539b240dc8aa82d29c4261153 (DEFAULT_TABLE table=default.Table_0e1d06e34184490d8c82c4c47f22fef4 [id=f512fbac07fb44cd8b231019d979c9fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:33.655344 751 tablet_service.cc:1514] Processing DeleteTablet for tablet 35a13d9c065348e08de66044c04d221d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:33 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:33.656782 1926 leader_election.cc:290] T 1a63f485caf54068abd6b50d7de3d14a P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Requested vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:33.656994 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a63f485caf54068abd6b50d7de3d14a" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f"
W20250113 02:51:33.657513 998 leader_election.cc:343] T 1a63f485caf54068abd6b50d7de3d14a P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:33.657924 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 582aef4539b240dc8aa82d29c4261153. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:33.658499 2109 raft_consensus.cc:2743] T aa5c172a69bf4ca79999f4b58df08de8 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:33.612000 2108 tablet.cc:1721] T 32cba8f3efc54e78b4df41c15df82eea P df69f48656b44bb890f76587ee69cc0f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20250113 02:51:33.648584 1196 rpcz_store.cc:276] 0113 02:51:32.587903 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:32.587959 (+ 56us) service_pool.cc:225] Handling call
0113 02:51:33.508320 (+920361us) ts_tablet_manager.cc:1145] Deleting tablet 6d63f0e5835540a78c82020b20a07dac
0113 02:51:33.508327 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:33.536061 (+ 27734us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:33.546167 (+ 10106us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:33.648493 (+102326us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":920162}
I20250113 02:51:33.660228 1196 tablet_replica.cc:331] T ddf79eec41214693859f526265a55492 P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:33.660368 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 789) took 2254 ms. Trace:
I20250113 02:51:33.660392 1196 raft_consensus.cc:2238] T ddf79eec41214693859f526265a55492 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:33.660526 1196 raft_consensus.cc:2267] T ddf79eec41214693859f526265a55492 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:33.660437 1198 rpcz_store.cc:276] 0113 02:51:31.406078 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:31.406290 (+ 212us) service_pool.cc:225] Handling call
0113 02:51:33.645430 (+2239140us) ts_tablet_manager.cc:1145] Deleting tablet f8a6eebb6b6d4b63903e68750e181707
0113 02:51:33.645438 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:33.660354 (+ 14916us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2239002}
W20250113 02:51:33.660844 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet f8a6eebb6b6d4b63903e68750e181707 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f8a6eebb6b6d4b63903e68750e181707 already in progress: creating tablet
I20250113 02:51:33.661017 1196 ts_tablet_manager.cc:1905] T ddf79eec41214693859f526265a55492 P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:33.661202 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 805) took 2019 ms. Trace:
I20250113 02:51:33.661280 1198 rpcz_store.cc:276] 0113 02:51:31.642017 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:31.642118 (+ 101us) service_pool.cc:225] Handling call
0113 02:51:33.661093 (+2018975us) ts_tablet_manager.cc:1145] Deleting tablet 88a776aa820c44b3a3854ca5022bd353
0113 02:51:33.661102 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:33.661183 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2018757}
W20250113 02:51:33.661662 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 88a776aa820c44b3a3854ca5022bd353 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 88a776aa820c44b3a3854ca5022bd353 already in progress: creating tablet
I20250113 02:51:33.661784 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet 6d63f0e5835540a78c82020b20a07dac (table default.Table_af3ebf543c9448b7ae473d721a23d47e [id=93c6f37549ce4544bc0cb4a267f44b00]) successfully deleted
W20250113 02:51:33.661993 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 8cc78bf7d6324b7ba10a1b1045923f2b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8cc78bf7d6324b7ba10a1b1045923f2b already in progress: creating tablet
W20250113 02:51:33.662241 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 1f128d9bf2924fb1bada3a8462818004 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1f128d9bf2924fb1bada3a8462818004 already in progress: creating tablet
W20250113 02:51:33.662437 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 7213e3284b4d4cb581d9fb946a9f87e9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7213e3284b4d4cb581d9fb946a9f87e9 already in progress: creating tablet
W20250113 02:51:33.662627 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet e6e8966da70746d7a10c7243ba2f6f87 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e6e8966da70746d7a10c7243ba2f6f87 already in progress: creating tablet
W20250113 02:51:33.662806 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet b86d035e503344e1a3bd85c55131274e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b86d035e503344e1a3bd85c55131274e already in progress: creating tablet
W20250113 02:51:33.663041 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 36726b40ac5145bfb371871b3fff079a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 36726b40ac5145bfb371871b3fff079a already in progress: creating tablet
I20250113 02:51:33.664580 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 806) took 2022 ms. Trace:
I20250113 02:51:33.664717 1198 rpcz_store.cc:276] 0113 02:51:31.642169 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:31.642225 (+ 56us) service_pool.cc:225] Handling call
0113 02:51:33.664467 (+2022242us) ts_tablet_manager.cc:1145] Deleting tablet 1d15c7a01f854f0aaabcd30ec3166e31
0113 02:51:33.664479 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:33.664563 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2022017}
W20250113 02:51:33.665244 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 1d15c7a01f854f0aaabcd30ec3166e31 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1d15c7a01f854f0aaabcd30ec3166e31 already in progress: creating tablet
I20250113 02:51:33.665493 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet ec8196505dab4e05bb6487b566cf3144
I20250113 02:51:33.665648 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 811) took 1881 ms. Trace:
I20250113 02:51:33.665742 1198 rpcz_store.cc:276] 0113 02:51:31.784670 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:31.821952 (+ 37282us) service_pool.cc:225] Handling call
0113 02:51:33.665538 (+1843586us) ts_tablet_manager.cc:1145] Deleting tablet f4379f7fc6b54e6785f27975d9030cce
0113 02:51:33.665548 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:33.665630 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1843376}
I20250113 02:51:33.666227 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_ea693041c5824d818e3bd15e28c7a3f5" } new_table_name: "default.Table_9a1871f22c9e42ddb9b84ac6d5c04096" modify_external_catalogs: true
I20250113 02:51:33.666352 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet ec8196505dab4e05bb6487b566cf3144 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:33 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:33.668401 1926 raft_consensus.cc:491] T a97efc70946c4af487e33a85abf6b90c P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:33.668578 1926 raft_consensus.cc:513] T a97efc70946c4af487e33a85abf6b90c P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:33.669483 751 tablet_service.cc:1514] Processing DeleteTablet for tablet ec8196505dab4e05bb6487b566cf3144 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:33 UTC) from {username='slave'} at 127.0.0.1:55398
W20250113 02:51:33.670670 995 leader_election.cc:343] T a97efc70946c4af487e33a85abf6b90c P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Not found: Tablet not found: a97efc70946c4af487e33a85abf6b90c
I20250113 02:51:33.671233 1926 leader_election.cc:290] T a97efc70946c4af487e33a85abf6b90c P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:33.672302 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a97efc70946c4af487e33a85abf6b90c" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:33.672834 998 leader_election.cc:343] T a97efc70946c4af487e33a85abf6b90c P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:33.672950 998 leader_election.cc:304] T a97efc70946c4af487e33a85abf6b90c P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4216a7f853d642b7bc3e5cff458f3aca; no voters: 1186fa869eb14812832fa6562531f196, df69f48656b44bb890f76587ee69cc0f
W20250113 02:51:33.669486 2103 log.cc:926] Time spent T 32cba8f3efc54e78b4df41c15df82eea P df69f48656b44bb890f76587ee69cc0f: Append to log took a long time: real 0.100s user 0.000s sys 0.000s
I20250113 02:51:33.674111 1926 raft_consensus.cc:2743] T a97efc70946c4af487e33a85abf6b90c P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:33.675731 455 catalog_manager.cc:6071] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: default.Table_24d9af3d0b0740d69c791fc96916ef0a [id=124669ccadaf49e19d2660b1495f9647] alter complete (version 3)
I20250113 02:51:33.677500 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_4557b7a5878e45d0860844f7e191bfcf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:33.678015 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4557b7a5878e45d0860844f7e191bfcf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:33.678028 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 818) took 1827 ms. Trace:
I20250113 02:51:33.678117 1198 rpcz_store.cc:276] 0113 02:51:31.850656 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:31.862408 (+ 11752us) service_pool.cc:225] Handling call
0113 02:51:33.677903 (+1815495us) ts_tablet_manager.cc:1145] Deleting tablet 883027967e0c442a91ac820210b9c6c7
0113 02:51:33.677914 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:33.678010 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1815315}
I20250113 02:51:33.679726 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_8db01e4eaddd49aeb8f955e75cabb671" } modify_external_catalogs: true
I20250113 02:51:33.679884 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_8db01e4eaddd49aeb8f955e75cabb671" } modify_external_catalogs: true
I20250113 02:51:33.680487 890 tablet_service.cc:1514] Processing DeleteTablet for tablet ec8196505dab4e05bb6487b566cf3144 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:33 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:33.680805 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 820) took 1814 ms. Trace:
I20250113 02:51:33.680889 1198 rpcz_store.cc:276] 0113 02:51:31.866362 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:31.870850 (+ 4488us) service_pool.cc:225] Handling call
0113 02:51:33.680703 (+1809853us) ts_tablet_manager.cc:1145] Deleting tablet 16a132384b0e4264a73d09a3ea133893
0113 02:51:33.680715 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:33.680787 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1809688}
I20250113 02:51:33.681237 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 847) took 1554 ms. Trace:
I20250113 02:51:33.681299 1198 rpcz_store.cc:276] 0113 02:51:32.126538 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:32.158206 (+ 31668us) service_pool.cc:225] Handling call
0113 02:51:33.681118 (+1522912us) ts_tablet_manager.cc:1145] Deleting tablet e266a9083e2f46a0a43861b7708dcdd5
0113 02:51:33.681126 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:33.681224 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1522729}
I20250113 02:51:33.681581 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 848) took 1554 ms. Trace:
W20250113 02:51:33.683818 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet f4379f7fc6b54e6785f27975d9030cce because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f4379f7fc6b54e6785f27975d9030cce already in progress: creating tablet
W20250113 02:51:33.684146 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 883027967e0c442a91ac820210b9c6c7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 883027967e0c442a91ac820210b9c6c7 already in progress: creating tablet
W20250113 02:51:33.684391 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 16a132384b0e4264a73d09a3ea133893 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 16a132384b0e4264a73d09a3ea133893 already in progress: creating tablet
W20250113 02:51:33.684640 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet e266a9083e2f46a0a43861b7708dcdd5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e266a9083e2f46a0a43861b7708dcdd5 already in progress: creating tablet
I20250113 02:51:33.685647 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet c826a0b8327f44f9a00e5afe3a36cd8a with delete_type TABLET_DATA_DELETED (replaced by tablet bc533518faa24dffa3306ff092e8d507) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:33.685813 1040 tablet_service.cc:1514] Processing DeleteTablet for tablet c1fb2b36c0504cb4ba32671ae4c620d1 with delete_type TABLET_DATA_DELETED (replaced by tablet fb2458e2044d49029ed2d6851db9de25) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:33.685942 1036 tablet_service.cc:1514] Processing DeleteTablet for tablet a97efc70946c4af487e33a85abf6b90c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:20 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:33.686232 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d4d8b5cc8ad4c9e935d8afaf5766e66 with delete_type TABLET_DATA_DELETED (replaced by tablet e57ac440265e4646bfc5d5d9ca4c35d2) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:33.686798 1039 tablet_service.cc:1514] Processing DeleteTablet for tablet 05bf5ab0df874dd5b6559f6d82069184 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:21 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:33.687417 1196 ts_tablet_manager.cc:1918] T ddf79eec41214693859f526265a55492 P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:33.685000 751 tablet_service.cc:1514] Processing DeleteTablet for tablet 42cf1e708644427f9d4dcd41ae2bcb37 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:22 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:33.687916 1196 log.cc:1198] T ddf79eec41214693859f526265a55492 P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/ddf79eec41214693859f526265a55492
I20250113 02:51:33.688230 746 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b07a985897641c3ac48021c2a4daedb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:30 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:33.691273 1192 tablet_bootstrap.cc:492] T f129bb3fbbfd492d98822be0a6626e44 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:33.691578 1192 ts_tablet_manager.cc:1397] T f129bb3fbbfd492d98822be0a6626e44 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.246s user 0.006s sys 0.004s
I20250113 02:51:33.681645 1198 rpcz_store.cc:276] 0113 02:51:32.126652 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:32.158404 (+ 31752us) service_pool.cc:225] Handling call
0113 02:51:33.681496 (+1523092us) ts_tablet_manager.cc:1145] Deleting tablet cfd1697a03494ad2aaa41a056e334ca7
0113 02:51:33.681504 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:33.681568 (+ 64us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1522965}
I20250113 02:51:33.692787 1198 tablet_replica.cc:331] T de4a58bf63054890808e2e2fe46e43d2 P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:33.693030 1198 raft_consensus.cc:2238] T de4a58bf63054890808e2e2fe46e43d2 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:33.693209 1198 raft_consensus.cc:2267] T de4a58bf63054890808e2e2fe46e43d2 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:33.693709 1198 ts_tablet_manager.cc:1905] T de4a58bf63054890808e2e2fe46e43d2 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250113 02:51:33.696199 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet cfd1697a03494ad2aaa41a056e334ca7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cfd1697a03494ad2aaa41a056e334ca7 already in progress: creating tablet
I20250113 02:51:33.701299 1194 tablet_bootstrap.cc:492] T 4c615b0b83d74fcfae0c51047dbfbbf4 P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:33.701483 1194 ts_tablet_manager.cc:1397] T 4c615b0b83d74fcfae0c51047dbfbbf4 P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.172s user 0.014s sys 0.005s
I20250113 02:51:33.699887 1192 raft_consensus.cc:357] T f129bb3fbbfd492d98822be0a6626e44 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:33.711362 1196 ts_tablet_manager.cc:1939] T ddf79eec41214693859f526265a55492 P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:33.704090 1192 raft_consensus.cc:383] T f129bb3fbbfd492d98822be0a6626e44 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:33.712087 1192 raft_consensus.cc:738] T f129bb3fbbfd492d98822be0a6626e44 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:33.713493 2113 consensus_queue.cc:1035] T 8337f3c82d654ecdac2cbc1ffa88ef05 P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 02:51:33.717391 1194 raft_consensus.cc:357] T 4c615b0b83d74fcfae0c51047dbfbbf4 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:33.717581 1194 raft_consensus.cc:383] T 4c615b0b83d74fcfae0c51047dbfbbf4 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:33.717633 1194 raft_consensus.cc:738] T 4c615b0b83d74fcfae0c51047dbfbbf4 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:33.717854 1194 consensus_queue.cc:260] T 4c615b0b83d74fcfae0c51047dbfbbf4 P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:33.718420 1194 ts_tablet_manager.cc:1428] T 4c615b0b83d74fcfae0c51047dbfbbf4 P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.017s user 0.001s sys 0.001s
I20250113 02:51:33.720804 1191 tablet_bootstrap.cc:492] T 1d15c7a01f854f0aaabcd30ec3166e31 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:33.721585 1191 ts_tablet_manager.cc:1397] T 1d15c7a01f854f0aaabcd30ec3166e31 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.094s user 0.000s sys 0.003s
I20250113 02:51:33.723024 1191 raft_consensus.cc:357] T 1d15c7a01f854f0aaabcd30ec3166e31 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:33.724455 1191 raft_consensus.cc:383] T 1d15c7a01f854f0aaabcd30ec3166e31 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:33.724661 1191 raft_consensus.cc:738] T 1d15c7a01f854f0aaabcd30ec3166e31 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:33.725288 1191 consensus_queue.cc:260] T 1d15c7a01f854f0aaabcd30ec3166e31 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:33.727404 1191 ts_tablet_manager.cc:1428] T 1d15c7a01f854f0aaabcd30ec3166e31 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.005s user 0.000s sys 0.001s
I20250113 02:51:33.729926 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet ddf79eec41214693859f526265a55492 (table default.Table_f5342954ee0548ec90d482ed62198240 [id=888ae0b48950407086e5e2faa7fcf506]) successfully deleted
I20250113 02:51:33.730559 1191 tablet_bootstrap.cc:492] T 33c665e149dd48e8868acd935de581bf P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:33.732043 1192 consensus_queue.cc:260] T f129bb3fbbfd492d98822be0a6626e44 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
W20250113 02:51:33.734725 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 9fff52e99c64462a8a73fccc3c22ca63 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9fff52e99c64462a8a73fccc3c22ca63 already in progress: creating tablet
I20250113 02:51:33.735373 1196 tablet_replica.cc:331] T 8adfa738c17a4169a5d99f6ceca75100 P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:33.735617 1196 raft_consensus.cc:2238] T 8adfa738c17a4169a5d99f6ceca75100 P df69f48656b44bb890f76587ee69cc0f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:33.735838 1196 raft_consensus.cc:2267] T 8adfa738c17a4169a5d99f6ceca75100 P df69f48656b44bb890f76587ee69cc0f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:33.736478 1196 ts_tablet_manager.cc:1905] T 8adfa738c17a4169a5d99f6ceca75100 P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250113 02:51:33.736901 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 89b6fbf30ca44488ad7969930f1356e2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 89b6fbf30ca44488ad7969930f1356e2 already in progress: creating tablet
I20250113 02:51:33.737474 1192 ts_tablet_manager.cc:1428] T f129bb3fbbfd492d98822be0a6626e44 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.041s user 0.001s sys 0.001s
I20250113 02:51:33.737761 1192 tablet_bootstrap.cc:492] T 3b1910496e0849b193e390a5ef1a6b2b P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:33.740185 1194 tablet_bootstrap.cc:492] T c94eedb5dce44e6d896192c93127903a P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
W20250113 02:51:33.740469 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e90242bf273c42beb57ef160991a89ed consensus state has no leader
I20250113 02:51:33.742019 1194 tablet_bootstrap.cc:654] T c94eedb5dce44e6d896192c93127903a P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:33.743155 453 catalog_manager.cc:5618] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467) does not have the latest schema for tablet 32cba8f3efc54e78b4df41c15df82eea (table default.Table_27e1ad9a94f8455f806f48768ef4ddac [id=8aff13d6d3c5483eadb596b705fe5151]). Expected version 5 got 0
I20250113 02:51:33.743386 453 catalog_manager.cc:5618] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467) does not have the latest schema for tablet 378d59d314a84ddb84cf2ab041c51cf1 (table default.Table_2794f9d2a3cc49f78de33a775c1619f1 [id=0c01441cec7043b687d6744e72a0e45d]). Expected version 1 got 0
I20250113 02:51:33.743639 454 catalog_manager.cc:5618] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095) does not have the latest schema for tablet 56b022c94847411785a7e9db3757a2d9 (table default.Table_182ee8e8d8854eeabef569bd48babd04 [id=40ca872c045b4349b63ba9046e31ed13]). Expected version 1 got 0
I20250113 02:51:33.747243 890 tablet_service.cc:1514] Processing DeleteTablet for tablet a77c38a394114d6c8a8da99d7eef7669 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:20 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:33.747388 892 tablet_service.cc:1514] Processing DeleteTablet for tablet a0ca2e3a3ce94f3aa20df64c89c7cf95 with delete_type TABLET_DATA_DELETED (replaced by tablet 2470b010b571444eba3f132323038f33) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:33.747886 1191 tablet_bootstrap.cc:654] T 33c665e149dd48e8868acd935de581bf P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:33.756302 1195 ts_tablet_manager.cc:1918] T 0fe5421dc3ea4be684cab084f13d8df0 P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:33.756417 1195 log.cc:1198] T 0fe5421dc3ea4be684cab084f13d8df0 P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/0fe5421dc3ea4be684cab084f13d8df0
I20250113 02:51:33.756695 1195 ts_tablet_manager.cc:1939] T 0fe5421dc3ea4be684cab084f13d8df0 P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:33.757000 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 891) took 1096 ms. Trace:
I20250113 02:51:33.757076 1195 rpcz_store.cc:276] 0113 02:51:32.660402 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:32.661213 (+ 811us) service_pool.cc:225] Handling call
0113 02:51:33.569610 (+908397us) ts_tablet_manager.cc:1145] Deleting tablet 0fe5421dc3ea4be684cab084f13d8df0
0113 02:51:33.569622 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:33.677412 (+107790us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:33.756201 (+ 78789us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:33.756982 (+ 781us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":907171}
I20250113 02:51:33.757418 1195 tablet_replica.cc:331] T 056bf1e67be04d179eace3d93282b145 P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:33.757579 1195 raft_consensus.cc:2238] T 056bf1e67be04d179eace3d93282b145 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:33.757716 1195 raft_consensus.cc:2267] T 056bf1e67be04d179eace3d93282b145 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:33.758167 1195 ts_tablet_manager.cc:1905] T 056bf1e67be04d179eace3d93282b145 P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:33.759783 448 catalog_manager.cc:5618] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467) does not have the latest schema for tablet 4c615b0b83d74fcfae0c51047dbfbbf4 (table default.Table_bf689f8eb8884e88a72e9b50598fc389 [id=a956f95f6dd94b578f6bd3cb3b97f9da]). Expected version 2 got 0
I20250113 02:51:33.760973 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_39421f9a919a4fe0851a559994412d3a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:33.761505 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_39421f9a919a4fe0851a559994412d3a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:33.762686 450 master_service.cc:946] ReplaceTablet: received request to replace tablet 0953b2f89e35402c9e9e84f3589c23ac from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:33.764573 450 master_service.cc:946] ReplaceTablet: received request to replace tablet 02bf7042d4ce4077851a44bbfb6c4b46 from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:33.764608 1196 ts_tablet_manager.cc:1918] T 8adfa738c17a4169a5d99f6ceca75100 P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250113 02:51:33.764701 1196 log.cc:1198] T 8adfa738c17a4169a5d99f6ceca75100 P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/8adfa738c17a4169a5d99f6ceca75100
I20250113 02:51:33.765094 2102 tablet.cc:1716] T 32cba8f3efc54e78b4df41c15df82eea P df69f48656b44bb890f76587ee69cc0f: Skipping requested alter to schema version 5, tablet already version 5
I20250113 02:51:33.765198 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet 0fe5421dc3ea4be684cab084f13d8df0 (table default.Table_ca04ad8d0d8b40d1ba6b147731579526 [id=beb83fe520f94241949a46d436e6374f]) successfully deleted
I20250113 02:51:33.768249 2024 raft_consensus.cc:491] T 378d59d314a84ddb84cf2ab041c51cf1 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:33.768466 2024 raft_consensus.cc:513] T 378d59d314a84ddb84cf2ab041c51cf1 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:33.769438 1062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "378d59d314a84ddb84cf2ab041c51cf1" candidate_uuid: "df69f48656b44bb890f76587ee69cc0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:33.769693 2024 leader_election.cc:290] T 378d59d314a84ddb84cf2ab041c51cf1 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587), 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
I20250113 02:51:33.770093 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "378d59d314a84ddb84cf2ab041c51cf1" candidate_uuid: "df69f48656b44bb890f76587ee69cc0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
I20250113 02:51:33.770706 851 leader_election.cc:304] T 378d59d314a84ddb84cf2ab041c51cf1 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df69f48656b44bb890f76587ee69cc0f; no voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca
I20250113 02:51:33.771184 2024 raft_consensus.cc:2743] T 378d59d314a84ddb84cf2ab041c51cf1 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:33.773646 1192 tablet_bootstrap.cc:654] T 3b1910496e0849b193e390a5ef1a6b2b P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:33.775059 1194 tablet_bootstrap.cc:492] T c94eedb5dce44e6d896192c93127903a P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:33.775238 1194 ts_tablet_manager.cc:1397] T c94eedb5dce44e6d896192c93127903a P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.035s user 0.001s sys 0.002s
I20250113 02:51:33.776111 1194 raft_consensus.cc:357] T c94eedb5dce44e6d896192c93127903a P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:33.776367 1194 raft_consensus.cc:383] T c94eedb5dce44e6d896192c93127903a P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:33.776453 1194 raft_consensus.cc:738] T c94eedb5dce44e6d896192c93127903a P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:33.776682 1194 consensus_queue.cc:260] T c94eedb5dce44e6d896192c93127903a P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:33.777194 1194 ts_tablet_manager.cc:1428] T c94eedb5dce44e6d896192c93127903a P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250113 02:51:33.775338 1198 ts_tablet_manager.cc:1918] T de4a58bf63054890808e2e2fe46e43d2 P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:33.778681 1198 log.cc:1198] T de4a58bf63054890808e2e2fe46e43d2 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/de4a58bf63054890808e2e2fe46e43d2
I20250113 02:51:33.779295 1198 ts_tablet_manager.cc:1939] T de4a58bf63054890808e2e2fe46e43d2 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:33.780198 1194 tablet_bootstrap.cc:492] T ff8ef6290698477391721fd65b9eeb28 P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:33.781802 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 850) took 1579 ms. Trace:
I20250113 02:51:33.782591 1198 rpcz_store.cc:276] 0113 02:51:32.202703 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:32.202776 (+ 73us) service_pool.cc:225] Handling call
0113 02:51:33.692746 (+1489970us) ts_tablet_manager.cc:1145] Deleting tablet de4a58bf63054890808e2e2fe46e43d2
0113 02:51:33.692754 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:33.711349 (+ 18595us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:33.775218 (+ 63869us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:33.781787 (+ 6569us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1489672}
W20250113 02:51:33.785254 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ea51f6e169ae49bd815a9496a1830d0a consensus state has no leader
I20250113 02:51:33.785640 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_08f7721ed9f1449ba1fbce603d01b13b" } new_table_name: "default.Table_2c6861906f3f4ea98e9d9f758dbf92cf" modify_external_catalogs: true
I20250113 02:51:33.786094 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_1871943d605842118e5ac9c6f594f30c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:33.786664 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1871943d605842118e5ac9c6f594f30c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:33.787142 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 21bcdd741104474bad7023426856d18f
I20250113 02:51:33.787891 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_2bf3c3c3eb364670a2d0633238a32fe6" } new_table_name: "default.Table_40c1445967024038bf8a25c42bf346ad" modify_external_catalogs: true
I20250113 02:51:33.788096 892 tablet_service.cc:1514] Processing DeleteTablet for tablet c94eedb5dce44e6d896192c93127903a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:20 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:33.789347 892 tablet_service.cc:1514] Processing DeleteTablet for tablet 21bcdd741104474bad7023426856d18f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:33 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:33.790552 747 tablet_service.cc:1514] Processing DeleteTablet for tablet 21bcdd741104474bad7023426856d18f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:33 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:33.792140 1196 ts_tablet_manager.cc:1939] T 8adfa738c17a4169a5d99f6ceca75100 P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:33.792409 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 851) took 1583 ms. Trace:
I20250113 02:51:33.793546 1198 rpcz_store.cc:276] 0113 02:51:32.209245 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:32.209306 (+ 61us) service_pool.cc:225] Handling call
0113 02:51:33.792276 (+1582970us) ts_tablet_manager.cc:1145] Deleting tablet cd6e48a720d549f2b5f30a61e1f5e017
0113 02:51:33.792288 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:33.792390 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1582693}
I20250113 02:51:33.794013 1194 tablet_bootstrap.cc:654] T ff8ef6290698477391721fd65b9eeb28 P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:33.794631 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet de4a58bf63054890808e2e2fe46e43d2 (table default.Table_5b0e60572cb54fcca8b3b5b94326ae3a [id=9f5add219a7c48da9a8afa966e22d1bd]) successfully deleted
I20250113 02:51:33.795771 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 854) took 1568 ms. Trace:
I20250113 02:51:33.795993 1198 rpcz_store.cc:276] 0113 02:51:32.227474 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:32.227538 (+ 64us) service_pool.cc:225] Handling call
0113 02:51:33.795651 (+1568113us) ts_tablet_manager.cc:1145] Deleting tablet 29b1444efd28483297bafc73a33e7a54
0113 02:51:33.795661 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:33.795757 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1567864}
W20250113 02:51:33.796321 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet cd6e48a720d549f2b5f30a61e1f5e017 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cd6e48a720d549f2b5f30a61e1f5e017 already in progress: creating tablet
I20250113 02:51:33.796525 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 853) took 1571 ms. Trace:
I20250113 02:51:33.796607 1198 rpcz_store.cc:276] 0113 02:51:32.225185 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:32.239133 (+ 13948us) service_pool.cc:225] Handling call
0113 02:51:33.796428 (+1557295us) ts_tablet_manager.cc:1145] Deleting tablet 87e6a4e4a8434bbcbbe70ddf634accba
0113 02:51:33.796438 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:33.796510 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1555018}
W20250113 02:51:33.796711 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 29b1444efd28483297bafc73a33e7a54 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 29b1444efd28483297bafc73a33e7a54 already in progress: creating tablet
W20250113 02:51:33.797430 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 87e6a4e4a8434bbcbbe70ddf634accba because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 87e6a4e4a8434bbcbbe70ddf634accba already in progress: creating tablet
W20250113 02:51:33.797753 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 58b0e9391d614b578974f0e29b020c16 consensus state has no leader
I20250113 02:51:33.793095 1041 tablet_service.cc:1514] Processing DeleteTablet for tablet 21bcdd741104474bad7023426856d18f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:33 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:33.799589 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_213c7f5ea7184ab5ab5d3699b4a684bd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:33.801112 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_213c7f5ea7184ab5ab5d3699b4a684bd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:33.794467 1192 tablet_bootstrap.cc:492] T 3b1910496e0849b193e390a5ef1a6b2b P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:33.802404 1192 ts_tablet_manager.cc:1397] T 3b1910496e0849b193e390a5ef1a6b2b P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.065s user 0.005s sys 0.000s
I20250113 02:51:33.802907 1195 ts_tablet_manager.cc:1918] T 056bf1e67be04d179eace3d93282b145 P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:33.804344 455 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_f6135d2bf1204fb5b661ea8a71c526a4" } modify_external_catalogs: true
I20250113 02:51:33.804508 455 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_f6135d2bf1204fb5b661ea8a71c526a4" } modify_external_catalogs: true
I20250113 02:51:33.806331 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 54aae7c131404facba9af3a3d143b4a4
I20250113 02:51:33.807459 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet 8adfa738c17a4169a5d99f6ceca75100 (table default.Table_9a9b1ced3bef4e648088e0781851128e [id=ae544735623e4cfd894dc9340aca177c]) successfully deleted
I20250113 02:51:33.807678 892 tablet_service.cc:1514] Processing DeleteTablet for tablet 54aae7c131404facba9af3a3d143b4a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:33 UTC) from {username='slave'} at 127.0.0.1:37366
W20250113 02:51:33.807940 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 12a28557b0cc42fdad927c86fcfa7b96 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 12a28557b0cc42fdad927c86fcfa7b96 already in progress: creating tablet
I20250113 02:51:33.808821 1041 tablet_service.cc:1514] Processing DeleteTablet for tablet 54aae7c131404facba9af3a3d143b4a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:33 UTC) from {username='slave'} at 127.0.0.1:60230
W20250113 02:51:33.809108 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 7b16286181684916ac84823f5fec563e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7b16286181684916ac84823f5fec563e already in progress: creating tablet
W20250113 02:51:33.809764 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet b16ff51101e0452ebc05374f0d1f4917 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b16ff51101e0452ebc05374f0d1f4917 already in progress: creating tablet
I20250113 02:51:33.809708 1192 raft_consensus.cc:357] T 3b1910496e0849b193e390a5ef1a6b2b P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:33.809903 1192 raft_consensus.cc:383] T 3b1910496e0849b193e390a5ef1a6b2b P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:33.809965 1192 raft_consensus.cc:738] T 3b1910496e0849b193e390a5ef1a6b2b P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:33.810227 1196 tablet_replica.cc:331] T 2778d7dc7d9c4843bbc9eb705cec71cd P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:33.810463 1196 raft_consensus.cc:2238] T 2778d7dc7d9c4843bbc9eb705cec71cd P df69f48656b44bb890f76587ee69cc0f [term 1 LEADER]: Raft consensus shutting down.
I20250113 02:51:33.810736 1196 pending_rounds.cc:62] T 2778d7dc7d9c4843bbc9eb705cec71cd P df69f48656b44bb890f76587ee69cc0f: Trying to abort 1 pending ops.
I20250113 02:51:33.810842 1196 pending_rounds.cc:69] T 2778d7dc7d9c4843bbc9eb705cec71cd P df69f48656b44bb890f76587ee69cc0f: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113673497282609152 op_type: NO_OP noop_request { }
I20250113 02:51:33.811002 1196 raft_consensus.cc:2883] T 2778d7dc7d9c4843bbc9eb705cec71cd P df69f48656b44bb890f76587ee69cc0f [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250113 02:51:33.811089 1198 tablet_replica.cc:331] T 2778d7dc7d9c4843bbc9eb705cec71cd P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:33.811152 1196 raft_consensus.cc:2267] T 2778d7dc7d9c4843bbc9eb705cec71cd P df69f48656b44bb890f76587ee69cc0f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:33.811465 1198 raft_consensus.cc:2238] T 2778d7dc7d9c4843bbc9eb705cec71cd P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:33.812551 1198 raft_consensus.cc:2267] T 2778d7dc7d9c4843bbc9eb705cec71cd P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:33.813261 1198 ts_tablet_manager.cc:1905] T 2778d7dc7d9c4843bbc9eb705cec71cd P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:33.813462 1196 ts_tablet_manager.cc:1905] T 2778d7dc7d9c4843bbc9eb705cec71cd P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:33.813529 1192 consensus_queue.cc:260] T 3b1910496e0849b193e390a5ef1a6b2b P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:33.815541 2112 raft_consensus.cc:491] T 51d9483ecacb413796d0fd9e7560eac1 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:33.815666 2112 raft_consensus.cc:513] T 51d9483ecacb413796d0fd9e7560eac1 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:33.816318 2112 leader_election.cc:290] T 51d9483ecacb413796d0fd9e7560eac1 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:33.816737 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51d9483ecacb413796d0fd9e7560eac1" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:33.817274 995 leader_election.cc:343] T 51d9483ecacb413796d0fd9e7560eac1 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Not found: Tablet not found: 51d9483ecacb413796d0fd9e7560eac1
W20250113 02:51:33.817384 998 leader_election.cc:343] T 51d9483ecacb413796d0fd9e7560eac1 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:33.817492 998 leader_election.cc:304] T 51d9483ecacb413796d0fd9e7560eac1 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4216a7f853d642b7bc3e5cff458f3aca; no voters: 1186fa869eb14812832fa6562531f196, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:33.817976 2112 raft_consensus.cc:2743] T 51d9483ecacb413796d0fd9e7560eac1 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:33.819806 1192 ts_tablet_manager.cc:1428] T 3b1910496e0849b193e390a5ef1a6b2b P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.017s user 0.001s sys 0.001s
I20250113 02:51:33.809123 747 tablet_service.cc:1514] Processing DeleteTablet for tablet 54aae7c131404facba9af3a3d143b4a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:33 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:33.820398 1195 log.cc:1198] T 056bf1e67be04d179eace3d93282b145 P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/056bf1e67be04d179eace3d93282b145
I20250113 02:51:33.820849 1192 tablet_bootstrap.cc:492] T 6666922d677a4c678f80d7c6c47fd2a5 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:33.821231 1195 ts_tablet_manager.cc:1939] T 056bf1e67be04d179eace3d93282b145 P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:33.821633 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 888) took 1161 ms. Trace:
W20250113 02:51:33.823659 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet cd5589a82daa4624b3e8f94a01ae70d9 consensus state has no leader
I20250113 02:51:33.823783 1195 rpcz_store.cc:276] 0113 02:51:32.660156 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:32.661753 (+ 1597us) service_pool.cc:225] Handling call
0113 02:51:33.757378 (+1095625us) ts_tablet_manager.cc:1145] Deleting tablet 056bf1e67be04d179eace3d93282b145
0113 02:51:33.757386 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:33.795582 (+ 38196us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:33.802793 (+ 7211us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:33.821619 (+ 18826us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1095510}
I20250113 02:51:33.824210 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet 056bf1e67be04d179eace3d93282b145 (table default.Table_f5f150e8da7a4848bfab67568975d66f [id=5a295e63ad43499f90b2993ac92a16cf]) successfully deleted
I20250113 02:51:33.826963 1192 tablet_bootstrap.cc:654] T 6666922d677a4c678f80d7c6c47fd2a5 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:33.828258 1195 tablet_replica.cc:331] T 6666922d677a4c678f80d7c6c47fd2a5 P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:33.828727 1195 raft_consensus.cc:2238] T 6666922d677a4c678f80d7c6c47fd2a5 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:33.829169 1195 raft_consensus.cc:2267] T 6666922d677a4c678f80d7c6c47fd2a5 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:33.832607 1195 ts_tablet_manager.cc:1905] T 6666922d677a4c678f80d7c6c47fd2a5 P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:33.833477 1194 tablet_bootstrap.cc:492] T ff8ef6290698477391721fd65b9eeb28 P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:33.833524 446 master_service.cc:946] ReplaceTablet: received request to replace tablet 6d4d8b5cc8ad4c9e935d8afaf5766e66 from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:33.833699 1194 ts_tablet_manager.cc:1397] T ff8ef6290698477391721fd65b9eeb28 P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.054s user 0.003s sys 0.002s
I20250113 02:51:33.834388 1194 raft_consensus.cc:357] T ff8ef6290698477391721fd65b9eeb28 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:33.834609 1194 raft_consensus.cc:383] T ff8ef6290698477391721fd65b9eeb28 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:33.834688 1194 raft_consensus.cc:738] T ff8ef6290698477391721fd65b9eeb28 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:33.834939 1194 consensus_queue.cc:260] T ff8ef6290698477391721fd65b9eeb28 P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:33.835417 1194 ts_tablet_manager.cc:1428] T ff8ef6290698477391721fd65b9eeb28 P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250113 02:51:33.836687 1194 tablet_bootstrap.cc:492] T f68f9f4076d9475d938e795e72bdd468 P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
W20250113 02:51:33.837088 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 61c79029dfb74f01aaac0f7f828880d0 consensus state has no leader
I20250113 02:51:33.838402 1194 tablet_bootstrap.cc:654] T f68f9f4076d9475d938e795e72bdd468 P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:33.840826 1191 tablet_bootstrap.cc:492] T 33c665e149dd48e8868acd935de581bf P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:33.841691 892 tablet_service.cc:1514] Processing DeleteTablet for tablet ff8ef6290698477391721fd65b9eeb28 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:20 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:33.842347 1191 ts_tablet_manager.cc:1397] T 33c665e149dd48e8868acd935de581bf P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.112s user 0.003s sys 0.003s
I20250113 02:51:33.843189 1191 raft_consensus.cc:357] T 33c665e149dd48e8868acd935de581bf P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:33.843575 1191 raft_consensus.cc:383] T 33c665e149dd48e8868acd935de581bf P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:33.843679 1191 raft_consensus.cc:738] T 33c665e149dd48e8868acd935de581bf P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:33.844060 1198 ts_tablet_manager.cc:1918] T 2778d7dc7d9c4843bbc9eb705cec71cd P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:33.844012 1191 consensus_queue.cc:260] T 33c665e149dd48e8868acd935de581bf P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:33.845413 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_be0e2878c7d74f7d8fcb4280e8cf6d0b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:33.845943 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_be0e2878c7d74f7d8fcb4280e8cf6d0b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:33.846946 1191 ts_tablet_manager.cc:1428] T 33c665e149dd48e8868acd935de581bf P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250113 02:51:33.849635 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_64430a2da8d04ce88ae813a23452ecb6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:33.850137 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_64430a2da8d04ce88ae813a23452ecb6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:33.851168 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_6a7ae5ff14af4fa0aacbafcc8a8dfc9e" } modify_external_catalogs: true
I20250113 02:51:33.851367 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_6a7ae5ff14af4fa0aacbafcc8a8dfc9e" } modify_external_catalogs: true
I20250113 02:51:33.851604 1198 log.cc:1198] T 2778d7dc7d9c4843bbc9eb705cec71cd P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/2778d7dc7d9c4843bbc9eb705cec71cd
I20250113 02:51:33.851925 1198 ts_tablet_manager.cc:1939] T 2778d7dc7d9c4843bbc9eb705cec71cd P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:33.852335 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 863) took 1547 ms. Trace:
I20250113 02:51:33.852514 1192 tablet_bootstrap.cc:492] T 6666922d677a4c678f80d7c6c47fd2a5 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:33.852640 1192 ts_tablet_manager.cc:1397] T 6666922d677a4c678f80d7c6c47fd2a5 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.032s user 0.001s sys 0.001s
I20250113 02:51:33.852459 1198 rpcz_store.cc:276] 0113 02:51:32.304915 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:32.305689 (+ 774us) service_pool.cc:225] Handling call
0113 02:51:33.811043 (+1505354us) ts_tablet_manager.cc:1145] Deleting tablet 2778d7dc7d9c4843bbc9eb705cec71cd
0113 02:51:33.811053 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:33.830889 (+ 19836us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:33.843946 (+ 13057us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:33.852316 (+ 8370us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1505136}
I20250113 02:51:33.853080 1198 tablet_replica.cc:331] T 7d3562a48c6e4badbbcf37640e91b5ba P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:33.853295 1198 raft_consensus.cc:2238] T 7d3562a48c6e4badbbcf37640e91b5ba P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:33.853365 1192 raft_consensus.cc:357] T 6666922d677a4c678f80d7c6c47fd2a5 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:33.853533 1192 raft_consensus.cc:383] T 6666922d677a4c678f80d7c6c47fd2a5 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:33.853484 1198 raft_consensus.cc:2267] T 7d3562a48c6e4badbbcf37640e91b5ba P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:33.853732 1195 ts_tablet_manager.cc:1918] T 6666922d677a4c678f80d7c6c47fd2a5 P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:33.854086 1198 ts_tablet_manager.cc:1905] T 7d3562a48c6e4badbbcf37640e91b5ba P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:33.854350 1195 log.cc:1198] T 6666922d677a4c678f80d7c6c47fd2a5 P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/6666922d677a4c678f80d7c6c47fd2a5
I20250113 02:51:33.855810 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet 2778d7dc7d9c4843bbc9eb705cec71cd (table default.Table_b3284070cce54277bed74120ca8f199b [id=4fafe716b4704c0c96514223a04e480a]) successfully deleted
I20250113 02:51:33.855532 1191 tablet_bootstrap.cc:492] T fb18c16fb3ba44ec8a190b88d3bbd1b1 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:33.853587 1192 raft_consensus.cc:738] T 6666922d677a4c678f80d7c6c47fd2a5 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:33.856319 1192 consensus_queue.cc:260] T 6666922d677a4c678f80d7c6c47fd2a5 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:33.856961 1192 ts_tablet_manager.cc:1428] T 6666922d677a4c678f80d7c6c47fd2a5 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250113 02:51:33.857221 1192 tablet_bootstrap.cc:492] T f56230a2c75d4763a54671c79358f4d9 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:33.857887 1191 tablet_bootstrap.cc:654] T fb18c16fb3ba44ec8a190b88d3bbd1b1 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:33.858474 1192 tablet_bootstrap.cc:654] T f56230a2c75d4763a54671c79358f4d9 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:33.858721 1195 ts_tablet_manager.cc:1939] T 6666922d677a4c678f80d7c6c47fd2a5 P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:33.858999 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 884) took 1199 ms. Trace:
I20250113 02:51:33.859068 1195 rpcz_store.cc:276] 0113 02:51:32.659707 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:32.668528 (+ 8821us) service_pool.cc:225] Handling call
0113 02:51:33.828209 (+1159681us) ts_tablet_manager.cc:1145] Deleting tablet 6666922d677a4c678f80d7c6c47fd2a5
0113 02:51:33.828220 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:33.841365 (+ 13145us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:33.853643 (+ 12278us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:33.858983 (+ 5340us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1159273}
I20250113 02:51:33.862118 1195 tablet_replica.cc:331] T 3b1910496e0849b193e390a5ef1a6b2b P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:33.862455 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet 6666922d677a4c678f80d7c6c47fd2a5 (table default.Table_03b7491a47674f7187f4952feb9b44e2 [id=6e014400813c4402891b32bcbe3555e7]) successfully deleted
I20250113 02:51:33.864393 1195 raft_consensus.cc:2238] T 3b1910496e0849b193e390a5ef1a6b2b P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:33.864552 1195 raft_consensus.cc:2267] T 3b1910496e0849b193e390a5ef1a6b2b P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:33.865020 1195 ts_tablet_manager.cc:1905] T 3b1910496e0849b193e390a5ef1a6b2b P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:33.869228 449 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_7bb55657ad1b451c86260845c932b23e" } modify_external_catalogs: true
I20250113 02:51:33.869412 449 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_7bb55657ad1b451c86260845c932b23e" } modify_external_catalogs: true
I20250113 02:51:33.870079 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet d52b9d0f60714a31918c1645178f3525
I20250113 02:51:33.871655 747 tablet_service.cc:1514] Processing DeleteTablet for tablet d52b9d0f60714a31918c1645178f3525 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:33 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:33.872030 1041 tablet_service.cc:1514] Processing DeleteTablet for tablet d52b9d0f60714a31918c1645178f3525 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:33 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:33.884900 2112 raft_consensus.cc:491] T 6d4d8b5cc8ad4c9e935d8afaf5766e66 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:33.885035 2112 raft_consensus.cc:513] T 6d4d8b5cc8ad4c9e935d8afaf5766e66 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:33.885619 2112 leader_election.cc:290] T 6d4d8b5cc8ad4c9e935d8afaf5766e66 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
W20250113 02:51:33.886253 995 leader_election.cc:343] T 6d4d8b5cc8ad4c9e935d8afaf5766e66 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Not found: Tablet not found: 6d4d8b5cc8ad4c9e935d8afaf5766e66
I20250113 02:51:33.887382 1198 ts_tablet_manager.cc:1918] T 7d3562a48c6e4badbbcf37640e91b5ba P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:33.887732 1198 log.cc:1198] T 7d3562a48c6e4badbbcf37640e91b5ba P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/7d3562a48c6e4badbbcf37640e91b5ba
I20250113 02:51:33.888132 1198 ts_tablet_manager.cc:1939] T 7d3562a48c6e4badbbcf37640e91b5ba P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:33.888468 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 860) took 1590 ms. Trace:
I20250113 02:51:33.888595 1198 rpcz_store.cc:276] 0113 02:51:32.297575 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:32.297845 (+ 270us) service_pool.cc:225] Handling call
0113 02:51:33.853034 (+1555189us) ts_tablet_manager.cc:1145] Deleting tablet 7d3562a48c6e4badbbcf37640e91b5ba
0113 02:51:33.853044 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:33.881723 (+ 28679us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:33.887302 (+ 5579us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:33.888455 (+ 1153us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1531313}
I20250113 02:51:33.889140 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet 7d3562a48c6e4badbbcf37640e91b5ba (table default.Table_06f81a97cb014a3bad594e88c8cf4733 [id=7c2bc6d4232f4585b84d71e3e1f85794]) successfully deleted
I20250113 02:51:33.889463 1198 tablet_replica.cc:331] T b0b267b0d9c54ba496c89f6cb287dffd P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:33.889669 1198 raft_consensus.cc:2238] T b0b267b0d9c54ba496c89f6cb287dffd P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:33.889873 1198 raft_consensus.cc:2267] T b0b267b0d9c54ba496c89f6cb287dffd P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:33.890579 1198 ts_tablet_manager.cc:1905] T b0b267b0d9c54ba496c89f6cb287dffd P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:33.890995 1192 tablet_bootstrap.cc:492] T f56230a2c75d4763a54671c79358f4d9 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:33.891119 1192 ts_tablet_manager.cc:1397] T f56230a2c75d4763a54671c79358f4d9 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.034s user 0.000s sys 0.004s
I20250113 02:51:33.891774 1192 raft_consensus.cc:357] T f56230a2c75d4763a54671c79358f4d9 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:33.891954 1192 raft_consensus.cc:383] T f56230a2c75d4763a54671c79358f4d9 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:33.892050 1192 raft_consensus.cc:738] T f56230a2c75d4763a54671c79358f4d9 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:33.892231 1192 consensus_queue.cc:260] T f56230a2c75d4763a54671c79358f4d9 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:33.893625 892 tablet_service.cc:1514] Processing DeleteTablet for tablet d52b9d0f60714a31918c1645178f3525 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:33 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:33.893847 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d4d8b5cc8ad4c9e935d8afaf5766e66" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:33.894942 998 leader_election.cc:343] T 6d4d8b5cc8ad4c9e935d8afaf5766e66 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:33.895041 998 leader_election.cc:304] T 6d4d8b5cc8ad4c9e935d8afaf5766e66 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4216a7f853d642b7bc3e5cff458f3aca; no voters: 1186fa869eb14812832fa6562531f196, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:33.896193 1195 ts_tablet_manager.cc:1918] T 3b1910496e0849b193e390a5ef1a6b2b P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:33.896368 1195 log.cc:1198] T 3b1910496e0849b193e390a5ef1a6b2b P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/3b1910496e0849b193e390a5ef1a6b2b
I20250113 02:51:33.896571 1198 ts_tablet_manager.cc:1918] T b0b267b0d9c54ba496c89f6cb287dffd P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:33.896723 1195 ts_tablet_manager.cc:1939] T 3b1910496e0849b193e390a5ef1a6b2b P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:33.896728 1198 log.cc:1198] T b0b267b0d9c54ba496c89f6cb287dffd P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/b0b267b0d9c54ba496c89f6cb287dffd
I20250113 02:51:33.897029 1198 ts_tablet_manager.cc:1939] T b0b267b0d9c54ba496c89f6cb287dffd P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:33.898605 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 883) took 1269 ms. Trace:
I20250113 02:51:33.898841 1195 rpcz_store.cc:276] 0113 02:51:32.628850 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:32.713690 (+ 84840us) service_pool.cc:225] Handling call
0113 02:51:33.862056 (+1148366us) ts_tablet_manager.cc:1145] Deleting tablet 3b1910496e0849b193e390a5ef1a6b2b
0113 02:51:33.862068 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:33.878682 (+ 16614us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:33.896096 (+ 17414us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:33.898577 (+ 2481us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1148171}
I20250113 02:51:33.899657 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet 3b1910496e0849b193e390a5ef1a6b2b (table default.Table_2f3720d8d897466388c3f9dde157d701 [id=8af6fc6e393d4cd3a5eb0671346df592]) successfully deleted
I20250113 02:51:33.899760 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 906) took 1149 ms. Trace:
I20250113 02:51:33.899858 1195 rpcz_store.cc:276] 0113 02:51:32.750580 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:32.750637 (+ 57us) service_pool.cc:225] Handling call
0113 02:51:33.899533 (+1148896us) ts_tablet_manager.cc:1145] Deleting tablet 1f128d9bf2924fb1bada3a8462818004
0113 02:51:33.899568 (+ 35us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:33.899717 (+ 149us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1148677}
W20250113 02:51:33.900439 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 1f128d9bf2924fb1bada3a8462818004 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1f128d9bf2924fb1bada3a8462818004 already in progress: creating tablet
I20250113 02:51:33.900552 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 907) took 1149 ms. Trace:
I20250113 02:51:33.900753 1195 rpcz_store.cc:276] 0113 02:51:32.750687 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:32.750893 (+ 206us) service_pool.cc:225] Handling call
0113 02:51:33.900361 (+1149468us) ts_tablet_manager.cc:1145] Deleting tablet e6e8966da70746d7a10c7243ba2f6f87
0113 02:51:33.900396 (+ 35us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:33.900536 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1149270}
W20250113 02:51:33.901255 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet e6e8966da70746d7a10c7243ba2f6f87 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e6e8966da70746d7a10c7243ba2f6f87 already in progress: creating tablet
I20250113 02:51:33.901309 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 905) took 1150 ms. Trace:
I20250113 02:51:33.901422 1195 rpcz_store.cc:276] 0113 02:51:32.750463 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:32.750963 (+ 500us) service_pool.cc:225] Handling call
0113 02:51:33.901155 (+1150192us) ts_tablet_manager.cc:1145] Deleting tablet 7213e3284b4d4cb581d9fb946a9f87e9
0113 02:51:33.901167 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:33.901293 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1149841}
W20250113 02:51:33.901918 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 7213e3284b4d4cb581d9fb946a9f87e9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7213e3284b4d4cb581d9fb946a9f87e9 already in progress: creating tablet
I20250113 02:51:33.901974 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 908) took 1143 ms. Trace:
I20250113 02:51:33.902089 1195 rpcz_store.cc:276] 0113 02:51:32.758795 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:32.758965 (+ 170us) service_pool.cc:225] Handling call
0113 02:51:33.901816 (+1142851us) ts_tablet_manager.cc:1145] Deleting tablet 8cc78bf7d6324b7ba10a1b1045923f2b
0113 02:51:33.901826 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:33.901960 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1142666}
W20250113 02:51:33.902549 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 8cc78bf7d6324b7ba10a1b1045923f2b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8cc78bf7d6324b7ba10a1b1045923f2b already in progress: creating tablet
I20250113 02:51:33.902618 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 913) took 1116 ms. Trace:
I20250113 02:51:33.902734 1195 rpcz_store.cc:276] 0113 02:51:32.785765 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:32.787141 (+ 1376us) service_pool.cc:225] Handling call
0113 02:51:33.902443 (+1115302us) ts_tablet_manager.cc:1145] Deleting tablet b86d035e503344e1a3bd85c55131274e
0113 02:51:33.902475 (+ 32us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:33.902602 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":903300}
I20250113 02:51:33.902858 2112 raft_consensus.cc:2743] T 6d4d8b5cc8ad4c9e935d8afaf5766e66 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250113 02:51:33.903223 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet b86d035e503344e1a3bd85c55131274e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b86d035e503344e1a3bd85c55131274e already in progress: creating tablet
I20250113 02:51:33.903432 1192 ts_tablet_manager.cc:1428] T f56230a2c75d4763a54671c79358f4d9 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.012s user 0.000s sys 0.002s
W20250113 02:51:33.903596 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 36726b40ac5145bfb371871b3fff079a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 36726b40ac5145bfb371871b3fff079a already in progress: creating tablet
I20250113 02:51:33.903723 1195 tablet_replica.cc:331] T 9fff52e99c64462a8a73fccc3c22ca63 P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:33.903827 916 raft_consensus.cc:3054] T 378d59d314a84ddb84cf2ab041c51cf1 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:33.904085 1195 raft_consensus.cc:2238] T 9fff52e99c64462a8a73fccc3c22ca63 P 1186fa869eb14812832fa6562531f196 [term 1 LEADER]: Raft consensus shutting down.
I20250113 02:51:33.904630 1195 raft_consensus.cc:2267] T 9fff52e99c64462a8a73fccc3c22ca63 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:33.904805 1192 tablet_bootstrap.cc:492] T b8b6ffe690fc459db78b2b3bc12671dd P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:33.905596 1195 ts_tablet_manager.cc:1905] T 9fff52e99c64462a8a73fccc3c22ca63 P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:33.906390 1192 tablet_bootstrap.cc:654] T b8b6ffe690fc459db78b2b3bc12671dd P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:33.907768 2024 raft_consensus.cc:491] T ff8ef6290698477391721fd65b9eeb28 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:33.907883 2024 raft_consensus.cc:513] T ff8ef6290698477391721fd65b9eeb28 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:33.908437 2024 leader_election.cc:290] T ff8ef6290698477391721fd65b9eeb28 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587), 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
I20250113 02:51:33.909402 447 master_service.cc:946] ReplaceTablet: received request to replace tablet 41fcde988f72496b824271fa36c1a226 from {username='slave'} at 127.0.0.1:49258
W20250113 02:51:33.909451 851 leader_election.cc:343] T ff8ef6290698477391721fd65b9eeb28 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Not found: Tablet not found: ff8ef6290698477391721fd65b9eeb28
W20250113 02:51:33.910456 852 leader_election.cc:343] T ff8ef6290698477391721fd65b9eeb28 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Not found: Tablet not found: ff8ef6290698477391721fd65b9eeb28
I20250113 02:51:33.910557 852 leader_election.cc:304] T ff8ef6290698477391721fd65b9eeb28 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df69f48656b44bb890f76587ee69cc0f; no voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca
I20250113 02:51:33.910946 2024 raft_consensus.cc:2743] T ff8ef6290698477391721fd65b9eeb28 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:33.911583 1194 tablet_bootstrap.cc:492] T f68f9f4076d9475d938e795e72bdd468 P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:33.911707 1194 ts_tablet_manager.cc:1397] T f68f9f4076d9475d938e795e72bdd468 P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.075s user 0.009s sys 0.002s
I20250113 02:51:33.911819 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 861) took 1607 ms. Trace:
I20250113 02:51:33.911939 1198 rpcz_store.cc:276] 0113 02:51:32.304718 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:32.307721 (+ 3003us) service_pool.cc:225] Handling call
0113 02:51:33.889422 (+1581701us) ts_tablet_manager.cc:1145] Deleting tablet b0b267b0d9c54ba496c89f6cb287dffd
0113 02:51:33.889431 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:33.893745 (+ 4314us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:33.896503 (+ 2758us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:33.911799 (+ 15296us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":13900,"tablet-delete.queue_time_us":1566985}
I20250113 02:51:33.912089 1191 tablet_bootstrap.cc:492] T fb18c16fb3ba44ec8a190b88d3bbd1b1 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:33.912490 1191 ts_tablet_manager.cc:1397] T fb18c16fb3ba44ec8a190b88d3bbd1b1 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.057s user 0.003s sys 0.000s
I20250113 02:51:33.912552 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet b0b267b0d9c54ba496c89f6cb287dffd (table default.Table_eaa71690c1224509a1b527415d09f340 [id=dae9b614d621428ab168fe6a978799a0]) successfully deleted
I20250113 02:51:33.912603 1194 raft_consensus.cc:357] T f68f9f4076d9475d938e795e72bdd468 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:33.912766 1194 raft_consensus.cc:383] T f68f9f4076d9475d938e795e72bdd468 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:33.912860 1198 tablet_replica.cc:331] T 23e04961f072440fa3e3f9485e40f79c P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:33.912837 1194 raft_consensus.cc:738] T f68f9f4076d9475d938e795e72bdd468 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:33.913031 1194 consensus_queue.cc:260] T f68f9f4076d9475d938e795e72bdd468 P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:33.913497 1194 ts_tablet_manager.cc:1428] T f68f9f4076d9475d938e795e72bdd468 P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250113 02:51:33.913735 1194 tablet_bootstrap.cc:492] T 0810b9671f824d0a84741bc1904df247 P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:33.914448 2024 raft_consensus.cc:491] T c94eedb5dce44e6d896192c93127903a P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:33.913486 1198 raft_consensus.cc:2238] T 23e04961f072440fa3e3f9485e40f79c P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:33.914569 2024 raft_consensus.cc:513] T c94eedb5dce44e6d896192c93127903a P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:33.915100 2024 leader_election.cc:290] T c94eedb5dce44e6d896192c93127903a P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587), 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
W20250113 02:51:33.916954 851 leader_election.cc:343] T c94eedb5dce44e6d896192c93127903a P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Not found: Tablet not found: c94eedb5dce44e6d896192c93127903a
I20250113 02:51:33.917140 964 heartbeater.cc:346] Connected to a master server at 127.0.104.60:42067
I20250113 02:51:33.917224 1194 tablet_bootstrap.cc:654] T 0810b9671f824d0a84741bc1904df247 P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:33.919040 1196 ts_tablet_manager.cc:1918] T 2778d7dc7d9c4843bbc9eb705cec71cd P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250113 02:51:33.919245 1196 log.cc:1198] T 2778d7dc7d9c4843bbc9eb705cec71cd P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/2778d7dc7d9c4843bbc9eb705cec71cd
I20250113 02:51:33.919663 1196 ts_tablet_manager.cc:1939] T 2778d7dc7d9c4843bbc9eb705cec71cd P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:33.920419 1196 tablet_replica.cc:331] T a77c38a394114d6c8a8da99d7eef7669 P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:33.920523 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet 2778d7dc7d9c4843bbc9eb705cec71cd (table default.Table_b3284070cce54277bed74120ca8f199b [id=4fafe716b4704c0c96514223a04e480a]) successfully deleted
I20250113 02:51:33.917789 1191 raft_consensus.cc:357] T fb18c16fb3ba44ec8a190b88d3bbd1b1 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:33.920696 1196 raft_consensus.cc:2238] T a77c38a394114d6c8a8da99d7eef7669 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
W20250113 02:51:33.920761 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 35a13d9c065348e08de66044c04d221d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 35a13d9c065348e08de66044c04d221d already in progress: creating tablet
I20250113 02:51:33.920835 1196 raft_consensus.cc:2267] T a77c38a394114d6c8a8da99d7eef7669 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
W20250113 02:51:33.921021 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet ec8196505dab4e05bb6487b566cf3144 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ec8196505dab4e05bb6487b566cf3144 already in progress: creating tablet
I20250113 02:51:33.920688 1191 raft_consensus.cc:383] T fb18c16fb3ba44ec8a190b88d3bbd1b1 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250113 02:51:33.921903 852 leader_election.cc:343] T c94eedb5dce44e6d896192c93127903a P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Not found: Tablet not found: c94eedb5dce44e6d896192c93127903a
I20250113 02:51:33.921487 1191 raft_consensus.cc:738] T fb18c16fb3ba44ec8a190b88d3bbd1b1 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:33.922026 852 leader_election.cc:304] T c94eedb5dce44e6d896192c93127903a P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df69f48656b44bb890f76587ee69cc0f; no voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca
I20250113 02:51:33.914819 1198 raft_consensus.cc:2267] T 23e04961f072440fa3e3f9485e40f79c P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:33.923326 2024 raft_consensus.cc:2743] T c94eedb5dce44e6d896192c93127903a P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:33.922900 1191 consensus_queue.cc:260] T fb18c16fb3ba44ec8a190b88d3bbd1b1 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:33.923863 453 catalog_manager.cc:5526] T 1a63f485caf54068abd6b50d7de3d14a P 4216a7f853d642b7bc3e5cff458f3aca reported cstate change: term changed from 0 to 1, leader changed from <none> to 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3). New cstate: current_term: 1 leader_uuid: "4216a7f853d642b7bc3e5cff458f3aca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } health_report { overall_health: UNKNOWN } } }
I20250113 02:51:33.924366 453 catalog_manager.cc:5618] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587) does not have the latest schema for tablet 1a63f485caf54068abd6b50d7de3d14a (table default.Table_f8a6812c7fb94ea69cd9bf775b914ea6 [id=71508871e2d5469a91aaf0e19b02b638]). Expected version 1 got 0
I20250113 02:51:33.921900 1196 ts_tablet_manager.cc:1905] T a77c38a394114d6c8a8da99d7eef7669 P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:33.924765 453 catalog_manager.cc:5526] T 6e1284c8aaba44b5810b0d06d36757eb P 4216a7f853d642b7bc3e5cff458f3aca reported cstate change: term changed from 0 to 1, leader changed from <none> to 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3). New cstate: current_term: 1 leader_uuid: "4216a7f853d642b7bc3e5cff458f3aca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } health_report { overall_health: HEALTHY } } }
I20250113 02:51:33.925287 453 catalog_manager.cc:5618] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587) does not have the latest schema for tablet 6e1284c8aaba44b5810b0d06d36757eb (table default.Table_bfc5031f68f64006aceefe2ee986223a [id=aa4679fc0d7148e3bef21342b3a0ea22]). Expected version 2 got 0
I20250113 02:51:33.925627 454 catalog_manager.cc:6071] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: default.Table_810d62357c794aeaa18da712725a338f [id=08db71e435a24e36bdb20f9a5f1cc096] alter complete (version 3)
I20250113 02:51:33.927417 742 tablet_service.cc:1514] Processing DeleteTablet for tablet 999fecb3fc65466cad21cc1ff5ef2d44 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:23 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:33.927610 2109 raft_consensus.cc:491] T 33c665e149dd48e8868acd935de581bf P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:33.927742 2109 raft_consensus.cc:513] T 33c665e149dd48e8868acd935de581bf P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:33.928352 2109 leader_election.cc:290] T 33c665e149dd48e8868acd935de581bf P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:33.928660 747 tablet_service.cc:1514] Processing DeleteTablet for tablet 0916b41c171a419791797559fbc2807d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:23 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:33.928889 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 02bf7042d4ce4077851a44bbfb6c4b46
I20250113 02:51:33.929342 450 catalog_manager.cc:6596] ReplaceTablet: tablet 02bf7042d4ce4077851a44bbfb6c4b46 deleted and replaced by tablet a894199f21aa4dba8a03583ea14a9092
I20250113 02:51:33.929294 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33c665e149dd48e8868acd935de581bf" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:33.929937 892 tablet_service.cc:1514] Processing DeleteTablet for tablet 02bf7042d4ce4077851a44bbfb6c4b46 with delete_type TABLET_DATA_DELETED (replaced by tablet a894199f21aa4dba8a03583ea14a9092) from {username='slave'} at 127.0.0.1:37366
W20250113 02:51:33.930824 711 leader_election.cc:343] T 33c665e149dd48e8868acd935de581bf P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:33.930824 747 tablet_service.cc:1514] Processing DeleteTablet for tablet 02bf7042d4ce4077851a44bbfb6c4b46 with delete_type TABLET_DATA_DELETED (replaced by tablet a894199f21aa4dba8a03583ea14a9092) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:33.933727 1191 ts_tablet_manager.cc:1428] T fb18c16fb3ba44ec8a190b88d3bbd1b1 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.021s user 0.003s sys 0.000s
I20250113 02:51:33.935732 1191 tablet_bootstrap.cc:492] T e70658c90555463498d3fc2e2f13ea7f P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:33.936501 820 heartbeater.cc:346] Connected to a master server at 127.0.104.60:42067
I20250113 02:51:33.922649 1198 ts_tablet_manager.cc:1905] T 23e04961f072440fa3e3f9485e40f79c P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:33.942291 454 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_64856bab03754bdc9ded6e123a7d1c7e" } new_table_name: "default.Table_4f5f9d0a5adb402598009f489797d7a1" modify_external_catalogs: true
I20250113 02:51:33.928817 1062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33c665e149dd48e8868acd935de581bf" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:33.930346 1041 tablet_service.cc:1514] Processing DeleteTablet for tablet 02bf7042d4ce4077851a44bbfb6c4b46 with delete_type TABLET_DATA_DELETED (replaced by tablet a894199f21aa4dba8a03583ea14a9092) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:33.937260 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet 99c8046f5d124ca49899df365114c69f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:23 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:33.939739 1039 tablet_service.cc:1514] Processing DeleteTablet for tablet 0539cbbe47e64c039ad39c4b52edcc4b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:27 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:33.939878 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet c7243d27a1c642269a3d0e67632e458c with delete_type TABLET_DATA_DELETED (replaced by tablet ed68a2b7ebcf4841bef54798dd3ed80d) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:33.939967 1036 tablet_service.cc:1514] Processing DeleteTablet for tablet 51d9483ecacb413796d0fd9e7560eac1 with delete_type TABLET_DATA_DELETED (replaced by tablet 8898e1e9efce453488fa4a4562a82127) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:33.948660 1191 tablet_bootstrap.cc:654] T e70658c90555463498d3fc2e2f13ea7f P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:33.948988 742 tablet_service.cc:1467] Processing CreateTablet for tablet 75116fcc95ea464d9dbc0129623f4c3d (DEFAULT_TABLE table=default.Table_c2fca6e58b3947bcacaeec643890f1f9 [id=5f664f114dbf474e80a4ecdd88216201]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:33.949352 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 75116fcc95ea464d9dbc0129623f4c3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:33.949853 751 tablet_service.cc:1467] Processing CreateTablet for tablet 4a3ba2cd77b8416db72d894b09cc1100 (DEFAULT_TABLE table=default.Table_a969335ba8de47a49dc4629865ee4a1a [id=12470ce35ef74aa58ee01ed82937163c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:33.950255 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a3ba2cd77b8416db72d894b09cc1100. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:33.950277 746 tablet_service.cc:1467] Processing CreateTablet for tablet 6887e74b88aa458f857728460521a96a (DEFAULT_TABLE table=default.Table_787937b2aa714be5b37053648c4240fe [id=591327669a2a4833937df0e6dd0b6008]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:33.950539 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6887e74b88aa458f857728460521a96a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:33.948637 747 tablet_service.cc:1467] Processing CreateTablet for tablet 85d13f8abbec484d82c4d0b86480531b (DEFAULT_TABLE table=default.Table_84475daaf7544a4b83c6846dffec23e0 [id=2c4597b2da774043a06332d53a500a53]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:33.953588 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 85d13f8abbec484d82c4d0b86480531b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:33.957042 1195 ts_tablet_manager.cc:1918] T 9fff52e99c64462a8a73fccc3c22ca63 P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250113 02:51:33.957890 1195 log.cc:1198] T 9fff52e99c64462a8a73fccc3c22ca63 P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/9fff52e99c64462a8a73fccc3c22ca63
I20250113 02:51:33.959867 1192 tablet_bootstrap.cc:492] T b8b6ffe690fc459db78b2b3bc12671dd P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:33.960155 1192 ts_tablet_manager.cc:1397] T b8b6ffe690fc459db78b2b3bc12671dd P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.055s user 0.001s sys 0.003s
W20250113 02:51:33.960660 709 leader_election.cc:343] T 33c665e149dd48e8868acd935de581bf P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:33.960772 709 leader_election.cc:304] T 33c665e149dd48e8868acd935de581bf P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:33.961179 2109 raft_consensus.cc:2743] T 33c665e149dd48e8868acd935de581bf P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:33.961658 1192 raft_consensus.cc:357] T b8b6ffe690fc459db78b2b3bc12671dd P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:33.961994 1192 raft_consensus.cc:383] T b8b6ffe690fc459db78b2b3bc12671dd P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:33.962177 1192 raft_consensus.cc:738] T b8b6ffe690fc459db78b2b3bc12671dd P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:33.962579 744 tablet_service.cc:1467] Processing CreateTablet for tablet 0269663b35f148c4a2a93cfa6132d0f8 (DEFAULT_TABLE table=default.Table_bf1d5374f8a54f8abe3314c2c9615fdf [id=47fa61abd6f14b16b52a694c72fb6eb2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:33.962762 738 tablet_service.cc:1467] Processing CreateTablet for tablet ae1df9a4958645ff9290c7fa89dca5cf (DEFAULT_TABLE table=default.Table_1e7d9a88187442b7bf4ae616cfd746d0 [id=89a1291143c248b0a77415b7b279e0a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:33.962891 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0269663b35f148c4a2a93cfa6132d0f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:33.963040 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae1df9a4958645ff9290c7fa89dca5cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:33.963001 754 tablet_service.cc:1467] Processing CreateTablet for tablet 141a8eb1a30f41549786117f9e2d9d5f (DEFAULT_TABLE table=default.Table_b4f66c4bb40c4135ac5516b28550f0d5 [id=2112c1b322a548de902edee2bcb72bee]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:33.963073 752 tablet_service.cc:1467] Processing CreateTablet for tablet efe5f5ffd2c8423686b4007ded447466 (DEFAULT_TABLE table=default.Table_1f3f594aac1e4cd2bcec522b01f7e746 [id=f43e5f7cd9824f60be8ead15a07b1e46]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:33.963315 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 141a8eb1a30f41549786117f9e2d9d5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:33.963315 743 tablet_service.cc:1467] Processing CreateTablet for tablet d2eb86388a464ee0840bd5dcde2f2189 (DEFAULT_TABLE table=default.Table_b05aa043ae0e44b5a2a881764be3dbf0 [id=b503f19f124c44208c633f7b520e3f8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:33.963553 739 tablet_service.cc:1467] Processing CreateTablet for tablet c8966b3b93144a30813bfdaf04f9c43a (DEFAULT_TABLE table=default.Table_3cfdcce7afbf48ee8cf67f1a4c839612 [id=be95e01bc4844382a7a5635a609f9c64]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:33.963380 736 tablet_service.cc:1467] Processing CreateTablet for tablet d147f3b1545d4807a397012205afd47c (DEFAULT_TABLE table=default.Table_c9adaef9de094549a9cbd3340c1b33b8 [id=ebecbbe340454a0390f85ff8ceea6417]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:33.965289 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_4e05dc66d38a47b982d5b2389cfc6582"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:33.965838 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4e05dc66d38a47b982d5b2389cfc6582 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:33.966274 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d2eb86388a464ee0840bd5dcde2f2189. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:33.968022 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_ceb8d423519441d595d613778ab75514"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:33.968115 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_f905559759c6440b9ae154f582126dc2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:33.968494 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f905559759c6440b9ae154f582126dc2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250113 02:51:33.968490 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ceb8d423519441d595d613778ab75514 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:33.962512 1192 consensus_queue.cc:260] T b8b6ffe690fc459db78b2b3bc12671dd P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:33.971312 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c8966b3b93144a30813bfdaf04f9c43a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:33.976372 736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d147f3b1545d4807a397012205afd47c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:33.977931 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet efe5f5ffd2c8423686b4007ded447466. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:33.980119 2109 raft_consensus.cc:491] T 0916b41c171a419791797559fbc2807d P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:33.980235 2109 raft_consensus.cc:513] T 0916b41c171a419791797559fbc2807d P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:33.980516 2112 raft_consensus.cc:491] T 05bf5ab0df874dd5b6559f6d82069184 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:33.980643 2112 raft_consensus.cc:513] T 05bf5ab0df874dd5b6559f6d82069184 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:33.980823 2109 leader_election.cc:290] T 0916b41c171a419791797559fbc2807d P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:33.981222 2112 leader_election.cc:290] T 05bf5ab0df874dd5b6559f6d82069184 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:33.981688 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0916b41c171a419791797559fbc2807d" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:33.982043 2024 raft_consensus.cc:491] T 4c615b0b83d74fcfae0c51047dbfbbf4 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:33.982156 2024 raft_consensus.cc:513] T 4c615b0b83d74fcfae0c51047dbfbbf4 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:33.982682 880 tablet_service.cc:1467] Processing CreateTablet for tablet d2eb86388a464ee0840bd5dcde2f2189 (DEFAULT_TABLE table=default.Table_b05aa043ae0e44b5a2a881764be3dbf0 [id=b503f19f124c44208c633f7b520e3f8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:33.982756 2024 leader_election.cc:290] T 4c615b0b83d74fcfae0c51047dbfbbf4 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587), 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
I20250113 02:51:33.982946 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05bf5ab0df874dd5b6559f6d82069184" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:33.983018 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d2eb86388a464ee0840bd5dcde2f2189. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:33.983074 890 tablet_service.cc:1467] Processing CreateTablet for tablet 75116fcc95ea464d9dbc0129623f4c3d (DEFAULT_TABLE table=default.Table_c2fca6e58b3947bcacaeec643890f1f9 [id=5f664f114dbf474e80a4ecdd88216201]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:33.983361 888 tablet_service.cc:1467] Processing CreateTablet for tablet 0269663b35f148c4a2a93cfa6132d0f8 (DEFAULT_TABLE table=default.Table_bf1d5374f8a54f8abe3314c2c9615fdf [id=47fa61abd6f14b16b52a694c72fb6eb2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:33.983598 883 tablet_service.cc:1467] Processing CreateTablet for tablet 6887e74b88aa458f857728460521a96a (DEFAULT_TABLE table=default.Table_787937b2aa714be5b37053648c4240fe [id=591327669a2a4833937df0e6dd0b6008]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:33.984174 887 tablet_service.cc:1467] Processing CreateTablet for tablet efe5f5ffd2c8423686b4007ded447466 (DEFAULT_TABLE table=default.Table_1f3f594aac1e4cd2bcec522b01f7e746 [id=f43e5f7cd9824f60be8ead15a07b1e46]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:33.984215 899 tablet_service.cc:1467] Processing CreateTablet for tablet 2cd3b62ecc774d2eb658ed34e05864e1 (DEFAULT_TABLE table=default.Table_d0642c34ba7d40c9888ec7521d3cb1e3 [id=d38e68ad1eab4262bb3d8aba3cd37b52]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:33.984560 891 tablet_service.cc:1467] Processing CreateTablet for tablet ae1df9a4958645ff9290c7fa89dca5cf (DEFAULT_TABLE table=default.Table_1e7d9a88187442b7bf4ae616cfd746d0 [id=89a1291143c248b0a77415b7b279e0a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:33.984593 895 tablet_service.cc:1467] Processing CreateTablet for tablet d147f3b1545d4807a397012205afd47c (DEFAULT_TABLE table=default.Table_c9adaef9de094549a9cbd3340c1b33b8 [id=ebecbbe340454a0390f85ff8ceea6417]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:33.984905 894 tablet_service.cc:1467] Processing CreateTablet for tablet 141a8eb1a30f41549786117f9e2d9d5f (DEFAULT_TABLE table=default.Table_b4f66c4bb40c4135ac5516b28550f0d5 [id=2112c1b322a548de902edee2bcb72bee]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:33.984930 892 tablet_service.cc:1467] Processing CreateTablet for tablet 85d13f8abbec484d82c4d0b86480531b (DEFAULT_TABLE table=default.Table_84475daaf7544a4b83c6846dffec23e0 [id=2c4597b2da774043a06332d53a500a53]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:33.985301 881 tablet_service.cc:1467] Processing CreateTablet for tablet c8966b3b93144a30813bfdaf04f9c43a (DEFAULT_TABLE table=default.Table_3cfdcce7afbf48ee8cf67f1a4c839612 [id=be95e01bc4844382a7a5635a609f9c64]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:33.985584 450 catalog_manager.cc:5618] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095) does not have the latest schema for tablet fb18c16fb3ba44ec8a190b88d3bbd1b1 (table default.Table_7a4628807cc44ebdadf927ff709de3a2 [id=75ca7cbd7b9548d784f9a77d0f0cb64f]). Expected version 1 got 0
I20250113 02:51:33.985800 885 tablet_service.cc:1467] Processing CreateTablet for tablet 4a3ba2cd77b8416db72d894b09cc1100 (DEFAULT_TABLE table=default.Table_a969335ba8de47a49dc4629865ee4a1a [id=12470ce35ef74aa58ee01ed82937163c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:33.986330 1040 tablet_service.cc:1467] Processing CreateTablet for tablet 0269663b35f148c4a2a93cfa6132d0f8 (DEFAULT_TABLE table=default.Table_bf1d5374f8a54f8abe3314c2c9615fdf [id=47fa61abd6f14b16b52a694c72fb6eb2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:33.983374 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 75116fcc95ea464d9dbc0129623f4c3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:33.986750 1032 tablet_service.cc:1467] Processing CreateTablet for tablet ae1df9a4958645ff9290c7fa89dca5cf (DEFAULT_TABLE table=default.Table_1e7d9a88187442b7bf4ae616cfd746d0 [id=89a1291143c248b0a77415b7b279e0a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:33.987195 1035 tablet_service.cc:1467] Processing CreateTablet for tablet d2eb86388a464ee0840bd5dcde2f2189 (DEFAULT_TABLE table=default.Table_b05aa043ae0e44b5a2a881764be3dbf0 [id=b503f19f124c44208c633f7b520e3f8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:33.987552 1030 tablet_service.cc:1467] Processing CreateTablet for tablet efe5f5ffd2c8423686b4007ded447466 (DEFAULT_TABLE table=default.Table_1f3f594aac1e4cd2bcec522b01f7e746 [id=f43e5f7cd9824f60be8ead15a07b1e46]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:33.987877 1038 tablet_service.cc:1467] Processing CreateTablet for tablet 141a8eb1a30f41549786117f9e2d9d5f (DEFAULT_TABLE table=default.Table_b4f66c4bb40c4135ac5516b28550f0d5 [id=2112c1b322a548de902edee2bcb72bee]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:33.988230 1029 tablet_service.cc:1467] Processing CreateTablet for tablet d147f3b1545d4807a397012205afd47c (DEFAULT_TABLE table=default.Table_c9adaef9de094549a9cbd3340c1b33b8 [id=ebecbbe340454a0390f85ff8ceea6417]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:33.988519 1024 tablet_service.cc:1467] Processing CreateTablet for tablet c8966b3b93144a30813bfdaf04f9c43a (DEFAULT_TABLE table=default.Table_3cfdcce7afbf48ee8cf67f1a4c839612 [id=be95e01bc4844382a7a5635a609f9c64]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:33.988839 1031 tablet_service.cc:1467] Processing CreateTablet for tablet 9351be9ce9b64f1bac13d34e83f17966 (DEFAULT_TABLE table=default.Table_fd78ad35561d4253aac07bd501c6aa30 [id=3ead3a19d2744e6a9fbf6c4849e84677]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:33.992210 1042 tablet_service.cc:1467] Processing CreateTablet for tablet 4a8f4920c9a44ed7a9ac5e3058a7d6ea (DEFAULT_TABLE table=default.Table_ddf6ee80ea4048b19e7ac7de9934b274 [id=68ebe66f06604fb3a9492c95f0d7dfa6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:33.992559 1034 tablet_service.cc:1467] Processing CreateTablet for tablet 80b9f58258ae49c1ab8d3be86411fabd (DEFAULT_TABLE table=default.Table_e3586ada32644653a92140e6ce9deed9 [id=0be3f772cbda4d9f852edb476009fa3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:33.992913 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6887e74b88aa458f857728460521a96a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:33.996152 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae1df9a4958645ff9290c7fa89dca5cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:33.996831 1023 tablet_service.cc:1467] Processing CreateTablet for tablet 75116fcc95ea464d9dbc0129623f4c3d (DEFAULT_TABLE table=default.Table_c2fca6e58b3947bcacaeec643890f1f9 [id=5f664f114dbf474e80a4ecdd88216201]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:33.997246 1036 tablet_service.cc:1467] Processing CreateTablet for tablet 85d13f8abbec484d82c4d0b86480531b (DEFAULT_TABLE table=default.Table_84475daaf7544a4b83c6846dffec23e0 [id=2c4597b2da774043a06332d53a500a53]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:33.998296 1192 ts_tablet_manager.cc:1428] T b8b6ffe690fc459db78b2b3bc12671dd P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.038s user 0.001s sys 0.001s
I20250113 02:51:33.998350 1041 tablet_service.cc:1467] Processing CreateTablet for tablet 2cd3b62ecc774d2eb658ed34e05864e1 (DEFAULT_TABLE table=default.Table_d0642c34ba7d40c9888ec7521d3cb1e3 [id=d38e68ad1eab4262bb3d8aba3cd37b52]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:33.998878 1026 tablet_service.cc:1467] Processing CreateTablet for tablet 4a3ba2cd77b8416db72d894b09cc1100 (DEFAULT_TABLE table=default.Table_a969335ba8de47a49dc4629865ee4a1a [id=12470ce35ef74aa58ee01ed82937163c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:34.000512 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet efe5f5ffd2c8423686b4007ded447466. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:34.000726 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c615b0b83d74fcfae0c51047dbfbbf4" candidate_uuid: "df69f48656b44bb890f76587ee69cc0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
W20250113 02:51:34.001525 711 leader_election.cc:343] T 0916b41c171a419791797559fbc2807d P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:34.003770 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0269663b35f148c4a2a93cfa6132d0f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:34.004696 1040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0269663b35f148c4a2a93cfa6132d0f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:34.006841 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d147f3b1545d4807a397012205afd47c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:34.010136 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae1df9a4958645ff9290c7fa89dca5cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:33.998843 1039 tablet_service.cc:1467] Processing CreateTablet for tablet 6887e74b88aa458f857728460521a96a (DEFAULT_TABLE table=default.Table_787937b2aa714be5b37053648c4240fe [id=591327669a2a4833937df0e6dd0b6008]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:34.012715 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6887e74b88aa458f857728460521a96a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:34.016260 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 85d13f8abbec484d82c4d0b86480531b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:34.016481 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 141a8eb1a30f41549786117f9e2d9d5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:34.016521 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d2eb86388a464ee0840bd5dcde2f2189. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:34.016721 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet efe5f5ffd2c8423686b4007ded447466. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:34.017671 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c8966b3b93144a30813bfdaf04f9c43a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:34.018038 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a3ba2cd77b8416db72d894b09cc1100. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:34.018517 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2cd3b62ecc774d2eb658ed34e05864e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:34.020174 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 141a8eb1a30f41549786117f9e2d9d5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:34.024241 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d147f3b1545d4807a397012205afd47c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:34.028187 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c8966b3b93144a30813bfdaf04f9c43a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:34.031348 1198 ts_tablet_manager.cc:1918] T 23e04961f072440fa3e3f9485e40f79c P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:34.031445 1198 log.cc:1198] T 23e04961f072440fa3e3f9485e40f79c P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/23e04961f072440fa3e3f9485e40f79c
I20250113 02:51:34.031798 1198 ts_tablet_manager.cc:1939] T 23e04961f072440fa3e3f9485e40f79c P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:34.032243 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9351be9ce9b64f1bac13d34e83f17966. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:34.032418 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 80b9f58258ae49c1ab8d3be86411fabd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:34.032660 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 75116fcc95ea464d9dbc0129623f4c3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:34.036082 1042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a8f4920c9a44ed7a9ac5e3058a7d6ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:34.036265 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2cd3b62ecc774d2eb658ed34e05864e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:34.036296 1192 tablet_bootstrap.cc:492] T 377ebf66bbef40988043f6200ae0893c P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:34.037918 1192 tablet_bootstrap.cc:654] T 377ebf66bbef40988043f6200ae0893c P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:34.040323 1036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 85d13f8abbec484d82c4d0b86480531b. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:34.042435 998 leader_election.cc:343] T 05bf5ab0df874dd5b6559f6d82069184 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:34.042843 1062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0916b41c171a419791797559fbc2807d" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
W20250113 02:51:34.043812 709 leader_election.cc:343] T 0916b41c171a419791797559fbc2807d P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:34.043958 709 leader_election.cc:304] T 0916b41c171a419791797559fbc2807d P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:34.044716 2109 raft_consensus.cc:2743] T 0916b41c171a419791797559fbc2807d P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:34.047569 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 9b2d0aa6f0e9456d95f4e48c360ab6e8
I20250113 02:51:34.048327 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet d2699bb529bf4f0f9d7f2668ffe4c9ff
I20250113 02:51:34.051085 738 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d15c7a01f854f0aaabcd30ec3166e31 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:31 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:34.051168 747 tablet_service.cc:1514] Processing DeleteTablet for tablet 33c665e149dd48e8868acd935de581bf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:20 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:34.051097 746 tablet_service.cc:1467] Processing CreateTablet for tablet cf85cf0ebf5e4fda9d8803ac05741b50 (DEFAULT_TABLE table=default.Table_d2d35cb51870448bbf935e20885ec960 [id=faf2aec59f834e5b8fd56d112e8c7020]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:34.051599 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cf85cf0ebf5e4fda9d8803ac05741b50. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:34.051528 754 tablet_service.cc:1467] Processing CreateTablet for tablet f7bac98bba2e4b8e9c72946c33bad17b (DEFAULT_TABLE table=default.Table_64b83cf4ac3548bd89fefb3779bd0e1a [id=c6c77d2667b443f3a7030fc98f8b8da5]), partition=RANGE (key) PARTITION UNBOUNDED
W20250113 02:51:34.051602 454 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9b9e7deda7334ff4a3429f70d9b3a7fc consensus state has no leader
I20250113 02:51:34.051851 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f7bac98bba2e4b8e9c72946c33bad17b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:34.052211 738 tablet_service.cc:1467] Processing CreateTablet for tablet 5f5315047bd346b99e51909438f9e60f (DEFAULT_TABLE table=default.Table_21eff11db1644c02bfa6ceb7d46e7eac [id=b45a2b0b9c26476da37d6cea04f9bacb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:34.052495 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f5315047bd346b99e51909438f9e60f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:34.053108 1926 raft_consensus.cc:491] T f56230a2c75d4763a54671c79358f4d9 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:34.053212 1926 raft_consensus.cc:513] T f56230a2c75d4763a54671c79358f4d9 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:34.055466 880 tablet_service.cc:1467] Processing CreateTablet for tablet cf85cf0ebf5e4fda9d8803ac05741b50 (DEFAULT_TABLE table=default.Table_d2d35cb51870448bbf935e20885ec960 [id=faf2aec59f834e5b8fd56d112e8c7020]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:34.055675 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_a632f9b27ba74e3b947fd6371082dde3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:34.056124 743 tablet_service.cc:1467] Processing CreateTablet for tablet ca1052d60b71464889a632ebe7885120 (DEFAULT_TABLE table=default.Table_deef0377723a44e4a4ed576829e0e763 [id=f61353d9514843eb9fe6e5414e387352]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:34.056121 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_1d41ead6635b49f380159a2312885b50" } modify_external_catalogs: true
W20250113 02:51:34.056361 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a632f9b27ba74e3b947fd6371082dde3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:34.056424 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca1052d60b71464889a632ebe7885120. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:34.056665 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_e2733f0c9d6a4e2b9f020c369a2e8e52"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:34.057132 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e2733f0c9d6a4e2b9f020c369a2e8e52 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:34.056314 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_1d41ead6635b49f380159a2312885b50" } modify_external_catalogs: true
I20250113 02:51:34.060135 2113 raft_consensus.cc:491] T c1fb2b36c0504cb4ba32671ae4c620d1 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:34.060181 744 tablet_service.cc:1467] Processing CreateTablet for tablet 4a8f4920c9a44ed7a9ac5e3058a7d6ea (DEFAULT_TABLE table=default.Table_ddf6ee80ea4048b19e7ac7de9934b274 [id=68ebe66f06604fb3a9492c95f0d7dfa6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:34.060258 2113 raft_consensus.cc:513] T c1fb2b36c0504cb4ba32671ae4c620d1 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:34.060672 890 tablet_service.cc:1467] Processing CreateTablet for tablet 9351be9ce9b64f1bac13d34e83f17966 (DEFAULT_TABLE table=default.Table_fd78ad35561d4253aac07bd501c6aa30 [id=3ead3a19d2744e6a9fbf6c4849e84677]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:34.061106 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9351be9ce9b64f1bac13d34e83f17966. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:34.061203 454 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_a71ef07a0c614223916ca517b610739d" } new_table_name: "default.Table_f4d4f601c3c44c599304d8eb7ceb30c9" modify_external_catalogs: true
I20250113 02:51:34.061990 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a8f4920c9a44ed7a9ac5e3058a7d6ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:34.063891 2024 raft_consensus.cc:491] T f68f9f4076d9475d938e795e72bdd468 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:34.073275 2109 raft_consensus.cc:491] T 4b07a985897641c3ac48021c2a4daedb P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:34.073410 2109 raft_consensus.cc:513] T 4b07a985897641c3ac48021c2a4daedb P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:34.073964 2109 leader_election.cc:290] T 4b07a985897641c3ac48021c2a4daedb P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:34.073964 738 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b2d0aa6f0e9456d95f4e48c360ab6e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:33 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:34.074792 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b07a985897641c3ac48021c2a4daedb" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:34.074558 751 tablet_service.cc:1467] Processing CreateTablet for tablet 2cd3b62ecc774d2eb658ed34e05864e1 (DEFAULT_TABLE table=default.Table_d0642c34ba7d40c9888ec7521d3cb1e3 [id=d38e68ad1eab4262bb3d8aba3cd37b52]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:34.075280 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2cd3b62ecc774d2eb658ed34e05864e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:34.075215 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b07a985897641c3ac48021c2a4daedb" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:34.075719 754 tablet_service.cc:1467] Processing CreateTablet for tablet c87118901d7f4790be6f17600bd7c946 (DEFAULT_TABLE table=default.Table_9ad6dc8c7c204c7ea01374d6d8c5844d [id=4919099d8b234d309c9910cbe4ff690f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:34.074911 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f56230a2c75d4763a54671c79358f4d9" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
I20250113 02:51:34.076223 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c87118901d7f4790be6f17600bd7c946. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:34.076292 775 raft_consensus.cc:2463] T f56230a2c75d4763a54671c79358f4d9 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 0.
I20250113 02:51:34.080209 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cf85cf0ebf5e4fda9d8803ac05741b50. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:34.064033 2024 raft_consensus.cc:513] T f68f9f4076d9475d938e795e72bdd468 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:34.080917 2024 leader_election.cc:290] T f68f9f4076d9475d938e795e72bdd468 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587), 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
W20250113 02:51:34.081985 851 leader_election.cc:343] T f68f9f4076d9475d938e795e72bdd468 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Not found: Tablet not found: f68f9f4076d9475d938e795e72bdd468
I20250113 02:51:34.082109 886 tablet_service.cc:1467] Processing CreateTablet for tablet 7e825fa063904d57b7c32c99e3ba7f46 (DEFAULT_TABLE table=default.Table_14aabaa1901741fe8c6f84fd8a397d10 [id=3548788a0bdf46a19328c3392fb1fd15]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:34.082470 896 tablet_service.cc:1467] Processing CreateTablet for tablet 5f5315047bd346b99e51909438f9e60f (DEFAULT_TABLE table=default.Table_21eff11db1644c02bfa6ceb7d46e7eac [id=b45a2b0b9c26476da37d6cea04f9bacb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:34.083297 897 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b2d0aa6f0e9456d95f4e48c360ab6e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:33 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:34.083395 898 tablet_service.cc:1514] Processing DeleteTablet for tablet d2699bb529bf4f0f9d7f2668ffe4c9ff with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:33 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:34.084215 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1fb2b36c0504cb4ba32671ae4c620d1" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:34.083706 884 tablet_service.cc:1467] Processing CreateTablet for tablet ca1052d60b71464889a632ebe7885120 (DEFAULT_TABLE table=default.Table_deef0377723a44e4a4ed576829e0e763 [id=f61353d9514843eb9fe6e5414e387352]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:34.085099 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca1052d60b71464889a632ebe7885120. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:34.085211 882 tablet_service.cc:1467] Processing CreateTablet for tablet c87118901d7f4790be6f17600bd7c946 (DEFAULT_TABLE table=default.Table_9ad6dc8c7c204c7ea01374d6d8c5844d [id=4919099d8b234d309c9910cbe4ff690f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:34.085589 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c87118901d7f4790be6f17600bd7c946. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:34.086776 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f5315047bd346b99e51909438f9e60f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:34.086789 1196 ts_tablet_manager.cc:1918] T a77c38a394114d6c8a8da99d7eef7669 P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:34.088824 1196 log.cc:1198] T a77c38a394114d6c8a8da99d7eef7669 P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/a77c38a394114d6c8a8da99d7eef7669
I20250113 02:51:34.089159 1196 ts_tablet_manager.cc:1939] T a77c38a394114d6c8a8da99d7eef7669 P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
W20250113 02:51:34.089409 711 leader_election.cc:343] T 4b07a985897641c3ac48021c2a4daedb P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:34.089578 1196 tablet_replica.cc:331] T a0ca2e3a3ce94f3aa20df64c89c7cf95 P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:34.089740 1196 raft_consensus.cc:2238] T a0ca2e3a3ce94f3aa20df64c89c7cf95 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:34.089877 1196 raft_consensus.cc:2267] T a0ca2e3a3ce94f3aa20df64c89c7cf95 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
W20250113 02:51:34.090310 998 leader_election.cc:343] T c1fb2b36c0504cb4ba32671ae4c620d1 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:34.089887 891 tablet_service.cc:1467] Processing CreateTablet for tablet 4a8f4920c9a44ed7a9ac5e3058a7d6ea (DEFAULT_TABLE table=default.Table_ddf6ee80ea4048b19e7ac7de9934b274 [id=68ebe66f06604fb3a9492c95f0d7dfa6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:34.090593 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet a77c38a394114d6c8a8da99d7eef7669 (table default.Table_649d3c3eaebd4d4da258735b332a4a75 [id=d1b33c3e180343f5b88dcb200e7c9dde]) successfully deleted
I20250113 02:51:34.090759 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a8f4920c9a44ed7a9ac5e3058a7d6ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:34.091257 883 tablet_service.cc:1467] Processing CreateTablet for tablet 80b9f58258ae49c1ab8d3be86411fabd (DEFAULT_TABLE table=default.Table_e3586ada32644653a92140e6ce9deed9 [id=0be3f772cbda4d9f852edb476009fa3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:34.091725 1196 ts_tablet_manager.cc:1905] T a0ca2e3a3ce94f3aa20df64c89c7cf95 P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:34.093084 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 80b9f58258ae49c1ab8d3be86411fabd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:34.093919 1926 leader_election.cc:290] T f56230a2c75d4763a54671c79358f4d9 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:34.094298 2010 raft_consensus.cc:491] T c7243d27a1c642269a3d0e67632e458c P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:34.094451 2010 raft_consensus.cc:513] T c7243d27a1c642269a3d0e67632e458c P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
W20250113 02:51:34.094645 852 leader_election.cc:343] T f68f9f4076d9475d938e795e72bdd468 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Not found: Tablet not found: f68f9f4076d9475d938e795e72bdd468
I20250113 02:51:34.094740 852 leader_election.cc:304] T f68f9f4076d9475d938e795e72bdd468 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df69f48656b44bb890f76587ee69cc0f; no voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca
I20250113 02:51:34.094997 2010 leader_election.cc:290] T c7243d27a1c642269a3d0e67632e458c P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:34.095125 2113 leader_election.cc:290] T c1fb2b36c0504cb4ba32671ae4c620d1 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:34.095184 2024 raft_consensus.cc:2743] T f68f9f4076d9475d938e795e72bdd468 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:34.095301 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a3ba2cd77b8416db72d894b09cc1100. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:34.095803 2109 raft_consensus.cc:491] T 1d15c7a01f854f0aaabcd30ec3166e31 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:34.095916 2109 raft_consensus.cc:513] T 1d15c7a01f854f0aaabcd30ec3166e31 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:34.096462 2109 leader_election.cc:290] T 1d15c7a01f854f0aaabcd30ec3166e31 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:34.112664 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 862) took 1807 ms. Trace:
I20250113 02:51:34.115873 746 tablet_service.cc:1514] Processing DeleteTablet for tablet d2699bb529bf4f0f9d7f2668ffe4c9ff with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:33 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:34.118607 742 tablet_service.cc:1467] Processing CreateTablet for tablet 7e825fa063904d57b7c32c99e3ba7f46 (DEFAULT_TABLE table=default.Table_14aabaa1901741fe8c6f84fd8a397d10 [id=3548788a0bdf46a19328c3392fb1fd15]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:34.119146 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7e825fa063904d57b7c32c99e3ba7f46. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:34.053802 1062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c615b0b83d74fcfae0c51047dbfbbf4" candidate_uuid: "df69f48656b44bb890f76587ee69cc0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
W20250113 02:51:34.127372 995 leader_election.cc:343] T 05bf5ab0df874dd5b6559f6d82069184 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Not found: Tablet not found: 05bf5ab0df874dd5b6559f6d82069184
I20250113 02:51:34.127532 995 leader_election.cc:304] T 05bf5ab0df874dd5b6559f6d82069184 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4216a7f853d642b7bc3e5cff458f3aca; no voters: 1186fa869eb14812832fa6562531f196, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:34.128190 2010 raft_consensus.cc:2743] T 05bf5ab0df874dd5b6559f6d82069184 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250113 02:51:34.128964 998 consensus_peers.cc:487] T 056bf1e67be04d179eace3d93282b145 P 4216a7f853d642b7bc3e5cff458f3aca -> Peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Couldn't send request to peer df69f48656b44bb890f76587ee69cc0f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250113 02:51:34.129335 995 leader_election.cc:343] T c1fb2b36c0504cb4ba32671ae4c620d1 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Not found: Tablet not found: c1fb2b36c0504cb4ba32671ae4c620d1
I20250113 02:51:34.129295 1198 rpcz_store.cc:276] 0113 02:51:32.304817 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:32.308216 (+ 3399us) service_pool.cc:225] Handling call
0113 02:51:33.912811 (+1604595us) ts_tablet_manager.cc:1145] Deleting tablet 23e04961f072440fa3e3f9485e40f79c
0113 02:51:33.912822 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:33.962066 (+ 49244us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:34.031280 (+ 69214us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:34.112638 (+ 81358us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1588705}
I20250113 02:51:34.129478 995 leader_election.cc:304] T c1fb2b36c0504cb4ba32671ae4c620d1 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4216a7f853d642b7bc3e5cff458f3aca; no voters: 1186fa869eb14812832fa6562531f196, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:34.129742 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 866) took 1729 ms. Trace:
I20250113 02:51:34.132685 995 leader_election.cc:304] T f56230a2c75d4763a54671c79358f4d9 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters:
W20250113 02:51:34.132985 995 leader_election.cc:343] T c7243d27a1c642269a3d0e67632e458c P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Not found: Tablet not found: c7243d27a1c642269a3d0e67632e458c
I20250113 02:51:34.134754 1042 tablet_service.cc:1514] Processing DeleteTablet for tablet 6666922d677a4c678f80d7c6c47fd2a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:28 UTC) from {username='slave'} at 127.0.0.1:60230
W20250113 02:51:34.137089 998 consensus_peers.cc:487] T 8adfa738c17a4169a5d99f6ceca75100 P 4216a7f853d642b7bc3e5cff458f3aca -> Peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Couldn't send request to peer df69f48656b44bb890f76587ee69cc0f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8adfa738c17a4169a5d99f6ceca75100. This is attempt 1: this message will repeat every 5th retry.
W20250113 02:51:34.137482 709 leader_election.cc:343] T 4b07a985897641c3ac48021c2a4daedb P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:34.137604 709 leader_election.cc:304] T 4b07a985897641c3ac48021c2a4daedb P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:34.138069 1062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d15c7a01f854f0aaabcd30ec3166e31" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:34.138844 2109 raft_consensus.cc:2743] T 4b07a985897641c3ac48021c2a4daedb P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250113 02:51:34.139932 709 leader_election.cc:343] T 1d15c7a01f854f0aaabcd30ec3166e31 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:34.141588 1040 tablet_service.cc:1467] Processing CreateTablet for tablet 5f5315047bd346b99e51909438f9e60f (DEFAULT_TABLE table=default.Table_21eff11db1644c02bfa6ceb7d46e7eac [id=b45a2b0b9c26476da37d6cea04f9bacb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:34.143368 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_ef81f67732f246589e2e2f2c8808dc00" } new_table_name: "default.Table_9209a23d3c264eb2ace750095618d82d" modify_external_catalogs: true
I20250113 02:51:34.145817 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet f56230a2c75d4763a54671c79358f4d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:22 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:34.146270 739 tablet_service.cc:1467] Processing CreateTablet for tablet 80b9f58258ae49c1ab8d3be86411fabd (DEFAULT_TABLE table=default.Table_e3586ada32644653a92140e6ce9deed9 [id=0be3f772cbda4d9f852edb476009fa3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:34.146695 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 80b9f58258ae49c1ab8d3be86411fabd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:34.148305 736 tablet_service.cc:1467] Processing CreateTablet for tablet 9351be9ce9b64f1bac13d34e83f17966 (DEFAULT_TABLE table=default.Table_fd78ad35561d4253aac07bd501c6aa30 [id=3ead3a19d2744e6a9fbf6c4849e84677]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:34.148387 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_9e652a1c2e5f47a8bd55590289e5836f" } new_table_name: "default.Table_a05992a3efc949979f04afb92236369a" modify_external_catalogs: true
I20250113 02:51:34.148628 736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9351be9ce9b64f1bac13d34e83f17966. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:34.148624 448 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_7f3a0392b4234e48835b8c3c5ec34b15" } modify_external_catalogs: true
I20250113 02:51:34.148846 448 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_7f3a0392b4234e48835b8c3c5ec34b15" } modify_external_catalogs: true
I20250113 02:51:34.151360 455 catalog_manager.cc:5618] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095) does not have the latest schema for tablet fb18c16fb3ba44ec8a190b88d3bbd1b1 (table default.Table_7a4628807cc44ebdadf927ff709de3a2 [id=75ca7cbd7b9548d784f9a77d0f0cb64f]). Expected version 1 got 0
I20250113 02:51:34.152303 452 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet 926a569407ab488cab8e68480e01a334
I20250113 02:51:34.152766 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_17633452a13c4859971518af3e51e69e" } modify_external_catalogs: true
I20250113 02:51:34.152801 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_f167644bb9424d71af7befe561e5f3f5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:34.153002 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_17633452a13c4859971518af3e51e69e" } modify_external_catalogs: true
W20250113 02:51:34.153295 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f167644bb9424d71af7befe561e5f3f5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:34.152171 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_0a1e1addf0f745e9801f12d1b02b0f0e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:34.154958 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0a1e1addf0f745e9801f12d1b02b0f0e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:34.155704 1042 tablet_service.cc:1514] Processing DeleteTablet for tablet f129bb3fbbfd492d98822be0a6626e44 with delete_type TABLET_DATA_DELETED (replaced by tablet 67baaab1e36e415381579d7385d4ab5a) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:34.138737 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7243d27a1c642269a3d0e67632e458c" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:34.139603 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f56230a2c75d4763a54671c79358f4d9" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:34.157214 889 tablet_service.cc:1467] Processing CreateTablet for tablet f7bac98bba2e4b8e9c72946c33bad17b (DEFAULT_TABLE table=default.Table_64b83cf4ac3548bd89fefb3779bd0e1a [id=c6c77d2667b443f3a7030fc98f8b8da5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:34.157620 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f7bac98bba2e4b8e9c72946c33bad17b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:34.157913 2113 raft_consensus.cc:2743] T c1fb2b36c0504cb4ba32671ae4c620d1 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:34.154388 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_2794f9d2a3cc49f78de33a775c1619f1" } new_table_name: "default.Table_49dafeaa3df04cb789a3e54f469973e8" modify_external_catalogs: true
I20250113 02:51:34.158972 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d15c7a01f854f0aaabcd30ec3166e31" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:34.159013 852 leader_election.cc:304] T 4c615b0b83d74fcfae0c51047dbfbbf4 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df69f48656b44bb890f76587ee69cc0f; no voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca
I20250113 02:51:34.159334 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_d8e5fcedafb14988b94b1114b38e35d3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:34.159651 2024 raft_consensus.cc:2743] T 4c615b0b83d74fcfae0c51047dbfbbf4 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250113 02:51:34.159718 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d8e5fcedafb14988b94b1114b38e35d3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250113 02:51:34.159951 711 leader_election.cc:343] T 1d15c7a01f854f0aaabcd30ec3166e31 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:34.160125 711 leader_election.cc:304] T 1d15c7a01f854f0aaabcd30ec3166e31 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:34.160624 2072 raft_consensus.cc:2743] T 1d15c7a01f854f0aaabcd30ec3166e31 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250113 02:51:34.162593 454 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 874e4392e5d946acae0bf9e3b3994735 consensus state has no leader
I20250113 02:51:34.164171 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_fef715653eb349d19cf7f16198cd7669"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:34.155583 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_e6725383626f4287a24babc4e253e7b1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:34.164981 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e6725383626f4287a24babc4e253e7b1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:34.168287 1195 ts_tablet_manager.cc:1939] T 9fff52e99c64462a8a73fccc3c22ca63 P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:34.168763 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet ec06c9f6a3bf46a9bcb90a85a82b6ff8
I20250113 02:51:34.169263 1195 tablet_replica.cc:331] T 8adfa738c17a4169a5d99f6ceca75100 P 1186fa869eb14812832fa6562531f196: stopping tablet replica
W20250113 02:51:34.169371 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9fa6f11e7a86410099069389bbdec6c3 consensus state has no leader
I20250113 02:51:34.169569 1195 raft_consensus.cc:2238] T 8adfa738c17a4169a5d99f6ceca75100 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:34.169744 1195 raft_consensus.cc:2267] T 8adfa738c17a4169a5d99f6ceca75100 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250113 02:51:34.169935 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fc28641fbd834d768aea272e0f988688 consensus state has no leader
W20250113 02:51:34.170394 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fef715653eb349d19cf7f16198cd7669 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:34.172257 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet 23e04961f072440fa3e3f9485e40f79c (table default.Table_4a0258998d4f413b80d701c74526b86d [id=6fd9215b11274a76b06813227889927b]) successfully deleted
I20250113 02:51:34.175745 1195 ts_tablet_manager.cc:1905] T 8adfa738c17a4169a5d99f6ceca75100 P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:34.184687 1191 tablet_bootstrap.cc:492] T e70658c90555463498d3fc2e2f13ea7f P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:34.184927 1191 ts_tablet_manager.cc:1397] T e70658c90555463498d3fc2e2f13ea7f P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.250s user 0.004s sys 0.000s
I20250113 02:51:34.189726 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet 9fff52e99c64462a8a73fccc3c22ca63 (table default.Table_9ef18ee71ad843c5921750a2f8478855 [id=bd8bd572ed524f2cae01f820f07e3c36]) successfully deleted
I20250113 02:51:34.189761 1030 tablet_service.cc:1467] Processing CreateTablet for tablet cf85cf0ebf5e4fda9d8803ac05741b50 (DEFAULT_TABLE table=default.Table_d2d35cb51870448bbf935e20885ec960 [id=faf2aec59f834e5b8fd56d112e8c7020]), partition=RANGE (key) PARTITION UNBOUNDED
W20250113 02:51:34.191026 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 89b6fbf30ca44488ad7969930f1356e2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 89b6fbf30ca44488ad7969930f1356e2 already in progress: creating tablet
I20250113 02:51:34.192207 751 tablet_service.cc:1514] Processing DeleteTablet for tablet ec06c9f6a3bf46a9bcb90a85a82b6ff8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:34 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:34.192459 1026 tablet_service.cc:1467] Processing CreateTablet for tablet 7e825fa063904d57b7c32c99e3ba7f46 (DEFAULT_TABLE table=default.Table_14aabaa1901741fe8c6f84fd8a397d10 [id=3548788a0bdf46a19328c3392fb1fd15]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:34.193034 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b1910496e0849b193e390a5ef1a6b2b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:26 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:34.193079 1041 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b2d0aa6f0e9456d95f4e48c360ab6e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:33 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:34.195096 1926 raft_consensus.cc:2798] T f56230a2c75d4763a54671c79358f4d9 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 02:51:34.195223 1926 raft_consensus.cc:491] T f56230a2c75d4763a54671c79358f4d9 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 02:51:34.195796 1035 tablet_service.cc:1467] Processing CreateTablet for tablet f7bac98bba2e4b8e9c72946c33bad17b (DEFAULT_TABLE table=default.Table_64b83cf4ac3548bd89fefb3779bd0e1a [id=c6c77d2667b443f3a7030fc98f8b8da5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:34.196141 1040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f5315047bd346b99e51909438f9e60f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:34.200335 2010 consensus_queue.cc:1035] T 0752fde3a30a4947a2a688c56ceadab5 P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.072s
I20250113 02:51:34.200680 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f7bac98bba2e4b8e9c72946c33bad17b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:34.200938 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet ec06c9f6a3bf46a9bcb90a85a82b6ff8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:34 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:34.203090 1023 tablet_service.cc:1467] Processing CreateTablet for tablet ca1052d60b71464889a632ebe7885120 (DEFAULT_TABLE table=default.Table_deef0377723a44e4a4ed576829e0e763 [id=f61353d9514843eb9fe6e5414e387352]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:34.203404 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca1052d60b71464889a632ebe7885120. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:34.205391 1042 tablet_service.cc:1467] Processing CreateTablet for tablet c87118901d7f4790be6f17600bd7c946 (DEFAULT_TABLE table=default.Table_9ad6dc8c7c204c7ea01374d6d8c5844d [id=4919099d8b234d309c9910cbe4ff690f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:34.205703 1042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c87118901d7f4790be6f17600bd7c946. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:34.210747 1926 raft_consensus.cc:3054] T f56230a2c75d4763a54671c79358f4d9 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:34.212008 1198 rpcz_store.cc:276] 0113 02:51:32.400616 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:32.455208 (+ 54592us) service_pool.cc:225] Handling call
0113 02:51:34.129607 (+1674399us) ts_tablet_manager.cc:1145] Deleting tablet 4cbc27d8ac2a46c58bffc33d8af17be2
0113 02:51:34.129616 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:34.129697 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1653586}
I20250113 02:51:34.212491 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet d2699bb529bf4f0f9d7f2668ffe4c9ff with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:33 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:34.213711 1926 raft_consensus.cc:513] T f56230a2c75d4763a54671c79358f4d9 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:34.214371 1926 leader_election.cc:290] T f56230a2c75d4763a54671c79358f4d9 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Requested vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:34.214511 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cf85cf0ebf5e4fda9d8803ac05741b50. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:34.217146 998 leader_election.cc:343] T f56230a2c75d4763a54671c79358f4d9 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:34.217139 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f56230a2c75d4763a54671c79358f4d9" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196"
I20250113 02:51:34.217342 775 raft_consensus.cc:3054] T f56230a2c75d4763a54671c79358f4d9 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Advancing to term 1
W20250113 02:51:34.217638 998 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:38467, user_credentials={real_user=slave}} blocked reactor thread for 51301.4us
I20250113 02:51:34.218194 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f56230a2c75d4763a54671c79358f4d9" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f"
I20250113 02:51:34.219442 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7e825fa063904d57b7c32c99e3ba7f46. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:34.219700 998 leader_election.cc:343] T c7243d27a1c642269a3d0e67632e458c P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:34.219790 998 leader_election.cc:304] T c7243d27a1c642269a3d0e67632e458c P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4216a7f853d642b7bc3e5cff458f3aca; no voters: 1186fa869eb14812832fa6562531f196, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:34.220676 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7e825fa063904d57b7c32c99e3ba7f46. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:34.221452 775 raft_consensus.cc:2463] T f56230a2c75d4763a54671c79358f4d9 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 1.
I20250113 02:51:34.221585 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 867) took 1798 ms. Trace:
I20250113 02:51:34.221655 1198 rpcz_store.cc:276] 0113 02:51:32.423023 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:32.424251 (+ 1228us) service_pool.cc:225] Handling call
0113 02:51:34.212315 (+1788064us) ts_tablet_manager.cc:1145] Deleting tablet 0f428c555442401285fb4eab8280b10b
0113 02:51:34.221472 (+ 9157us) spinlock_profiling.cc:243] Waited 8.96 ms on lock 0x61800000fd18. stack: 00007f0bad094fd6 00007f0bad094d90 00007f0bac07448d 000000000096d544 000000000096d50d 000000000096ceda 00007f0bad07d0fd 00007f0bad07e16e 00007f0bc04b1c7d 00007f0bc042dd43 00007f0bbc6cf276 00007f0bbc6d05f9 00007f0bbc703584 00007f0bbc6e4690 00007f0bbc6e441d 0000000000a71ecc
0113 02:51:34.221481 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:34.221572 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":20613376,"tablet-delete.queue_time_us":1728032}
I20250113 02:51:34.222144 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 868) took 1741 ms. Trace:
I20250113 02:51:34.222223 1198 rpcz_store.cc:276] 0113 02:51:32.480541 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:32.480592 (+ 51us) service_pool.cc:225] Handling call
0113 02:51:34.222031 (+1741439us) ts_tablet_manager.cc:1145] Deleting tablet fd0b8592997d4a61b8ca854ecbb67c55
0113 02:51:34.222043 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:34.222129 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1730529}
I20250113 02:51:34.222551 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 884) took 1480 ms. Trace:
I20250113 02:51:34.222620 1198 rpcz_store.cc:276] 0113 02:51:32.742494 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:32.742639 (+ 145us) service_pool.cc:225] Handling call
0113 02:51:34.222452 (+1479813us) ts_tablet_manager.cc:1145] Deleting tablet 7213e3284b4d4cb581d9fb946a9f87e9
0113 02:51:34.222464 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:34.222534 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1479633}
I20250113 02:51:34.222939 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 885) took 1480 ms. Trace:
I20250113 02:51:34.223009 1198 rpcz_store.cc:276] 0113 02:51:32.742595 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:32.742859 (+ 264us) service_pool.cc:225] Handling call
0113 02:51:34.222834 (+1479975us) ts_tablet_manager.cc:1145] Deleting tablet 1f128d9bf2924fb1bada3a8462818004
0113 02:51:34.222843 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:34.222926 (+ 83us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1479868}
I20250113 02:51:34.223318 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 886) took 1478 ms. Trace:
I20250113 02:51:34.223387 1198 rpcz_store.cc:276] 0113 02:51:32.744951 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:32.745008 (+ 57us) service_pool.cc:225] Handling call
0113 02:51:34.223218 (+1478210us) ts_tablet_manager.cc:1145] Deleting tablet e6e8966da70746d7a10c7243ba2f6f87
0113 02:51:34.223228 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:34.223302 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1478060}
I20250113 02:51:34.223707 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 887) took 1471 ms. Trace:
I20250113 02:51:34.223784 1198 rpcz_store.cc:276] 0113 02:51:32.751866 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:32.751924 (+ 58us) service_pool.cc:225] Handling call
0113 02:51:34.223617 (+1471693us) ts_tablet_manager.cc:1145] Deleting tablet 8cc78bf7d6324b7ba10a1b1045923f2b
0113 02:51:34.223625 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:34.223692 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1471484}
I20250113 02:51:34.227010 1041 tablet_service.cc:1514] Processing DeleteTablet for tablet b8b6ffe690fc459db78b2b3bc12671dd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:23 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:34.227380 2072 raft_consensus.cc:491] T 56b022c94847411785a7e9db3757a2d9 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:34.227526 2072 raft_consensus.cc:513] T 56b022c94847411785a7e9db3757a2d9 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:34.228250 2072 leader_election.cc:290] T 56b022c94847411785a7e9db3757a2d9 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:34.228374 2072 raft_consensus.cc:491] T 999fecb3fc65466cad21cc1ff5ef2d44 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:34.228468 2072 raft_consensus.cc:513] T 999fecb3fc65466cad21cc1ff5ef2d44 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:34.228991 2072 leader_election.cc:290] T 999fecb3fc65466cad21cc1ff5ef2d44 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:34.229934 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 892) took 1452 ms. Trace:
I20250113 02:51:34.230019 1198 rpcz_store.cc:276] 0113 02:51:32.777831 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:32.777998 (+ 167us) service_pool.cc:225] Handling call
0113 02:51:34.229834 (+1451836us) ts_tablet_manager.cc:1145] Deleting tablet b86d035e503344e1a3bd85c55131274e
0113 02:51:34.229843 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:34.229921 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1451615}
I20250113 02:51:34.230053 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "999fecb3fc65466cad21cc1ff5ef2d44" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:34.230226 1198 tablet_replica.cc:331] T 0a7e7713bc8648378b668d8b769ac927 P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:34.230373 1198 raft_consensus.cc:2238] T 0a7e7713bc8648378b668d8b769ac927 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:34.230762 995 leader_election.cc:304] T f56230a2c75d4763a54671c79358f4d9 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters:
W20250113 02:51:34.233670 711 leader_election.cc:343] T 999fecb3fc65466cad21cc1ff5ef2d44 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:34.235145 2010 raft_consensus.cc:2743] T c7243d27a1c642269a3d0e67632e458c P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:34.235513 1196 ts_tablet_manager.cc:1918] T a0ca2e3a3ce94f3aa20df64c89c7cf95 P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:34.235528 2010 raft_consensus.cc:2798] T f56230a2c75d4763a54671c79358f4d9 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Leader election won for term 1
I20250113 02:51:34.235646 2010 raft_consensus.cc:695] T f56230a2c75d4763a54671c79358f4d9 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: Becoming Leader. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Running, Role: LEADER
I20250113 02:51:34.235839 1196 log.cc:1198] T a0ca2e3a3ce94f3aa20df64c89c7cf95 P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/a0ca2e3a3ce94f3aa20df64c89c7cf95
I20250113 02:51:34.235909 2010 consensus_queue.cc:237] T f56230a2c75d4763a54671c79358f4d9 P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:34.236186 1196 ts_tablet_manager.cc:1939] T a0ca2e3a3ce94f3aa20df64c89c7cf95 P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:34.237336 1198 raft_consensus.cc:2267] T 0a7e7713bc8648378b668d8b769ac927 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:34.237866 1198 ts_tablet_manager.cc:1905] T 0a7e7713bc8648378b668d8b769ac927 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:34.238276 880 tablet_service.cc:1514] Processing DeleteTablet for tablet ec06c9f6a3bf46a9bcb90a85a82b6ff8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:34 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:34.242508 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56b022c94847411785a7e9db3757a2d9" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:34.244273 711 leader_election.cc:343] T 56b022c94847411785a7e9db3757a2d9 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
W20250113 02:51:34.252084 998 leader_election.cc:343] T f56230a2c75d4763a54671c79358f4d9 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:34.252642 1062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56b022c94847411785a7e9db3757a2d9" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:34.253187 1062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "999fecb3fc65466cad21cc1ff5ef2d44" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
W20250113 02:51:34.253602 709 leader_election.cc:343] T 56b022c94847411785a7e9db3757a2d9 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:34.253717 709 leader_election.cc:304] T 56b022c94847411785a7e9db3757a2d9 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
W20250113 02:51:34.254066 709 leader_election.cc:343] T 999fecb3fc65466cad21cc1ff5ef2d44 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:34.254159 709 leader_election.cc:304] T 999fecb3fc65466cad21cc1ff5ef2d44 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:34.254462 2109 raft_consensus.cc:2743] T 56b022c94847411785a7e9db3757a2d9 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:34.254599 2109 raft_consensus.cc:2743] T 999fecb3fc65466cad21cc1ff5ef2d44 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250113 02:51:34.262588 441 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 9fa6f11e7a86410099069389bbdec6c3 (table default.Table_9209a23d3c264eb2ace750095618d82d [id=123edf937eae4e50a175e8ea641c97c5], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 9fa6f11e7a86410099069389bbdec6c3 consensus state has no leader
I20250113 02:51:34.268425 447 catalog_manager.cc:5618] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467) does not have the latest schema for tablet 378d59d314a84ddb84cf2ab041c51cf1 (table default.Table_49dafeaa3df04cb789a3e54f469973e8 [id=0c01441cec7043b687d6744e72a0e45d]). Expected version 2 got 0
W20250113 02:51:34.271258 995 consensus_peers.cc:487] T 056bf1e67be04d179eace3d93282b145 P 4216a7f853d642b7bc3e5cff458f3aca -> Peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Couldn't send request to peer 1186fa869eb14812832fa6562531f196. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 056bf1e67be04d179eace3d93282b145. This is attempt 1: this message will repeat every 5th retry.
W20250113 02:51:34.272918 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 4cbc27d8ac2a46c58bffc33d8af17be2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4cbc27d8ac2a46c58bffc33d8af17be2 already in progress: creating tablet
W20250113 02:51:34.273299 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 0f428c555442401285fb4eab8280b10b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0f428c555442401285fb4eab8280b10b already in progress: creating tablet
W20250113 02:51:34.273522 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet fd0b8592997d4a61b8ca854ecbb67c55 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fd0b8592997d4a61b8ca854ecbb67c55 already in progress: creating tablet
I20250113 02:51:34.273566 1926 raft_consensus.cc:491] T 3b1910496e0849b193e390a5ef1a6b2b P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250113 02:51:34.273753 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 7213e3284b4d4cb581d9fb946a9f87e9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7213e3284b4d4cb581d9fb946a9f87e9 already in progress: creating tablet
I20250113 02:51:34.273670 1926 raft_consensus.cc:513] T 3b1910496e0849b193e390a5ef1a6b2b P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
W20250113 02:51:34.273941 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 1f128d9bf2924fb1bada3a8462818004 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1f128d9bf2924fb1bada3a8462818004 already in progress: creating tablet
W20250113 02:51:34.274106 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet e6e8966da70746d7a10c7243ba2f6f87 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e6e8966da70746d7a10c7243ba2f6f87 already in progress: creating tablet
I20250113 02:51:34.274277 1926 leader_election.cc:290] T 3b1910496e0849b193e390a5ef1a6b2b P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
W20250113 02:51:34.274407 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 8cc78bf7d6324b7ba10a1b1045923f2b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8cc78bf7d6324b7ba10a1b1045923f2b already in progress: creating tablet
W20250113 02:51:34.274719 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet b86d035e503344e1a3bd85c55131274e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b86d035e503344e1a3bd85c55131274e already in progress: creating tablet
I20250113 02:51:34.276350 447 catalog_manager.cc:5618] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467) does not have the latest schema for tablet 378d59d314a84ddb84cf2ab041c51cf1 (table default.Table_49dafeaa3df04cb789a3e54f469973e8 [id=0c01441cec7043b687d6744e72a0e45d]). Expected version 2 got 0
W20250113 02:51:34.278026 998 consensus_peers.cc:487] T 0752fde3a30a4947a2a688c56ceadab5 P 4216a7f853d642b7bc3e5cff458f3aca -> Peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Couldn't send request to peer df69f48656b44bb890f76587ee69cc0f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250113 02:51:34.278669 995 leader_election.cc:343] T 3b1910496e0849b193e390a5ef1a6b2b P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Not found: Tablet not found: 3b1910496e0849b193e390a5ef1a6b2b
I20250113 02:51:34.278715 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b1910496e0849b193e390a5ef1a6b2b" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:34.282775 998 leader_election.cc:343] T 3b1910496e0849b193e390a5ef1a6b2b P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:34.282907 998 leader_election.cc:304] T 3b1910496e0849b193e390a5ef1a6b2b P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4216a7f853d642b7bc3e5cff458f3aca; no voters: 1186fa869eb14812832fa6562531f196, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:34.283421 2112 raft_consensus.cc:2743] T 3b1910496e0849b193e390a5ef1a6b2b P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:34.285585 1191 raft_consensus.cc:357] T e70658c90555463498d3fc2e2f13ea7f P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:34.285851 1191 raft_consensus.cc:383] T e70658c90555463498d3fc2e2f13ea7f P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:34.285912 1191 raft_consensus.cc:738] T e70658c90555463498d3fc2e2f13ea7f P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:34.286159 1191 consensus_queue.cc:260] T e70658c90555463498d3fc2e2f13ea7f P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:34.286777 1191 ts_tablet_manager.cc:1428] T e70658c90555463498d3fc2e2f13ea7f P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.102s user 0.001s sys 0.000s
I20250113 02:51:34.287094 1191 tablet_bootstrap.cc:492] T eebad045e7bb43fa9ca9c9af8b3b7ea5 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:34.287371 1196 tablet_replica.cc:331] T c94eedb5dce44e6d896192c93127903a P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:34.287714 1196 raft_consensus.cc:2238] T c94eedb5dce44e6d896192c93127903a P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:34.291321 884 tablet_service.cc:1514] Processing DeleteTablet for tablet f68f9f4076d9475d938e795e72bdd468 with delete_type TABLET_DATA_DELETED (replaced by tablet f2c6c8c17d6e499d84ad7c0c23889cc4) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:34.290395 449 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_c59e24fdd40849b48b0b75b0123cd6ed" } modify_external_catalogs: true
I20250113 02:51:34.291771 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_a7ed8a8a7d00492b8fc543baeb433a27" } new_table_name: "default.Table_fd46797b5cb645e0a712aaec33089be6" modify_external_catalogs: true
I20250113 02:51:34.292083 447 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_e1594759ceea473eb9f01fff92b93b10" } modify_external_catalogs: true
I20250113 02:51:34.292321 1196 raft_consensus.cc:2267] T c94eedb5dce44e6d896192c93127903a P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:34.292294 447 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_e1594759ceea473eb9f01fff92b93b10" } modify_external_catalogs: true
I20250113 02:51:34.292891 1196 ts_tablet_manager.cc:1905] T c94eedb5dce44e6d896192c93127903a P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:34.293285 449 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_c59e24fdd40849b48b0b75b0123cd6ed" } modify_external_catalogs: true
I20250113 02:51:34.293731 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet a0ca2e3a3ce94f3aa20df64c89c7cf95 (table default.Table_f08525e2a08e48ec85ef55287d0bac49 [id=96ff1ee9327f41a5a1bb2757b188254a]) successfully deleted
W20250113 02:51:34.303090 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet be5d6907a02f41be8eab71de051a6d3b consensus state has no leader
I20250113 02:51:34.303092 1198 ts_tablet_manager.cc:1918] T 0a7e7713bc8648378b668d8b769ac927 P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:34.303200 1198 log.cc:1198] T 0a7e7713bc8648378b668d8b769ac927 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/0a7e7713bc8648378b668d8b769ac927
I20250113 02:51:34.303468 1198 ts_tablet_manager.cc:1939] T 0a7e7713bc8648378b668d8b769ac927 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:34.303825 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 894) took 1393 ms. Trace:
I20250113 02:51:34.303967 1198 rpcz_store.cc:276] 0113 02:51:32.910858 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:32.911649 (+ 791us) service_pool.cc:225] Handling call
0113 02:51:34.230191 (+1318542us) ts_tablet_manager.cc:1145] Deleting tablet 0a7e7713bc8648378b668d8b769ac927
0113 02:51:34.230197 (+ 6us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:34.268406 (+ 38209us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:34.302997 (+ 34591us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:34.303809 (+ 812us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4954,"tablet-delete.queue_time_us":1318331}
I20250113 02:51:34.304478 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet 0a7e7713bc8648378b668d8b769ac927 (table default.Table_93a7b6abed0446a4baf21e5e7c8f0c1b [id=9fd4d65e7b9d443a82f06a1f660a7275]) successfully deleted
I20250113 02:51:34.311723 1191 tablet_bootstrap.cc:654] T eebad045e7bb43fa9ca9c9af8b3b7ea5 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
W20250113 02:51:34.313302 995 consensus_peers.cc:487] T 8adfa738c17a4169a5d99f6ceca75100 P 4216a7f853d642b7bc3e5cff458f3aca -> Peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Couldn't send request to peer 1186fa869eb14812832fa6562531f196. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250113 02:51:34.314004 742 tablet_service.cc:1467] Processing CreateTablet for tablet 78bf9b924b144837b6e17b54086d7908 (DEFAULT_TABLE table=default.Table_64430a2da8d04ce88ae813a23452ecb6 [id=0354002efedc4902902ec83180ebad87]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:34.314254 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 6e208f83f3384a78a959909b4ac690c0
I20250113 02:51:34.314431 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 78bf9b924b144837b6e17b54086d7908. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:34.315127 736 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e208f83f3384a78a959909b4ac690c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:34 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:34.315701 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet db486978b33a45df90025988b084aecd
I20250113 02:51:34.316373 736 tablet_service.cc:1514] Processing DeleteTablet for tablet db486978b33a45df90025988b084aecd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:34 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:34.319180 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 1c77033ca393469e9324d2406a65a5e9
I20250113 02:51:34.320835 736 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c77033ca393469e9324d2406a65a5e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:34 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:34.321990 1030 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e208f83f3384a78a959909b4ac690c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:34 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:34.321954 736 tablet_service.cc:1467] Processing CreateTablet for tablet 4d9dcaabdc524bdeb01b1c842e15d1ce (DEFAULT_TABLE table=default.Table_be0e2878c7d74f7d8fcb4280e8cf6d0b [id=e0d823580816424ebb4c4a3232c5fe48]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:34.322372 1030 tablet_service.cc:1514] Processing DeleteTablet for tablet db486978b33a45df90025988b084aecd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:34 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:34.322429 736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d9dcaabdc524bdeb01b1c842e15d1ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:34.322587 1026 tablet_service.cc:1467] Processing CreateTablet for tablet 78bf9b924b144837b6e17b54086d7908 (DEFAULT_TABLE table=default.Table_64430a2da8d04ce88ae813a23452ecb6 [id=0354002efedc4902902ec83180ebad87]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:34.322875 1030 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c77033ca393469e9324d2406a65a5e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:34 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:34.323097 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 78bf9b924b144837b6e17b54086d7908. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:34.326020 914 raft_consensus.cc:3054] T 4c615b0b83d74fcfae0c51047dbfbbf4 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:34.327152 887 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e208f83f3384a78a959909b4ac690c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:34 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:34.327361 1023 tablet_service.cc:1467] Processing CreateTablet for tablet 4d9dcaabdc524bdeb01b1c842e15d1ce (DEFAULT_TABLE table=default.Table_be0e2878c7d74f7d8fcb4280e8cf6d0b [id=e0d823580816424ebb4c4a3232c5fe48]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:34.328034 887 tablet_service.cc:1467] Processing CreateTablet for tablet 4d9dcaabdc524bdeb01b1c842e15d1ce (DEFAULT_TABLE table=default.Table_be0e2878c7d74f7d8fcb4280e8cf6d0b [id=e0d823580816424ebb4c4a3232c5fe48]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:34.327476 2141 tablet.cc:1721] T 378d59d314a84ddb84cf2ab041c51cf1 P 1186fa869eb14812832fa6562531f196: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250113 02:51:34.328476 889 tablet_service.cc:1514] Processing DeleteTablet for tablet db486978b33a45df90025988b084aecd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:34 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:34.328492 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d9dcaabdc524bdeb01b1c842e15d1ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:34.329226 889 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c77033ca393469e9324d2406a65a5e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:34 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:34.330809 885 tablet_service.cc:1467] Processing CreateTablet for tablet 78bf9b924b144837b6e17b54086d7908 (DEFAULT_TABLE table=default.Table_64430a2da8d04ce88ae813a23452ecb6 [id=0354002efedc4902902ec83180ebad87]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:34.339160 2144 tablet.cc:1721] T 378d59d314a84ddb84cf2ab041c51cf1 P df69f48656b44bb890f76587ee69cc0f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250113 02:51:34.345592 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d9dcaabdc524bdeb01b1c842e15d1ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:34.345927 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 78bf9b924b144837b6e17b54086d7908. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:34.348949 447 catalog_manager.cc:5618] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467) does not have the latest schema for tablet 4c615b0b83d74fcfae0c51047dbfbbf4 (table default.Table_bf689f8eb8884e88a72e9b50598fc389 [id=a956f95f6dd94b578f6bd3cb3b97f9da]). Expected version 2 got 0
I20250113 02:51:34.349783 1192 tablet_bootstrap.cc:492] T 377ebf66bbef40988043f6200ae0893c P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:34.351449 1192 ts_tablet_manager.cc:1397] T 377ebf66bbef40988043f6200ae0893c P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.315s user 0.023s sys 0.044s
I20250113 02:51:34.352295 739 tablet_service.cc:1467] Processing CreateTablet for tablet 04dd182ad3564f34989932e463e4d9e4 (DEFAULT_TABLE table=default.Table_213c7f5ea7184ab5ab5d3699b4a684bd [id=c9542032d5a341b2b5d94d3a2e3b4ae3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:34.352774 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 04dd182ad3564f34989932e463e4d9e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:34.353237 447 catalog_manager.cc:5618] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467) does not have the latest schema for tablet 4c615b0b83d74fcfae0c51047dbfbbf4 (table default.Table_bf689f8eb8884e88a72e9b50598fc389 [id=a956f95f6dd94b578f6bd3cb3b97f9da]). Expected version 2 got 0
I20250113 02:51:34.357162 1035 tablet_service.cc:1467] Processing CreateTablet for tablet 04dd182ad3564f34989932e463e4d9e4 (DEFAULT_TABLE table=default.Table_213c7f5ea7184ab5ab5d3699b4a684bd [id=c9542032d5a341b2b5d94d3a2e3b4ae3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:34.358412 751 tablet_service.cc:1467] Processing CreateTablet for tablet 128f74fdc98f4135b39dbf23f1d987b0 (DEFAULT_TABLE table=default.Table_1871943d605842118e5ac9c6f594f30c [id=8522c951c28c41e5b859ec2df35529dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:34.358788 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 128f74fdc98f4135b39dbf23f1d987b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:34.358479 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 04dd182ad3564f34989932e463e4d9e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:34.359611 2140 tablet.cc:1721] T 378d59d314a84ddb84cf2ab041c51cf1 P 4216a7f853d642b7bc3e5cff458f3aca: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250113 02:51:34.359901 887 tablet_service.cc:1467] Processing CreateTablet for tablet 128f74fdc98f4135b39dbf23f1d987b0 (DEFAULT_TABLE table=default.Table_1871943d605842118e5ac9c6f594f30c [id=8522c951c28c41e5b859ec2df35529dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:34.364818 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 128f74fdc98f4135b39dbf23f1d987b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:34.364197 885 tablet_service.cc:1467] Processing CreateTablet for tablet 04dd182ad3564f34989932e463e4d9e4 (DEFAULT_TABLE table=default.Table_213c7f5ea7184ab5ab5d3699b4a684bd [id=c9542032d5a341b2b5d94d3a2e3b4ae3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:34.366458 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 04dd182ad3564f34989932e463e4d9e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:34.372797 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 897) took 1355 ms. Trace:
I20250113 02:51:34.372900 1198 rpcz_store.cc:276] 0113 02:51:33.016899 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:33.016971 (+ 72us) service_pool.cc:225] Handling call
0113 02:51:34.372670 (+1355699us) ts_tablet_manager.cc:1145] Deleting tablet 36726b40ac5145bfb371871b3fff079a
0113 02:51:34.372680 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:34.372778 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1345588}
I20250113 02:51:34.373250 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 899) took 1121 ms. Trace:
I20250113 02:51:34.373323 1198 rpcz_store.cc:276] 0113 02:51:33.252084 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:33.252153 (+ 69us) service_pool.cc:225] Handling call
0113 02:51:34.373158 (+1121005us) ts_tablet_manager.cc:1145] Deleting tablet 89b6fbf30ca44488ad7969930f1356e2
0113 02:51:34.373168 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:34.373237 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1120844}
I20250113 02:51:34.373564 1198 tablet_replica.cc:331] T 056bf1e67be04d179eace3d93282b145 P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:34.373745 1198 raft_consensus.cc:2238] T 056bf1e67be04d179eace3d93282b145 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: Raft consensus shutting down.
I20250113 02:51:34.374048 1198 pending_rounds.cc:62] T 056bf1e67be04d179eace3d93282b145 P 4216a7f853d642b7bc3e5cff458f3aca: Trying to abort 1 pending ops.
I20250113 02:51:34.374105 1198 pending_rounds.cc:69] T 056bf1e67be04d179eace3d93282b145 P 4216a7f853d642b7bc3e5cff458f3aca: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113673494717513728 op_type: NO_OP noop_request { }
I20250113 02:51:34.374204 1198 raft_consensus.cc:2883] T 056bf1e67be04d179eace3d93282b145 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250113 02:51:34.374305 1198 raft_consensus.cc:2267] T 056bf1e67be04d179eace3d93282b145 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:34.374812 1198 ts_tablet_manager.cc:1905] T 056bf1e67be04d179eace3d93282b145 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250113 02:51:34.380040 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 36726b40ac5145bfb371871b3fff079a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 36726b40ac5145bfb371871b3fff079a already in progress: creating tablet
I20250113 02:51:34.380946 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_5bc5d0b055904e50b74e0981675f4abb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:34.383872 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_e906a17932314e12bd60df6ea31b8881"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:34.385722 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_b7c9a4329c0a4aac98cdb053b3921b63"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:34.386752 446 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_a969335ba8de47a49dc4629865ee4a1a" } modify_external_catalogs: true
I20250113 02:51:34.387812 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_ab3c82460efc4216a97966ab66413b5a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:34.388067 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_c8ea3525cf2847649a201509a00a5d9f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:34.387930 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 89b6fbf30ca44488ad7969930f1356e2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 89b6fbf30ca44488ad7969930f1356e2 already in progress: creating tablet
I20250113 02:51:34.391607 1192 raft_consensus.cc:357] T 377ebf66bbef40988043f6200ae0893c P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:34.391793 1192 raft_consensus.cc:383] T 377ebf66bbef40988043f6200ae0893c P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:34.391853 1192 raft_consensus.cc:738] T 377ebf66bbef40988043f6200ae0893c P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
W20250113 02:51:34.392122 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5bc5d0b055904e50b74e0981675f4abb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:34.392118 1192 consensus_queue.cc:260] T 377ebf66bbef40988043f6200ae0893c P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
W20250113 02:51:34.392305 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c8ea3525cf2847649a201509a00a5d9f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:34.392621 1192 ts_tablet_manager.cc:1428] T 377ebf66bbef40988043f6200ae0893c P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.041s user 0.001s sys 0.000s
I20250113 02:51:34.392863 1192 tablet_bootstrap.cc:492] T e468b652f71f44e3945f8d919ac0e1a8 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:34.394256 1192 tablet_bootstrap.cc:654] T e468b652f71f44e3945f8d919ac0e1a8 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
W20250113 02:51:34.397997 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e906a17932314e12bd60df6ea31b8881 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:34.399664 2142 tablet.cc:1721] T 4c615b0b83d74fcfae0c51047dbfbbf4 P df69f48656b44bb890f76587ee69cc0f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20250113 02:51:34.400082 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b7c9a4329c0a4aac98cdb053b3921b63 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:34.401578 446 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_a969335ba8de47a49dc4629865ee4a1a" } modify_external_catalogs: true
I20250113 02:51:34.402707 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_a35695d3f16144fdb1aa00394d72eb14" } new_table_name: "default.Table_4edd5e339c0a437c88823ed8388a400d" modify_external_catalogs: true
W20250113 02:51:34.403740 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ab3c82460efc4216a97966ab66413b5a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:34.404320 1195 ts_tablet_manager.cc:1918] T 8adfa738c17a4169a5d99f6ceca75100 P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250113 02:51:34.404423 1195 log.cc:1198] T 8adfa738c17a4169a5d99f6ceca75100 P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/8adfa738c17a4169a5d99f6ceca75100
I20250113 02:51:34.404773 1195 ts_tablet_manager.cc:1939] T 8adfa738c17a4169a5d99f6ceca75100 P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:34.405381 448 catalog_manager.cc:6071] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: default.Table_49dafeaa3df04cb789a3e54f469973e8 [id=0c01441cec7043b687d6744e72a0e45d] alter complete (version 2)
I20250113 02:51:34.405437 1194 tablet_bootstrap.cc:492] T 0810b9671f824d0a84741bc1904df247 P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:34.405560 1194 ts_tablet_manager.cc:1397] T 0810b9671f824d0a84741bc1904df247 P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.492s user 0.022s sys 0.016s
I20250113 02:51:34.405812 449 catalog_manager.cc:5618] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095) does not have the latest schema for tablet 0752fde3a30a4947a2a688c56ceadab5 (table default.Table_da3e49fa1e654145a05858466ec2260f [id=94704429b49b46e38a48258a450d2705]). Expected version 1 got 0
I20250113 02:51:34.406395 1194 raft_consensus.cc:357] T 0810b9671f824d0a84741bc1904df247 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:34.406682 1194 raft_consensus.cc:383] T 0810b9671f824d0a84741bc1904df247 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:34.406785 1194 raft_consensus.cc:738] T 0810b9671f824d0a84741bc1904df247 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:34.407037 1194 consensus_queue.cc:260] T 0810b9671f824d0a84741bc1904df247 P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:34.407697 1194 ts_tablet_manager.cc:1428] T 0810b9671f824d0a84741bc1904df247 P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250113 02:51:34.416440 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 919) took 1143 ms. Trace:
I20250113 02:51:34.416579 448 catalog_manager.cc:5618] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095) does not have the latest schema for tablet 0752fde3a30a4947a2a688c56ceadab5 (table default.Table_da3e49fa1e654145a05858466ec2260f [id=94704429b49b46e38a48258a450d2705]). Expected version 1 got 0
I20250113 02:51:34.416541 1195 rpcz_store.cc:276] 0113 02:51:33.272717 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:33.273175 (+ 458us) service_pool.cc:225] Handling call
0113 02:51:34.169220 (+896045us) ts_tablet_manager.cc:1145] Deleting tablet 8adfa738c17a4169a5d99f6ceca75100
0113 02:51:34.169227 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:34.318240 (+149013us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:34.404197 (+ 85957us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:34.416409 (+ 12212us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":895509}
I20250113 02:51:34.417028 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet 8adfa738c17a4169a5d99f6ceca75100 (table default.Table_9a9b1ced3bef4e648088e0781851128e [id=ae544735623e4cfd894dc9340aca177c]) successfully deleted
I20250113 02:51:34.418428 746 tablet_service.cc:1514] Processing DeleteTablet for tablet e70658c90555463498d3fc2e2f13ea7f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:23 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:34.418567 1030 tablet_service.cc:1467] Processing CreateTablet for tablet 128f74fdc98f4135b39dbf23f1d987b0 (DEFAULT_TABLE table=default.Table_1871943d605842118e5ac9c6f594f30c [id=8522c951c28c41e5b859ec2df35529dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:34.418943 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 128f74fdc98f4135b39dbf23f1d987b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:34.422870 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 4a3ba2cd77b8416db72d894b09cc1100
I20250113 02:51:34.423569 746 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a3ba2cd77b8416db72d894b09cc1100 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:34 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:34.424533 1038 tablet_service.cc:1467] Processing CreateTablet for tablet 6770a0cb45fb420d9fbf24d9a5fe4256 (DEFAULT_TABLE table=default.Table_39421f9a919a4fe0851a559994412d3a [id=581289610ec443d1916ef1daa0010f65]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:34.424873 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6770a0cb45fb420d9fbf24d9a5fe4256. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:34.428184 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f3a8c7a25aca4496919386396626de00 consensus state has no leader
I20250113 02:51:34.428663 751 tablet_service.cc:1467] Processing CreateTablet for tablet 6770a0cb45fb420d9fbf24d9a5fe4256 (DEFAULT_TABLE table=default.Table_39421f9a919a4fe0851a559994412d3a [id=581289610ec443d1916ef1daa0010f65]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:34.429116 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6770a0cb45fb420d9fbf24d9a5fe4256. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:34.430522 889 tablet_service.cc:1467] Processing CreateTablet for tablet 2002c69206ec468aad1b44f1b5b0d3b2 (DEFAULT_TABLE table=default.Table_9b26cd6ed9944430a1228f552a7c3d95 [id=f313b4a543574e4b936c0eb13d6d2568]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:34.431064 894 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a3ba2cd77b8416db72d894b09cc1100 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:34 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:34.431995 1030 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a3ba2cd77b8416db72d894b09cc1100 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:34 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:34.433096 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_5ddfdff2cc8c4d12a1c4552e9a302027" } modify_external_catalogs: true
I20250113 02:51:34.433326 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_5ddfdff2cc8c4d12a1c4552e9a302027" } modify_external_catalogs: true
I20250113 02:51:34.433607 1195 tablet_replica.cc:331] T f56230a2c75d4763a54671c79358f4d9 P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:34.433558 890 tablet_service.cc:1467] Processing CreateTablet for tablet 6770a0cb45fb420d9fbf24d9a5fe4256 (DEFAULT_TABLE table=default.Table_39421f9a919a4fe0851a559994412d3a [id=581289610ec443d1916ef1daa0010f65]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:34.434089 1195 raft_consensus.cc:2238] T f56230a2c75d4763a54671c79358f4d9 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:34.434005 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_b3f0fce26a8d48f8a7660c9edba52f3d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:34.434253 1195 raft_consensus.cc:2267] T f56230a2c75d4763a54671c79358f4d9 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250113 02:51:34.434530 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b3f0fce26a8d48f8a7660c9edba52f3d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:34.434773 1195 ts_tablet_manager.cc:1905] T f56230a2c75d4763a54671c79358f4d9 P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:34.435910 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_c6708c29033245ec913eacfe9c3255c2" } new_table_name: "default.Table_34e676c9d2fb4f7ba8a20c74bc14eccf" modify_external_catalogs: true
I20250113 02:51:34.437356 449 catalog_manager.cc:5618] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587) does not have the latest schema for tablet 377ebf66bbef40988043f6200ae0893c (table default.Table_5ddfdff2cc8c4d12a1c4552e9a302027 [id=8b35e389de9f4917904e87b43618737e]). Expected version 1 got 0
I20250113 02:51:34.437822 449 catalog_manager.cc:5618] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587) does not have the latest schema for tablet 0752fde3a30a4947a2a688c56ceadab5 (table default.Table_da3e49fa1e654145a05858466ec2260f [id=94704429b49b46e38a48258a450d2705]). Expected version 1 got 0
I20250113 02:51:34.445475 2024 raft_consensus.cc:491] T 0810b9671f824d0a84741bc1904df247 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:34.445605 2024 raft_consensus.cc:513] T 0810b9671f824d0a84741bc1904df247 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:34.446269 2024 leader_election.cc:290] T 0810b9671f824d0a84741bc1904df247 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587), 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
I20250113 02:51:34.447029 1196 ts_tablet_manager.cc:1918] T c94eedb5dce44e6d896192c93127903a P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:34.447140 1196 log.cc:1198] T c94eedb5dce44e6d896192c93127903a P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/c94eedb5dce44e6d896192c93127903a
W20250113 02:51:34.447409 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet dd566a49d3ff4d3889b80d8d2ccbee0e consensus state has no leader
I20250113 02:51:34.447491 1196 ts_tablet_manager.cc:1939] T c94eedb5dce44e6d896192c93127903a P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
W20250113 02:51:34.447582 852 leader_election.cc:343] T 0810b9671f824d0a84741bc1904df247 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Not found: Tablet not found: 0810b9671f824d0a84741bc1904df247
I20250113 02:51:34.449189 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 377ebf66bbef40988043f6200ae0893c
W20250113 02:51:34.449887 851 leader_election.cc:343] T 0810b9671f824d0a84741bc1904df247 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Not found: Tablet not found: 0810b9671f824d0a84741bc1904df247
I20250113 02:51:34.449988 851 leader_election.cc:304] T 0810b9671f824d0a84741bc1904df247 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df69f48656b44bb890f76587ee69cc0f; no voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca
I20250113 02:51:34.451126 736 tablet_service.cc:1467] Processing CreateTablet for tablet 2002c69206ec468aad1b44f1b5b0d3b2 (DEFAULT_TABLE table=default.Table_9b26cd6ed9944430a1228f552a7c3d95 [id=f313b4a543574e4b936c0eb13d6d2568]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:34.451493 736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2002c69206ec468aad1b44f1b5b0d3b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:34.451643 1198 ts_tablet_manager.cc:1918] T 056bf1e67be04d179eace3d93282b145 P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250113 02:51:34.451745 1198 log.cc:1198] T 056bf1e67be04d179eace3d93282b145 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/056bf1e67be04d179eace3d93282b145
I20250113 02:51:34.451959 2024 raft_consensus.cc:2743] T 0810b9671f824d0a84741bc1904df247 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:34.452150 1198 ts_tablet_manager.cc:1939] T 056bf1e67be04d179eace3d93282b145 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:34.452867 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 901) took 1193 ms. Trace:
I20250113 02:51:34.457289 739 tablet_service.cc:1467] Processing CreateTablet for tablet f426887763ed48e3bc87f59d80986672 (DEFAULT_TABLE table=default.Table_8a9caf2b686f4017948ebdd7f3fe8f58 [id=e53f92f1ff4d47fa90651da781293731]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:34.457726 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f426887763ed48e3bc87f59d80986672. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:34.463192 1194 tablet_bootstrap.cc:492] T b4806b18a4194f9c95350d89ba62e13a P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:34.463495 1191 tablet_bootstrap.cc:492] T eebad045e7bb43fa9ca9c9af8b3b7ea5 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:34.463672 1191 ts_tablet_manager.cc:1397] T eebad045e7bb43fa9ca9c9af8b3b7ea5 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.177s user 0.007s sys 0.000s
I20250113 02:51:34.464514 1191 raft_consensus.cc:357] T eebad045e7bb43fa9ca9c9af8b3b7ea5 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:34.464776 1191 raft_consensus.cc:383] T eebad045e7bb43fa9ca9c9af8b3b7ea5 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:34.464828 1194 tablet_bootstrap.cc:654] T b4806b18a4194f9c95350d89ba62e13a P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:34.464895 1191 raft_consensus.cc:738] T eebad045e7bb43fa9ca9c9af8b3b7ea5 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:34.465173 1191 consensus_queue.cc:260] T eebad045e7bb43fa9ca9c9af8b3b7ea5 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:34.465391 746 tablet_service.cc:1467] Processing CreateTablet for tablet 2c49f1eefd2041b4ae617a4cb35d6f15 (DEFAULT_TABLE table=default.Table_4557b7a5878e45d0860844f7e191bfcf [id=ca60f9dab0db4b2f9fbbd9125039ab2b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:34.465742 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c49f1eefd2041b4ae617a4cb35d6f15. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:34.467592 1191 ts_tablet_manager.cc:1428] T eebad045e7bb43fa9ca9c9af8b3b7ea5 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20250113 02:51:34.467949 1191 tablet_bootstrap.cc:492] T 7527e6e0b65e4a43b7f48899a34cdf7f P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:34.468537 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_27e1ad9a94f8455f806f48768ef4ddac" } new_table_name: "default.Table_3d65267002364c8a9175ccb2223cac44" modify_external_catalogs: true
I20250113 02:51:34.469055 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_4c6f3deafe73412a9496041fea5f2f69"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:34.469632 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4c6f3deafe73412a9496041fea5f2f69 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:34.469700 1191 tablet_bootstrap.cc:654] T 7527e6e0b65e4a43b7f48899a34cdf7f P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:34.470045 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_521ea6e222cb450797a9d20bc8112797"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:34.470537 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_521ea6e222cb450797a9d20bc8112797 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:34.471583 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_2d36ffc5be5043fcb20b054d5283120d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:34.472044 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2d36ffc5be5043fcb20b054d5283120d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:34.472220 894 tablet_service.cc:1514] Processing DeleteTablet for tablet 377ebf66bbef40988043f6200ae0893c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:34 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:34.473268 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_69a9d7c8990e42c68549f893afb31c50" } new_table_name: "default.Table_f830fa2aaadc44bbb73b8eb61545aabc" modify_external_catalogs: true
I20250113 02:51:34.473284 894 tablet_service.cc:1467] Processing CreateTablet for tablet f426887763ed48e3bc87f59d80986672 (DEFAULT_TABLE table=default.Table_8a9caf2b686f4017948ebdd7f3fe8f58 [id=e53f92f1ff4d47fa90651da781293731]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:34.473999 884 tablet_service.cc:1467] Processing CreateTablet for tablet 2c49f1eefd2041b4ae617a4cb35d6f15 (DEFAULT_TABLE table=default.Table_4557b7a5878e45d0860844f7e191bfcf [id=ca60f9dab0db4b2f9fbbd9125039ab2b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:34.474213 752 tablet_service.cc:1514] Processing DeleteTablet for tablet eebad045e7bb43fa9ca9c9af8b3b7ea5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:21 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:34.458977 2010 raft_consensus.cc:491] T f129bb3fbbfd492d98822be0a6626e44 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:34.463060 1030 tablet_service.cc:1467] Processing CreateTablet for tablet f426887763ed48e3bc87f59d80986672 (DEFAULT_TABLE table=default.Table_8a9caf2b686f4017948ebdd7f3fe8f58 [id=e53f92f1ff4d47fa90651da781293731]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:34.474390 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_f407d41e445c4422901d17d1a02a31b2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:34.474728 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f426887763ed48e3bc87f59d80986672. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:34.474786 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f407d41e445c4422901d17d1a02a31b2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:34.474737 2010 raft_consensus.cc:513] T f129bb3fbbfd492d98822be0a6626e44 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:34.476177 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet 377ebf66bbef40988043f6200ae0893c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:34 UTC) from {username='slave'} at 127.0.0.1:60230
W20250113 02:51:34.476229 995 leader_election.cc:343] T f129bb3fbbfd492d98822be0a6626e44 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Not found: Tablet not found: f129bb3fbbfd492d98822be0a6626e44
I20250113 02:51:34.476646 2010 leader_election.cc:290] T f129bb3fbbfd492d98822be0a6626e44 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:34.476760 1023 tablet_service.cc:1467] Processing CreateTablet for tablet 2002c69206ec468aad1b44f1b5b0d3b2 (DEFAULT_TABLE table=default.Table_9b26cd6ed9944430a1228f552a7c3d95 [id=f313b4a543574e4b936c0eb13d6d2568]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:34.477432 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2002c69206ec468aad1b44f1b5b0d3b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:34.474390 1198 rpcz_store.cc:276] 0113 02:51:33.259415 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:33.259558 (+ 143us) service_pool.cc:225] Handling call
0113 02:51:34.373523 (+1113965us) ts_tablet_manager.cc:1145] Deleting tablet 056bf1e67be04d179eace3d93282b145
0113 02:51:34.373532 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:34.413794 (+ 40262us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:34.451551 (+ 37757us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:34.452848 (+ 1297us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1113803}
I20250113 02:51:34.480793 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet 056bf1e67be04d179eace3d93282b145 (table default.Table_f5f150e8da7a4848bfab67568975d66f [id=5a295e63ad43499f90b2993ac92a16cf]) successfully deleted
I20250113 02:51:34.481125 1198 tablet_replica.cc:331] T 42fad91fe1b64a21a2c8412a2585af47 P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:34.481353 1198 raft_consensus.cc:2238] T 42fad91fe1b64a21a2c8412a2585af47 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:34.481588 1198 raft_consensus.cc:2267] T 42fad91fe1b64a21a2c8412a2585af47 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus is shut down!
W20250113 02:51:34.482309 1992 log.cc:926] Time spent T 00000000000000000000000000000000 P 45e9d43ecac0401fb2c18fe5eaf089b5: Append to log took a long time: real 0.066s user 0.000s sys 0.003s
I20250113 02:51:34.482648 742 tablet_service.cc:1514] Processing DeleteTablet for tablet 377ebf66bbef40988043f6200ae0893c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:34 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:34.485242 2141 tablet.cc:1721] T 0752fde3a30a4947a2a688c56ceadab5 P 1186fa869eb14812832fa6562531f196: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250113 02:51:34.487617 1030 tablet_service.cc:1467] Processing CreateTablet for tablet 2c49f1eefd2041b4ae617a4cb35d6f15 (DEFAULT_TABLE table=default.Table_4557b7a5878e45d0860844f7e191bfcf [id=ca60f9dab0db4b2f9fbbd9125039ab2b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:34.488090 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c49f1eefd2041b4ae617a4cb35d6f15. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:34.488962 1191 tablet_bootstrap.cc:492] T 7527e6e0b65e4a43b7f48899a34cdf7f P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:34.489130 1191 ts_tablet_manager.cc:1397] T 7527e6e0b65e4a43b7f48899a34cdf7f P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.000s
I20250113 02:51:34.489450 746 tablet_service.cc:1467] Processing CreateTablet for tablet a894199f21aa4dba8a03583ea14a9092 (DEFAULT_TABLE table=default.Table_4078d475fb2e4e52ad184b9318ec9c7d [id=6e6f2eaf51294f6a8a1228cc66b89575]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:34.489600 742 tablet_service.cc:1467] Processing CreateTablet for tablet 5a0ad1e232e848469585eb463bea8d20 (DEFAULT_TABLE table=default.Table_759338d187b64a61aea6fc11366d3877 [id=493148f498a34ae493461adaf676f7b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:34.490854 1198 ts_tablet_manager.cc:1905] T 42fad91fe1b64a21a2c8412a2585af47 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:34.491362 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a894199f21aa4dba8a03583ea14a9092. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:34.491919 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5a0ad1e232e848469585eb463bea8d20. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:34.490792 2140 tablet.cc:1721] T 0752fde3a30a4947a2a688c56ceadab5 P 4216a7f853d642b7bc3e5cff458f3aca: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250113 02:51:34.497566 892 tablet_service.cc:1514] Processing DeleteTablet for tablet 0810b9671f824d0a84741bc1904df247 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:20 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:34.498055 892 tablet_service.cc:1467] Processing CreateTablet for tablet 5a0ad1e232e848469585eb463bea8d20 (DEFAULT_TABLE table=default.Table_759338d187b64a61aea6fc11366d3877 [id=493148f498a34ae493461adaf676f7b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:34.502622 1195 ts_tablet_manager.cc:1918] T f56230a2c75d4763a54671c79358f4d9 P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:34.503197 1195 log.cc:1198] T f56230a2c75d4763a54671c79358f4d9 P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/f56230a2c75d4763a54671c79358f4d9
I20250113 02:51:34.503702 1195 ts_tablet_manager.cc:1939] T f56230a2c75d4763a54671c79358f4d9 P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:34.505498 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f129bb3fbbfd492d98822be0a6626e44" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:34.505111 1035 tablet_service.cc:1467] Processing CreateTablet for tablet 5a0ad1e232e848469585eb463bea8d20 (DEFAULT_TABLE table=default.Table_759338d187b64a61aea6fc11366d3877 [id=493148f498a34ae493461adaf676f7b6]), partition=RANGE (key) PARTITION UNBOUNDED
W20250113 02:51:34.506417 998 leader_election.cc:343] T f129bb3fbbfd492d98822be0a6626e44 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:34.506536 998 leader_election.cc:304] T f129bb3fbbfd492d98822be0a6626e44 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4216a7f853d642b7bc3e5cff458f3aca; no voters: 1186fa869eb14812832fa6562531f196, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:34.506863 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5a0ad1e232e848469585eb463bea8d20. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:34.507143 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f426887763ed48e3bc87f59d80986672. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:34.507648 1192 tablet_bootstrap.cc:492] T e468b652f71f44e3945f8d919ac0e1a8 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:34.507764 1192 ts_tablet_manager.cc:1397] T e468b652f71f44e3945f8d919ac0e1a8 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.115s user 0.001s sys 0.005s
I20250113 02:51:34.508695 1192 raft_consensus.cc:357] T e468b652f71f44e3945f8d919ac0e1a8 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:34.508909 1192 raft_consensus.cc:383] T e468b652f71f44e3945f8d919ac0e1a8 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:34.508966 1192 raft_consensus.cc:738] T e468b652f71f44e3945f8d919ac0e1a8 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:34.509204 1192 consensus_queue.cc:260] T e468b652f71f44e3945f8d919ac0e1a8 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:34.509460 2113 raft_consensus.cc:2743] T f129bb3fbbfd492d98822be0a6626e44 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:34.512264 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2002c69206ec468aad1b44f1b5b0d3b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:34.515074 1192 ts_tablet_manager.cc:1428] T e468b652f71f44e3945f8d919ac0e1a8 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20250113 02:51:34.520108 1192 tablet_bootstrap.cc:492] T 0fe5421dc3ea4be684cab084f13d8df0 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:34.521776 1192 tablet_bootstrap.cc:654] T 0fe5421dc3ea4be684cab084f13d8df0 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:34.522251 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6770a0cb45fb420d9fbf24d9a5fe4256. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:34.522538 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5a0ad1e232e848469585eb463bea8d20. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:34.522840 1191 raft_consensus.cc:357] T 7527e6e0b65e4a43b7f48899a34cdf7f P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:34.523226 1191 raft_consensus.cc:383] T 7527e6e0b65e4a43b7f48899a34cdf7f P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:34.523281 1191 raft_consensus.cc:738] T 7527e6e0b65e4a43b7f48899a34cdf7f P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:34.525409 1191 consensus_queue.cc:260] T 7527e6e0b65e4a43b7f48899a34cdf7f P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:34.525600 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c49f1eefd2041b4ae617a4cb35d6f15. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:34.527690 1191 ts_tablet_manager.cc:1428] T 7527e6e0b65e4a43b7f48899a34cdf7f P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.038s user 0.001s sys 0.000s
I20250113 02:51:34.527966 1191 tablet_bootstrap.cc:492] T a92f9b1c9c1a4cbdbf0cc7b2a3f2275b P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:34.529984 1030 tablet_service.cc:1514] Processing DeleteTablet for tablet e468b652f71f44e3945f8d919ac0e1a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:29 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:34.529628 1191 tablet_bootstrap.cc:654] T a92f9b1c9c1a4cbdbf0cc7b2a3f2275b P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:34.536628 449 catalog_manager.cc:5618] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095) does not have the latest schema for tablet 7527e6e0b65e4a43b7f48899a34cdf7f (table default.Table_c8b93363989c4e109540af6bef426f73 [id=b7846a8ebee84005b1b5d259560a26bd]). Expected version 1 got 0
I20250113 02:51:34.537628 1192 tablet_bootstrap.cc:492] T 0fe5421dc3ea4be684cab084f13d8df0 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:34.538141 1192 ts_tablet_manager.cc:1397] T 0fe5421dc3ea4be684cab084f13d8df0 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.018s user 0.001s sys 0.002s
I20250113 02:51:34.539021 1192 raft_consensus.cc:357] T 0fe5421dc3ea4be684cab084f13d8df0 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:34.540717 447 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_2f1a7bb2c4cd4ead95b175e1d3545f20" } modify_external_catalogs: true
I20250113 02:51:34.540839 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_43fca7e946094914badafec91de2b53e" } modify_external_catalogs: true
I20250113 02:51:34.540943 447 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_2f1a7bb2c4cd4ead95b175e1d3545f20" } modify_external_catalogs: true
I20250113 02:51:34.541055 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_43fca7e946094914badafec91de2b53e" } modify_external_catalogs: true
I20250113 02:51:34.542503 1192 raft_consensus.cc:383] T 0fe5421dc3ea4be684cab084f13d8df0 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:34.542614 1192 raft_consensus.cc:738] T 0fe5421dc3ea4be684cab084f13d8df0 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:34.542727 1196 tablet_replica.cc:331] T ff8ef6290698477391721fd65b9eeb28 P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:34.542932 1196 raft_consensus.cc:2238] T ff8ef6290698477391721fd65b9eeb28 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:34.543020 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 920) took 1165 ms. Trace:
I20250113 02:51:34.543092 1196 raft_consensus.cc:2267] T ff8ef6290698477391721fd65b9eeb28 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:34.543110 1195 rpcz_store.cc:276] 0113 02:51:33.377786 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:33.379373 (+ 1587us) service_pool.cc:225] Handling call
0113 02:51:34.433419 (+1054046us) ts_tablet_manager.cc:1145] Deleting tablet f56230a2c75d4763a54671c79358f4d9
0113 02:51:34.433568 (+ 149us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:34.437715 (+ 4147us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:34.502533 (+ 64818us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:34.542999 (+ 40466us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1053853}
I20250113 02:51:34.543509 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 921) took 1162 ms. Trace:
I20250113 02:51:34.543610 1196 ts_tablet_manager.cc:1905] T ff8ef6290698477391721fd65b9eeb28 P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:34.543601 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet f56230a2c75d4763a54671c79358f4d9 (table default.Table_fb08fe0ed5cd4aaebce3169c264441cb [id=34b84bee300d4b30b1d426b9f26e6e4a]) successfully deleted
I20250113 02:51:34.543651 1195 rpcz_store.cc:276] 0113 02:51:33.380563 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:33.380619 (+ 56us) service_pool.cc:225] Handling call
0113 02:51:34.543394 (+1162775us) ts_tablet_manager.cc:1145] Deleting tablet 12a28557b0cc42fdad927c86fcfa7b96
0113 02:51:34.543404 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:34.543493 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1162567}
I20250113 02:51:34.543226 1192 consensus_queue.cc:260] T 0fe5421dc3ea4be684cab084f13d8df0 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
W20250113 02:51:34.544148 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 12a28557b0cc42fdad927c86fcfa7b96 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 12a28557b0cc42fdad927c86fcfa7b96 already in progress: creating tablet
I20250113 02:51:34.545178 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 922) took 1148 ms. Trace:
I20250113 02:51:34.545254 1195 rpcz_store.cc:276] 0113 02:51:33.396759 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:33.397847 (+ 1088us) service_pool.cc:225] Handling call
0113 02:51:34.545040 (+1147193us) ts_tablet_manager.cc:1145] Deleting tablet 7b16286181684916ac84823f5fec563e
0113 02:51:34.545048 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:34.545145 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1145265}
I20250113 02:51:34.545382 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet c94eedb5dce44e6d896192c93127903a (table default.Table_616eda5ca6ce4155b20351b6b8b000f5 [id=e0cdd94a9fc947fc8a9555150b7272f0]) successfully deleted
W20250113 02:51:34.545627 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 7b16286181684916ac84823f5fec563e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7b16286181684916ac84823f5fec563e already in progress: creating tablet
W20250113 02:51:34.546211 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet b16ff51101e0452ebc05374f0d1f4917 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b16ff51101e0452ebc05374f0d1f4917 already in progress: creating tablet
W20250113 02:51:34.546761 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 35a13d9c065348e08de66044c04d221d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 35a13d9c065348e08de66044c04d221d already in progress: creating tablet
W20250113 02:51:34.547313 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet ec8196505dab4e05bb6487b566cf3144 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ec8196505dab4e05bb6487b566cf3144 already in progress: creating tablet
I20250113 02:51:34.550909 1192 ts_tablet_manager.cc:1428] T 0fe5421dc3ea4be684cab084f13d8df0 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.013s user 0.001s sys 0.001s
I20250113 02:51:34.552233 1192 tablet_bootstrap.cc:492] T 42cf1e708644427f9d4dcd41ae2bcb37 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:34.553797 1192 tablet_bootstrap.cc:654] T 42cf1e708644427f9d4dcd41ae2bcb37 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:34.557052 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_cf3667be494e4680bd98beb9c0e1fcf2" } new_table_name: "default.Table_aa7e1ca42f92492b836ab88a775ba54e" modify_external_catalogs: true
I20250113 02:51:34.558383 1195 tablet_replica.cc:331] T 42cf1e708644427f9d4dcd41ae2bcb37 P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:34.558578 1195 raft_consensus.cc:2238] T 42cf1e708644427f9d4dcd41ae2bcb37 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:34.541240 1198 ts_tablet_manager.cc:1918] T 42fad91fe1b64a21a2c8412a2585af47 P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:34.558743 1195 raft_consensus.cc:2267] T 42cf1e708644427f9d4dcd41ae2bcb37 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:34.557380 1030 tablet_service.cc:1467] Processing CreateTablet for tablet a894199f21aa4dba8a03583ea14a9092 (DEFAULT_TABLE table=default.Table_4078d475fb2e4e52ad184b9318ec9c7d [id=6e6f2eaf51294f6a8a1228cc66b89575]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:34.559119 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a894199f21aa4dba8a03583ea14a9092. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:34.559298 1195 ts_tablet_manager.cc:1905] T 42cf1e708644427f9d4dcd41ae2bcb37 P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:34.563686 2072 raft_consensus.cc:491] T e70658c90555463498d3fc2e2f13ea7f P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:34.563802 2072 raft_consensus.cc:513] T e70658c90555463498d3fc2e2f13ea7f P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:34.564435 2072 leader_election.cc:290] T e70658c90555463498d3fc2e2f13ea7f P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:34.565130 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e70658c90555463498d3fc2e2f13ea7f" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:34.565694 711 leader_election.cc:343] T e70658c90555463498d3fc2e2f13ea7f P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:34.566293 1062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e70658c90555463498d3fc2e2f13ea7f" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
W20250113 02:51:34.566794 709 leader_election.cc:343] T e70658c90555463498d3fc2e2f13ea7f P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:34.566907 709 leader_election.cc:304] T e70658c90555463498d3fc2e2f13ea7f P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:34.567241 2072 raft_consensus.cc:2743] T e70658c90555463498d3fc2e2f13ea7f P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:34.567468 1198 log.cc:1198] T 42fad91fe1b64a21a2c8412a2585af47 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/42fad91fe1b64a21a2c8412a2585af47
I20250113 02:51:34.567834 1198 ts_tablet_manager.cc:1939] T 42fad91fe1b64a21a2c8412a2585af47 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:34.568567 1194 tablet_bootstrap.cc:492] T b4806b18a4194f9c95350d89ba62e13a P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:34.568733 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 902) took 1309 ms. Trace:
I20250113 02:51:34.568789 1194 ts_tablet_manager.cc:1397] T b4806b18a4194f9c95350d89ba62e13a P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.106s user 0.003s sys 0.000s
I20250113 02:51:34.568970 1198 rpcz_store.cc:276] 0113 02:51:33.259511 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:33.259750 (+ 239us) service_pool.cc:225] Handling call
0113 02:51:34.481080 (+1221330us) ts_tablet_manager.cc:1145] Deleting tablet 42fad91fe1b64a21a2c8412a2585af47
0113 02:51:34.481090 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:34.506740 (+ 25650us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:34.541138 (+ 34398us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:34.568694 (+ 27556us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1221226}
I20250113 02:51:34.569680 1194 raft_consensus.cc:357] T b4806b18a4194f9c95350d89ba62e13a P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:34.569943 1194 raft_consensus.cc:383] T b4806b18a4194f9c95350d89ba62e13a P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:34.569681 2010 raft_consensus.cc:491] T 6666922d677a4c678f80d7c6c47fd2a5 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:34.570173 1198 tablet_replica.cc:331] T 77cf8b0c824f401aba1880eb7fd1807b P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:34.570106 2010 raft_consensus.cc:513] T 6666922d677a4c678f80d7c6c47fd2a5 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:34.570345 1198 raft_consensus.cc:2238] T 77cf8b0c824f401aba1880eb7fd1807b P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:34.570389 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet 42fad91fe1b64a21a2c8412a2585af47 (table default.Table_639e38f37be1466e9c2ce92fa1c2d890 [id=290b815241d648b6b62b8a70d2efd1c9]) successfully deleted
I20250113 02:51:34.570474 1198 raft_consensus.cc:2267] T 77cf8b0c824f401aba1880eb7fd1807b P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:34.570640 1194 raft_consensus.cc:738] T b4806b18a4194f9c95350d89ba62e13a P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:34.570741 2010 leader_election.cc:290] T 6666922d677a4c678f80d7c6c47fd2a5 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:34.570957 1198 ts_tablet_manager.cc:1905] T 77cf8b0c824f401aba1880eb7fd1807b P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:34.570878 1194 consensus_queue.cc:260] T b4806b18a4194f9c95350d89ba62e13a P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:34.571393 1194 ts_tablet_manager.cc:1428] T b4806b18a4194f9c95350d89ba62e13a P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
W20250113 02:51:34.572284 995 leader_election.cc:343] T 6666922d677a4c678f80d7c6c47fd2a5 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Not found: Tablet not found: 6666922d677a4c678f80d7c6c47fd2a5
I20250113 02:51:34.572530 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6666922d677a4c678f80d7c6c47fd2a5" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:34.572999 998 leader_election.cc:343] T 6666922d677a4c678f80d7c6c47fd2a5 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:34.573103 998 leader_election.cc:304] T 6666922d677a4c678f80d7c6c47fd2a5 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4216a7f853d642b7bc3e5cff458f3aca; no voters: 1186fa869eb14812832fa6562531f196, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:34.574007 2010 raft_consensus.cc:2743] T 6666922d677a4c678f80d7c6c47fd2a5 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:34.575908 1194 tablet_bootstrap.cc:492] T 11950e996c814590bc9f3f22c0354902 P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:34.578012 2010 raft_consensus.cc:491] T 0fe5421dc3ea4be684cab084f13d8df0 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:34.578296 2010 raft_consensus.cc:513] T 0fe5421dc3ea4be684cab084f13d8df0 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
W20250113 02:51:34.582966 995 leader_election.cc:343] T 0fe5421dc3ea4be684cab084f13d8df0 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Not found: Tablet not found: 0fe5421dc3ea4be684cab084f13d8df0
W20250113 02:51:34.545625 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 21bcdd741104474bad7023426856d18f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 21bcdd741104474bad7023426856d18f already in progress: creating tablet
I20250113 02:51:34.585713 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fe5421dc3ea4be684cab084f13d8df0" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:34.586278 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 54aae7c131404facba9af3a3d143b4a4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 54aae7c131404facba9af3a3d143b4a4 already in progress: creating tablet
W20250113 02:51:34.586395 998 leader_election.cc:343] T 0fe5421dc3ea4be684cab084f13d8df0 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:34.586494 998 leader_election.cc:304] T 0fe5421dc3ea4be684cab084f13d8df0 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4216a7f853d642b7bc3e5cff458f3aca; no voters: 1186fa869eb14812832fa6562531f196, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:34.586764 2113 raft_consensus.cc:2743] T 0fe5421dc3ea4be684cab084f13d8df0 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:34.587373 2010 leader_election.cc:290] T 0fe5421dc3ea4be684cab084f13d8df0 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:34.589000 1194 tablet_bootstrap.cc:654] T 11950e996c814590bc9f3f22c0354902 P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:34.588898 890 tablet_service.cc:1467] Processing CreateTablet for tablet a894199f21aa4dba8a03583ea14a9092 (DEFAULT_TABLE table=default.Table_4078d475fb2e4e52ad184b9318ec9c7d [id=6e6f2eaf51294f6a8a1228cc66b89575]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:34.595206 1191 tablet_bootstrap.cc:492] T a92f9b1c9c1a4cbdbf0cc7b2a3f2275b P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:34.595400 1191 ts_tablet_manager.cc:1397] T a92f9b1c9c1a4cbdbf0cc7b2a3f2275b P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.067s user 0.003s sys 0.003s
I20250113 02:51:34.595638 1198 ts_tablet_manager.cc:1918] T 77cf8b0c824f401aba1880eb7fd1807b P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:34.596273 1191 raft_consensus.cc:357] T a92f9b1c9c1a4cbdbf0cc7b2a3f2275b P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:34.596575 1191 raft_consensus.cc:383] T a92f9b1c9c1a4cbdbf0cc7b2a3f2275b P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:34.596729 1191 raft_consensus.cc:738] T a92f9b1c9c1a4cbdbf0cc7b2a3f2275b P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:34.597182 1191 consensus_queue.cc:260] T a92f9b1c9c1a4cbdbf0cc7b2a3f2275b P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:34.598119 1191 ts_tablet_manager.cc:1428] T a92f9b1c9c1a4cbdbf0cc7b2a3f2275b P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250113 02:51:34.600256 1191 tablet_bootstrap.cc:492] T 3025935f9743442eb3e6f2db22eb52ad P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:34.605590 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a894199f21aa4dba8a03583ea14a9092. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:34.610872 1196 ts_tablet_manager.cc:1918] T ff8ef6290698477391721fd65b9eeb28 P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:34.611102 1196 log.cc:1198] T ff8ef6290698477391721fd65b9eeb28 P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/ff8ef6290698477391721fd65b9eeb28
I20250113 02:51:34.611655 1196 ts_tablet_manager.cc:1939] T ff8ef6290698477391721fd65b9eeb28 P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:34.612356 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet ff8ef6290698477391721fd65b9eeb28 (table default.Table_6c7330704479422a817543c4cbbd70d3 [id=7e176c2354964cdd8e6b724f6bb84b99]) successfully deleted
I20250113 02:51:34.612934 2072 raft_consensus.cc:491] T eebad045e7bb43fa9ca9c9af8b3b7ea5 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:34.613072 2072 raft_consensus.cc:513] T eebad045e7bb43fa9ca9c9af8b3b7ea5 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:34.613695 2072 leader_election.cc:290] T eebad045e7bb43fa9ca9c9af8b3b7ea5 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:34.614454 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eebad045e7bb43fa9ca9c9af8b3b7ea5" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:34.614960 711 leader_election.cc:343] T eebad045e7bb43fa9ca9c9af8b3b7ea5 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:34.605018 2113 consensus_queue.cc:1035] T 1a63f485caf54068abd6b50d7de3d14a P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 02:51:34.615293 1198 log.cc:1198] T 77cf8b0c824f401aba1880eb7fd1807b P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/77cf8b0c824f401aba1880eb7fd1807b
I20250113 02:51:34.614619 1062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eebad045e7bb43fa9ca9c9af8b3b7ea5" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:34.616034 1107 heartbeater.cc:346] Connected to a master server at 127.0.104.60:42067
I20250113 02:51:34.616109 1198 ts_tablet_manager.cc:1939] T 77cf8b0c824f401aba1880eb7fd1807b P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:34.616851 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 900) took 1357 ms. Trace:
W20250113 02:51:34.617719 709 leader_election.cc:343] T eebad045e7bb43fa9ca9c9af8b3b7ea5 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:34.617842 709 leader_election.cc:304] T eebad045e7bb43fa9ca9c9af8b3b7ea5 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:34.618532 2072 raft_consensus.cc:2743] T eebad045e7bb43fa9ca9c9af8b3b7ea5 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:34.619192 1191 tablet_bootstrap.cc:654] T 3025935f9743442eb3e6f2db22eb52ad P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:34.623481 1195 ts_tablet_manager.cc:1918] T 42cf1e708644427f9d4dcd41ae2bcb37 P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:34.624189 1195 log.cc:1198] T 42cf1e708644427f9d4dcd41ae2bcb37 P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/42cf1e708644427f9d4dcd41ae2bcb37
I20250113 02:51:34.624956 1194 tablet_bootstrap.cc:492] T 11950e996c814590bc9f3f22c0354902 P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:34.625197 1194 ts_tablet_manager.cc:1397] T 11950e996c814590bc9f3f22c0354902 P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.049s user 0.003s sys 0.000s
I20250113 02:51:34.626197 1194 raft_consensus.cc:357] T 11950e996c814590bc9f3f22c0354902 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:34.626540 1194 raft_consensus.cc:383] T 11950e996c814590bc9f3f22c0354902 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:34.626677 1194 raft_consensus.cc:738] T 11950e996c814590bc9f3f22c0354902 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:34.627009 1194 consensus_queue.cc:260] T 11950e996c814590bc9f3f22c0354902 P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:34.627610 1195 ts_tablet_manager.cc:1939] T 42cf1e708644427f9d4dcd41ae2bcb37 P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:34.630064 1195 tablet_replica.cc:331] T 4b07a985897641c3ac48021c2a4daedb P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:34.630383 1194 ts_tablet_manager.cc:1428] T 11950e996c814590bc9f3f22c0354902 P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250113 02:51:34.630769 1194 tablet_bootstrap.cc:492] T efccefd6787943469131492741ee2bd5 P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:34.630813 1195 raft_consensus.cc:2238] T 4b07a985897641c3ac48021c2a4daedb P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:34.631199 1195 raft_consensus.cc:2267] T 4b07a985897641c3ac48021c2a4daedb P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:34.632565 1195 ts_tablet_manager.cc:1905] T 4b07a985897641c3ac48021c2a4daedb P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:34.627377 1198 rpcz_store.cc:276] 0113 02:51:33.259323 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:33.261873 (+ 2550us) service_pool.cc:225] Handling call
0113 02:51:34.570127 (+1308254us) ts_tablet_manager.cc:1145] Deleting tablet 77cf8b0c824f401aba1880eb7fd1807b
0113 02:51:34.570136 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:34.592775 (+ 22639us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:34.595554 (+ 2779us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:34.616833 (+ 21279us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1308088}
I20250113 02:51:34.634349 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet 77cf8b0c824f401aba1880eb7fd1807b (table default.Table_ea18ad8d07b34ee4b3ef07b8f0431cfd [id=6069bb344e624c3bb8ca97e6c7fe23cc]) successfully deleted
I20250113 02:51:34.634853 1198 tablet_replica.cc:331] T 9fff52e99c64462a8a73fccc3c22ca63 P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:34.636214 1198 raft_consensus.cc:2238] T 9fff52e99c64462a8a73fccc3c22ca63 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:34.636749 1198 raft_consensus.cc:2267] T 9fff52e99c64462a8a73fccc3c22ca63 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:34.637639 1198 ts_tablet_manager.cc:1905] T 9fff52e99c64462a8a73fccc3c22ca63 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250113 02:51:34.641422 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet d52b9d0f60714a31918c1645178f3525 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d52b9d0f60714a31918c1645178f3525 already in progress: creating tablet
I20250113 02:51:34.641770 1191 tablet_bootstrap.cc:492] T 3025935f9743442eb3e6f2db22eb52ad P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:34.642164 1191 ts_tablet_manager.cc:1397] T 3025935f9743442eb3e6f2db22eb52ad P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.043s user 0.008s sys 0.000s
W20250113 02:51:34.642189 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 02bf7042d4ce4077851a44bbfb6c4b46 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 02bf7042d4ce4077851a44bbfb6c4b46 already in progress: creating tablet
W20250113 02:51:34.642760 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 9b2d0aa6f0e9456d95f4e48c360ab6e8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9b2d0aa6f0e9456d95f4e48c360ab6e8 already in progress: creating tablet
I20250113 02:51:34.644220 1191 raft_consensus.cc:357] T 3025935f9743442eb3e6f2db22eb52ad P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:34.644688 1191 raft_consensus.cc:383] T 3025935f9743442eb3e6f2db22eb52ad P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:34.644821 1191 raft_consensus.cc:738] T 3025935f9743442eb3e6f2db22eb52ad P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:34.645313 1191 consensus_queue.cc:260] T 3025935f9743442eb3e6f2db22eb52ad P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:34.647291 1191 ts_tablet_manager.cc:1428] T 3025935f9743442eb3e6f2db22eb52ad P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20250113 02:51:34.648523 1191 tablet_bootstrap.cc:492] T 7213e3284b4d4cb581d9fb946a9f87e9 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
W20250113 02:51:34.649276 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet d2699bb529bf4f0f9d7f2668ffe4c9ff because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d2699bb529bf4f0f9d7f2668ffe4c9ff already in progress: creating tablet
W20250113 02:51:34.656011 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet ec06c9f6a3bf46a9bcb90a85a82b6ff8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ec06c9f6a3bf46a9bcb90a85a82b6ff8 already in progress: creating tablet
I20250113 02:51:34.658819 1191 tablet_bootstrap.cc:654] T 7213e3284b4d4cb581d9fb946a9f87e9 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:34.664168 1196 tablet_replica.cc:331] T f68f9f4076d9475d938e795e72bdd468 P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:34.664505 1196 raft_consensus.cc:2238] T f68f9f4076d9475d938e795e72bdd468 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:34.664741 1196 raft_consensus.cc:2267] T f68f9f4076d9475d938e795e72bdd468 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:34.665377 1196 ts_tablet_manager.cc:1905] T f68f9f4076d9475d938e795e72bdd468 P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:34.666980 1195 ts_tablet_manager.cc:1918] T 4b07a985897641c3ac48021c2a4daedb P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:34.667186 1195 log.cc:1198] T 4b07a985897641c3ac48021c2a4daedb P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/4b07a985897641c3ac48021c2a4daedb
I20250113 02:51:34.667793 1195 ts_tablet_manager.cc:1939] T 4b07a985897641c3ac48021c2a4daedb P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:34.668752 1194 tablet_bootstrap.cc:654] T efccefd6787943469131492741ee2bd5 P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:34.670096 1198 ts_tablet_manager.cc:1918] T 9fff52e99c64462a8a73fccc3c22ca63 P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250113 02:51:34.672569 1198 log.cc:1198] T 9fff52e99c64462a8a73fccc3c22ca63 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/9fff52e99c64462a8a73fccc3c22ca63
I20250113 02:51:34.673123 1198 ts_tablet_manager.cc:1939] T 9fff52e99c64462a8a73fccc3c22ca63 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:34.674366 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 898) took 1425 ms. Trace:
I20250113 02:51:34.675526 2072 raft_consensus.cc:491] T fb18c16fb3ba44ec8a190b88d3bbd1b1 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:34.675652 2072 raft_consensus.cc:513] T fb18c16fb3ba44ec8a190b88d3bbd1b1 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:34.676441 2072 leader_election.cc:290] T fb18c16fb3ba44ec8a190b88d3bbd1b1 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:34.676352 1198 rpcz_store.cc:276] 0113 02:51:33.251892 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:33.268625 (+ 16733us) service_pool.cc:225] Handling call
0113 02:51:34.634787 (+1366162us) ts_tablet_manager.cc:1145] Deleting tablet 9fff52e99c64462a8a73fccc3c22ca63
0113 02:51:34.634803 (+ 16us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:34.642147 (+ 7344us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:34.669990 (+ 27843us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:34.674345 (+ 4355us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1365971}
I20250113 02:51:34.677177 1062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb18c16fb3ba44ec8a190b88d3bbd1b1" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:34.677321 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb18c16fb3ba44ec8a190b88d3bbd1b1" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:34.677721 709 leader_election.cc:343] T fb18c16fb3ba44ec8a190b88d3bbd1b1 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:34.678037 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet 9fff52e99c64462a8a73fccc3c22ca63 (table default.Table_9ef18ee71ad843c5921750a2f8478855 [id=bd8bd572ed524f2cae01f820f07e3c36]) successfully deleted
I20250113 02:51:34.678963 1198 tablet_replica.cc:331] T 8adfa738c17a4169a5d99f6ceca75100 P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
W20250113 02:51:34.678951 711 leader_election.cc:343] T fb18c16fb3ba44ec8a190b88d3bbd1b1 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:34.679060 711 leader_election.cc:304] T fb18c16fb3ba44ec8a190b88d3bbd1b1 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:34.679361 1198 raft_consensus.cc:2238] T 8adfa738c17a4169a5d99f6ceca75100 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: Raft consensus shutting down.
I20250113 02:51:34.679527 2109 raft_consensus.cc:2743] T fb18c16fb3ba44ec8a190b88d3bbd1b1 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:34.679683 1198 raft_consensus.cc:2267] T 8adfa738c17a4169a5d99f6ceca75100 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:34.680244 1198 ts_tablet_manager.cc:1905] T 8adfa738c17a4169a5d99f6ceca75100 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250113 02:51:34.682976 998 consensus_peers.cc:487] T 6e1284c8aaba44b5810b0d06d36757eb P 4216a7f853d642b7bc3e5cff458f3aca -> Peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Couldn't send request to peer df69f48656b44bb890f76587ee69cc0f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250113 02:51:34.686085 439 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.104.1:40095, user_credentials={real_user=slave}} blocked reactor thread for 138800us
I20250113 02:51:34.687608 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet 42cf1e708644427f9d4dcd41ae2bcb37 (table default.Table_54963a1a7ff24d5a99644441a755f5de [id=acf16114671d4a3eb7ed4a0ffc984fef]) successfully deleted
I20250113 02:51:34.687817 1196 ts_tablet_manager.cc:1918] T f68f9f4076d9475d938e795e72bdd468 P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250113 02:51:34.689522 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 17fc320eb01f4ac2b9361aeab5b452fc consensus state has no leader
I20250113 02:51:34.692133 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet 0fe5421dc3ea4be684cab084f13d8df0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:24 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:34.692947 449 catalog_manager.cc:5618] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095) does not have the latest schema for tablet 7527e6e0b65e4a43b7f48899a34cdf7f (table default.Table_c8b93363989c4e109540af6bef426f73 [id=b7846a8ebee84005b1b5d259560a26bd]). Expected version 1 got 0
I20250113 02:51:34.694736 1194 tablet_bootstrap.cc:492] T efccefd6787943469131492741ee2bd5 P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:34.698751 1194 ts_tablet_manager.cc:1397] T efccefd6787943469131492741ee2bd5 P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.068s user 0.002s sys 0.002s
I20250113 02:51:34.699928 1194 raft_consensus.cc:357] T efccefd6787943469131492741ee2bd5 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:34.700235 1194 raft_consensus.cc:383] T efccefd6787943469131492741ee2bd5 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:34.700346 1194 raft_consensus.cc:738] T efccefd6787943469131492741ee2bd5 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:34.700590 1194 consensus_queue.cc:260] T efccefd6787943469131492741ee2bd5 P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:34.701148 1194 ts_tablet_manager.cc:1428] T efccefd6787943469131492741ee2bd5 P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250113 02:51:34.702304 451 master_service.cc:946] ReplaceTablet: received request to replace tablet e13267498917432ba92b6badb76f5d73 from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:34.703696 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_a7cd30c14bce41f0aa51afc99452fddd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:34.704878 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a7cd30c14bce41f0aa51afc99452fddd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:34.706542 964 heartbeater.cc:346] Connected to a master server at 127.0.104.60:42067
I20250113 02:51:34.709148 885 tablet_service.cc:1514] Processing DeleteTablet for tablet 11950e996c814590bc9f3f22c0354902 with delete_type TABLET_DATA_DELETED (replaced by tablet 00d4381876234f16b19336523640bbc8) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:34.709645 885 tablet_service.cc:1514] Processing DeleteTablet for tablet efccefd6787943469131492741ee2bd5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:21 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:34.709836 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_035c14601d094e78bc0f136d22321b4d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:34.710675 1198 ts_tablet_manager.cc:1918] T 8adfa738c17a4169a5d99f6ceca75100 P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20250113 02:51:34.711076 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_035c14601d094e78bc0f136d22321b4d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:34.712029 454 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_4557b7a5878e45d0860844f7e191bfcf" } modify_external_catalogs: true
I20250113 02:51:34.712572 1194 tablet_bootstrap.cc:492] T 1cfbcd682b944c89847b40fe69f0a75e P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:34.713483 1195 tablet_replica.cc:331] T 999fecb3fc65466cad21cc1ff5ef2d44 P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:34.715934 1194 tablet_bootstrap.cc:654] T 1cfbcd682b944c89847b40fe69f0a75e P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:34.715734 1195 raft_consensus.cc:2238] T 999fecb3fc65466cad21cc1ff5ef2d44 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:34.716532 1198 log.cc:1198] T 8adfa738c17a4169a5d99f6ceca75100 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/8adfa738c17a4169a5d99f6ceca75100
I20250113 02:51:34.716835 1195 raft_consensus.cc:2267] T 999fecb3fc65466cad21cc1ff5ef2d44 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:34.713608 454 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_4557b7a5878e45d0860844f7e191bfcf" } modify_external_catalogs: true
I20250113 02:51:34.716131 1196 log.cc:1198] T f68f9f4076d9475d938e795e72bdd468 P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/f68f9f4076d9475d938e795e72bdd468
I20250113 02:51:34.717468 1198 ts_tablet_manager.cc:1939] T 8adfa738c17a4169a5d99f6ceca75100 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:34.717708 1196 ts_tablet_manager.cc:1939] T f68f9f4076d9475d938e795e72bdd468 P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:34.719592 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_d9a1e87da55c409281a423318555570c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:34.720753 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet f68f9f4076d9475d938e795e72bdd468 (table default.Table_92ea541283614df981d96c5b74362f1d [id=9e58548d1a184cc0b362f148c44005ed]) successfully deleted
W20250113 02:51:34.720867 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d9a1e87da55c409281a423318555570c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250113 02:51:34.721599 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 6e208f83f3384a78a959909b4ac690c0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6e208f83f3384a78a959909b4ac690c0 already in progress: creating tablet
I20250113 02:51:34.722337 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 903) took 1450 ms. Trace:
I20250113 02:51:34.723336 1198 rpcz_store.cc:276] 0113 02:51:33.271782 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:33.271831 (+ 49us) service_pool.cc:225] Handling call
0113 02:51:34.678920 (+1407089us) ts_tablet_manager.cc:1145] Deleting tablet 8adfa738c17a4169a5d99f6ceca75100
0113 02:51:34.678929 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:34.692169 (+ 13240us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:34.710569 (+ 18400us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:34.722314 (+ 11745us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1406937}
I20250113 02:51:34.724112 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet 8adfa738c17a4169a5d99f6ceca75100 (table default.Table_9a9b1ced3bef4e648088e0781851128e [id=ae544735623e4cfd894dc9340aca177c]) successfully deleted
I20250113 02:51:34.724234 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 904) took 1343 ms. Trace:
I20250113 02:51:34.724305 1198 rpcz_store.cc:276] 0113 02:51:33.380861 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:33.380918 (+ 57us) service_pool.cc:225] Handling call
0113 02:51:34.724111 (+1343193us) ts_tablet_manager.cc:1145] Deleting tablet 12a28557b0cc42fdad927c86fcfa7b96
0113 02:51:34.724121 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:34.724218 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1328208}
W20250113 02:51:34.724689 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 12a28557b0cc42fdad927c86fcfa7b96 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 12a28557b0cc42fdad927c86fcfa7b96 already in progress: creating tablet
I20250113 02:51:34.728397 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 905) took 1330 ms. Trace:
W20250113 02:51:34.728588 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet db486978b33a45df90025988b084aecd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet db486978b33a45df90025988b084aecd already in progress: creating tablet
I20250113 02:51:34.728830 1198 rpcz_store.cc:276] 0113 02:51:33.397562 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:33.397619 (+ 57us) service_pool.cc:225] Handling call
0113 02:51:34.728287 (+1330668us) ts_tablet_manager.cc:1145] Deleting tablet 7b16286181684916ac84823f5fec563e
0113 02:51:34.728298 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:34.728380 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1330489}
W20250113 02:51:34.729311 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 1c77033ca393469e9324d2406a65a5e9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1c77033ca393469e9324d2406a65a5e9 already in progress: creating tablet
I20250113 02:51:34.729380 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 920) took 1134 ms. Trace:
W20250113 02:51:34.729460 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 7b16286181684916ac84823f5fec563e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7b16286181684916ac84823f5fec563e already in progress: creating tablet
I20250113 02:51:34.729449 1198 rpcz_store.cc:276] 0113 02:51:33.594834 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:33.594885 (+ 51us) service_pool.cc:225] Handling call
0113 02:51:34.729280 (+1134395us) ts_tablet_manager.cc:1145] Deleting tablet b16ff51101e0452ebc05374f0d1f4917
0113 02:51:34.729289 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:34.729366 (+ 77us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1134226}
W20250113 02:51:34.729777 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet b16ff51101e0452ebc05374f0d1f4917 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b16ff51101e0452ebc05374f0d1f4917 already in progress: creating tablet
W20250113 02:51:34.730019 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 4a3ba2cd77b8416db72d894b09cc1100 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4a3ba2cd77b8416db72d894b09cc1100 already in progress: creating tablet
W20250113 02:51:34.730695 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 377ebf66bbef40988043f6200ae0893c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 377ebf66bbef40988043f6200ae0893c already in progress: creating tablet
I20250113 02:51:34.731050 1196 tablet_replica.cc:331] T 0810b9671f824d0a84741bc1904df247 P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:34.731345 1196 raft_consensus.cc:2238] T 0810b9671f824d0a84741bc1904df247 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:34.736141 1196 raft_consensus.cc:2267] T 0810b9671f824d0a84741bc1904df247 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:34.736416 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 922) took 1091 ms. Trace:
I20250113 02:51:34.736845 1196 ts_tablet_manager.cc:1905] T 0810b9671f824d0a84741bc1904df247 P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:34.736747 1198 rpcz_store.cc:276] 0113 02:51:33.645216 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:33.646765 (+ 1549us) service_pool.cc:225] Handling call
0113 02:51:34.736272 (+1089507us) ts_tablet_manager.cc:1145] Deleting tablet 35a13d9c065348e08de66044c04d221d
0113 02:51:34.736283 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:34.736396 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1089335}
W20250113 02:51:34.737210 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 35a13d9c065348e08de66044c04d221d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 35a13d9c065348e08de66044c04d221d already in progress: creating tablet
I20250113 02:51:34.737416 891 tablet_service.cc:1514] Processing DeleteTablet for tablet b4806b18a4194f9c95350d89ba62e13a with delete_type TABLET_DATA_DELETED (replaced by tablet 146dad5c977f464496f7d21611030dad) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:34.738286 1194 tablet_bootstrap.cc:492] T 1cfbcd682b944c89847b40fe69f0a75e P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:34.738510 1194 ts_tablet_manager.cc:1397] T 1cfbcd682b944c89847b40fe69f0a75e P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.026s user 0.005s sys 0.000s
I20250113 02:51:34.739414 1192 tablet_bootstrap.cc:492] T 42cf1e708644427f9d4dcd41ae2bcb37 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:34.739545 1192 ts_tablet_manager.cc:1397] T 42cf1e708644427f9d4dcd41ae2bcb37 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.187s user 0.006s sys 0.022s
I20250113 02:51:34.739401 1194 raft_consensus.cc:357] T 1cfbcd682b944c89847b40fe69f0a75e P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:34.739848 1194 raft_consensus.cc:383] T 1cfbcd682b944c89847b40fe69f0a75e P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:34.740008 1194 raft_consensus.cc:738] T 1cfbcd682b944c89847b40fe69f0a75e P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:34.740272 1192 raft_consensus.cc:357] T 42cf1e708644427f9d4dcd41ae2bcb37 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:34.740451 1192 raft_consensus.cc:383] T 42cf1e708644427f9d4dcd41ae2bcb37 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:34.740504 1192 raft_consensus.cc:738] T 42cf1e708644427f9d4dcd41ae2bcb37 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:34.740693 1192 consensus_queue.cc:260] T 42cf1e708644427f9d4dcd41ae2bcb37 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:34.734373 449 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_915411462fb84dd7b917caa2f3a31498" } modify_external_catalogs: true
I20250113 02:51:34.740975 449 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_915411462fb84dd7b917caa2f3a31498" } modify_external_catalogs: true
I20250113 02:51:34.741662 1194 consensus_queue.cc:260] T 1cfbcd682b944c89847b40fe69f0a75e P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:34.742815 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_b2f523343ede4ce5950a27d55c16eac2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:34.743283 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b2f523343ede4ce5950a27d55c16eac2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:34.746380 1194 ts_tablet_manager.cc:1428] T 1cfbcd682b944c89847b40fe69f0a75e P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20250113 02:51:34.746634 1192 ts_tablet_manager.cc:1428] T 42cf1e708644427f9d4dcd41ae2bcb37 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20250113 02:51:34.749513 1192 tablet_bootstrap.cc:492] T aa5c172a69bf4ca79999f4b58df08de8 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:34.750979 1194 tablet_bootstrap.cc:492] T 840773ee80c7447097ea000b106fc89d P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:34.752738 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 924) took 1086 ms. Trace:
I20250113 02:51:34.752849 1198 rpcz_store.cc:276] 0113 02:51:33.666200 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:33.666307 (+ 107us) service_pool.cc:225] Handling call
0113 02:51:34.751781 (+1085474us) ts_tablet_manager.cc:1145] Deleting tablet ec8196505dab4e05bb6487b566cf3144
0113 02:51:34.752617 (+ 836us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:34.752721 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1085312}
I20250113 02:51:34.753326 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 370c53f565df4e6f915e54a2fd2bf2ce
I20250113 02:51:34.753580 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet df87e25053a74990959f85288a85217b
I20250113 02:51:34.753798 1198 tablet_replica.cc:331] T c826a0b8327f44f9a00e5afe3a36cd8a P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:34.753979 1198 raft_consensus.cc:2238] T c826a0b8327f44f9a00e5afe3a36cd8a P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:34.754114 1198 raft_consensus.cc:2267] T c826a0b8327f44f9a00e5afe3a36cd8a P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:34.754216 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_13fd4df9c207466cb85cd8ed9062722c" } new_table_name: "default.Table_9cb972a609224de5a8d9bc404f5a193a" modify_external_catalogs: true
I20250113 02:51:34.754427 2010 raft_consensus.cc:491] T 42cf1e708644427f9d4dcd41ae2bcb37 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:34.754560 2010 raft_consensus.cc:513] T 42cf1e708644427f9d4dcd41ae2bcb37 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:34.755170 2010 leader_election.cc:290] T 42cf1e708644427f9d4dcd41ae2bcb37 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:34.755205 452 catalog_manager.cc:5618] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467) does not have the latest schema for tablet 1cfbcd682b944c89847b40fe69f0a75e (table default.Table_32e88abaae96492fbcce39fb2cd7c221 [id=2b84deb07e5c4ab8b550fe42945d7b71]). Expected version 2 got 0
I20250113 02:51:34.755676 1192 tablet_bootstrap.cc:654] T aa5c172a69bf4ca79999f4b58df08de8 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:34.755930 885 tablet_service.cc:1514] Processing DeleteTablet for tablet df87e25053a74990959f85288a85217b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:34 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:34.756722 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_e2733f0c9d6a4e2b9f020c369a2e8e52" } new_table_name: "default.Table_165edade161e458c81a066ac578d832c" modify_external_catalogs: true
I20250113 02:51:34.756865 1198 ts_tablet_manager.cc:1905] T c826a0b8327f44f9a00e5afe3a36cd8a P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:34.758298 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42cf1e708644427f9d4dcd41ae2bcb37" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:34.758929 998 leader_election.cc:343] T 42cf1e708644427f9d4dcd41ae2bcb37 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
W20250113 02:51:34.759369 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet ec8196505dab4e05bb6487b566cf3144 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ec8196505dab4e05bb6487b566cf3144 already in progress: creating tablet
I20250113 02:51:34.759819 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet 370c53f565df4e6f915e54a2fd2bf2ce with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:34 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:34.759908 1030 tablet_service.cc:1514] Processing DeleteTablet for tablet df87e25053a74990959f85288a85217b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:34 UTC) from {username='slave'} at 127.0.0.1:60230
W20250113 02:51:34.761756 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f24cda32b6e947c7983ae2f1f9a64c23 consensus state has no leader
I20250113 02:51:34.761909 1194 tablet_bootstrap.cc:654] T 840773ee80c7447097ea000b106fc89d P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:34.762547 1198 ts_tablet_manager.cc:1918] T c826a0b8327f44f9a00e5afe3a36cd8a P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:34.762646 1198 log.cc:1198] T c826a0b8327f44f9a00e5afe3a36cd8a P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/c826a0b8327f44f9a00e5afe3a36cd8a
I20250113 02:51:34.762912 446 catalog_manager.cc:5618] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467) does not have the latest schema for tablet 1cfbcd682b944c89847b40fe69f0a75e (table default.Table_32e88abaae96492fbcce39fb2cd7c221 [id=2b84deb07e5c4ab8b550fe42945d7b71]). Expected version 2 got 0
I20250113 02:51:34.762954 1198 ts_tablet_manager.cc:1939] T c826a0b8327f44f9a00e5afe3a36cd8a P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:34.763247 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 927) took 1077 ms. Trace:
I20250113 02:51:34.763332 1198 rpcz_store.cc:276] 0113 02:51:33.685345 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:33.685607 (+ 262us) service_pool.cc:225] Handling call
0113 02:51:34.753759 (+1068152us) ts_tablet_manager.cc:1145] Deleting tablet c826a0b8327f44f9a00e5afe3a36cd8a
0113 02:51:34.753768 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:34.759824 (+ 6056us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:34.762473 (+ 2649us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:34.763229 (+ 756us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1067999}
I20250113 02:51:34.763610 1198 tablet_replica.cc:331] T c1fb2b36c0504cb4ba32671ae4c620d1 P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:34.763659 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_ae7b078c404f4196ac5289edb6ac5b3f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:34.764804 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet c826a0b8327f44f9a00e5afe3a36cd8a (table default.Table_03babf194cc448f296512a09b2e3fe3e [id=64b42698782f41a589577af20b33501e]) successfully deleted
W20250113 02:51:34.765448 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ae7b078c404f4196ac5289edb6ac5b3f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:34.770563 1195 ts_tablet_manager.cc:1905] T 999fecb3fc65466cad21cc1ff5ef2d44 P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250113 02:51:34.771989 995 leader_election.cc:343] T 42cf1e708644427f9d4dcd41ae2bcb37 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Not found: Tablet not found: 42cf1e708644427f9d4dcd41ae2bcb37
I20250113 02:51:34.772100 995 leader_election.cc:304] T 42cf1e708644427f9d4dcd41ae2bcb37 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4216a7f853d642b7bc3e5cff458f3aca; no voters: 1186fa869eb14812832fa6562531f196, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:34.772591 891 tablet_service.cc:1514] Processing DeleteTablet for tablet 370c53f565df4e6f915e54a2fd2bf2ce with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:34 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:34.772886 2113 raft_consensus.cc:2743] T 42cf1e708644427f9d4dcd41ae2bcb37 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:34.773057 2113 consensus_queue.cc:1035] T 6e1284c8aaba44b5810b0d06d36757eb P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 02:51:34.780551 1191 tablet_bootstrap.cc:492] T 7213e3284b4d4cb581d9fb946a9f87e9 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:34.780696 1191 ts_tablet_manager.cc:1397] T 7213e3284b4d4cb581d9fb946a9f87e9 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.132s user 0.010s sys 0.000s
I20250113 02:51:34.781999 1191 raft_consensus.cc:357] T 7213e3284b4d4cb581d9fb946a9f87e9 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:34.782400 1191 raft_consensus.cc:383] T 7213e3284b4d4cb581d9fb946a9f87e9 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:34.782461 1191 raft_consensus.cc:738] T 7213e3284b4d4cb581d9fb946a9f87e9 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:34.782749 1194 tablet_bootstrap.cc:492] T 840773ee80c7447097ea000b106fc89d P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:34.782722 1191 consensus_queue.cc:260] T 7213e3284b4d4cb581d9fb946a9f87e9 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:34.783113 1194 ts_tablet_manager.cc:1397] T 840773ee80c7447097ea000b106fc89d P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.032s user 0.005s sys 0.001s
I20250113 02:51:34.783509 1191 ts_tablet_manager.cc:1428] T 7213e3284b4d4cb581d9fb946a9f87e9 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250113 02:51:34.783815 1191 tablet_bootstrap.cc:492] T 0b119704bc774ca9b4155c9619d4765b P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:34.790288 1195 ts_tablet_manager.cc:1918] T 999fecb3fc65466cad21cc1ff5ef2d44 P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:34.790402 1195 log.cc:1198] T 999fecb3fc65466cad21cc1ff5ef2d44 P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/999fecb3fc65466cad21cc1ff5ef2d44
I20250113 02:51:34.784552 1194 raft_consensus.cc:357] T 840773ee80c7447097ea000b106fc89d P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:34.790668 1195 ts_tablet_manager.cc:1939] T 999fecb3fc65466cad21cc1ff5ef2d44 P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:34.790766 1194 raft_consensus.cc:383] T 840773ee80c7447097ea000b106fc89d P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:34.790915 1194 raft_consensus.cc:738] T 840773ee80c7447097ea000b106fc89d P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:34.791231 1194 consensus_queue.cc:260] T 840773ee80c7447097ea000b106fc89d P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:34.791807 1194 ts_tablet_manager.cc:1428] T 840773ee80c7447097ea000b106fc89d P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20250113 02:51:34.792407 1194 tablet_bootstrap.cc:492] T 41fcde988f72496b824271fa36c1a226 P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:34.794107 1194 tablet_bootstrap.cc:654] T 41fcde988f72496b824271fa36c1a226 P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:34.800159 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet 4b07a985897641c3ac48021c2a4daedb (table default.Table_066f136aea8141fa91862f7285fbaac2 [id=fa29f9bbea034fafad86ad86aa5775eb]) successfully deleted
W20250113 02:51:34.800503 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 21bcdd741104474bad7023426856d18f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 21bcdd741104474bad7023426856d18f already in progress: creating tablet
W20250113 02:51:34.800750 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 54aae7c131404facba9af3a3d143b4a4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 54aae7c131404facba9af3a3d143b4a4 already in progress: creating tablet
W20250113 02:51:34.801017 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet d52b9d0f60714a31918c1645178f3525 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d52b9d0f60714a31918c1645178f3525 already in progress: creating tablet
I20250113 02:51:34.802505 751 tablet_service.cc:1514] Processing DeleteTablet for tablet 370c53f565df4e6f915e54a2fd2bf2ce with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:34 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:34.802858 751 tablet_service.cc:1514] Processing DeleteTablet for tablet df87e25053a74990959f85288a85217b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:34 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:34.800719 1196 ts_tablet_manager.cc:1918] T 0810b9671f824d0a84741bc1904df247 P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:34.805851 1196 log.cc:1198] T 0810b9671f824d0a84741bc1904df247 P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/0810b9671f824d0a84741bc1904df247
I20250113 02:51:34.804324 1195 tablet_replica.cc:331] T 0916b41c171a419791797559fbc2807d P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:34.806779 1196 ts_tablet_manager.cc:1939] T 0810b9671f824d0a84741bc1904df247 P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:34.807463 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet 0810b9671f824d0a84741bc1904df247 (table default.Table_796801d17a3c458693693af6e39cf0f5 [id=df24b379d986402187cec33190501012]) successfully deleted
I20250113 02:51:34.807825 1196 tablet_replica.cc:331] T 11950e996c814590bc9f3f22c0354902 P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:34.808106 1196 raft_consensus.cc:2238] T 11950e996c814590bc9f3f22c0354902 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:34.808341 1196 raft_consensus.cc:2267] T 11950e996c814590bc9f3f22c0354902 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:34.809108 1196 ts_tablet_manager.cc:1905] T 11950e996c814590bc9f3f22c0354902 P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:34.810977 1191 tablet_bootstrap.cc:654] T 0b119704bc774ca9b4155c9619d4765b P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:34.815199 890 tablet_service.cc:1514] Processing DeleteTablet for tablet 840773ee80c7447097ea000b106fc89d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:19 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:34.820403 453 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_220afbc6ffc34d63a882ab817a21087f" } modify_external_catalogs: true
I20250113 02:51:34.820664 453 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_220afbc6ffc34d63a882ab817a21087f" } modify_external_catalogs: true
I20250113 02:51:34.822835 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_603d7765bb3442899a83e29f0cc0cac3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:34.823575 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet 999fecb3fc65466cad21cc1ff5ef2d44 (table default.Table_991d8c72ce924153ad119a440b54fb1c [id=71e76dffaa6f4b0682422afed1749a64]) successfully deleted
W20250113 02:51:34.823895 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_603d7765bb3442899a83e29f0cc0cac3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:34.825928 1195 raft_consensus.cc:2238] T 0916b41c171a419791797559fbc2807d P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:34.826663 1195 raft_consensus.cc:2267] T 0916b41c171a419791797559fbc2807d P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:34.827704 1192 tablet_bootstrap.cc:492] T aa5c172a69bf4ca79999f4b58df08de8 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:34.827848 1192 ts_tablet_manager.cc:1397] T aa5c172a69bf4ca79999f4b58df08de8 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.078s user 0.001s sys 0.003s
I20250113 02:51:34.828668 1192 raft_consensus.cc:357] T aa5c172a69bf4ca79999f4b58df08de8 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:34.828914 1192 raft_consensus.cc:383] T aa5c172a69bf4ca79999f4b58df08de8 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:34.828981 1192 raft_consensus.cc:738] T aa5c172a69bf4ca79999f4b58df08de8 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:34.829210 1192 consensus_queue.cc:260] T aa5c172a69bf4ca79999f4b58df08de8 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
W20250113 02:51:34.829416 998 consensus_peers.cc:487] T 1a63f485caf54068abd6b50d7de3d14a P 4216a7f853d642b7bc3e5cff458f3aca -> Peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Couldn't send request to peer df69f48656b44bb890f76587ee69cc0f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250113 02:51:34.831861 1192 ts_tablet_manager.cc:1428] T aa5c172a69bf4ca79999f4b58df08de8 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20250113 02:51:34.832144 1192 tablet_bootstrap.cc:492] T 4b07a985897641c3ac48021c2a4daedb P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:34.834271 1196 ts_tablet_manager.cc:1918] T 11950e996c814590bc9f3f22c0354902 P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:34.834393 1196 log.cc:1198] T 11950e996c814590bc9f3f22c0354902 P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/11950e996c814590bc9f3f22c0354902
I20250113 02:51:34.834682 1196 ts_tablet_manager.cc:1939] T 11950e996c814590bc9f3f22c0354902 P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:34.835144 1196 tablet_replica.cc:331] T efccefd6787943469131492741ee2bd5 P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:34.835280 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet 11950e996c814590bc9f3f22c0354902 (table default.Table_325c103d1e1349e4896118881852e462 [id=2e275f7f92ac440ca3ea6e29c5566f00]) successfully deleted
I20250113 02:51:34.835865 1194 tablet_bootstrap.cc:492] T 41fcde988f72496b824271fa36c1a226 P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:34.837476 1194 ts_tablet_manager.cc:1397] T 41fcde988f72496b824271fa36c1a226 P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.045s user 0.002s sys 0.004s
I20250113 02:51:34.838266 1194 raft_consensus.cc:357] T 41fcde988f72496b824271fa36c1a226 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:34.838518 1194 raft_consensus.cc:383] T 41fcde988f72496b824271fa36c1a226 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:34.838616 1194 raft_consensus.cc:738] T 41fcde988f72496b824271fa36c1a226 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:34.838866 1194 consensus_queue.cc:260] T 41fcde988f72496b824271fa36c1a226 P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:34.839612 1194 ts_tablet_manager.cc:1428] T 41fcde988f72496b824271fa36c1a226 P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250113 02:51:34.839854 1196 raft_consensus.cc:2238] T efccefd6787943469131492741ee2bd5 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:34.840075 1196 raft_consensus.cc:2267] T efccefd6787943469131492741ee2bd5 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:34.841583 1194 tablet_bootstrap.cc:492] T 9fff52e99c64462a8a73fccc3c22ca63 P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:34.843809 1194 tablet_bootstrap.cc:654] T 9fff52e99c64462a8a73fccc3c22ca63 P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:34.845489 1196 ts_tablet_manager.cc:1905] T efccefd6787943469131492741ee2bd5 P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:34.847987 2140 tablet.cc:1721] T 32cba8f3efc54e78b4df41c15df82eea P 4216a7f853d642b7bc3e5cff458f3aca: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 6
I20250113 02:51:34.850852 1191 tablet_bootstrap.cc:492] T 0b119704bc774ca9b4155c9619d4765b P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:34.851123 1191 ts_tablet_manager.cc:1397] T 0b119704bc774ca9b4155c9619d4765b P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.067s user 0.007s sys 0.000s
I20250113 02:51:34.849236 890 tablet_service.cc:1514] Processing DeleteTablet for tablet 41fcde988f72496b824271fa36c1a226 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:23 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:34.852094 2142 tablet.cc:1721] T 32cba8f3efc54e78b4df41c15df82eea P df69f48656b44bb890f76587ee69cc0f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 6
I20250113 02:51:34.853560 1191 raft_consensus.cc:357] T 0b119704bc774ca9b4155c9619d4765b P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:34.853760 1191 raft_consensus.cc:383] T 0b119704bc774ca9b4155c9619d4765b P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:34.853818 1191 raft_consensus.cc:738] T 0b119704bc774ca9b4155c9619d4765b P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:34.854063 1191 consensus_queue.cc:260] T 0b119704bc774ca9b4155c9619d4765b P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:34.854738 1191 ts_tablet_manager.cc:1428] T 0b119704bc774ca9b4155c9619d4765b P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250113 02:51:34.855170 1191 tablet_bootstrap.cc:492] T df87e25053a74990959f85288a85217b P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:34.856864 1191 tablet_bootstrap.cc:654] T df87e25053a74990959f85288a85217b P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
W20250113 02:51:34.858814 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c5d77927b1b24ffebb21c9f10e5cd440 consensus state has no leader
W20250113 02:51:34.859263 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f59dc0fa43dd4eb5932a1ad8a5a08349 has no consensus state
I20250113 02:51:34.859256 1196 ts_tablet_manager.cc:1918] T efccefd6787943469131492741ee2bd5 P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:34.859236 2141 tablet.cc:1721] T 32cba8f3efc54e78b4df41c15df82eea P 1186fa869eb14812832fa6562531f196: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 6
I20250113 02:51:34.859401 1196 log.cc:1198] T efccefd6787943469131492741ee2bd5 P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/efccefd6787943469131492741ee2bd5
I20250113 02:51:34.859687 1196 ts_tablet_manager.cc:1939] T efccefd6787943469131492741ee2bd5 P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:34.860208 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet efccefd6787943469131492741ee2bd5 (table default.Table_faf2fafcef874bf6a5798ee6b19ef704 [id=bcdd761e38934a2e9197b9c1b9165b27]) successfully deleted
I20250113 02:51:34.865411 1192 tablet_bootstrap.cc:654] T 4b07a985897641c3ac48021c2a4daedb P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:34.867470 450 catalog_manager.cc:6071] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: default.Table_da3e49fa1e654145a05858466ec2260f [id=94704429b49b46e38a48258a450d2705] alter complete (version 1)
I20250113 02:51:34.868326 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet d565b77f90784d87b64996772c938085
I20250113 02:51:34.869251 890 tablet_service.cc:1514] Processing DeleteTablet for tablet d565b77f90784d87b64996772c938085 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:34 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:34.869894 1194 tablet_bootstrap.cc:492] T 9fff52e99c64462a8a73fccc3c22ca63 P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:34.870079 1194 ts_tablet_manager.cc:1397] T 9fff52e99c64462a8a73fccc3c22ca63 P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.029s user 0.004s sys 0.000s
I20250113 02:51:34.870488 1198 raft_consensus.cc:2238] T c1fb2b36c0504cb4ba32671ae4c620d1 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:34.870698 1198 raft_consensus.cc:2267] T c1fb2b36c0504cb4ba32671ae4c620d1 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:34.872267 1030 tablet_service.cc:1514] Processing DeleteTablet for tablet d565b77f90784d87b64996772c938085 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:34 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:34.870882 1194 raft_consensus.cc:357] T 9fff52e99c64462a8a73fccc3c22ca63 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:34.873265 1194 raft_consensus.cc:383] T 9fff52e99c64462a8a73fccc3c22ca63 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:34.873386 1194 raft_consensus.cc:738] T 9fff52e99c64462a8a73fccc3c22ca63 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:34.873692 1194 consensus_queue.cc:260] T 9fff52e99c64462a8a73fccc3c22ca63 P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:34.874320 1194 ts_tablet_manager.cc:1428] T 9fff52e99c64462a8a73fccc3c22ca63 P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250113 02:51:34.874626 1194 tablet_bootstrap.cc:492] T de4a58bf63054890808e2e2fe46e43d2 P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:34.876489 1194 tablet_bootstrap.cc:654] T de4a58bf63054890808e2e2fe46e43d2 P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:34.877348 1198 ts_tablet_manager.cc:1905] T c1fb2b36c0504cb4ba32671ae4c620d1 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:34.879237 751 tablet_service.cc:1514] Processing DeleteTablet for tablet d565b77f90784d87b64996772c938085 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:34 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:34.887647 1196 tablet_replica.cc:331] T b4806b18a4194f9c95350d89ba62e13a P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:34.887938 1196 raft_consensus.cc:2238] T b4806b18a4194f9c95350d89ba62e13a P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:34.888188 1196 raft_consensus.cc:2267] T b4806b18a4194f9c95350d89ba62e13a P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:34.889070 1196 ts_tablet_manager.cc:1905] T b4806b18a4194f9c95350d89ba62e13a P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:34.895164 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 726e68108f5b4de4a51ffdf778ef825d
I20250113 02:51:34.895401 1194 tablet_bootstrap.cc:492] T de4a58bf63054890808e2e2fe46e43d2 P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:34.895530 1194 ts_tablet_manager.cc:1397] T de4a58bf63054890808e2e2fe46e43d2 P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.000s
I20250113 02:51:34.896338 751 tablet_service.cc:1514] Processing DeleteTablet for tablet 726e68108f5b4de4a51ffdf778ef825d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:34 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:34.896338 1030 tablet_service.cc:1514] Processing DeleteTablet for tablet 726e68108f5b4de4a51ffdf778ef825d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:34 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:34.896348 1194 raft_consensus.cc:357] T de4a58bf63054890808e2e2fe46e43d2 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:34.896661 1194 raft_consensus.cc:383] T de4a58bf63054890808e2e2fe46e43d2 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:34.896766 1194 raft_consensus.cc:738] T de4a58bf63054890808e2e2fe46e43d2 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:34.897097 890 tablet_service.cc:1514] Processing DeleteTablet for tablet 726e68108f5b4de4a51ffdf778ef825d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:34 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:34.897037 1194 consensus_queue.cc:260] T de4a58bf63054890808e2e2fe46e43d2 P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:34.897768 1194 ts_tablet_manager.cc:1428] T de4a58bf63054890808e2e2fe46e43d2 P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250113 02:51:34.897720 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_fd1e5342502a4cd9b3f8078ae35cbd52"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:34.898279 1194 tablet_bootstrap.cc:492] T 092892a36e0e47cb8d2184c2bbde95d7 P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:34.898471 449 catalog_manager.cc:5618] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095) does not have the latest schema for tablet 1a63f485caf54068abd6b50d7de3d14a (table default.Table_f8a6812c7fb94ea69cd9bf775b914ea6 [id=71508871e2d5469a91aaf0e19b02b638]). Expected version 1 got 0
I20250113 02:51:34.898761 449 catalog_manager.cc:5618] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095) does not have the latest schema for tablet 6e1284c8aaba44b5810b0d06d36757eb (table default.Table_bfc5031f68f64006aceefe2ee986223a [id=aa4679fc0d7148e3bef21342b3a0ea22]). Expected version 2 got 0
I20250113 02:51:34.900195 1194 tablet_bootstrap.cc:654] T 092892a36e0e47cb8d2184c2bbde95d7 P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:34.900851 751 tablet_service.cc:1514] Processing DeleteTablet for tablet a92f9b1c9c1a4cbdbf0cc7b2a3f2275b with delete_type TABLET_DATA_DELETED (replaced by tablet bc6a7322c0c34adb85457750526d8955) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:34.901242 450 catalog_manager.cc:5618] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587) does not have the latest schema for tablet aa5c172a69bf4ca79999f4b58df08de8 (table default.Table_3f1e7146fed34d53896f138e43a4d26e [id=5dceb76e1dbe422ab98d4da5bf6c45a0]). Expected version 1 got 0
I20250113 02:51:34.902117 450 catalog_manager.cc:5618] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587) does not have the latest schema for tablet 1a63f485caf54068abd6b50d7de3d14a (table default.Table_f8a6812c7fb94ea69cd9bf775b914ea6 [id=71508871e2d5469a91aaf0e19b02b638]). Expected version 1 got 0
I20250113 02:51:34.902170 751 tablet_service.cc:1514] Processing DeleteTablet for tablet 7213e3284b4d4cb581d9fb946a9f87e9 with delete_type TABLET_DATA_DELETED (replaced by tablet 7c443454994a47f2bfa7e2ec7ac576e0) from {username='slave'} at 127.0.0.1:55398
W20250113 02:51:34.902315 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fd1e5342502a4cd9b3f8078ae35cbd52 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:34.902650 751 tablet_service.cc:1514] Processing DeleteTablet for tablet 3025935f9743442eb3e6f2db22eb52ad with delete_type TABLET_DATA_DELETED (replaced by tablet 5d2d14548b734b1fbe57d68c0ea3e2f8) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:34.904726 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_d2b687ff8f614fa3ae9aeb8ca9a54531"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:34.905308 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d2b687ff8f614fa3ae9aeb8ca9a54531 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:34.909936 1191 tablet_bootstrap.cc:492] T df87e25053a74990959f85288a85217b P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:34.910065 1191 ts_tablet_manager.cc:1397] T df87e25053a74990959f85288a85217b P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.055s user 0.000s sys 0.003s
I20250113 02:51:34.909976 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_ee56f8e59e4f40efbaf24ff31dcf0920" } new_table_name: "default.Table_a7b637f0a6a244fdb2172e2015816fce" modify_external_catalogs: true
I20250113 02:51:34.910760 2072 raft_consensus.cc:491] T 3025935f9743442eb3e6f2db22eb52ad P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:34.910876 2072 raft_consensus.cc:513] T 3025935f9743442eb3e6f2db22eb52ad P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:34.911543 2072 leader_election.cc:290] T 3025935f9743442eb3e6f2db22eb52ad P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:34.912228 1062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3025935f9743442eb3e6f2db22eb52ad" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:34.912526 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3025935f9743442eb3e6f2db22eb52ad" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:34.913066 709 leader_election.cc:343] T 3025935f9743442eb3e6f2db22eb52ad P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:34.913264 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_66000626bf85477d9623d439bad71554"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:34.913511 711 leader_election.cc:343] T 3025935f9743442eb3e6f2db22eb52ad P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:34.913604 711 leader_election.cc:304] T 3025935f9743442eb3e6f2db22eb52ad P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
W20250113 02:51:34.913772 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_66000626bf85477d9623d439bad71554 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:34.913990 2072 raft_consensus.cc:2743] T 3025935f9743442eb3e6f2db22eb52ad P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:34.914512 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_182ee8e8d8854eeabef569bd48babd04" } new_table_name: "default.Table_6461b062100240d59a5d91a87a74510c" modify_external_catalogs: true
I20250113 02:51:34.909026 1196 ts_tablet_manager.cc:1918] T b4806b18a4194f9c95350d89ba62e13a P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:34.915316 1196 log.cc:1198] T b4806b18a4194f9c95350d89ba62e13a P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/b4806b18a4194f9c95350d89ba62e13a
I20250113 02:51:34.915802 1196 ts_tablet_manager.cc:1939] T b4806b18a4194f9c95350d89ba62e13a P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:34.916862 1196 tablet_replica.cc:331] T 840773ee80c7447097ea000b106fc89d P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:34.917327 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet b4806b18a4194f9c95350d89ba62e13a (table default.Table_4369a143c72643b084df58d25314c6dd [id=36ff239c08a54be99ca307354810fff3]) successfully deleted
W20250113 02:51:34.917589 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet df87e25053a74990959f85288a85217b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet df87e25053a74990959f85288a85217b already in progress: creating tablet
W20250113 02:51:34.917819 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 370c53f565df4e6f915e54a2fd2bf2ce because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 370c53f565df4e6f915e54a2fd2bf2ce already in progress: creating tablet
I20250113 02:51:34.918246 2147 raft_consensus.cc:491] T de4a58bf63054890808e2e2fe46e43d2 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:34.920838 1196 raft_consensus.cc:2238] T 840773ee80c7447097ea000b106fc89d P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:34.920785 2147 raft_consensus.cc:513] T de4a58bf63054890808e2e2fe46e43d2 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:34.921312 1196 raft_consensus.cc:2267] T 840773ee80c7447097ea000b106fc89d P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:34.921653 1195 ts_tablet_manager.cc:1905] T 0916b41c171a419791797559fbc2807d P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:34.921739 2147 leader_election.cc:290] T de4a58bf63054890808e2e2fe46e43d2 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:34.922446 1196 ts_tablet_manager.cc:1905] T 840773ee80c7447097ea000b106fc89d P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:34.923341 1191 raft_consensus.cc:357] T df87e25053a74990959f85288a85217b P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:34.923554 1191 raft_consensus.cc:383] T df87e25053a74990959f85288a85217b P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250113 02:51:34.922999 852 leader_election.cc:343] T de4a58bf63054890808e2e2fe46e43d2 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Not found: Tablet not found: de4a58bf63054890808e2e2fe46e43d2
I20250113 02:51:34.923617 1191 raft_consensus.cc:738] T df87e25053a74990959f85288a85217b P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:34.923465 1194 tablet_bootstrap.cc:492] T 092892a36e0e47cb8d2184c2bbde95d7 P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
W20250113 02:51:34.923676 851 leader_election.cc:343] T de4a58bf63054890808e2e2fe46e43d2 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Not found: Tablet not found: de4a58bf63054890808e2e2fe46e43d2
I20250113 02:51:34.923883 1194 ts_tablet_manager.cc:1397] T 092892a36e0e47cb8d2184c2bbde95d7 P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.026s user 0.003s sys 0.002s
I20250113 02:51:34.923837 851 leader_election.cc:304] T de4a58bf63054890808e2e2fe46e43d2 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df69f48656b44bb890f76587ee69cc0f; no voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca
I20250113 02:51:34.923818 1191 consensus_queue.cc:260] T df87e25053a74990959f85288a85217b P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:34.924311 2147 raft_consensus.cc:2743] T de4a58bf63054890808e2e2fe46e43d2 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:34.924607 1194 raft_consensus.cc:357] T 092892a36e0e47cb8d2184c2bbde95d7 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:34.924872 1194 raft_consensus.cc:383] T 092892a36e0e47cb8d2184c2bbde95d7 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:34.924968 1194 raft_consensus.cc:738] T 092892a36e0e47cb8d2184c2bbde95d7 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:34.925218 1194 consensus_queue.cc:260] T 092892a36e0e47cb8d2184c2bbde95d7 P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:34.925757 1194 ts_tablet_manager.cc:1428] T 092892a36e0e47cb8d2184c2bbde95d7 P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250113 02:51:34.928192 1194 tablet_bootstrap.cc:492] T be4b2a8a3927410eaf12e7592646e4af P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:34.929435 1191 ts_tablet_manager.cc:1428] T df87e25053a74990959f85288a85217b P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.019s user 0.002s sys 0.000s
I20250113 02:51:34.929925 1194 tablet_bootstrap.cc:654] T be4b2a8a3927410eaf12e7592646e4af P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:34.932390 1195 ts_tablet_manager.cc:1918] T 0916b41c171a419791797559fbc2807d P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:34.932657 1195 log.cc:1198] T 0916b41c171a419791797559fbc2807d P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/0916b41c171a419791797559fbc2807d
I20250113 02:51:34.933279 1195 ts_tablet_manager.cc:1939] T 0916b41c171a419791797559fbc2807d P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:34.937072 2010 raft_consensus.cc:491] T b8b6ffe690fc459db78b2b3bc12671dd P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:34.937199 1196 ts_tablet_manager.cc:1918] T 840773ee80c7447097ea000b106fc89d P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:34.937284 1196 log.cc:1198] T 840773ee80c7447097ea000b106fc89d P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/840773ee80c7447097ea000b106fc89d
I20250113 02:51:34.937297 2010 raft_consensus.cc:513] T b8b6ffe690fc459db78b2b3bc12671dd P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:34.937909 2010 leader_election.cc:290] T b8b6ffe690fc459db78b2b3bc12671dd P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:34.938571 1198 ts_tablet_manager.cc:1918] T c1fb2b36c0504cb4ba32671ae4c620d1 P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:34.938674 1198 log.cc:1198] T c1fb2b36c0504cb4ba32671ae4c620d1 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/c1fb2b36c0504cb4ba32671ae4c620d1
I20250113 02:51:34.938980 1198 ts_tablet_manager.cc:1939] T c1fb2b36c0504cb4ba32671ae4c620d1 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:34.939235 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8b6ffe690fc459db78b2b3bc12671dd" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:34.939944 998 leader_election.cc:343] T b8b6ffe690fc459db78b2b3bc12671dd P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
W20250113 02:51:34.940400 995 leader_election.cc:343] T b8b6ffe690fc459db78b2b3bc12671dd P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Not found: Tablet not found: b8b6ffe690fc459db78b2b3bc12671dd
I20250113 02:51:34.940486 995 leader_election.cc:304] T b8b6ffe690fc459db78b2b3bc12671dd P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4216a7f853d642b7bc3e5cff458f3aca; no voters: 1186fa869eb14812832fa6562531f196, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:34.940894 2010 raft_consensus.cc:2743] T b8b6ffe690fc459db78b2b3bc12671dd P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:34.946326 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_9c662bce442d44bd86f62065a4325cf9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:34.946893 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9c662bce442d44bd86f62065a4325cf9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:34.947947 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 928) took 1262 ms. Trace:
I20250113 02:51:34.948102 1198 rpcz_store.cc:276] 0113 02:51:33.685423 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:33.685787 (+ 364us) service_pool.cc:225] Handling call
0113 02:51:34.763570 (+1077783us) ts_tablet_manager.cc:1145] Deleting tablet c1fb2b36c0504cb4ba32671ae4c620d1
0113 02:51:34.763580 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:34.924164 (+160584us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:34.938486 (+ 14322us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:34.947874 (+ 9388us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7424,"tablet-delete.queue_time_us":1077675}
I20250113 02:51:34.948546 1196 ts_tablet_manager.cc:1939] T 840773ee80c7447097ea000b106fc89d P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:34.949005 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet c1fb2b36c0504cb4ba32671ae4c620d1 (table default.Table_53a85211f49241c4af531e479ecdc0cb [id=ff8b884b04954ff48a7c8a72df232e8a]) successfully deleted
I20250113 02:51:34.949152 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet 840773ee80c7447097ea000b106fc89d (table default.Table_5aafd46e1cbf4e03a5b87797d5d97bdd [id=b6d8d0de849a4d3c988b8fd468b14d62]) successfully deleted
I20250113 02:51:34.949564 1198 tablet_replica.cc:331] T a97efc70946c4af487e33a85abf6b90c P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:34.949820 1198 raft_consensus.cc:2238] T a97efc70946c4af487e33a85abf6b90c P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:34.950083 1196 tablet_replica.cc:331] T 41fcde988f72496b824271fa36c1a226 P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:34.950084 1198 raft_consensus.cc:2267] T a97efc70946c4af487e33a85abf6b90c P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:34.950248 1196 raft_consensus.cc:2238] T 41fcde988f72496b824271fa36c1a226 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:34.950379 1196 raft_consensus.cc:2267] T 41fcde988f72496b824271fa36c1a226 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:34.950841 1198 ts_tablet_manager.cc:1905] T a97efc70946c4af487e33a85abf6b90c P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:34.953263 1196 ts_tablet_manager.cc:1905] T 41fcde988f72496b824271fa36c1a226 P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:34.960503 1191 tablet_bootstrap.cc:492] T 4eb5f1944cb44fd5a40ca92a6a3d7214 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:34.962756 1191 tablet_bootstrap.cc:654] T 4eb5f1944cb44fd5a40ca92a6a3d7214 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:34.969712 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 947) took 1042 ms. Trace:
I20250113 02:51:34.969834 1195 rpcz_store.cc:276] 0113 02:51:33.926799 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:33.928606 (+ 1807us) service_pool.cc:225] Handling call
0113 02:51:34.804086 (+875480us) ts_tablet_manager.cc:1145] Deleting tablet 0916b41c171a419791797559fbc2807d
0113 02:51:34.804185 (+ 99us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:34.927121 (+122936us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:34.932269 (+ 5148us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:34.969689 (+ 37420us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":875292}
I20250113 02:51:34.970377 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 949) took 1039 ms. Trace:
I20250113 02:51:34.970463 1195 rpcz_store.cc:276] 0113 02:51:33.930706 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:33.930775 (+ 69us) service_pool.cc:225] Handling call
0113 02:51:34.970229 (+1039454us) ts_tablet_manager.cc:1145] Deleting tablet 02bf7042d4ce4077851a44bbfb6c4b46
0113 02:51:34.970241 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:34.970353 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1039249}
I20250113 02:51:34.970785 1195 tablet_replica.cc:331] T 1d15c7a01f854f0aaabcd30ec3166e31 P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:34.971019 1195 raft_consensus.cc:2238] T 1d15c7a01f854f0aaabcd30ec3166e31 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:34.971169 1195 raft_consensus.cc:2267] T 1d15c7a01f854f0aaabcd30ec3166e31 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:34.971378 1196 ts_tablet_manager.cc:1918] T 41fcde988f72496b824271fa36c1a226 P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:34.971482 1196 log.cc:1198] T 41fcde988f72496b824271fa36c1a226 P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/41fcde988f72496b824271fa36c1a226
I20250113 02:51:34.971764 1196 ts_tablet_manager.cc:1939] T 41fcde988f72496b824271fa36c1a226 P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:34.971833 1195 ts_tablet_manager.cc:1905] T 1d15c7a01f854f0aaabcd30ec3166e31 P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:34.972355 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet 41fcde988f72496b824271fa36c1a226 (table default.Table_2caed919541e43c78a858513a2128f50 [id=26ba87b9bd384b99a016b29fe5b5babd]) successfully deleted
I20250113 02:51:34.972365 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet 0916b41c171a419791797559fbc2807d (table default.Table_8be7ebd7e1984b979fce6e7aac1f576e [id=8231428f86ba4a6388c55c3c06f6b194]) successfully deleted
W20250113 02:51:34.972760 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 02bf7042d4ce4077851a44bbfb6c4b46 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 02bf7042d4ce4077851a44bbfb6c4b46 already in progress: creating tablet
W20250113 02:51:34.977681 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet d565b77f90784d87b64996772c938085 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d565b77f90784d87b64996772c938085 already in progress: creating tablet
W20250113 02:51:34.978646 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 726e68108f5b4de4a51ffdf778ef825d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 726e68108f5b4de4a51ffdf778ef825d already in progress: creating tablet
I20250113 02:51:34.981559 1192 tablet_bootstrap.cc:492] T 4b07a985897641c3ac48021c2a4daedb P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:34.981760 1192 ts_tablet_manager.cc:1397] T 4b07a985897641c3ac48021c2a4daedb P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.150s user 0.011s sys 0.021s
I20250113 02:51:34.982252 1195 ts_tablet_manager.cc:1918] T 1d15c7a01f854f0aaabcd30ec3166e31 P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:34.985107 1192 raft_consensus.cc:357] T 4b07a985897641c3ac48021c2a4daedb P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:34.990517 1198 ts_tablet_manager.cc:1918] T a97efc70946c4af487e33a85abf6b90c P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:34.990751 1195 log.cc:1198] T 1d15c7a01f854f0aaabcd30ec3166e31 P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/1d15c7a01f854f0aaabcd30ec3166e31
I20250113 02:51:34.991930 1195 ts_tablet_manager.cc:1939] T 1d15c7a01f854f0aaabcd30ec3166e31 P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:34.992534 453 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_bee5bcef1f534c6c812e6075291ff90d" } modify_external_catalogs: true
I20250113 02:51:34.992583 446 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_23c1655ab4cf4a30b67185ccf4fbb76d" } modify_external_catalogs: true
I20250113 02:51:34.992769 453 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_bee5bcef1f534c6c812e6075291ff90d" } modify_external_catalogs: true
I20250113 02:51:34.994170 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet 1d15c7a01f854f0aaabcd30ec3166e31 (table default.Table_21648ca05a3344848aeac2ad300f7400 [id=fe31ebf2077b43a285155bef4c983d75]) successfully deleted
I20250113 02:51:34.995018 446 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_23c1655ab4cf4a30b67185ccf4fbb76d" } modify_external_catalogs: true
I20250113 02:51:34.996327 1198 log.cc:1198] T a97efc70946c4af487e33a85abf6b90c P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/a97efc70946c4af487e33a85abf6b90c
I20250113 02:51:34.996686 1195 tablet_replica.cc:331] T 33c665e149dd48e8868acd935de581bf P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:34.997097 1198 ts_tablet_manager.cc:1939] T a97efc70946c4af487e33a85abf6b90c P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:34.997500 1195 raft_consensus.cc:2238] T 33c665e149dd48e8868acd935de581bf P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:34.998571 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 929) took 1313 ms. Trace:
I20250113 02:51:34.998840 1195 raft_consensus.cc:2267] T 33c665e149dd48e8868acd935de581bf P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:34.999572 1195 ts_tablet_manager.cc:1905] T 33c665e149dd48e8868acd935de581bf P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250113 02:51:35.000501 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c368b5b0c3a940c4b4efb64c946375c3 consensus state has no leader
I20250113 02:51:35.000897 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_e9dd2f1b0ade4fc6a1db103979189b11" } new_table_name: "default.Table_1b00bd87905943a480ae8c6344417112" modify_external_catalogs: true
W20250113 02:51:35.002419 451 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 56b022c94847411785a7e9db3757a2d9 consensus state has no leader
I20250113 02:51:35.004076 1198 rpcz_store.cc:276] 0113 02:51:33.685513 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:33.685917 (+ 404us) service_pool.cc:225] Handling call
0113 02:51:34.949507 (+1263590us) ts_tablet_manager.cc:1145] Deleting tablet a97efc70946c4af487e33a85abf6b90c
0113 02:51:34.949521 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:34.957772 (+ 8251us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:34.990423 (+ 32651us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:34.998551 (+ 8128us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1263488}
I20250113 02:51:35.005318 1198 tablet_replica.cc:331] T 6d4d8b5cc8ad4c9e935d8afaf5766e66 P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:35.006053 1198 raft_consensus.cc:2238] T 6d4d8b5cc8ad4c9e935d8afaf5766e66 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:35.006539 1198 raft_consensus.cc:2267] T 6d4d8b5cc8ad4c9e935d8afaf5766e66 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:35.007140 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet a97efc70946c4af487e33a85abf6b90c (table default.Table_a5e21e7ff75b4d70a6944712fb3d1392 [id=d29a78ceccaa42808de9c2736d3eb4ac]) successfully deleted
I20250113 02:51:35.012526 1198 ts_tablet_manager.cc:1905] T 6d4d8b5cc8ad4c9e935d8afaf5766e66 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:35.013077 2010 raft_consensus.cc:491] T 377ebf66bbef40988043f6200ae0893c P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:35.013242 2010 raft_consensus.cc:513] T 377ebf66bbef40988043f6200ae0893c P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:35.013851 2010 leader_election.cc:290] T 377ebf66bbef40988043f6200ae0893c P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:35.014722 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "377ebf66bbef40988043f6200ae0893c" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
I20250113 02:51:35.014935 774 raft_consensus.cc:2463] T 377ebf66bbef40988043f6200ae0893c P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 0.
I20250113 02:51:35.014894 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_4b9ceec9d6854b32894186e0c19dfe0f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:35.015486 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4b9ceec9d6854b32894186e0c19dfe0f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:35.015499 995 leader_election.cc:304] T 377ebf66bbef40988043f6200ae0893c P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters:
I20250113 02:51:35.015823 2010 raft_consensus.cc:2798] T 377ebf66bbef40988043f6200ae0893c P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 02:51:35.015893 2010 raft_consensus.cc:491] T 377ebf66bbef40988043f6200ae0893c P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 02:51:35.015964 2010 raft_consensus.cc:3054] T 377ebf66bbef40988043f6200ae0893c P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:35.016746 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "377ebf66bbef40988043f6200ae0893c" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:35.017241 998 leader_election.cc:343] T 377ebf66bbef40988043f6200ae0893c P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:35.018503 2010 raft_consensus.cc:513] T 377ebf66bbef40988043f6200ae0893c P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:35.020298 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "377ebf66bbef40988043f6200ae0893c" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196"
I20250113 02:51:35.020483 774 raft_consensus.cc:3054] T 377ebf66bbef40988043f6200ae0893c P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:34.985853 1192 raft_consensus.cc:383] T 4b07a985897641c3ac48021c2a4daedb P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:35.021406 1192 raft_consensus.cc:738] T 4b07a985897641c3ac48021c2a4daedb P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:35.021773 1192 consensus_queue.cc:260] T 4b07a985897641c3ac48021c2a4daedb P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:35.022619 1192 ts_tablet_manager.cc:1428] T 4b07a985897641c3ac48021c2a4daedb P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.041s user 0.001s sys 0.001s
I20250113 02:51:35.023180 774 raft_consensus.cc:2463] T 377ebf66bbef40988043f6200ae0893c P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 1.
I20250113 02:51:35.024585 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_ea97918d1c344fdfb9491dbe8439ac69"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:35.025147 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ea97918d1c344fdfb9491dbe8439ac69 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:35.025192 1192 tablet_bootstrap.cc:492] T 999fecb3fc65466cad21cc1ff5ef2d44 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:35.026815 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_ba66360c176446ee945a533bd8358296"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:35.027293 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ba66360c176446ee945a533bd8358296 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:35.027896 1192 tablet_bootstrap.cc:654] T 999fecb3fc65466cad21cc1ff5ef2d44 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:35.028309 995 leader_election.cc:304] T 377ebf66bbef40988043f6200ae0893c P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters:
I20250113 02:51:35.028589 2113 raft_consensus.cc:2798] T 377ebf66bbef40988043f6200ae0893c P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Leader election won for term 1
I20250113 02:51:35.028689 2113 raft_consensus.cc:695] T 377ebf66bbef40988043f6200ae0893c P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: Becoming Leader. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Running, Role: LEADER
I20250113 02:51:35.028964 2113 consensus_queue.cc:237] T 377ebf66bbef40988043f6200ae0893c P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:35.037357 1195 ts_tablet_manager.cc:1918] T 33c665e149dd48e8868acd935de581bf P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:35.037901 1195 log.cc:1198] T 33c665e149dd48e8868acd935de581bf P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/33c665e149dd48e8868acd935de581bf
I20250113 02:51:35.038522 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "377ebf66bbef40988043f6200ae0893c" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f"
W20250113 02:51:35.039103 998 leader_election.cc:343] T 377ebf66bbef40988043f6200ae0893c P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:35.039515 2010 leader_election.cc:290] T 377ebf66bbef40988043f6200ae0893c P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Requested vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:35.039537 1195 ts_tablet_manager.cc:1939] T 33c665e149dd48e8868acd935de581bf P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:35.042475 1191 tablet_bootstrap.cc:492] T 4eb5f1944cb44fd5a40ca92a6a3d7214 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:35.042500 1192 tablet_bootstrap.cc:492] T 999fecb3fc65466cad21cc1ff5ef2d44 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:35.042716 1192 ts_tablet_manager.cc:1397] T 999fecb3fc65466cad21cc1ff5ef2d44 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.002s
I20250113 02:51:35.042922 1191 ts_tablet_manager.cc:1397] T 4eb5f1944cb44fd5a40ca92a6a3d7214 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.083s user 0.006s sys 0.006s
I20250113 02:51:35.044256 1192 raft_consensus.cc:357] T 999fecb3fc65466cad21cc1ff5ef2d44 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:35.044546 1192 raft_consensus.cc:383] T 999fecb3fc65466cad21cc1ff5ef2d44 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:35.044677 1192 raft_consensus.cc:738] T 999fecb3fc65466cad21cc1ff5ef2d44 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:35.045094 1192 consensus_queue.cc:260] T 999fecb3fc65466cad21cc1ff5ef2d44 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:35.043781 1191 raft_consensus.cc:357] T 4eb5f1944cb44fd5a40ca92a6a3d7214 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:35.045920 1191 raft_consensus.cc:383] T 4eb5f1944cb44fd5a40ca92a6a3d7214 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:35.046121 1191 raft_consensus.cc:738] T 4eb5f1944cb44fd5a40ca92a6a3d7214 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:35.046542 1191 consensus_queue.cc:260] T 4eb5f1944cb44fd5a40ca92a6a3d7214 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:35.047880 1192 ts_tablet_manager.cc:1428] T 999fecb3fc65466cad21cc1ff5ef2d44 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20250113 02:51:35.049224 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet 33c665e149dd48e8868acd935de581bf (table default.Table_ec91552e358547e99d908cef4bcb7c36 [id=0ebda43313f249d6b35f5beda3d5fc0e]) successfully deleted
I20250113 02:51:35.049839 1192 tablet_bootstrap.cc:492] T 1d15c7a01f854f0aaabcd30ec3166e31 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
W20250113 02:51:35.050690 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 9b2d0aa6f0e9456d95f4e48c360ab6e8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9b2d0aa6f0e9456d95f4e48c360ab6e8 already in progress: creating tablet
W20250113 02:51:35.051736 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet d2699bb529bf4f0f9d7f2668ffe4c9ff because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d2699bb529bf4f0f9d7f2668ffe4c9ff already in progress: creating tablet
W20250113 02:51:35.053014 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet ec06c9f6a3bf46a9bcb90a85a82b6ff8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ec06c9f6a3bf46a9bcb90a85a82b6ff8 already in progress: creating tablet
W20250113 02:51:35.054031 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 6e208f83f3384a78a959909b4ac690c0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6e208f83f3384a78a959909b4ac690c0 already in progress: creating tablet
W20250113 02:51:35.055223 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet db486978b33a45df90025988b084aecd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet db486978b33a45df90025988b084aecd already in progress: creating tablet
W20250113 02:51:35.056912 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 1c77033ca393469e9324d2406a65a5e9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1c77033ca393469e9324d2406a65a5e9 already in progress: creating tablet
I20250113 02:51:35.057476 1195 tablet_replica.cc:331] T e70658c90555463498d3fc2e2f13ea7f P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:35.058408 1195 raft_consensus.cc:2238] T e70658c90555463498d3fc2e2f13ea7f P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:35.059263 1195 raft_consensus.cc:2267] T e70658c90555463498d3fc2e2f13ea7f P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:35.060490 1192 tablet_bootstrap.cc:654] T 1d15c7a01f854f0aaabcd30ec3166e31 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:35.060909 1194 tablet_bootstrap.cc:492] T be4b2a8a3927410eaf12e7592646e4af P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:35.061087 1194 ts_tablet_manager.cc:1397] T be4b2a8a3927410eaf12e7592646e4af P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.133s user 0.006s sys 0.035s
I20250113 02:51:35.061740 1194 raft_consensus.cc:357] T be4b2a8a3927410eaf12e7592646e4af P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:35.061976 1194 raft_consensus.cc:383] T be4b2a8a3927410eaf12e7592646e4af P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:35.062057 1194 raft_consensus.cc:738] T be4b2a8a3927410eaf12e7592646e4af P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:35.062186 1195 ts_tablet_manager.cc:1905] T e70658c90555463498d3fc2e2f13ea7f P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:35.062301 1194 consensus_queue.cc:260] T be4b2a8a3927410eaf12e7592646e4af P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:35.062884 1194 ts_tablet_manager.cc:1428] T be4b2a8a3927410eaf12e7592646e4af P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250113 02:51:35.064522 1194 tablet_bootstrap.cc:492] T 7d3562a48c6e4badbbcf37640e91b5ba P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:35.065716 1194 tablet_bootstrap.cc:654] T 7d3562a48c6e4badbbcf37640e91b5ba P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:35.075942 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet d5bd09a1ede644c59e400e49c8125793
I20250113 02:51:35.076764 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_38d715a0346f4b3db579e626066b5d49"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:35.077312 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_38d715a0346f4b3db579e626066b5d49 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250113 02:51:35.078747 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 905542fb0dce4bbb8852b74d94c7b2a6 consensus state has no leader
I20250113 02:51:35.079144 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_f2c94bca6ad3435e9d28d82c98214064" } new_table_name: "default.Table_4c5e2e1008b54019a243ffe1ec72c020" modify_external_catalogs: true
I20250113 02:51:35.081811 890 tablet_service.cc:1514] Processing DeleteTablet for tablet d5bd09a1ede644c59e400e49c8125793 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:34 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:35.082083 1196 tablet_replica.cc:331] T d5bd09a1ede644c59e400e49c8125793 P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:35.082100 751 tablet_service.cc:1514] Processing DeleteTablet for tablet d5bd09a1ede644c59e400e49c8125793 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:34 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:35.082252 1196 raft_consensus.cc:2238] T d5bd09a1ede644c59e400e49c8125793 P df69f48656b44bb890f76587ee69cc0f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:35.082409 1196 raft_consensus.cc:2267] T d5bd09a1ede644c59e400e49c8125793 P df69f48656b44bb890f76587ee69cc0f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:35.082572 2072 raft_consensus.cc:491] T 7213e3284b4d4cb581d9fb946a9f87e9 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:35.082700 2072 raft_consensus.cc:513] T 7213e3284b4d4cb581d9fb946a9f87e9 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:35.082912 1196 ts_tablet_manager.cc:1905] T d5bd09a1ede644c59e400e49c8125793 P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:35.083346 2072 leader_election.cc:290] T 7213e3284b4d4cb581d9fb946a9f87e9 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:35.084471 1030 tablet_service.cc:1514] Processing DeleteTablet for tablet d5bd09a1ede644c59e400e49c8125793 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:34 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:35.084934 1062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7213e3284b4d4cb581d9fb946a9f87e9" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
W20250113 02:51:35.085851 709 leader_election.cc:343] T 7213e3284b4d4cb581d9fb946a9f87e9 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:35.087210 964 heartbeater.cc:346] Connected to a master server at 127.0.104.60:42067
I20250113 02:51:35.089212 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7213e3284b4d4cb581d9fb946a9f87e9" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:35.089478 1196 ts_tablet_manager.cc:1918] T d5bd09a1ede644c59e400e49c8125793 P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250113 02:51:35.089619 1196 log.cc:1198] T d5bd09a1ede644c59e400e49c8125793 P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/d5bd09a1ede644c59e400e49c8125793
W20250113 02:51:35.089735 711 leader_election.cc:343] T 7213e3284b4d4cb581d9fb946a9f87e9 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:35.089850 711 leader_election.cc:304] T 7213e3284b4d4cb581d9fb946a9f87e9 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:35.090133 1196 ts_tablet_manager.cc:1939] T d5bd09a1ede644c59e400e49c8125793 P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:35.090287 2072 raft_consensus.cc:2743] T 7213e3284b4d4cb581d9fb946a9f87e9 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:35.093109 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_4a8b3f444ec345ad8a0c424f291e6d77" } new_table_name: "default.Table_c9eb2fbeba3849d38743dd775d3f12c0" modify_external_catalogs: true
I20250113 02:51:35.093547 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_b030b1ebb66e4b1482e9a6d2efdb7c8a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:35.094038 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b030b1ebb66e4b1482e9a6d2efdb7c8a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:35.094625 448 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_6754736c569e4f55848afc146fe41b74" } modify_external_catalogs: true
I20250113 02:51:35.094791 448 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_6754736c569e4f55848afc146fe41b74" } modify_external_catalogs: true
I20250113 02:51:35.096741 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet d5bd09a1ede644c59e400e49c8125793 (table default.Table_bee5bcef1f534c6c812e6075291ff90d [id=e0ced6f3dc0049eaa0e76974916ff63c]) successfully deleted
I20250113 02:51:35.102377 1198 ts_tablet_manager.cc:1918] T 6d4d8b5cc8ad4c9e935d8afaf5766e66 P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:35.102507 1198 log.cc:1198] T 6d4d8b5cc8ad4c9e935d8afaf5766e66 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/6d4d8b5cc8ad4c9e935d8afaf5766e66
I20250113 02:51:35.102836 1198 ts_tablet_manager.cc:1939] T 6d4d8b5cc8ad4c9e935d8afaf5766e66 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:35.114698 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 925) took 1429 ms. Trace:
I20250113 02:51:35.114816 1198 rpcz_store.cc:276] 0113 02:51:33.685185 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:33.686187 (+ 1002us) service_pool.cc:225] Handling call
0113 02:51:35.005267 (+1319080us) ts_tablet_manager.cc:1145] Deleting tablet 6d4d8b5cc8ad4c9e935d8afaf5766e66
0113 02:51:35.005279 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:35.059441 (+ 54162us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:35.102291 (+ 42850us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:35.114673 (+ 12382us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":15744,"tablet-delete.queue_time_us":1318898}
I20250113 02:51:35.115139 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_14183012578a4660ad28c7c483de871d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:35.115676 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_14183012578a4660ad28c7c483de871d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:35.116811 1194 tablet_bootstrap.cc:492] T 7d3562a48c6e4badbbcf37640e91b5ba P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:35.117004 1194 ts_tablet_manager.cc:1397] T 7d3562a48c6e4badbbcf37640e91b5ba P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.053s user 0.001s sys 0.001s
I20250113 02:51:35.117344 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet 6d4d8b5cc8ad4c9e935d8afaf5766e66 (table default.Table_54119752ebc147b8b34429f1c38b71d9 [id=2ab34074f7624b419c1cd36e9a09b4c6]) successfully deleted
I20250113 02:51:35.117688 1194 raft_consensus.cc:357] T 7d3562a48c6e4badbbcf37640e91b5ba P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:35.117951 1194 raft_consensus.cc:383] T 7d3562a48c6e4badbbcf37640e91b5ba P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:35.118036 1194 raft_consensus.cc:738] T 7d3562a48c6e4badbbcf37640e91b5ba P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:35.118284 1194 consensus_queue.cc:260] T 7d3562a48c6e4badbbcf37640e91b5ba P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:35.118773 1194 ts_tablet_manager.cc:1428] T 7d3562a48c6e4badbbcf37640e91b5ba P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250113 02:51:35.121206 1198 tablet_replica.cc:331] T 05bf5ab0df874dd5b6559f6d82069184 P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:35.121455 1198 raft_consensus.cc:2238] T 05bf5ab0df874dd5b6559f6d82069184 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:35.121727 1198 raft_consensus.cc:2267] T 05bf5ab0df874dd5b6559f6d82069184 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:35.122330 1198 ts_tablet_manager.cc:1905] T 05bf5ab0df874dd5b6559f6d82069184 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250113 02:51:35.123303 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f1237f7b04af4ebbb39b03aa3c78a499 consensus state has no leader
I20250113 02:51:35.123656 447 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_3d65267002364c8a9175ccb2223cac44" } modify_external_catalogs: true
I20250113 02:51:35.123809 447 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_3d65267002364c8a9175ccb2223cac44" } modify_external_catalogs: true
I20250113 02:51:35.124619 455 master_service.cc:946] ReplaceTablet: received request to replace tablet f105f8b32a654e4ca154e8fc0cb96091 from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:35.130952 1191 ts_tablet_manager.cc:1428] T 4eb5f1944cb44fd5a40ca92a6a3d7214 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.088s user 0.015s sys 0.000s
I20250113 02:51:35.134186 1198 ts_tablet_manager.cc:1918] T 05bf5ab0df874dd5b6559f6d82069184 P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:35.137157 1194 tablet_bootstrap.cc:492] T 23e04961f072440fa3e3f9485e40f79c P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:35.138232 820 heartbeater.cc:346] Connected to a master server at 127.0.104.60:42067
I20250113 02:51:35.139051 1194 tablet_bootstrap.cc:654] T 23e04961f072440fa3e3f9485e40f79c P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:35.137527 1191 tablet_bootstrap.cc:492] T 9438472464304da9a0c7658d4f0736b6 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:35.140055 451 master_service.cc:946] ReplaceTablet: received request to replace tablet 1c77033ca393469e9324d2406a65a5e9 from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:35.141086 1191 tablet_bootstrap.cc:654] T 9438472464304da9a0c7658d4f0736b6 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:35.141709 1198 log.cc:1198] T 05bf5ab0df874dd5b6559f6d82069184 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/05bf5ab0df874dd5b6559f6d82069184
I20250113 02:51:35.142119 1198 ts_tablet_manager.cc:1939] T 05bf5ab0df874dd5b6559f6d82069184 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:35.144722 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 926) took 1459 ms. Trace:
I20250113 02:51:35.144865 1198 rpcz_store.cc:276] 0113 02:51:33.685274 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:33.686759 (+ 1485us) service_pool.cc:225] Handling call
0113 02:51:35.121104 (+1434345us) ts_tablet_manager.cc:1145] Deleting tablet 05bf5ab0df874dd5b6559f6d82069184
0113 02:51:35.121114 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:35.128675 (+ 7561us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:35.134069 (+ 5394us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:35.144702 (+ 10633us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1434152}
I20250113 02:51:35.145395 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet 05bf5ab0df874dd5b6559f6d82069184 (table default.Table_5175e117e3f34778a5ef456bed6b5e28 [id=5c9d31a6ab2445ad949f45abffdf4cc7]) successfully deleted
I20250113 02:51:35.145853 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 930) took 1356 ms. Trace:
I20250113 02:51:35.145990 1198 rpcz_store.cc:276] 0113 02:51:33.789801 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:33.793039 (+ 3238us) service_pool.cc:225] Handling call
0113 02:51:35.145733 (+1352694us) ts_tablet_manager.cc:1145] Deleting tablet 21bcdd741104474bad7023426856d18f
0113 02:51:35.145744 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:35.145837 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1346918}
W20250113 02:51:35.146518 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 21bcdd741104474bad7023426856d18f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 21bcdd741104474bad7023426856d18f already in progress: creating tablet
I20250113 02:51:35.147089 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 931) took 1338 ms. Trace:
I20250113 02:51:35.150177 1198 rpcz_store.cc:276] 0113 02:51:33.808660 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:33.808772 (+ 112us) service_pool.cc:225] Handling call
0113 02:51:35.146945 (+1338173us) ts_tablet_manager.cc:1145] Deleting tablet 54aae7c131404facba9af3a3d143b4a4
0113 02:51:35.146955 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:35.147074 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1337931}
I20250113 02:51:35.150683 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 932) took 1278 ms. Trace:
W20250113 02:51:35.150880 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 54aae7c131404facba9af3a3d143b4a4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 54aae7c131404facba9af3a3d143b4a4 already in progress: creating tablet
I20250113 02:51:35.150851 1198 rpcz_store.cc:276] 0113 02:51:33.871886 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:33.871940 (+ 54us) service_pool.cc:225] Handling call
0113 02:51:35.150580 (+1278640us) ts_tablet_manager.cc:1145] Deleting tablet d52b9d0f60714a31918c1645178f3525
0113 02:51:35.150589 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:35.150669 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1278414}
W20250113 02:51:35.151466 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet d52b9d0f60714a31918c1645178f3525 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d52b9d0f60714a31918c1645178f3525 already in progress: creating tablet
I20250113 02:51:35.151841 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 933) took 1221 ms. Trace:
I20250113 02:51:35.151998 1198 rpcz_store.cc:276] 0113 02:51:33.930254 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:33.930305 (+ 51us) service_pool.cc:225] Handling call
0113 02:51:35.151744 (+1221439us) ts_tablet_manager.cc:1145] Deleting tablet 02bf7042d4ce4077851a44bbfb6c4b46
0113 02:51:35.151753 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:35.151826 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1204818}
W20250113 02:51:35.152455 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 02bf7042d4ce4077851a44bbfb6c4b46 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 02bf7042d4ce4077851a44bbfb6c4b46 already in progress: creating tablet
I20250113 02:51:35.152770 1198 tablet_replica.cc:331] T 99c8046f5d124ca49899df365114c69f P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:35.153076 1198 raft_consensus.cc:2238] T 99c8046f5d124ca49899df365114c69f P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:35.153282 1198 raft_consensus.cc:2267] T 99c8046f5d124ca49899df365114c69f P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:35.153606 1191 tablet_bootstrap.cc:492] T 9438472464304da9a0c7658d4f0736b6 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:35.153741 1191 ts_tablet_manager.cc:1397] T 9438472464304da9a0c7658d4f0736b6 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.000s
I20250113 02:51:35.153993 1198 ts_tablet_manager.cc:1905] T 99c8046f5d124ca49899df365114c69f P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:35.154373 1191 raft_consensus.cc:357] T 9438472464304da9a0c7658d4f0736b6 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:35.154562 1191 raft_consensus.cc:383] T 9438472464304da9a0c7658d4f0736b6 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:35.154625 1191 raft_consensus.cc:738] T 9438472464304da9a0c7658d4f0736b6 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:35.154868 1191 consensus_queue.cc:260] T 9438472464304da9a0c7658d4f0736b6 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:35.155319 1191 ts_tablet_manager.cc:1428] T 9438472464304da9a0c7658d4f0736b6 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250113 02:51:35.155802 1191 tablet_bootstrap.cc:492] T d8bb60fdd06946739b75b255303d9d24 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:35.157105 1191 tablet_bootstrap.cc:654] T d8bb60fdd06946739b75b255303d9d24 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:35.166263 2072 raft_consensus.cc:491] T 0b119704bc774ca9b4155c9619d4765b P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:35.166397 2072 raft_consensus.cc:513] T 0b119704bc774ca9b4155c9619d4765b P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:35.166975 2072 leader_election.cc:290] T 0b119704bc774ca9b4155c9619d4765b P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:35.167654 1191 tablet_bootstrap.cc:492] T d8bb60fdd06946739b75b255303d9d24 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:35.167621 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b119704bc774ca9b4155c9619d4765b" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:35.167779 1191 ts_tablet_manager.cc:1397] T d8bb60fdd06946739b75b255303d9d24 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.003s
I20250113 02:51:35.168001 1198 ts_tablet_manager.cc:1918] T 99c8046f5d124ca49899df365114c69f P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:35.168150 1198 log.cc:1198] T 99c8046f5d124ca49899df365114c69f P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/99c8046f5d124ca49899df365114c69f
W20250113 02:51:35.168267 711 leader_election.cc:343] T 0b119704bc774ca9b4155c9619d4765b P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:35.168387 1191 raft_consensus.cc:357] T d8bb60fdd06946739b75b255303d9d24 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:35.168550 1191 raft_consensus.cc:383] T d8bb60fdd06946739b75b255303d9d24 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:35.168596 1191 raft_consensus.cc:738] T d8bb60fdd06946739b75b255303d9d24 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:35.168880 1062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b119704bc774ca9b4155c9619d4765b" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:35.168970 1198 ts_tablet_manager.cc:1939] T 99c8046f5d124ca49899df365114c69f P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
W20250113 02:51:35.169368 709 leader_election.cc:343] T 0b119704bc774ca9b4155c9619d4765b P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:35.169467 709 leader_election.cc:304] T 0b119704bc774ca9b4155c9619d4765b P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:35.169840 2072 raft_consensus.cc:2743] T 0b119704bc774ca9b4155c9619d4765b P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:35.170512 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 934) took 1233 ms. Trace:
I20250113 02:51:35.175283 1198 rpcz_store.cc:276] 0113 02:51:33.936717 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:33.937217 (+ 500us) service_pool.cc:225] Handling call
0113 02:51:35.152725 (+1215508us) ts_tablet_manager.cc:1145] Deleting tablet 99c8046f5d124ca49899df365114c69f
0113 02:51:35.152736 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:35.165034 (+ 12298us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:35.167884 (+ 2850us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:35.170492 (+ 2608us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1205736}
I20250113 02:51:35.175468 452 catalog_manager.cc:5618] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095) does not have the latest schema for tablet 1a63f485caf54068abd6b50d7de3d14a (table default.Table_f8a6812c7fb94ea69cd9bf775b914ea6 [id=71508871e2d5469a91aaf0e19b02b638]). Expected version 1 got 0
I20250113 02:51:35.175774 452 catalog_manager.cc:5618] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095) does not have the latest schema for tablet 6e1284c8aaba44b5810b0d06d36757eb (table default.Table_bfc5031f68f64006aceefe2ee986223a [id=aa4679fc0d7148e3bef21342b3a0ea22]). Expected version 2 got 0
I20250113 02:51:35.175895 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet 99c8046f5d124ca49899df365114c69f (table default.Table_7805d72c367b4ef8aa585de05f3d9b90 [id=d4ce999a9a7b47ada946f259228771bb]) successfully deleted
I20250113 02:51:35.176002 452 catalog_manager.cc:5618] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095) does not have the latest schema for tablet 0b119704bc774ca9b4155c9619d4765b (table default.Table_b95a9b1d12284c9aac701a8ef9c93482 [id=843aabcff7074acd98e5801b21282faa]). Expected version 1 got 0
I20250113 02:51:35.176123 1191 consensus_queue.cc:260] T d8bb60fdd06946739b75b255303d9d24 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:35.176330 1198 tablet_replica.cc:331] T 0539cbbe47e64c039ad39c4b52edcc4b P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:35.176519 450 catalog_manager.cc:6071] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: default.Table_3d65267002364c8a9175ccb2223cac44 [id=8aff13d6d3c5483eadb596b705fe5151] alter complete (version 6)
I20250113 02:51:35.176762 1191 ts_tablet_manager.cc:1428] T d8bb60fdd06946739b75b255303d9d24 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20250113 02:51:35.177060 1191 tablet_bootstrap.cc:492] T f21eb3fa0f414816a29e3d4560cdc0fe P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:35.177556 890 tablet_service.cc:1514] Processing DeleteTablet for tablet 9fff52e99c64462a8a73fccc3c22ca63 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:33 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:35.177820 1196 tablet_replica.cc:331] T 9fff52e99c64462a8a73fccc3c22ca63 P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:35.178004 1196 raft_consensus.cc:2238] T 9fff52e99c64462a8a73fccc3c22ca63 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:35.178138 1196 raft_consensus.cc:2267] T 9fff52e99c64462a8a73fccc3c22ca63 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:35.178803 1196 ts_tablet_manager.cc:1905] T 9fff52e99c64462a8a73fccc3c22ca63 P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:35.180668 1191 tablet_bootstrap.cc:654] T f21eb3fa0f414816a29e3d4560cdc0fe P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:35.181425 2147 raft_consensus.cc:491] T 1cfbcd682b944c89847b40fe69f0a75e P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:35.181530 2147 raft_consensus.cc:513] T 1cfbcd682b944c89847b40fe69f0a75e P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:35.182132 2147 leader_election.cc:290] T 1cfbcd682b944c89847b40fe69f0a75e P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587), 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
I20250113 02:51:35.182600 450 catalog_manager.cc:5618] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467) does not have the latest schema for tablet be4b2a8a3927410eaf12e7592646e4af (table default.Table_24d9af3d0b0740d69c791fc96916ef0a [id=124669ccadaf49e19d2660b1495f9647]). Expected version 3 got 0
I20250113 02:51:35.184118 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cfbcd682b944c89847b40fe69f0a75e" candidate_uuid: "df69f48656b44bb890f76587ee69cc0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
I20250113 02:51:35.186468 891 tablet_service.cc:1514] Processing DeleteTablet for tablet de4a58bf63054890808e2e2fe46e43d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:20 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:35.186640 885 tablet_service.cc:1514] Processing DeleteTablet for tablet 092892a36e0e47cb8d2184c2bbde95d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:22 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:35.187211 890 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d3562a48c6e4badbbcf37640e91b5ba with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:21 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:35.193126 1196 ts_tablet_manager.cc:1918] T 9fff52e99c64462a8a73fccc3c22ca63 P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:35.193292 1196 log.cc:1198] T 9fff52e99c64462a8a73fccc3c22ca63 P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/9fff52e99c64462a8a73fccc3c22ca63
I20250113 02:51:35.194064 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 32cba8f3efc54e78b4df41c15df82eea
I20250113 02:51:35.194326 1196 ts_tablet_manager.cc:1939] T 9fff52e99c64462a8a73fccc3c22ca63 P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:35.195392 751 tablet_service.cc:1514] Processing DeleteTablet for tablet 32cba8f3efc54e78b4df41c15df82eea with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:35 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:35.180716 2113 raft_consensus.cc:491] T 4b07a985897641c3ac48021c2a4daedb P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:35.185297 1062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cfbcd682b944c89847b40fe69f0a75e" candidate_uuid: "df69f48656b44bb890f76587ee69cc0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:35.196220 2113 raft_consensus.cc:513] T 4b07a985897641c3ac48021c2a4daedb P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:35.196921 2113 leader_election.cc:290] T 4b07a985897641c3ac48021c2a4daedb P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:35.185935 1030 tablet_service.cc:1514] Processing DeleteTablet for tablet 42cf1e708644427f9d4dcd41ae2bcb37 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:22 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:35.197050 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet 9fff52e99c64462a8a73fccc3c22ca63 (table default.Table_9ef18ee71ad843c5921750a2f8478855 [id=bd8bd572ed524f2cae01f820f07e3c36]) successfully deleted
I20250113 02:51:35.197448 890 tablet_service.cc:1514] Processing DeleteTablet for tablet 32cba8f3efc54e78b4df41c15df82eea with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:35 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:35.198124 852 leader_election.cc:304] T 1cfbcd682b944c89847b40fe69f0a75e P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df69f48656b44bb890f76587ee69cc0f; no voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca
I20250113 02:51:35.198372 2147 raft_consensus.cc:2743] T 1cfbcd682b944c89847b40fe69f0a75e P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:35.198486 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_022b76ef1f0a4679b59c414d4c77b135"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:35.199018 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_022b76ef1f0a4679b59c414d4c77b135 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:35.199143 1030 tablet_service.cc:1514] Processing DeleteTablet for tablet 32cba8f3efc54e78b4df41c15df82eea with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:35 UTC) from {username='slave'} at 127.0.0.1:60230
W20250113 02:51:35.200085 995 leader_election.cc:343] T 4b07a985897641c3ac48021c2a4daedb P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Not found: Tablet not found: 4b07a985897641c3ac48021c2a4daedb
I20250113 02:51:35.200292 1194 tablet_bootstrap.cc:492] T 23e04961f072440fa3e3f9485e40f79c P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:35.200327 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b07a985897641c3ac48021c2a4daedb" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:35.200945 998 leader_election.cc:343] T 4b07a985897641c3ac48021c2a4daedb P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:35.201048 998 leader_election.cc:304] T 4b07a985897641c3ac48021c2a4daedb P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4216a7f853d642b7bc3e5cff458f3aca; no voters: 1186fa869eb14812832fa6562531f196, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:35.201436 2113 raft_consensus.cc:2743] T 4b07a985897641c3ac48021c2a4daedb P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:35.201409 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_18b709d327d2407ea3909a1ec7091d92"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:35.201921 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_18b709d327d2407ea3909a1ec7091d92 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:35.202682 1196 tablet_replica.cc:331] T de4a58bf63054890808e2e2fe46e43d2 P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:35.207028 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_9c662bce442d44bd86f62065a4325cf9" } new_table_name: "default.Table_2f511897104f42e3a51b770d16d7073d" modify_external_catalogs: true
I20250113 02:51:35.207572 449 catalog_manager.cc:5618] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587) does not have the latest schema for tablet 6e1284c8aaba44b5810b0d06d36757eb (table default.Table_bfc5031f68f64006aceefe2ee986223a [id=aa4679fc0d7148e3bef21342b3a0ea22]). Expected version 2 got 0
I20250113 02:51:35.208987 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_294d733d092b4d9ca0ddd79b52024853" } new_table_name: "default.Table_ce5d67ac15bc4d89a173d58c766a259c" modify_external_catalogs: true
I20250113 02:51:35.173965 1192 tablet_bootstrap.cc:492] T 1d15c7a01f854f0aaabcd30ec3166e31 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:35.209478 1192 ts_tablet_manager.cc:1397] T 1d15c7a01f854f0aaabcd30ec3166e31 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.161s user 0.005s sys 0.004s
I20250113 02:51:35.208899 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_ac79159d146b433ebdea04d6746fb7d4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:35.211490 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ac79159d146b433ebdea04d6746fb7d4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:35.210994 1192 raft_consensus.cc:357] T 1d15c7a01f854f0aaabcd30ec3166e31 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:35.214699 1192 raft_consensus.cc:383] T 1d15c7a01f854f0aaabcd30ec3166e31 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:35.215230 1192 raft_consensus.cc:738] T 1d15c7a01f854f0aaabcd30ec3166e31 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:35.219280 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet 999fecb3fc65466cad21cc1ff5ef2d44 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:23 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:35.219465 1030 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b07a985897641c3ac48021c2a4daedb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:30 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:35.220636 450 catalog_manager.cc:5618] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095) does not have the latest schema for tablet 9438472464304da9a0c7658d4f0736b6 (table default.Table_acdc5be1660742d7a356474b86e8b35f [id=aaf6558a15124f2b8eb9f01d692b845d]). Expected version 2 got 0
I20250113 02:51:35.222066 751 tablet_service.cc:1514] Processing DeleteTablet for tablet d8bb60fdd06946739b75b255303d9d24 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:26 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:35.223067 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_93aa0e80539449feacb8085adce4b4ea"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:35.223739 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_93aa0e80539449feacb8085adce4b4ea in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:35.219488 1192 consensus_queue.cc:260] T 1d15c7a01f854f0aaabcd30ec3166e31 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:35.226662 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 2c49f1eefd2041b4ae617a4cb35d6f15
I20250113 02:51:35.227707 751 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c49f1eefd2041b4ae617a4cb35d6f15 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:34 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:35.229693 1192 ts_tablet_manager.cc:1428] T 1d15c7a01f854f0aaabcd30ec3166e31 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.020s user 0.001s sys 0.001s
W20250113 02:51:35.231871 995 consensus_peers.cc:487] T f56230a2c75d4763a54671c79358f4d9 P 4216a7f853d642b7bc3e5cff458f3aca -> Peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Couldn't send request to peer 1186fa869eb14812832fa6562531f196. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f56230a2c75d4763a54671c79358f4d9. This is attempt 1: this message will repeat every 5th retry.
I20250113 02:51:35.237303 1030 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c49f1eefd2041b4ae617a4cb35d6f15 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:34 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:35.237766 1192 tablet_bootstrap.cc:492] T 0916b41c171a419791797559fbc2807d P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:35.239712 1195 ts_tablet_manager.cc:1918] T e70658c90555463498d3fc2e2f13ea7f P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:35.241223 751 tablet_service.cc:1467] Processing CreateTablet for tablet 71befed8b5c74b41b199919b82325891 (DEFAULT_TABLE table=default.Table_5bc5d0b055904e50b74e0981675f4abb [id=80cd7e75d4e041b593c9307e47b532f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:35.241587 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 71befed8b5c74b41b199919b82325891. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:35.247432 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 7fb282707196444690352dea3eb1387f
I20250113 02:51:35.248203 742 tablet_service.cc:1514] Processing DeleteTablet for tablet 7fb282707196444690352dea3eb1387f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:35 UTC) from {username='slave'} at 127.0.0.1:55398
W20250113 02:51:35.248858 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 51e44b812fb84b8cb9e5ebe98fb187e4 has no consensus state
I20250113 02:51:35.249545 2072 raft_consensus.cc:491] T 7527e6e0b65e4a43b7f48899a34cdf7f P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:35.249672 2072 raft_consensus.cc:513] T 7527e6e0b65e4a43b7f48899a34cdf7f P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:35.250334 2072 leader_election.cc:290] T 7527e6e0b65e4a43b7f48899a34cdf7f P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:35.251166 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet 7fb282707196444690352dea3eb1387f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:35 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:35.251379 1030 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d15c7a01f854f0aaabcd30ec3166e31 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:31 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:35.251487 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7527e6e0b65e4a43b7f48899a34cdf7f" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
W20250113 02:51:35.252280 709 leader_election.cc:343] T 7527e6e0b65e4a43b7f48899a34cdf7f P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
W20250113 02:51:35.260335 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fe8f075d4c9a4f9abe372bfdb81a829c consensus state has no leader
I20250113 02:51:35.268299 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet f105f8b32a654e4ca154e8fc0cb96091
I20250113 02:51:35.268908 455 catalog_manager.cc:6596] ReplaceTablet: tablet f105f8b32a654e4ca154e8fc0cb96091 deleted and replaced by tablet 65e09181ee4c4061a7b5a5353be440cf
I20250113 02:51:35.269788 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet f9269967dea94fc3a337546ad0842e11
I20250113 02:51:35.272897 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet f9269967dea94fc3a337546ad0842e11 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:34 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:35.273269 1030 tablet_service.cc:1514] Processing DeleteTablet for tablet f105f8b32a654e4ca154e8fc0cb96091 with delete_type TABLET_DATA_DELETED (replaced by tablet 65e09181ee4c4061a7b5a5353be440cf) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:35.277284 2010 raft_consensus.cc:491] T e468b652f71f44e3945f8d919ac0e1a8 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:35.277438 2010 raft_consensus.cc:513] T e468b652f71f44e3945f8d919ac0e1a8 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:35.278190 2010 leader_election.cc:290] T e468b652f71f44e3945f8d919ac0e1a8 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
W20250113 02:51:35.284725 439 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet fe8f075d4c9a4f9abe372bfdb81a829c (table default.Table_ce5d67ac15bc4d89a173d58c766a259c [id=d37981b613bd494685de3e48d382f485], current schema version=3) failed: Not found: failed to reset TS proxy: No leader found: Tablet fe8f075d4c9a4f9abe372bfdb81a829c consensus state has no leader
I20250113 02:51:35.285126 455 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_aa7e1ca42f92492b836ab88a775ba54e" } modify_external_catalogs: true
I20250113 02:51:35.285311 455 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_aa7e1ca42f92492b836ab88a775ba54e" } modify_external_catalogs: true
I20250113 02:51:35.287204 446 catalog_manager.cc:5618] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587) does not have the latest schema for tablet 6e1284c8aaba44b5810b0d06d36757eb (table default.Table_bfc5031f68f64006aceefe2ee986223a [id=aa4679fc0d7148e3bef21342b3a0ea22]). Expected version 2 got 0
I20250113 02:51:35.287935 446 catalog_manager.cc:5618] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587) does not have the latest schema for tablet 1a63f485caf54068abd6b50d7de3d14a (table default.Table_f8a6812c7fb94ea69cd9bf775b914ea6 [id=71508871e2d5469a91aaf0e19b02b638]). Expected version 1 got 0
I20250113 02:51:35.288702 446 catalog_manager.cc:5618] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587) does not have the latest schema for tablet 1cfbcd682b944c89847b40fe69f0a75e (table default.Table_32e88abaae96492fbcce39fb2cd7c221 [id=2b84deb07e5c4ab8b550fe42945d7b71]). Expected version 2 got 0
I20250113 02:51:35.294553 2141 tablet.cc:1721] T 41bcb2eafff14fa8a040982c723f99c1 P 1186fa869eb14812832fa6562531f196: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250113 02:51:35.295290 2191 raft_consensus.cc:491] T d8bb60fdd06946739b75b255303d9d24 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:35.295472 2191 raft_consensus.cc:513] T d8bb60fdd06946739b75b255303d9d24 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:35.295744 742 tablet_service.cc:1514] Processing DeleteTablet for tablet f9269967dea94fc3a337546ad0842e11 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:34 UTC) from {username='slave'} at 127.0.0.1:55398
W20250113 02:51:35.296166 995 leader_election.cc:343] T e468b652f71f44e3945f8d919ac0e1a8 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Not found: Tablet not found: e468b652f71f44e3945f8d919ac0e1a8
I20250113 02:51:35.299335 2140 tablet.cc:1721] T 1a63f485caf54068abd6b50d7de3d14a P 4216a7f853d642b7bc3e5cff458f3aca: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250113 02:51:35.299716 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_1cfa6c2edbe9497ba84ffc25ce7b1573"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:35.300470 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1cfa6c2edbe9497ba84ffc25ce7b1573 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:35.300871 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_fa2b5d3af3ce4b3fae8bbaa614a7d6c0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:35.301362 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fa2b5d3af3ce4b3fae8bbaa614a7d6c0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:35.302119 2141 tablet.cc:1721] T 1a63f485caf54068abd6b50d7de3d14a P 1186fa869eb14812832fa6562531f196: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250113 02:51:35.302749 751 tablet_service.cc:1514] Processing DeleteTablet for tablet f105f8b32a654e4ca154e8fc0cb96091 with delete_type TABLET_DATA_DELETED (replaced by tablet 65e09181ee4c4061a7b5a5353be440cf) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:35.306280 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_05b6401543ad445d8ba261889e75c3f8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:35.306856 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_05b6401543ad445d8ba261889e75c3f8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:35.308714 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_0a1e1addf0f745e9801f12d1b02b0f0e" } new_table_name: "default.Table_2b653446c62d4fdaae71cfff4b0e1391" modify_external_catalogs: true
I20250113 02:51:35.311957 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet f24cda32b6e947c7983ae2f1f9a64c23
I20250113 02:51:35.314783 751 tablet_service.cc:1514] Processing DeleteTablet for tablet f24cda32b6e947c7983ae2f1f9a64c23 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:35 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:35.315016 1030 tablet_service.cc:1514] Processing DeleteTablet for tablet f24cda32b6e947c7983ae2f1f9a64c23 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:35 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:35.240479 1195 log.cc:1198] T e70658c90555463498d3fc2e2f13ea7f P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/e70658c90555463498d3fc2e2f13ea7f
I20250113 02:51:35.315685 1195 ts_tablet_manager.cc:1939] T e70658c90555463498d3fc2e2f13ea7f P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:35.318486 452 master_service.cc:946] ReplaceTablet: received request to replace tablet fb2458e2044d49029ed2d6851db9de25 from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:35.318526 1062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8bb60fdd06946739b75b255303d9d24" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:35.318604 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_2ebcc2867e6e41d1aaf0f53f9582a11f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:35.319000 709 leader_election.cc:343] T d8bb60fdd06946739b75b255303d9d24 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
W20250113 02:51:35.319106 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2ebcc2867e6e41d1aaf0f53f9582a11f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:35.319271 2191 leader_election.cc:290] T d8bb60fdd06946739b75b255303d9d24 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:35.321724 1030 tablet_service.cc:1467] Processing CreateTablet for tablet 71befed8b5c74b41b199919b82325891 (DEFAULT_TABLE table=default.Table_5bc5d0b055904e50b74e0981675f4abb [id=80cd7e75d4e041b593c9307e47b532f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:35.322115 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 71befed8b5c74b41b199919b82325891. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:35.325059 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet e70658c90555463498d3fc2e2f13ea7f (table default.Table_2565c0551f9943a084ace8f7e463e59d [id=fe6f8d302c3e489d8bb832503fd05642]) successfully deleted
I20250113 02:51:35.326015 742 tablet_service.cc:1467] Processing CreateTablet for tablet 4a654e8085fd49729aef1d343ceaf789 (DEFAULT_TABLE table=default.Table_b7c9a4329c0a4aac98cdb053b3921b63 [id=987fbcc3ae054bb083ece4d4364a89f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:35.326390 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a654e8085fd49729aef1d343ceaf789. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:35.334053 454 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_21eff11db1644c02bfa6ceb7d46e7eac" } new_table_name: "default.Table_220329606556483ba605d7e7cdf05e1b" modify_external_catalogs: true
I20250113 02:51:35.334379 751 tablet_service.cc:1467] Processing CreateTablet for tablet e0cdc5e4f4ca4f30b634e4dd2bf862d9 (DEFAULT_TABLE table=default.Table_b3f0fce26a8d48f8a7660c9edba52f3d [id=56a7ee1dc17440c0ae8e77113c7cbf4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:35.334749 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e0cdc5e4f4ca4f30b634e4dd2bf862d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:35.335554 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_6b6db917d3974c9eab95631eb01b7a6e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:35.336073 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6b6db917d3974c9eab95631eb01b7a6e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:35.341573 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_771ecccb82dd4b5ba2f748c98f40eeff" } new_table_name: "default.Table_0fea98afc97f400a8fee7d8258d7b37f" modify_external_catalogs: true
I20250113 02:51:35.342854 447 catalog_manager.cc:6071] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: default.Table_c9eb2fbeba3849d38743dd775d3f12c0 [id=2f9412fc42494fa68bee8ae23550beec] alter complete (version 1)
I20250113 02:51:35.345222 1195 tablet_replica.cc:331] T eebad045e7bb43fa9ca9c9af8b3b7ea5 P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:35.345429 1195 raft_consensus.cc:2238] T eebad045e7bb43fa9ca9c9af8b3b7ea5 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:35.345587 1195 raft_consensus.cc:2267] T eebad045e7bb43fa9ca9c9af8b3b7ea5 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:35.346081 1195 ts_tablet_manager.cc:1905] T eebad045e7bb43fa9ca9c9af8b3b7ea5 P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250113 02:51:35.346072 421 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.104.2:38467, user_credentials={real_user=slave}} blocked reactor thread for 79194us
W20250113 02:51:35.347383 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e7d6b565f4cc4d1884343de41b380d99 consensus state has no leader
W20250113 02:51:35.350730 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 4a3ba2cd77b8416db72d894b09cc1100 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4a3ba2cd77b8416db72d894b09cc1100 already in progress: creating tablet
I20250113 02:51:35.355580 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_7c44942328c24dc9818e392f0b40a975"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:35.355762 1030 tablet_service.cc:1467] Processing CreateTablet for tablet e0cdc5e4f4ca4f30b634e4dd2bf862d9 (DEFAULT_TABLE table=default.Table_b3f0fce26a8d48f8a7660c9edba52f3d [id=56a7ee1dc17440c0ae8e77113c7cbf4e]), partition=RANGE (key) PARTITION UNBOUNDED
W20250113 02:51:35.356323 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7c44942328c24dc9818e392f0b40a975 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:35.356740 1035 tablet_service.cc:1467] Processing CreateTablet for tablet 4a654e8085fd49729aef1d343ceaf789 (DEFAULT_TABLE table=default.Table_b7c9a4329c0a4aac98cdb053b3921b63 [id=987fbcc3ae054bb083ece4d4364a89f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:35.356827 1023 tablet_service.cc:1467] Processing CreateTablet for tablet 01371679919c4daaad9451a7279898e5 (DEFAULT_TABLE table=default.Table_e6725383626f4287a24babc4e253e7b1 [id=40a9506c1722415087b299aafcc49e45]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:35.357208 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e0cdc5e4f4ca4f30b634e4dd2bf862d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:35.357542 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 01371679919c4daaad9451a7279898e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:35.357586 742 tablet_service.cc:1467] Processing CreateTablet for tablet 01371679919c4daaad9451a7279898e5 (DEFAULT_TABLE table=default.Table_e6725383626f4287a24babc4e253e7b1 [id=40a9506c1722415087b299aafcc49e45]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:35.357915 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 01371679919c4daaad9451a7279898e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:35.360080 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a654e8085fd49729aef1d343ceaf789. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:35.374177 2190 raft_consensus.cc:491] T 9438472464304da9a0c7658d4f0736b6 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:35.374403 2190 raft_consensus.cc:513] T 9438472464304da9a0c7658d4f0736b6 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:35.375681 2190 leader_election.cc:290] T 9438472464304da9a0c7658d4f0736b6 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:35.376118 1062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9438472464304da9a0c7658d4f0736b6" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
W20250113 02:51:35.376770 709 leader_election.cc:343] T 9438472464304da9a0c7658d4f0736b6 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:35.377025 1198 raft_consensus.cc:2238] T 0539cbbe47e64c039ad39c4b52edcc4b P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:35.200425 1194 ts_tablet_manager.cc:1397] T 23e04961f072440fa3e3f9485e40f79c P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.063s user 0.001s sys 0.019s
I20250113 02:51:35.377921 1194 raft_consensus.cc:357] T 23e04961f072440fa3e3f9485e40f79c P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:35.378103 1194 raft_consensus.cc:383] T 23e04961f072440fa3e3f9485e40f79c P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:35.378154 1194 raft_consensus.cc:738] T 23e04961f072440fa3e3f9485e40f79c P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:35.378338 1194 consensus_queue.cc:260] T 23e04961f072440fa3e3f9485e40f79c P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:35.378722 1194 ts_tablet_manager.cc:1428] T 23e04961f072440fa3e3f9485e40f79c P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250113 02:51:35.379035 1194 tablet_bootstrap.cc:492] T bffec36d567544c5a1c87e64a7e9730b P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:35.227314 890 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c49f1eefd2041b4ae617a4cb35d6f15 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:34 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:35.239724 885 tablet_service.cc:1467] Processing CreateTablet for tablet 71befed8b5c74b41b199919b82325891 (DEFAULT_TABLE table=default.Table_5bc5d0b055904e50b74e0981675f4abb [id=80cd7e75d4e041b593c9307e47b532f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:35.248221 891 tablet_service.cc:1514] Processing DeleteTablet for tablet 7fb282707196444690352dea3eb1387f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:35 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:35.251802 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7527e6e0b65e4a43b7f48899a34cdf7f" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:35.271148 889 tablet_service.cc:1514] Processing DeleteTablet for tablet f105f8b32a654e4ca154e8fc0cb96091 with delete_type TABLET_DATA_DELETED (replaced by tablet 65e09181ee4c4061a7b5a5353be440cf) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:35.271283 894 tablet_service.cc:1514] Processing DeleteTablet for tablet f9269967dea94fc3a337546ad0842e11 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:34 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:35.279012 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e468b652f71f44e3945f8d919ac0e1a8" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:35.291340 917 raft_consensus.cc:3054] T be4b2a8a3927410eaf12e7592646e4af P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:35.381127 1194 tablet_bootstrap.cc:654] T bffec36d567544c5a1c87e64a7e9730b P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:35.383776 1198 raft_consensus.cc:2267] T 0539cbbe47e64c039ad39c4b52edcc4b P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:35.313220 887 tablet_service.cc:1514] Processing DeleteTablet for tablet f24cda32b6e947c7983ae2f1f9a64c23 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:35 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:35.321260 892 tablet_service.cc:1467] Processing CreateTablet for tablet e0cdc5e4f4ca4f30b634e4dd2bf862d9 (DEFAULT_TABLE table=default.Table_b3f0fce26a8d48f8a7660c9edba52f3d [id=56a7ee1dc17440c0ae8e77113c7cbf4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:35.322907 884 tablet_service.cc:1467] Processing CreateTablet for tablet 4a654e8085fd49729aef1d343ceaf789 (DEFAULT_TABLE table=default.Table_b7c9a4329c0a4aac98cdb053b3921b63 [id=987fbcc3ae054bb083ece4d4364a89f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:35.355571 882 tablet_service.cc:1467] Processing CreateTablet for tablet 01371679919c4daaad9451a7279898e5 (DEFAULT_TABLE table=default.Table_e6725383626f4287a24babc4e253e7b1 [id=40a9506c1722415087b299aafcc49e45]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:35.356259 881 tablet_service.cc:1467] Processing CreateTablet for tablet 8ce65d004b5c448b99ef6502fb600c0c (DEFAULT_TABLE table=default.Table_fef715653eb349d19cf7f16198cd7669 [id=81c218b721d14e78a31347b8c32a9cf3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:35.384862 1196 raft_consensus.cc:2238] T de4a58bf63054890808e2e2fe46e43d2 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:35.385056 1196 raft_consensus.cc:2267] T de4a58bf63054890808e2e2fe46e43d2 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:35.385708 1198 ts_tablet_manager.cc:1905] T 0539cbbe47e64c039ad39c4b52edcc4b P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:35.386291 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 71befed8b5c74b41b199919b82325891. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:35.396530 711 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.2:38467, user_credentials={real_user=slave}} blocked reactor thread for 140081us
W20250113 02:51:35.396729 454 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5f5315047bd346b99e51909438f9e60f consensus state has no leader
I20250113 02:51:35.397193 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet fb2458e2044d49029ed2d6851db9de25
I20250113 02:51:35.397708 452 catalog_manager.cc:6596] ReplaceTablet: tablet fb2458e2044d49029ed2d6851db9de25 deleted and replaced by tablet 32e3dc0b7c044415a6343368ffa08552
I20250113 02:51:35.398221 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a654e8085fd49729aef1d343ceaf789. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:35.400192 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 01371679919c4daaad9451a7279898e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:35.400720 2140 tablet.cc:1721] T 41bcb2eafff14fa8a040982c723f99c1 P 4216a7f853d642b7bc3e5cff458f3aca: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250113 02:51:35.399483 736 tablet_service.cc:1514] Processing DeleteTablet for tablet fb2458e2044d49029ed2d6851db9de25 with delete_type TABLET_DATA_DELETED (replaced by tablet 32e3dc0b7c044415a6343368ffa08552) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:35.404152 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet fb2458e2044d49029ed2d6851db9de25 with delete_type TABLET_DATA_DELETED (replaced by tablet 32e3dc0b7c044415a6343368ffa08552) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:35.404232 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ce65d004b5c448b99ef6502fb600c0c. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:35.406816 711 leader_election.cc:343] T 7527e6e0b65e4a43b7f48899a34cdf7f P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:35.406941 711 leader_election.cc:304] T 7527e6e0b65e4a43b7f48899a34cdf7f P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:35.408190 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e0cdc5e4f4ca4f30b634e4dd2bf862d9. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:35.408334 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a6503d0e37cf4675b5d721975c1245b7 consensus state has no leader
I20250113 02:51:35.408918 736 tablet_service.cc:1467] Processing CreateTablet for tablet ff645eb0f8fa4744b7ba5cd03ce21729 (DEFAULT_TABLE table=default.Table_a632f9b27ba74e3b947fd6371082dde3 [id=1f1f8cbca4d54972b263aff6d9d4c3f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:35.409411 736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff645eb0f8fa4744b7ba5cd03ce21729. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:35.412211 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9438472464304da9a0c7658d4f0736b6" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:35.416242 1023 tablet_service.cc:1467] Processing CreateTablet for tablet 8ce65d004b5c448b99ef6502fb600c0c (DEFAULT_TABLE table=default.Table_fef715653eb349d19cf7f16198cd7669 [id=81c218b721d14e78a31347b8c32a9cf3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:35.416532 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8bb60fdd06946739b75b255303d9d24" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:35.416657 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ce65d004b5c448b99ef6502fb600c0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:35.421983 2191 raft_consensus.cc:2743] T 7527e6e0b65e4a43b7f48899a34cdf7f P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:35.422616 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_ec65488dc1334a33a56ba81d2bda391c" } modify_external_catalogs: true
I20250113 02:51:35.423367 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_ec65488dc1334a33a56ba81d2bda391c" } modify_external_catalogs: true
W20250113 02:51:35.423542 711 leader_election.cc:343] T 9438472464304da9a0c7658d4f0736b6 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:35.424294 711 leader_election.cc:304] T 9438472464304da9a0c7658d4f0736b6 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
W20250113 02:51:35.425850 711 leader_election.cc:343] T d8bb60fdd06946739b75b255303d9d24 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:35.426221 1035 tablet_service.cc:1467] Processing CreateTablet for tablet ff645eb0f8fa4744b7ba5cd03ce21729 (DEFAULT_TABLE table=default.Table_a632f9b27ba74e3b947fd6371082dde3 [id=1f1f8cbca4d54972b263aff6d9d4c3f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:35.426640 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff645eb0f8fa4744b7ba5cd03ce21729. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:35.428485 1196 ts_tablet_manager.cc:1905] T de4a58bf63054890808e2e2fe46e43d2 P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:35.428786 711 leader_election.cc:304] T d8bb60fdd06946739b75b255303d9d24 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
W20250113 02:51:35.429689 998 leader_election.cc:343] T e468b652f71f44e3945f8d919ac0e1a8 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:35.429821 998 leader_election.cc:304] T e468b652f71f44e3945f8d919ac0e1a8 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4216a7f853d642b7bc3e5cff458f3aca; no voters: 1186fa869eb14812832fa6562531f196, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:35.430220 2112 raft_consensus.cc:2743] T e468b652f71f44e3945f8d919ac0e1a8 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:35.430950 885 tablet_service.cc:1514] Processing DeleteTablet for tablet fb2458e2044d49029ed2d6851db9de25 with delete_type TABLET_DATA_DELETED (replaced by tablet 32e3dc0b7c044415a6343368ffa08552) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:35.431319 884 tablet_service.cc:1467] Processing CreateTablet for tablet ff645eb0f8fa4744b7ba5cd03ce21729 (DEFAULT_TABLE table=default.Table_a632f9b27ba74e3b947fd6371082dde3 [id=1f1f8cbca4d54972b263aff6d9d4c3f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:35.431389 453 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_34e676c9d2fb4f7ba8a20c74bc14eccf" } modify_external_catalogs: true
I20250113 02:51:35.431666 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff645eb0f8fa4744b7ba5cd03ce21729. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:35.432150 453 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_34e676c9d2fb4f7ba8a20c74bc14eccf" } modify_external_catalogs: true
I20250113 02:51:35.432525 2191 raft_consensus.cc:2743] T d8bb60fdd06946739b75b255303d9d24 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:35.426069 2072 raft_consensus.cc:2743] T 9438472464304da9a0c7658d4f0736b6 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:35.432233 742 tablet_service.cc:1467] Processing CreateTablet for tablet f4acbe59400247678d2441b1b844d976 (DEFAULT_TABLE table=default.Table_4e05dc66d38a47b982d5b2389cfc6582 [id=89ef50570cfa44acb5be993ea493ca79]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:35.435554 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f4acbe59400247678d2441b1b844d976. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:35.436079 882 tablet_service.cc:1514] Processing DeleteTablet for tablet 23e04961f072440fa3e3f9485e40f79c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:20 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:35.436386 885 tablet_service.cc:1467] Processing CreateTablet for tablet f4acbe59400247678d2441b1b844d976 (DEFAULT_TABLE table=default.Table_4e05dc66d38a47b982d5b2389cfc6582 [id=89ef50570cfa44acb5be993ea493ca79]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:35.436703 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f4acbe59400247678d2441b1b844d976. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:35.436623 751 tablet_service.cc:1467] Processing CreateTablet for tablet 8ce65d004b5c448b99ef6502fb600c0c (DEFAULT_TABLE table=default.Table_fef715653eb349d19cf7f16198cd7669 [id=81c218b721d14e78a31347b8c32a9cf3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:35.438061 1026 tablet_service.cc:1467] Processing CreateTablet for tablet f4acbe59400247678d2441b1b844d976 (DEFAULT_TABLE table=default.Table_4e05dc66d38a47b982d5b2389cfc6582 [id=89ef50570cfa44acb5be993ea493ca79]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:35.438170 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet fa9052eea538439a9cb4c527e97897a4
I20250113 02:51:35.438417 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f4acbe59400247678d2441b1b844d976. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:35.439471 1038 tablet_service.cc:1514] Processing DeleteTablet for tablet fa9052eea538439a9cb4c527e97897a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:35 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:35.442719 736 tablet_service.cc:1514] Processing DeleteTablet for tablet fa9052eea538439a9cb4c527e97897a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:35 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:35.437727 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ce65d004b5c448b99ef6502fb600c0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:35.444144 882 tablet_service.cc:1467] Processing CreateTablet for tablet c94a5d112a424188b70422d08c6b295c (DEFAULT_TABLE table=default.Table_f905559759c6440b9ae154f582126dc2 [id=3330f5f4527f4fa2b05ffea6028b6ccf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:35.444449 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c94a5d112a424188b70422d08c6b295c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:35.445055 1107 heartbeater.cc:346] Connected to a master server at 127.0.104.60:42067
I20250113 02:51:35.445470 451 master_service.cc:946] ReplaceTablet: received request to replace tablet 130f8540b9a6477e84bb9595e7c39994 from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:35.450568 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_85ede9aae0dd4e3abefcc25b836e9d0b" } new_table_name: "default.Table_49ba362618104736951a80e9988c5645" modify_external_catalogs: true
I20250113 02:51:35.450680 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_2607e08cf8a043f7aaa54cc016d105b2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:35.451228 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2607e08cf8a043f7aaa54cc016d105b2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250113 02:51:35.452051 998 consensus_peers.cc:487] T f56230a2c75d4763a54671c79358f4d9 P 4216a7f853d642b7bc3e5cff458f3aca -> Peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Couldn't send request to peer df69f48656b44bb890f76587ee69cc0f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250113 02:51:35.452723 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_d0763569196645ab8a5c97710bb3600d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:35.453223 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d0763569196645ab8a5c97710bb3600d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:35.453522 736 tablet_service.cc:1467] Processing CreateTablet for tablet c94a5d112a424188b70422d08c6b295c (DEFAULT_TABLE table=default.Table_f905559759c6440b9ae154f582126dc2 [id=3330f5f4527f4fa2b05ffea6028b6ccf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:35.453879 736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c94a5d112a424188b70422d08c6b295c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:35.454771 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_307a6dea2b2949119bdcfb0dd2c722c0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:35.455204 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_307a6dea2b2949119bdcfb0dd2c722c0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:35.429353 2195 raft_consensus.cc:491] T 23e04961f072440fa3e3f9485e40f79c P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:35.456014 2195 raft_consensus.cc:513] T 23e04961f072440fa3e3f9485e40f79c P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:35.456441 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_b0050f363a17412aa670daf91361c7bb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:35.456866 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b0050f363a17412aa670daf91361c7bb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250113 02:51:35.457475 851 leader_election.cc:343] T 23e04961f072440fa3e3f9485e40f79c P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Not found: Tablet not found: 23e04961f072440fa3e3f9485e40f79c
I20250113 02:51:35.457967 2199 tablet.cc:1721] T 41bcb2eafff14fa8a040982c723f99c1 P df69f48656b44bb890f76587ee69cc0f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250113 02:51:35.458142 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet dd566a49d3ff4d3889b80d8d2ccbee0e
I20250113 02:51:35.459038 2147 raft_consensus.cc:491] T be4b2a8a3927410eaf12e7592646e4af P df69f48656b44bb890f76587ee69cc0f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4216a7f853d642b7bc3e5cff458f3aca)
I20250113 02:51:35.459159 2147 raft_consensus.cc:513] T be4b2a8a3927410eaf12e7592646e4af P df69f48656b44bb890f76587ee69cc0f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:35.459751 1038 tablet_service.cc:1514] Processing DeleteTablet for tablet dd566a49d3ff4d3889b80d8d2ccbee0e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:35 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:35.459791 2147 leader_election.cc:290] T be4b2a8a3927410eaf12e7592646e4af P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:35.460438 1062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be4b2a8a3927410eaf12e7592646e4af" candidate_uuid: "df69f48656b44bb890f76587ee69cc0f" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:35.461042 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_abe5b12ac76741b194dceecebd1e173c" } modify_external_catalogs: true
I20250113 02:51:35.461206 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_abe5b12ac76741b194dceecebd1e173c" } modify_external_catalogs: true
I20250113 02:51:35.462790 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be4b2a8a3927410eaf12e7592646e4af" candidate_uuid: "df69f48656b44bb890f76587ee69cc0f" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
I20250113 02:51:35.463303 851 leader_election.cc:304] T be4b2a8a3927410eaf12e7592646e4af P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df69f48656b44bb890f76587ee69cc0f; no voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca
I20250113 02:51:35.463637 2147 raft_consensus.cc:2743] T be4b2a8a3927410eaf12e7592646e4af P df69f48656b44bb890f76587ee69cc0f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250113 02:51:35.464079 884 tablet_service.cc:1514] Processing DeleteTablet for tablet dd566a49d3ff4d3889b80d8d2ccbee0e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:35 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:35.464318 885 tablet_service.cc:1514] Processing DeleteTablet for tablet fa9052eea538439a9cb4c527e97897a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:35 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:35.465724 746 tablet_service.cc:1514] Processing DeleteTablet for tablet dd566a49d3ff4d3889b80d8d2ccbee0e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:35 UTC) from {username='slave'} at 127.0.0.1:55398
W20250113 02:51:35.466409 852 leader_election.cc:343] T 23e04961f072440fa3e3f9485e40f79c P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Not found: Tablet not found: 23e04961f072440fa3e3f9485e40f79c
I20250113 02:51:35.466517 852 leader_election.cc:304] T 23e04961f072440fa3e3f9485e40f79c P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df69f48656b44bb890f76587ee69cc0f; no voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca
I20250113 02:51:35.466737 2147 raft_consensus.cc:2743] T 23e04961f072440fa3e3f9485e40f79c P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:35.466889 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_8d74f8c0d6b24538b60e7a61451ad264" } new_table_name: "default.Table_4cb5f7e419c94840b25631909069c3ac" modify_external_catalogs: true
I20250113 02:51:35.468438 2195 leader_election.cc:290] T 23e04961f072440fa3e3f9485e40f79c P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
W20250113 02:51:35.469230 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 166f58d45b7b443ea54087e4ab48eaba consensus state has no leader
I20250113 02:51:35.469561 453 catalog_manager.cc:5618] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467) does not have the latest schema for tablet be4b2a8a3927410eaf12e7592646e4af (table default.Table_24d9af3d0b0740d69c791fc96916ef0a [id=124669ccadaf49e19d2660b1495f9647]). Expected version 3 got 0
I20250113 02:51:35.470382 1196 ts_tablet_manager.cc:1918] T de4a58bf63054890808e2e2fe46e43d2 P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:35.471760 1196 log.cc:1198] T de4a58bf63054890808e2e2fe46e43d2 P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/de4a58bf63054890808e2e2fe46e43d2
I20250113 02:51:35.472669 1196 ts_tablet_manager.cc:1939] T de4a58bf63054890808e2e2fe46e43d2 P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:35.473501 1196 tablet_replica.cc:331] T 092892a36e0e47cb8d2184c2bbde95d7 P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:35.473469 452 catalog_manager.cc:5618] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467) does not have the latest schema for tablet be4b2a8a3927410eaf12e7592646e4af (table default.Table_24d9af3d0b0740d69c791fc96916ef0a [id=124669ccadaf49e19d2660b1495f9647]). Expected version 3 got 0
I20250113 02:51:35.473845 1196 raft_consensus.cc:2238] T 092892a36e0e47cb8d2184c2bbde95d7 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:35.474213 1196 raft_consensus.cc:2267] T 092892a36e0e47cb8d2184c2bbde95d7 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:35.474844 1196 ts_tablet_manager.cc:1905] T 092892a36e0e47cb8d2184c2bbde95d7 P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:35.476130 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet de4a58bf63054890808e2e2fe46e43d2 (table default.Table_5b0e60572cb54fcca8b3b5b94326ae3a [id=9f5add219a7c48da9a8afa966e22d1bd]) successfully deleted
I20250113 02:51:35.476702 455 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_9b26cd6ed9944430a1228f552a7c3d95" } modify_external_catalogs: true
I20250113 02:51:35.476929 455 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_9b26cd6ed9944430a1228f552a7c3d95" } modify_external_catalogs: true
I20250113 02:51:35.476969 885 tablet_service.cc:1467] Processing CreateTablet for tablet 1376cf72589d43048cb829e03240c8da (DEFAULT_TABLE table=default.Table_f167644bb9424d71af7befe561e5f3f5 [id=eaa20262431c4345b21fd0bca63a2855]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:35.477352 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1376cf72589d43048cb829e03240c8da. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:35.478940 746 tablet_service.cc:1467] Processing CreateTablet for tablet 1376cf72589d43048cb829e03240c8da (DEFAULT_TABLE table=default.Table_f167644bb9424d71af7befe561e5f3f5 [id=eaa20262431c4345b21fd0bca63a2855]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:35.479305 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1376cf72589d43048cb829e03240c8da. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:35.482820 1038 tablet_service.cc:1467] Processing CreateTablet for tablet c94a5d112a424188b70422d08c6b295c (DEFAULT_TABLE table=default.Table_f905559759c6440b9ae154f582126dc2 [id=3330f5f4527f4fa2b05ffea6028b6ccf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:35.483186 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c94a5d112a424188b70422d08c6b295c. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:35.491243 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet afaf16655a0b48098e26c771bc2473cb consensus state has no leader
I20250113 02:51:35.492017 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 2002c69206ec468aad1b44f1b5b0d3b2
I20250113 02:51:35.492753 884 tablet_service.cc:1514] Processing DeleteTablet for tablet 2002c69206ec468aad1b44f1b5b0d3b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:35 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:35.493283 746 tablet_service.cc:1514] Processing DeleteTablet for tablet 2002c69206ec468aad1b44f1b5b0d3b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:35 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:35.493497 1196 ts_tablet_manager.cc:1918] T 092892a36e0e47cb8d2184c2bbde95d7 P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:35.493839 885 tablet_service.cc:1467] Processing CreateTablet for tablet b9e51df79f1d42738e26b37823a8f1a5 (DEFAULT_TABLE table=default.Table_e906a17932314e12bd60df6ea31b8881 [id=f4dd1ca7e7f34e0e8da81a2e88e0d8da]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:35.494482 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b9e51df79f1d42738e26b37823a8f1a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:35.495524 1036 tablet_service.cc:1467] Processing CreateTablet for tablet 1376cf72589d43048cb829e03240c8da (DEFAULT_TABLE table=default.Table_f167644bb9424d71af7befe561e5f3f5 [id=eaa20262431c4345b21fd0bca63a2855]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:35.496029 1041 tablet_service.cc:1514] Processing DeleteTablet for tablet 2002c69206ec468aad1b44f1b5b0d3b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:35 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:35.497103 2072 raft_consensus.cc:491] T a92f9b1c9c1a4cbdbf0cc7b2a3f2275b P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:35.497251 2072 raft_consensus.cc:513] T a92f9b1c9c1a4cbdbf0cc7b2a3f2275b P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:35.498312 1062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a92f9b1c9c1a4cbdbf0cc7b2a3f2275b" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
W20250113 02:51:35.498886 709 leader_election.cc:343] T a92f9b1c9c1a4cbdbf0cc7b2a3f2275b P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:35.499068 2072 leader_election.cc:290] T a92f9b1c9c1a4cbdbf0cc7b2a3f2275b P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:35.500072 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a92f9b1c9c1a4cbdbf0cc7b2a3f2275b" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:35.500541 711 leader_election.cc:343] T a92f9b1c9c1a4cbdbf0cc7b2a3f2275b P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:35.500674 711 leader_election.cc:304] T a92f9b1c9c1a4cbdbf0cc7b2a3f2275b P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:35.501085 2072 raft_consensus.cc:2743] T a92f9b1c9c1a4cbdbf0cc7b2a3f2275b P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:35.499965 446 catalog_manager.cc:6071] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: default.Table_f8a6812c7fb94ea69cd9bf775b914ea6 [id=71508871e2d5469a91aaf0e19b02b638] alter complete (version 1)
I20250113 02:51:35.501549 746 tablet_service.cc:1467] Processing CreateTablet for tablet b9e51df79f1d42738e26b37823a8f1a5 (DEFAULT_TABLE table=default.Table_e906a17932314e12bd60df6ea31b8881 [id=f4dd1ca7e7f34e0e8da81a2e88e0d8da]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:35.501991 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b9e51df79f1d42738e26b37823a8f1a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:35.508527 2199 tablet.cc:1721] T be4b2a8a3927410eaf12e7592646e4af P df69f48656b44bb890f76587ee69cc0f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250113 02:51:35.510954 1195 ts_tablet_manager.cc:1918] T eebad045e7bb43fa9ca9c9af8b3b7ea5 P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:35.511214 1195 log.cc:1198] T eebad045e7bb43fa9ca9c9af8b3b7ea5 P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/eebad045e7bb43fa9ca9c9af8b3b7ea5
I20250113 02:51:35.511819 1196 log.cc:1198] T 092892a36e0e47cb8d2184c2bbde95d7 P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/092892a36e0e47cb8d2184c2bbde95d7
I20250113 02:51:35.512250 1196 ts_tablet_manager.cc:1939] T 092892a36e0e47cb8d2184c2bbde95d7 P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:35.512925 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet 092892a36e0e47cb8d2184c2bbde95d7 (table default.Table_03fafac4f68e4eb3be2ea41493d9e006 [id=d4bf205fb0464358af61bb3fb889c687]) successfully deleted
I20250113 02:51:35.514163 885 tablet_service.cc:1467] Processing CreateTablet for tablet c3bab84ac06c4b868475868edae3625a (DEFAULT_TABLE table=default.Table_ceb8d423519441d595d613778ab75514 [id=9fe419f25ec04d11902957063d32cefa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:35.514550 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c3bab84ac06c4b868475868edae3625a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:35.515751 446 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_9a4ca43a32a04beabb2108aae12c3b96" } modify_external_catalogs: true
I20250113 02:51:35.516017 446 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_9a4ca43a32a04beabb2108aae12c3b96" } modify_external_catalogs: true
I20250113 02:51:35.519730 1194 tablet_bootstrap.cc:492] T bffec36d567544c5a1c87e64a7e9730b P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:35.519948 1194 ts_tablet_manager.cc:1397] T bffec36d567544c5a1c87e64a7e9730b P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.141s user 0.007s sys 0.004s
I20250113 02:51:35.520151 1196 tablet_replica.cc:331] T 7d3562a48c6e4badbbcf37640e91b5ba P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:35.520345 1196 raft_consensus.cc:2238] T 7d3562a48c6e4badbbcf37640e91b5ba P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:35.520496 1196 raft_consensus.cc:2267] T 7d3562a48c6e4badbbcf37640e91b5ba P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:35.521101 1194 raft_consensus.cc:357] T bffec36d567544c5a1c87e64a7e9730b P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:35.521322 1194 raft_consensus.cc:383] T bffec36d567544c5a1c87e64a7e9730b P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:35.522503 1194 raft_consensus.cc:738] T bffec36d567544c5a1c87e64a7e9730b P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:35.522745 1194 consensus_queue.cc:260] T bffec36d567544c5a1c87e64a7e9730b P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:35.528915 1041 tablet_service.cc:1467] Processing CreateTablet for tablet b9e51df79f1d42738e26b37823a8f1a5 (DEFAULT_TABLE table=default.Table_e906a17932314e12bd60df6ea31b8881 [id=f4dd1ca7e7f34e0e8da81a2e88e0d8da]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:35.530351 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet c548a79f394d4ab88a309ca6400a05a2
I20250113 02:51:35.531361 739 tablet_service.cc:1514] Processing DeleteTablet for tablet c548a79f394d4ab88a309ca6400a05a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:35 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:35.531610 736 tablet_service.cc:1467] Processing CreateTablet for tablet c3bab84ac06c4b868475868edae3625a (DEFAULT_TABLE table=default.Table_ceb8d423519441d595d613778ab75514 [id=9fe419f25ec04d11902957063d32cefa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:35.532011 736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c3bab84ac06c4b868475868edae3625a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:35.536196 1040 tablet_service.cc:1467] Processing CreateTablet for tablet c3bab84ac06c4b868475868edae3625a (DEFAULT_TABLE table=default.Table_ceb8d423519441d595d613778ab75514 [id=9fe419f25ec04d11902957063d32cefa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:35.536978 1042 tablet_service.cc:1514] Processing DeleteTablet for tablet c548a79f394d4ab88a309ca6400a05a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:35 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:35.538983 1036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1376cf72589d43048cb829e03240c8da. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:35.543284 1195 ts_tablet_manager.cc:1939] T eebad045e7bb43fa9ca9c9af8b3b7ea5 P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:35.544075 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 990) took 1070 ms. Trace:
I20250113 02:51:35.544158 1195 rpcz_store.cc:276] 0113 02:51:34.474108 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:34.474167 (+ 59us) service_pool.cc:225] Handling call
0113 02:51:35.345169 (+871002us) ts_tablet_manager.cc:1145] Deleting tablet eebad045e7bb43fa9ca9c9af8b3b7ea5
0113 02:51:35.345178 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:35.442908 (+ 97730us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:35.510809 (+ 67901us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:35.544053 (+ 33244us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":870827}
I20250113 02:51:35.544454 1195 tablet_replica.cc:331] T 377ebf66bbef40988043f6200ae0893c P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:35.544611 1195 raft_consensus.cc:2238] T 377ebf66bbef40988043f6200ae0893c P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:35.544746 1195 raft_consensus.cc:2267] T 377ebf66bbef40988043f6200ae0893c P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:35.544797 752 tablet_service.cc:1467] Processing CreateTablet for tablet ff5e40047f9e43fbb980b5fedfb36ade (DEFAULT_TABLE table=default.Table_ab3c82460efc4216a97966ab66413b5a [id=5c447f893b1d4eaabbfd64605328b2f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:35.545145 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff5e40047f9e43fbb980b5fedfb36ade. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:35.545248 1195 ts_tablet_manager.cc:1905] T 377ebf66bbef40988043f6200ae0893c P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:35.546525 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet eebad045e7bb43fa9ca9c9af8b3b7ea5 (table default.Table_abd20980e42242719c1529668d68128b [id=043d63161d5e4454af0782a50d64d256]) successfully deleted
I20250113 02:51:35.548270 744 tablet_service.cc:1467] Processing CreateTablet for tablet 9952030ad9294b85a25e1241b4929258 (DEFAULT_TABLE table=default.Table_c8ea3525cf2847649a201509a00a5d9f [id=be2cfd4e531042188251836d955d1fc3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:35.548565 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9952030ad9294b85a25e1241b4929258. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:35.544620 739 tablet_service.cc:1467] Processing CreateTablet for tablet 045167a6122a4560b298aae2358a485c (DEFAULT_TABLE table=default.Table_d8e5fcedafb14988b94b1114b38e35d3 [id=e04e27290ec440c99f784d778e08951c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:35.549543 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 045167a6122a4560b298aae2358a485c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:35.552618 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_c7728cb38c5b4dcc8cd76e68b6a24a20"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:35.553402 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_787937b2aa714be5b37053648c4240fe" } new_table_name: "default.Table_dc40c4f395d846c7aa489903545cfc12" modify_external_catalogs: true
I20250113 02:51:35.554725 452 master_service.cc:946] ReplaceTablet: received request to replace tablet 78bf9b924b144837b6e17b54086d7908 from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:35.556223 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_4f3d23a03f104bd7acd23395be7568c1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:35.556815 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4f3d23a03f104bd7acd23395be7568c1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:35.557823 743 tablet_service.cc:1467] Processing CreateTablet for tablet f59dc0fa43dd4eb5932a1ad8a5a08349 (DEFAULT_TABLE table=default.Table_165edade161e458c81a066ac578d832c [id=da026c039e4346e8abdde115f44f0ece]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:35.558161 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f59dc0fa43dd4eb5932a1ad8a5a08349. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:35.563467 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c7728cb38c5b4dcc8cd76e68b6a24a20 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:35.566246 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_86e441512733438dae0893de5acd5c92"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:35.566851 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_86e441512733438dae0893de5acd5c92 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:35.571182 1026 tablet_service.cc:1467] Processing CreateTablet for tablet ff5e40047f9e43fbb980b5fedfb36ade (DEFAULT_TABLE table=default.Table_ab3c82460efc4216a97966ab66413b5a [id=5c447f893b1d4eaabbfd64605328b2f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:35.571234 1042 tablet_service.cc:1467] Processing CreateTablet for tablet 9952030ad9294b85a25e1241b4929258 (DEFAULT_TABLE table=default.Table_c8ea3525cf2847649a201509a00a5d9f [id=be2cfd4e531042188251836d955d1fc3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:35.571580 1042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9952030ad9294b85a25e1241b4929258. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:35.571545 1036 tablet_service.cc:1467] Processing CreateTablet for tablet 045167a6122a4560b298aae2358a485c (DEFAULT_TABLE table=default.Table_d8e5fcedafb14988b94b1114b38e35d3 [id=e04e27290ec440c99f784d778e08951c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:35.571820 1036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 045167a6122a4560b298aae2358a485c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:35.572052 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b9e51df79f1d42738e26b37823a8f1a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:35.576188 1040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c3bab84ac06c4b868475868edae3625a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:35.579964 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 51a7cb7e7e2c49949c3be328005ce468
I20250113 02:51:35.584697 1191 tablet_bootstrap.cc:492] T f21eb3fa0f414816a29e3d4560cdc0fe P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:35.585265 1191 ts_tablet_manager.cc:1397] T f21eb3fa0f414816a29e3d4560cdc0fe P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.408s user 0.056s sys 0.072s
I20250113 02:51:35.586095 1191 raft_consensus.cc:357] T f21eb3fa0f414816a29e3d4560cdc0fe P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:35.586313 1191 raft_consensus.cc:383] T f21eb3fa0f414816a29e3d4560cdc0fe P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:35.586385 1191 raft_consensus.cc:738] T f21eb3fa0f414816a29e3d4560cdc0fe P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:35.586776 1191 consensus_queue.cc:260] T f21eb3fa0f414816a29e3d4560cdc0fe P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:35.587437 1191 ts_tablet_manager.cc:1428] T f21eb3fa0f414816a29e3d4560cdc0fe P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250113 02:51:35.587961 1191 tablet_bootstrap.cc:492] T 30ca8a716bfc4ab789b216e17567785e P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:35.529207 1196 ts_tablet_manager.cc:1905] T 7d3562a48c6e4badbbcf37640e91b5ba P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:35.589648 1191 tablet_bootstrap.cc:654] T 30ca8a716bfc4ab789b216e17567785e P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:35.530207 884 tablet_service.cc:1467] Processing CreateTablet for tablet 045167a6122a4560b298aae2358a485c (DEFAULT_TABLE table=default.Table_d8e5fcedafb14988b94b1114b38e35d3 [id=e04e27290ec440c99f784d778e08951c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:35.592427 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 045167a6122a4560b298aae2358a485c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:35.594370 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff5e40047f9e43fbb980b5fedfb36ade. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:35.530968 882 tablet_service.cc:1514] Processing DeleteTablet for tablet c548a79f394d4ab88a309ca6400a05a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:35 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:35.540433 892 tablet_service.cc:1467] Processing CreateTablet for tablet ff5e40047f9e43fbb980b5fedfb36ade (DEFAULT_TABLE table=default.Table_ab3c82460efc4216a97966ab66413b5a [id=5c447f893b1d4eaabbfd64605328b2f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:35.542028 881 tablet_service.cc:1467] Processing CreateTablet for tablet 9952030ad9294b85a25e1241b4929258 (DEFAULT_TABLE table=default.Table_c8ea3525cf2847649a201509a00a5d9f [id=be2cfd4e531042188251836d955d1fc3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:35.552482 917 raft_consensus.cc:3054] T 1cfbcd682b944c89847b40fe69f0a75e P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:35.556654 887 tablet_service.cc:1467] Processing CreateTablet for tablet f59dc0fa43dd4eb5932a1ad8a5a08349 (DEFAULT_TABLE table=default.Table_165edade161e458c81a066ac578d832c [id=da026c039e4346e8abdde115f44f0ece]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:35.599514 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff5e40047f9e43fbb980b5fedfb36ade. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:35.599715 1194 ts_tablet_manager.cc:1428] T bffec36d567544c5a1c87e64a7e9730b P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.080s user 0.002s sys 0.000s
I20250113 02:51:35.602540 1192 tablet_bootstrap.cc:654] T 0916b41c171a419791797559fbc2807d P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:35.586254 894 tablet_service.cc:1514] Processing DeleteTablet for tablet 51a7cb7e7e2c49949c3be328005ce468 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:35 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:35.605082 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_6ca7cb0180f844548bfb870bb68aaeb9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:35.605656 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6ca7cb0180f844548bfb870bb68aaeb9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250113 02:51:35.607112 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6887e74b88aa458f857728460521a96a consensus state has no leader
I20250113 02:51:35.607546 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 78bf9b924b144837b6e17b54086d7908
I20250113 02:51:35.608039 452 catalog_manager.cc:6596] ReplaceTablet: tablet 78bf9b924b144837b6e17b54086d7908 deleted and replaced by tablet 49c2e648a49f4034a521f38e7d98edc4
I20250113 02:51:35.608646 894 tablet_service.cc:1514] Processing DeleteTablet for tablet 78bf9b924b144837b6e17b54086d7908 with delete_type TABLET_DATA_DELETED (replaced by tablet 49c2e648a49f4034a521f38e7d98edc4) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:35.610304 1196 ts_tablet_manager.cc:1918] T 7d3562a48c6e4badbbcf37640e91b5ba P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:35.612289 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_ab30cf183b59419689df6d79621cd053" } new_table_name: "default.Table_3a224acfb9184dd295d9e7321324a60e" modify_external_catalogs: true
I20250113 02:51:35.614704 894 tablet_service.cc:1514] Processing DeleteTablet for tablet bffec36d567544c5a1c87e64a7e9730b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:21 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:35.616211 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9952030ad9294b85a25e1241b4929258. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:35.619819 1198 ts_tablet_manager.cc:1918] T 0539cbbe47e64c039ad39c4b52edcc4b P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:35.619935 1198 log.cc:1198] T 0539cbbe47e64c039ad39c4b52edcc4b P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/0539cbbe47e64c039ad39c4b52edcc4b
I20250113 02:51:35.620184 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f59dc0fa43dd4eb5932a1ad8a5a08349. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:35.620280 1198 ts_tablet_manager.cc:1939] T 0539cbbe47e64c039ad39c4b52edcc4b P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:35.621768 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_6a6779b59f8541b086c5d17a372fd0c0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:35.622316 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6a6779b59f8541b086c5d17a372fd0c0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:35.625936 1040 tablet_service.cc:1514] Processing DeleteTablet for tablet 78bf9b924b144837b6e17b54086d7908 with delete_type TABLET_DATA_DELETED (replaced by tablet 49c2e648a49f4034a521f38e7d98edc4) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:35.628374 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 935) took 1691 ms. Trace:
I20250113 02:51:35.629233 1198 rpcz_store.cc:276] 0113 02:51:33.936948 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:33.939689 (+ 2741us) service_pool.cc:225] Handling call
0113 02:51:35.176267 (+1236578us) ts_tablet_manager.cc:1145] Deleting tablet 0539cbbe47e64c039ad39c4b52edcc4b
0113 02:51:35.176281 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:35.534193 (+357912us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:35.619728 (+ 85535us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:35.628356 (+ 8628us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1229206}
I20250113 02:51:35.629850 1198 tablet_replica.cc:331] T c7243d27a1c642269a3d0e67632e458c P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:35.630323 1198 raft_consensus.cc:2238] T c7243d27a1c642269a3d0e67632e458c P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:35.630836 1198 raft_consensus.cc:2267] T c7243d27a1c642269a3d0e67632e458c P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:35.610469 1196 log.cc:1198] T 7d3562a48c6e4badbbcf37640e91b5ba P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/7d3562a48c6e4badbbcf37640e91b5ba
I20250113 02:51:35.632915 1041 tablet_service.cc:1514] Processing DeleteTablet for tablet 51a7cb7e7e2c49949c3be328005ce468 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:35 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:35.633018 1196 ts_tablet_manager.cc:1939] T 7d3562a48c6e4badbbcf37640e91b5ba P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:35.633700 1198 ts_tablet_manager.cc:1905] T c7243d27a1c642269a3d0e67632e458c P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:35.637853 751 tablet_service.cc:1514] Processing DeleteTablet for tablet f21eb3fa0f414816a29e3d4560cdc0fe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:30 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:35.638187 746 tablet_service.cc:1514] Processing DeleteTablet for tablet 78bf9b924b144837b6e17b54086d7908 with delete_type TABLET_DATA_DELETED (replaced by tablet 49c2e648a49f4034a521f38e7d98edc4) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:35.638414 742 tablet_service.cc:1514] Processing DeleteTablet for tablet 51a7cb7e7e2c49949c3be328005ce468 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:35 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:35.643431 455 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_174390617a294f1090362caca3ab703f" } modify_external_catalogs: true
I20250113 02:51:35.643721 455 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_174390617a294f1090362caca3ab703f" } modify_external_catalogs: true
I20250113 02:51:35.645867 1194 tablet_bootstrap.cc:492] T 8337f3c82d654ecdac2cbc1ffa88ef05 P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:35.647053 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet 7d3562a48c6e4badbbcf37640e91b5ba (table default.Table_06f81a97cb014a3bad594e88c8cf4733 [id=7c2bc6d4232f4585b84d71e3e1f85794]) successfully deleted
I20250113 02:51:35.643594 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_7698d677addf448fa62c659dad28b1e4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:35.647500 881 tablet_service.cc:1467] Processing CreateTablet for tablet e7d6b565f4cc4d1884343de41b380d99 (DEFAULT_TABLE table=default.Table_2b653446c62d4fdaae71cfff4b0e1391 [id=469b0fd3dba5431d9f142f077b1aa78f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:35.647900 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e7d6b565f4cc4d1884343de41b380d99. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:35.648243 1194 tablet_bootstrap.cc:654] T 8337f3c82d654ecdac2cbc1ffa88ef05 P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
W20250113 02:51:35.649389 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7698d677addf448fa62c659dad28b1e4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:35.651831 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_bb7ff740ee724b3999eeec918a03b1ae" } new_table_name: "default.Table_e0a43ddf2e9c4ac099dc34a3c330ef1b" modify_external_catalogs: true
I20250113 02:51:35.653497 1196 tablet_replica.cc:331] T 32cba8f3efc54e78b4df41c15df82eea P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:35.653745 1196 raft_consensus.cc:2238] T 32cba8f3efc54e78b4df41c15df82eea P df69f48656b44bb890f76587ee69cc0f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:35.654003 1196 raft_consensus.cc:2267] T 32cba8f3efc54e78b4df41c15df82eea P df69f48656b44bb890f76587ee69cc0f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:35.656811 1192 tablet_bootstrap.cc:492] T 0916b41c171a419791797559fbc2807d P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:35.656940 1192 ts_tablet_manager.cc:1397] T 0916b41c171a419791797559fbc2807d P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.420s user 0.009s sys 0.000s
I20250113 02:51:35.657637 1192 raft_consensus.cc:357] T 0916b41c171a419791797559fbc2807d P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:35.657807 1192 raft_consensus.cc:383] T 0916b41c171a419791797559fbc2807d P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:35.657850 1192 raft_consensus.cc:738] T 0916b41c171a419791797559fbc2807d P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:35.658072 1192 consensus_queue.cc:260] T 0916b41c171a419791797559fbc2807d P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:35.659889 448 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_c05a2a3b8f804fb5bde195c57b87e094" } modify_external_catalogs: true
I20250113 02:51:35.660089 448 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_c05a2a3b8f804fb5bde195c57b87e094" } modify_external_catalogs: true
I20250113 02:51:35.662305 2112 raft_consensus.cc:491] T 1d15c7a01f854f0aaabcd30ec3166e31 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:35.662423 2112 raft_consensus.cc:513] T 1d15c7a01f854f0aaabcd30ec3166e31 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:35.662994 2112 leader_election.cc:290] T 1d15c7a01f854f0aaabcd30ec3166e31 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:35.663666 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d15c7a01f854f0aaabcd30ec3166e31" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:35.664235 998 leader_election.cc:343] T 1d15c7a01f854f0aaabcd30ec3166e31 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:35.664958 1192 ts_tablet_manager.cc:1428] T 0916b41c171a419791797559fbc2807d P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20250113 02:51:35.669276 1192 tablet_bootstrap.cc:492] T 56b022c94847411785a7e9db3757a2d9 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
W20250113 02:51:35.670691 995 leader_election.cc:343] T 1d15c7a01f854f0aaabcd30ec3166e31 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Not found: Tablet not found: 1d15c7a01f854f0aaabcd30ec3166e31
I20250113 02:51:35.670807 995 leader_election.cc:304] T 1d15c7a01f854f0aaabcd30ec3166e31 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4216a7f853d642b7bc3e5cff458f3aca; no voters: 1186fa869eb14812832fa6562531f196, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:35.671036 2112 raft_consensus.cc:2743] T 1d15c7a01f854f0aaabcd30ec3166e31 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:35.671590 1192 tablet_bootstrap.cc:654] T 56b022c94847411785a7e9db3757a2d9 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:35.671729 742 tablet_service.cc:1467] Processing CreateTablet for tablet e7d6b565f4cc4d1884343de41b380d99 (DEFAULT_TABLE table=default.Table_2b653446c62d4fdaae71cfff4b0e1391 [id=469b0fd3dba5431d9f142f077b1aa78f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:35.672219 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e7d6b565f4cc4d1884343de41b380d99. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:35.672672 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet 0539cbbe47e64c039ad39c4b52edcc4b (table default.Table_d1b93410e6504e71a32397040f50b464 [id=817479ce0ac04442a0f89e6115d24ba4]) successfully deleted
I20250113 02:51:35.673755 1026 tablet_service.cc:1467] Processing CreateTablet for tablet e7d6b565f4cc4d1884343de41b380d99 (DEFAULT_TABLE table=default.Table_2b653446c62d4fdaae71cfff4b0e1391 [id=469b0fd3dba5431d9f142f077b1aa78f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:35.677536 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_9e8c1b27c2f74c919d2e9224a1db676a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:35.678110 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9e8c1b27c2f74c919d2e9224a1db676a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:35.678488 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_41aed46571c449228b1f98fb1bae0b4d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:35.678932 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_41aed46571c449228b1f98fb1bae0b4d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:35.679828 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_b95dec5993da4c6fb06409ef6cc17a5b" } new_table_name: "default.Table_7bc3fe3fa752416b9fd940d2940f30fc" modify_external_catalogs: true
I20250113 02:51:35.683379 1192 tablet_bootstrap.cc:492] T 56b022c94847411785a7e9db3757a2d9 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:35.683530 1192 ts_tablet_manager.cc:1397] T 56b022c94847411785a7e9db3757a2d9 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.001s
I20250113 02:51:35.684345 1192 raft_consensus.cc:357] T 56b022c94847411785a7e9db3757a2d9 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:35.684535 1192 raft_consensus.cc:383] T 56b022c94847411785a7e9db3757a2d9 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:35.684592 1192 raft_consensus.cc:738] T 56b022c94847411785a7e9db3757a2d9 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:35.685420 1191 tablet_bootstrap.cc:492] T 30ca8a716bfc4ab789b216e17567785e P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:35.685748 1191 ts_tablet_manager.cc:1397] T 30ca8a716bfc4ab789b216e17567785e P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.098s user 0.002s sys 0.009s
I20250113 02:51:35.687937 1191 raft_consensus.cc:357] T 30ca8a716bfc4ab789b216e17567785e P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:35.688477 1191 raft_consensus.cc:383] T 30ca8a716bfc4ab789b216e17567785e P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:35.688719 1191 raft_consensus.cc:738] T 30ca8a716bfc4ab789b216e17567785e P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:35.688920 1041 tablet_service.cc:1467] Processing CreateTablet for tablet f59dc0fa43dd4eb5932a1ad8a5a08349 (DEFAULT_TABLE table=default.Table_165edade161e458c81a066ac578d832c [id=da026c039e4346e8abdde115f44f0ece]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:35.689431 1191 consensus_queue.cc:260] T 30ca8a716bfc4ab789b216e17567785e P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:35.694813 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e7d6b565f4cc4d1884343de41b380d99. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:35.695185 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f59dc0fa43dd4eb5932a1ad8a5a08349. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:35.700886 1196 ts_tablet_manager.cc:1905] T 32cba8f3efc54e78b4df41c15df82eea P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:35.705564 1191 ts_tablet_manager.cc:1428] T 30ca8a716bfc4ab789b216e17567785e P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.020s user 0.001s sys 0.002s
I20250113 02:51:35.707195 1192 consensus_queue.cc:260] T 56b022c94847411785a7e9db3757a2d9 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:35.713593 1195 ts_tablet_manager.cc:1918] T 377ebf66bbef40988043f6200ae0893c P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:35.713716 1195 log.cc:1198] T 377ebf66bbef40988043f6200ae0893c P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/377ebf66bbef40988043f6200ae0893c
I20250113 02:51:35.714000 1195 ts_tablet_manager.cc:1939] T 377ebf66bbef40988043f6200ae0893c P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:35.714727 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 986) took 1265 ms. Trace:
I20250113 02:51:35.714810 1195 rpcz_store.cc:276] 0113 02:51:34.449767 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:34.482583 (+ 32816us) service_pool.cc:225] Handling call
0113 02:51:35.544398 (+1061815us) ts_tablet_manager.cc:1145] Deleting tablet 377ebf66bbef40988043f6200ae0893c
0113 02:51:35.544418 (+ 20us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:35.682597 (+138179us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:35.713492 (+ 30895us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:35.714707 (+ 1215us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1061499}
I20250113 02:51:35.715296 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet 377ebf66bbef40988043f6200ae0893c (table default.Table_5ddfdff2cc8c4d12a1c4552e9a302027 [id=8b35e389de9f4917904e87b43618737e]) successfully deleted
W20250113 02:51:35.715988 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 370c53f565df4e6f915e54a2fd2bf2ce because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 370c53f565df4e6f915e54a2fd2bf2ce already in progress: creating tablet
I20250113 02:51:35.716264 1195 tablet_replica.cc:331] T df87e25053a74990959f85288a85217b P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:35.716436 1195 raft_consensus.cc:2238] T df87e25053a74990959f85288a85217b P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:35.716575 1195 raft_consensus.cc:2267] T df87e25053a74990959f85288a85217b P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:35.717067 1195 ts_tablet_manager.cc:1905] T df87e25053a74990959f85288a85217b P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:35.716897 1198 ts_tablet_manager.cc:1918] T c7243d27a1c642269a3d0e67632e458c P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:35.718247 1198 log.cc:1198] T c7243d27a1c642269a3d0e67632e458c P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/c7243d27a1c642269a3d0e67632e458c
I20250113 02:51:35.719682 1192 ts_tablet_manager.cc:1428] T 56b022c94847411785a7e9db3757a2d9 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.036s user 0.004s sys 0.000s
I20250113 02:51:35.720588 2112 raft_consensus.cc:491] T 0916b41c171a419791797559fbc2807d P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:35.720876 2112 raft_consensus.cc:513] T 0916b41c171a419791797559fbc2807d P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:35.721565 2112 leader_election.cc:290] T 0916b41c171a419791797559fbc2807d P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:35.722190 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0916b41c171a419791797559fbc2807d" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:35.722913 998 leader_election.cc:343] T 0916b41c171a419791797559fbc2807d P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
W20250113 02:51:35.723320 995 leader_election.cc:343] T 0916b41c171a419791797559fbc2807d P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Not found: Tablet not found: 0916b41c171a419791797559fbc2807d
I20250113 02:51:35.723426 995 leader_election.cc:304] T 0916b41c171a419791797559fbc2807d P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4216a7f853d642b7bc3e5cff458f3aca; no voters: 1186fa869eb14812832fa6562531f196, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:35.723579 1198 ts_tablet_manager.cc:1939] T c7243d27a1c642269a3d0e67632e458c P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:35.723865 2112 raft_consensus.cc:2743] T 0916b41c171a419791797559fbc2807d P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:35.732394 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 936) took 1795 ms. Trace:
I20250113 02:51:35.741824 1192 tablet_bootstrap.cc:492] T a92f9b1c9c1a4cbdbf0cc7b2a3f2275b P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:35.735939 1198 rpcz_store.cc:276] 0113 02:51:33.937038 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:33.939834 (+ 2796us) service_pool.cc:225] Handling call
0113 02:51:35.629807 (+1689973us) ts_tablet_manager.cc:1145] Deleting tablet c7243d27a1c642269a3d0e67632e458c
0113 02:51:35.629816 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:35.656278 (+ 26462us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:35.716032 (+ 59754us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:35.732105 (+ 16073us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1682710}
I20250113 02:51:35.744856 1194 tablet_bootstrap.cc:492] T 8337f3c82d654ecdac2cbc1ffa88ef05 P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:35.745239 1194 ts_tablet_manager.cc:1397] T 8337f3c82d654ecdac2cbc1ffa88ef05 P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.099s user 0.006s sys 0.006s
I20250113 02:51:35.746547 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet c7243d27a1c642269a3d0e67632e458c (table default.Table_7157800e639b45d8bb947e53d4319cfc [id=39997cddb4684b6c984eeeb48d715002]) successfully deleted
I20250113 02:51:35.747638 1194 raft_consensus.cc:357] T 8337f3c82d654ecdac2cbc1ffa88ef05 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:35.748448 1194 raft_consensus.cc:383] T 8337f3c82d654ecdac2cbc1ffa88ef05 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:35.748679 1194 raft_consensus.cc:738] T 8337f3c82d654ecdac2cbc1ffa88ef05 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:35.750064 2112 raft_consensus.cc:491] T 999fecb3fc65466cad21cc1ff5ef2d44 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:35.750207 2112 raft_consensus.cc:513] T 999fecb3fc65466cad21cc1ff5ef2d44 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:35.750897 2112 leader_election.cc:290] T 999fecb3fc65466cad21cc1ff5ef2d44 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:35.751648 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "999fecb3fc65466cad21cc1ff5ef2d44" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:35.752305 998 leader_election.cc:343] T 999fecb3fc65466cad21cc1ff5ef2d44 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:35.749989 1194 consensus_queue.cc:260] T 8337f3c82d654ecdac2cbc1ffa88ef05 P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
W20250113 02:51:35.752724 995 leader_election.cc:343] T 999fecb3fc65466cad21cc1ff5ef2d44 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Not found: Tablet not found: 999fecb3fc65466cad21cc1ff5ef2d44
I20250113 02:51:35.752815 995 leader_election.cc:304] T 999fecb3fc65466cad21cc1ff5ef2d44 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4216a7f853d642b7bc3e5cff458f3aca; no voters: 1186fa869eb14812832fa6562531f196, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:35.753360 2112 raft_consensus.cc:2743] T 999fecb3fc65466cad21cc1ff5ef2d44 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:35.755810 1198 tablet_replica.cc:331] T 51d9483ecacb413796d0fd9e7560eac1 P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:35.758805 2072 raft_consensus.cc:491] T 4eb5f1944cb44fd5a40ca92a6a3d7214 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:35.758952 2072 raft_consensus.cc:513] T 4eb5f1944cb44fd5a40ca92a6a3d7214 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:35.759611 2072 leader_election.cc:290] T 4eb5f1944cb44fd5a40ca92a6a3d7214 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:35.760072 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4eb5f1944cb44fd5a40ca92a6a3d7214" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:35.760140 1062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4eb5f1944cb44fd5a40ca92a6a3d7214" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
W20250113 02:51:35.760550 711 leader_election.cc:343] T 4eb5f1944cb44fd5a40ca92a6a3d7214 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
W20250113 02:51:35.760679 709 leader_election.cc:343] T 4eb5f1944cb44fd5a40ca92a6a3d7214 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:35.760785 709 leader_election.cc:304] T 4eb5f1944cb44fd5a40ca92a6a3d7214 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:35.761202 2072 raft_consensus.cc:2743] T 4eb5f1944cb44fd5a40ca92a6a3d7214 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:35.713491 1191 tablet_bootstrap.cc:492] T f1e32e09a8244957a99bdfea5812ff31 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:35.765540 1191 tablet_bootstrap.cc:654] T f1e32e09a8244957a99bdfea5812ff31 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:35.764947 1194 ts_tablet_manager.cc:1428] T 8337f3c82d654ecdac2cbc1ffa88ef05 P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.019s user 0.003s sys 0.001s
I20250113 02:51:35.765033 1198 raft_consensus.cc:2238] T 51d9483ecacb413796d0fd9e7560eac1 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:35.770233 2112 raft_consensus.cc:491] T aa5c172a69bf4ca79999f4b58df08de8 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:35.770915 2112 raft_consensus.cc:513] T aa5c172a69bf4ca79999f4b58df08de8 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:35.770762 1194 tablet_bootstrap.cc:492] T b0b267b0d9c54ba496c89f6cb287dffd P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:35.770465 1196 ts_tablet_manager.cc:1918] T 32cba8f3efc54e78b4df41c15df82eea P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20250113 02:51:35.771550 2112 leader_election.cc:290] T aa5c172a69bf4ca79999f4b58df08de8 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:35.771682 1196 log.cc:1198] T 32cba8f3efc54e78b4df41c15df82eea P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/32cba8f3efc54e78b4df41c15df82eea
I20250113 02:51:35.772419 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5c172a69bf4ca79999f4b58df08de8" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:35.772464 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5c172a69bf4ca79999f4b58df08de8" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
I20250113 02:51:35.772670 774 raft_consensus.cc:2463] T aa5c172a69bf4ca79999f4b58df08de8 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 0.
W20250113 02:51:35.773281 998 leader_election.cc:343] T aa5c172a69bf4ca79999f4b58df08de8 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:35.773732 995 leader_election.cc:304] T aa5c172a69bf4ca79999f4b58df08de8 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters: df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:35.774165 2112 raft_consensus.cc:2798] T aa5c172a69bf4ca79999f4b58df08de8 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 02:51:35.774240 2112 raft_consensus.cc:491] T aa5c172a69bf4ca79999f4b58df08de8 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 02:51:35.774298 2112 raft_consensus.cc:3054] T aa5c172a69bf4ca79999f4b58df08de8 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:35.774837 1196 ts_tablet_manager.cc:1939] T 32cba8f3efc54e78b4df41c15df82eea P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:35.776726 2112 raft_consensus.cc:513] T aa5c172a69bf4ca79999f4b58df08de8 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:35.777853 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet 32cba8f3efc54e78b4df41c15df82eea (table default.Table_3d65267002364c8a9175ccb2223cac44 [id=8aff13d6d3c5483eadb596b705fe5151]) successfully deleted
I20250113 02:51:35.777845 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5c172a69bf4ca79999f4b58df08de8" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196"
I20250113 02:51:35.778017 774 raft_consensus.cc:3054] T aa5c172a69bf4ca79999f4b58df08de8 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Advancing to term 1
W20250113 02:51:35.779646 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 2c49f1eefd2041b4ae617a4cb35d6f15 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2c49f1eefd2041b4ae617a4cb35d6f15 already in progress: creating tablet
I20250113 02:51:35.780839 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5c172a69bf4ca79999f4b58df08de8" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f"
W20250113 02:51:35.780974 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 7fb282707196444690352dea3eb1387f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7fb282707196444690352dea3eb1387f already in progress: creating tablet
W20250113 02:51:35.781585 998 leader_election.cc:343] T aa5c172a69bf4ca79999f4b58df08de8 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:35.781874 2112 leader_election.cc:290] T aa5c172a69bf4ca79999f4b58df08de8 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Requested vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
W20250113 02:51:35.782042 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet f105f8b32a654e4ca154e8fc0cb96091 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f105f8b32a654e4ca154e8fc0cb96091 already in progress: creating tablet
I20250113 02:51:35.783004 774 raft_consensus.cc:2463] T aa5c172a69bf4ca79999f4b58df08de8 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 1.
W20250113 02:51:35.783324 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet f9269967dea94fc3a337546ad0842e11 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f9269967dea94fc3a337546ad0842e11 already in progress: creating tablet
I20250113 02:51:35.783803 995 leader_election.cc:304] T aa5c172a69bf4ca79999f4b58df08de8 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters: df69f48656b44bb890f76587ee69cc0f
W20250113 02:51:35.784650 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet f24cda32b6e947c7983ae2f1f9a64c23 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f24cda32b6e947c7983ae2f1f9a64c23 already in progress: creating tablet
I20250113 02:51:35.785091 2112 raft_consensus.cc:2798] T aa5c172a69bf4ca79999f4b58df08de8 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Leader election won for term 1
W20250113 02:51:35.786001 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet fb2458e2044d49029ed2d6851db9de25 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fb2458e2044d49029ed2d6851db9de25 already in progress: creating tablet
I20250113 02:51:35.786747 1196 tablet_replica.cc:331] T 23e04961f072440fa3e3f9485e40f79c P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:35.787607 2112 raft_consensus.cc:695] T aa5c172a69bf4ca79999f4b58df08de8 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: Becoming Leader. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Running, Role: LEADER
I20250113 02:51:35.787956 1196 raft_consensus.cc:2238] T 23e04961f072440fa3e3f9485e40f79c P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:35.788825 1191 tablet_bootstrap.cc:492] T f1e32e09a8244957a99bdfea5812ff31 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:35.788975 1191 ts_tablet_manager.cc:1397] T f1e32e09a8244957a99bdfea5812ff31 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.076s user 0.003s sys 0.000s
I20250113 02:51:35.788975 1196 raft_consensus.cc:2267] T 23e04961f072440fa3e3f9485e40f79c P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:35.790238 1191 raft_consensus.cc:357] T f1e32e09a8244957a99bdfea5812ff31 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:35.790427 1191 raft_consensus.cc:383] T f1e32e09a8244957a99bdfea5812ff31 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:35.790510 1191 raft_consensus.cc:738] T f1e32e09a8244957a99bdfea5812ff31 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:35.790735 1191 consensus_queue.cc:260] T f1e32e09a8244957a99bdfea5812ff31 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:35.791234 1191 ts_tablet_manager.cc:1428] T f1e32e09a8244957a99bdfea5812ff31 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250113 02:51:35.792465 1191 tablet_bootstrap.cc:492] T b0113b16110e4787ac1c4676b2c8bd1e P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:35.792290 1196 ts_tablet_manager.cc:1905] T 23e04961f072440fa3e3f9485e40f79c P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:35.790141 2112 consensus_queue.cc:237] T aa5c172a69bf4ca79999f4b58df08de8 P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:35.769682 1198 raft_consensus.cc:2267] T 51d9483ecacb413796d0fd9e7560eac1 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:35.801642 1191 tablet_bootstrap.cc:654] T b0113b16110e4787ac1c4676b2c8bd1e P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:35.802053 1195 ts_tablet_manager.cc:1918] T df87e25053a74990959f85288a85217b P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:35.803089 1195 log.cc:1198] T df87e25053a74990959f85288a85217b P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/df87e25053a74990959f85288a85217b
I20250113 02:51:35.803474 1195 ts_tablet_manager.cc:1939] T df87e25053a74990959f85288a85217b P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
W20250113 02:51:35.807742 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 42a8e01bb71f49b09c83a8f5c905f38d consensus state has no leader
I20250113 02:51:35.808061 449 catalog_manager.cc:5618] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467) does not have the latest schema for tablet 1cfbcd682b944c89847b40fe69f0a75e (table default.Table_32e88abaae96492fbcce39fb2cd7c221 [id=2b84deb07e5c4ab8b550fe42945d7b71]). Expected version 2 got 0
I20250113 02:51:35.810089 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_9d1fda1b56c74fbb80a73da524dbbe7a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:35.810811 742 tablet_service.cc:1514] Processing DeleteTablet for tablet 30ca8a716bfc4ab789b216e17567785e with delete_type TABLET_DATA_DELETED (replaced by tablet 5f7c91d8c018455e847e5969f7b5e1cc) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:35.812971 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 994) took 1010 ms. Trace:
I20250113 02:51:35.813062 1195 rpcz_store.cc:276] 0113 02:51:34.802665 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:34.802825 (+ 160us) service_pool.cc:225] Handling call
0113 02:51:35.716224 (+913399us) ts_tablet_manager.cc:1145] Deleting tablet df87e25053a74990959f85288a85217b
0113 02:51:35.716233 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:35.719894 (+ 3661us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:35.801951 (+ 82057us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:35.812952 (+ 11001us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":913278}
I20250113 02:51:35.813495 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet df87e25053a74990959f85288a85217b (table default.Table_2f1a7bb2c4cd4ead95b175e1d3545f20 [id=b2c8431e06ac46be9840edf78b6c19a3]) successfully deleted
I20250113 02:51:35.813863 1194 tablet_bootstrap.cc:654] T b0b267b0d9c54ba496c89f6cb287dffd P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
W20250113 02:51:35.820441 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet d565b77f90784d87b64996772c938085 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d565b77f90784d87b64996772c938085 already in progress: creating tablet
I20250113 02:51:35.820505 742 tablet_service.cc:1514] Processing DeleteTablet for tablet f1e32e09a8244957a99bdfea5812ff31 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:25 UTC) from {username='slave'} at 127.0.0.1:55398
W20250113 02:51:35.821038 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 726e68108f5b4de4a51ffdf778ef825d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 726e68108f5b4de4a51ffdf778ef825d already in progress: creating tablet
I20250113 02:51:35.821338 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 29c17fd90d774c029a35e77888358067
I20250113 02:51:35.821924 742 tablet_service.cc:1514] Processing DeleteTablet for tablet 29c17fd90d774c029a35e77888358067 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:35 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:35.822167 1195 tablet_replica.cc:331] T a92f9b1c9c1a4cbdbf0cc7b2a3f2275b P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:35.822070 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_2b473336a0b14ae6a2cff5109f945c4f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:35.822494 1195 raft_consensus.cc:2238] T a92f9b1c9c1a4cbdbf0cc7b2a3f2275b P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus shutting down.
W20250113 02:51:35.822634 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2b473336a0b14ae6a2cff5109f945c4f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:35.823064 1195 raft_consensus.cc:2267] T a92f9b1c9c1a4cbdbf0cc7b2a3f2275b P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:35.824091 881 tablet_service.cc:1514] Processing DeleteTablet for tablet 29c17fd90d774c029a35e77888358067 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:35 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:35.824321 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet b2a23259f2c448cc8a904a13a291d6ca
I20250113 02:51:35.824930 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_fd46797b5cb645e0a712aaec33089be6" } new_table_name: "default.Table_d6208e261d8b4ccaae513508a6ff1e72" modify_external_catalogs: true
I20250113 02:51:35.826311 881 tablet_service.cc:1514] Processing DeleteTablet for tablet b2a23259f2c448cc8a904a13a291d6ca with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:35 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:35.826622 742 tablet_service.cc:1514] Processing DeleteTablet for tablet b2a23259f2c448cc8a904a13a291d6ca with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:35 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:35.827427 1195 ts_tablet_manager.cc:1905] T a92f9b1c9c1a4cbdbf0cc7b2a3f2275b P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:35.827574 1041 tablet_service.cc:1514] Processing DeleteTablet for tablet b2a23259f2c448cc8a904a13a291d6ca with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:35 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:35.827603 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet 29c17fd90d774c029a35e77888358067 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:35 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:35.827672 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet 0916b41c171a419791797559fbc2807d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:23 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:35.829463 450 catalog_manager.cc:5618] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467) does not have the latest schema for tablet 1cfbcd682b944c89847b40fe69f0a75e (table default.Table_32e88abaae96492fbcce39fb2cd7c221 [id=2b84deb07e5c4ab8b550fe42945d7b71]). Expected version 2 got 0
I20250113 02:51:35.829857 449 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_0bba725d39ff4d44a742e905c275724b" } modify_external_catalogs: true
I20250113 02:51:35.830075 449 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_0bba725d39ff4d44a742e905c275724b" } modify_external_catalogs: true
I20250113 02:51:35.830619 1196 ts_tablet_manager.cc:1918] T 23e04961f072440fa3e3f9485e40f79c P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:35.830580 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_4c6f3deafe73412a9496041fea5f2f69" } modify_external_catalogs: true
I20250113 02:51:35.830739 1196 log.cc:1198] T 23e04961f072440fa3e3f9485e40f79c P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/23e04961f072440fa3e3f9485e40f79c
I20250113 02:51:35.830752 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_4c6f3deafe73412a9496041fea5f2f69" } modify_external_catalogs: true
I20250113 02:51:35.831060 1196 ts_tablet_manager.cc:1939] T 23e04961f072440fa3e3f9485e40f79c P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:35.831593 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet 23e04961f072440fa3e3f9485e40f79c (table default.Table_4a0258998d4f413b80d701c74526b86d [id=6fd9215b11274a76b06813227889927b]) successfully deleted
W20250113 02:51:35.831832 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet dd566a49d3ff4d3889b80d8d2ccbee0e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dd566a49d3ff4d3889b80d8d2ccbee0e already in progress: creating tablet
W20250113 02:51:35.832088 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet fa9052eea538439a9cb4c527e97897a4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fa9052eea538439a9cb4c527e97897a4 already in progress: creating tablet
W20250113 02:51:35.832274 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 2002c69206ec468aad1b44f1b5b0d3b2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2002c69206ec468aad1b44f1b5b0d3b2 already in progress: creating tablet
I20250113 02:51:35.832396 1196 tablet_replica.cc:331] T bffec36d567544c5a1c87e64a7e9730b P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
W20250113 02:51:35.832459 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet c548a79f394d4ab88a309ca6400a05a2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c548a79f394d4ab88a309ca6400a05a2 already in progress: creating tablet
I20250113 02:51:35.832587 1196 raft_consensus.cc:2238] T bffec36d567544c5a1c87e64a7e9730b P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
W20250113 02:51:35.832659 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 51a7cb7e7e2c49949c3be328005ce468 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 51a7cb7e7e2c49949c3be328005ce468 already in progress: creating tablet
I20250113 02:51:35.832746 1196 raft_consensus.cc:2267] T bffec36d567544c5a1c87e64a7e9730b P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
W20250113 02:51:35.832857 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 78bf9b924b144837b6e17b54086d7908 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 78bf9b924b144837b6e17b54086d7908 already in progress: creating tablet
W20250113 02:51:35.834389 998 consensus_peers.cc:487] T 377ebf66bbef40988043f6200ae0893c P 4216a7f853d642b7bc3e5cff458f3aca -> Peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Couldn't send request to peer df69f48656b44bb890f76587ee69cc0f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250113 02:51:35.835150 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9d1fda1b56c74fbb80a73da524dbbe7a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:35.839783 450 master_service.cc:946] ReplaceTablet: received request to replace tablet cb897004cbfb4a0283b42d06509a9ee5 from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:35.844760 1191 tablet_bootstrap.cc:492] T b0113b16110e4787ac1c4676b2c8bd1e P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:35.845002 1191 ts_tablet_manager.cc:1397] T b0113b16110e4787ac1c4676b2c8bd1e P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.053s user 0.003s sys 0.000s
I20250113 02:51:35.845878 1191 raft_consensus.cc:357] T b0113b16110e4787ac1c4676b2c8bd1e P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:35.846170 1191 raft_consensus.cc:383] T b0113b16110e4787ac1c4676b2c8bd1e P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:35.846276 1191 raft_consensus.cc:738] T b0113b16110e4787ac1c4676b2c8bd1e P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:35.846616 1191 consensus_queue.cc:260] T b0113b16110e4787ac1c4676b2c8bd1e P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:35.847426 1194 tablet_bootstrap.cc:492] T b0b267b0d9c54ba496c89f6cb287dffd P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:35.847594 1194 ts_tablet_manager.cc:1397] T b0b267b0d9c54ba496c89f6cb287dffd P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.080s user 0.004s sys 0.008s
I20250113 02:51:35.848363 1194 raft_consensus.cc:357] T b0b267b0d9c54ba496c89f6cb287dffd P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:35.849007 1191 ts_tablet_manager.cc:1428] T b0113b16110e4787ac1c4676b2c8bd1e P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20250113 02:51:35.849316 1191 tablet_bootstrap.cc:492] T 9e736211eb3c4536b9fd1d90d2d83838 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:35.850104 1194 raft_consensus.cc:383] T b0b267b0d9c54ba496c89f6cb287dffd P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:35.850281 1194 raft_consensus.cc:738] T b0b267b0d9c54ba496c89f6cb287dffd P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:35.850638 1194 consensus_queue.cc:260] T b0b267b0d9c54ba496c89f6cb287dffd P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:35.851919 1191 tablet_bootstrap.cc:654] T 9e736211eb3c4536b9fd1d90d2d83838 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:35.852567 1194 ts_tablet_manager.cc:1428] T b0b267b0d9c54ba496c89f6cb287dffd P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20250113 02:51:35.854933 1198 ts_tablet_manager.cc:1905] T 51d9483ecacb413796d0fd9e7560eac1 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:35.857630 454 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_bf689f8eb8884e88a72e9b50598fc389" } modify_external_catalogs: true
I20250113 02:51:35.857841 454 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_bf689f8eb8884e88a72e9b50598fc389" } modify_external_catalogs: true
W20250113 02:51:35.859158 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6d659f76c66d47118edbf80c0fa28d14 consensus state has no leader
I20250113 02:51:35.859512 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_d6b0044411fc480ebcb1a635d4438a07"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:35.860172 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_0f386e0d294749fba0d3c68b7bcf0f46"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:35.860639 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0f386e0d294749fba0d3c68b7bcf0f46 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:35.861996 1194 tablet_bootstrap.cc:492] T 0752fde3a30a4947a2a688c56ceadab5 P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:35.863651 1194 tablet_bootstrap.cc:654] T 0752fde3a30a4947a2a688c56ceadab5 P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
W20250113 02:51:35.863626 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d6b0044411fc480ebcb1a635d4438a07 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:35.863770 742 tablet_service.cc:1514] Processing DeleteTablet for tablet b0113b16110e4787ac1c4676b2c8bd1e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:22 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:35.869087 964 heartbeater.cc:346] Connected to a master server at 127.0.104.60:42067
I20250113 02:51:35.870563 1192 tablet_bootstrap.cc:654] T a92f9b1c9c1a4cbdbf0cc7b2a3f2275b P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:35.871402 2072 raft_consensus.cc:491] T 30ca8a716bfc4ab789b216e17567785e P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:35.871536 2072 raft_consensus.cc:513] T 30ca8a716bfc4ab789b216e17567785e P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:35.872238 2072 leader_election.cc:290] T 30ca8a716bfc4ab789b216e17567785e P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:35.872916 1062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30ca8a716bfc4ab789b216e17567785e" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:35.872956 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30ca8a716bfc4ab789b216e17567785e" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:35.873472 709 leader_election.cc:343] T 30ca8a716bfc4ab789b216e17567785e P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
W20250113 02:51:35.873847 711 leader_election.cc:343] T 30ca8a716bfc4ab789b216e17567785e P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:35.873953 711 leader_election.cc:304] T 30ca8a716bfc4ab789b216e17567785e P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:35.874389 2072 raft_consensus.cc:2743] T 30ca8a716bfc4ab789b216e17567785e P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:35.877723 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet d8e81a6fef794f26bdb017e16af8b30c
I20250113 02:51:35.880134 742 tablet_service.cc:1514] Processing DeleteTablet for tablet d8e81a6fef794f26bdb017e16af8b30c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:35 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:35.882360 448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_eba6422342a742499b1c54520d991881" } new_table_name: "default.Table_f98f15e1cb6a43348c4438096a7f9ea4" modify_external_catalogs: true
W20250113 02:51:35.884166 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet eba80548f52f4c32a098d1540c9cded0 consensus state has no leader
I20250113 02:51:35.884538 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_c91c831eacb14ed9bbb303b6f7fcd6ca"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:35.884930 1191 tablet_bootstrap.cc:492] T 9e736211eb3c4536b9fd1d90d2d83838 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:35.885063 1191 ts_tablet_manager.cc:1397] T 9e736211eb3c4536b9fd1d90d2d83838 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.036s user 0.001s sys 0.002s
W20250113 02:51:35.885089 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c91c831eacb14ed9bbb303b6f7fcd6ca in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:35.885182 449 master_service.cc:946] ReplaceTablet: received request to replace tablet bffec36d567544c5a1c87e64a7e9730b from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:35.885771 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_b1bd611e962141f19470cb3296e4bd00"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:35.885898 1191 raft_consensus.cc:357] T 9e736211eb3c4536b9fd1d90d2d83838 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:35.886065 1191 raft_consensus.cc:383] T 9e736211eb3c4536b9fd1d90d2d83838 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:35.886126 1191 raft_consensus.cc:738] T 9e736211eb3c4536b9fd1d90d2d83838 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:35.886329 1191 consensus_queue.cc:260] T 9e736211eb3c4536b9fd1d90d2d83838 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:35.886693 881 tablet_service.cc:1514] Processing DeleteTablet for tablet d8e81a6fef794f26bdb017e16af8b30c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:35 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:35.886852 1191 ts_tablet_manager.cc:1428] T 9e736211eb3c4536b9fd1d90d2d83838 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250113 02:51:35.887758 881 tablet_service.cc:1514] Processing DeleteTablet for tablet b0b267b0d9c54ba496c89f6cb287dffd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:25 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:35.888777 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet d8e81a6fef794f26bdb017e16af8b30c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:35 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:35.887224 451 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet af993aa9671b408aa7e0613acf8c0e8f
I20250113 02:51:35.887579 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet cb897004cbfb4a0283b42d06509a9ee5
I20250113 02:51:35.889695 450 catalog_manager.cc:6596] ReplaceTablet: tablet cb897004cbfb4a0283b42d06509a9ee5 deleted and replaced by tablet 0e2d8726a2ab489cae80fb1685d6951a
I20250113 02:51:35.890444 742 tablet_service.cc:1514] Processing DeleteTablet for tablet cb897004cbfb4a0283b42d06509a9ee5 with delete_type TABLET_DATA_DELETED (replaced by tablet 0e2d8726a2ab489cae80fb1685d6951a) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:35.890461 881 tablet_service.cc:1514] Processing DeleteTablet for tablet cb897004cbfb4a0283b42d06509a9ee5 with delete_type TABLET_DATA_DELETED (replaced by tablet 0e2d8726a2ab489cae80fb1685d6951a) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:35.891952 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet cb897004cbfb4a0283b42d06509a9ee5 with delete_type TABLET_DATA_DELETED (replaced by tablet 0e2d8726a2ab489cae80fb1685d6951a) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:35.892444 450 catalog_manager.cc:5618] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587) does not have the latest schema for tablet 56b022c94847411785a7e9db3757a2d9 (table default.Table_6461b062100240d59a5d91a87a74510c [id=40ca872c045b4349b63ba9046e31ed13]). Expected version 2 got 0
W20250113 02:51:35.892855 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet be5d6907a02f41be8eab71de051a6d3b consensus state has no leader
I20250113 02:51:35.892695 450 catalog_manager.cc:5526] T aa5c172a69bf4ca79999f4b58df08de8 P 4216a7f853d642b7bc3e5cff458f3aca reported cstate change: term changed from 0 to 1, leader changed from <none> to 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3). New cstate: current_term: 1 leader_uuid: "4216a7f853d642b7bc3e5cff458f3aca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } health_report { overall_health: HEALTHY } } }
I20250113 02:51:35.893097 450 catalog_manager.cc:5618] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587) does not have the latest schema for tablet aa5c172a69bf4ca79999f4b58df08de8 (table default.Table_3f1e7146fed34d53896f138e43a4d26e [id=5dceb76e1dbe422ab98d4da5bf6c45a0]). Expected version 1 got 0
I20250113 02:51:35.893954 1191 tablet_bootstrap.cc:492] T dd566a49d3ff4d3889b80d8d2ccbee0e P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
W20250113 02:51:35.886217 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b1bd611e962141f19470cb3296e4bd00 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:35.895588 1191 tablet_bootstrap.cc:654] T dd566a49d3ff4d3889b80d8d2ccbee0e P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:35.899936 742 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e736211eb3c4536b9fd1d90d2d83838 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:24 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:35.902566 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 4c615b0b83d74fcfae0c51047dbfbbf4
I20250113 02:51:35.903700 881 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c615b0b83d74fcfae0c51047dbfbbf4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:35 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:35.903774 742 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c615b0b83d74fcfae0c51047dbfbbf4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:35 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:35.904735 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c615b0b83d74fcfae0c51047dbfbbf4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:35 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:35.919867 1196 ts_tablet_manager.cc:1905] T bffec36d567544c5a1c87e64a7e9730b P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:35.925567 1194 tablet_bootstrap.cc:492] T 0752fde3a30a4947a2a688c56ceadab5 P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:35.925813 1194 ts_tablet_manager.cc:1397] T 0752fde3a30a4947a2a688c56ceadab5 P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.064s user 0.007s sys 0.005s
I20250113 02:51:35.926707 1194 raft_consensus.cc:357] T 0752fde3a30a4947a2a688c56ceadab5 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:35.927090 1194 raft_consensus.cc:383] T 0752fde3a30a4947a2a688c56ceadab5 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:35.927193 1194 raft_consensus.cc:738] T 0752fde3a30a4947a2a688c56ceadab5 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:35.927495 1194 consensus_queue.cc:260] T 0752fde3a30a4947a2a688c56ceadab5 P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:35.928115 1194 ts_tablet_manager.cc:1428] T 0752fde3a30a4947a2a688c56ceadab5 P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250113 02:51:35.930119 1198 ts_tablet_manager.cc:1918] T 51d9483ecacb413796d0fd9e7560eac1 P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:35.931775 1195 ts_tablet_manager.cc:1918] T a92f9b1c9c1a4cbdbf0cc7b2a3f2275b P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:35.932191 455 catalog_manager.cc:5618] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467) does not have the latest schema for tablet 0752fde3a30a4947a2a688c56ceadab5 (table default.Table_da3e49fa1e654145a05858466ec2260f [id=94704429b49b46e38a48258a450d2705]). Expected version 1 got 0
I20250113 02:51:35.932508 1194 tablet_bootstrap.cc:492] T c508724edea147e5bf6e20bca2f50397 P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:35.933202 2195 raft_consensus.cc:491] T 0752fde3a30a4947a2a688c56ceadab5 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:35.933357 2195 raft_consensus.cc:513] T 0752fde3a30a4947a2a688c56ceadab5 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:35.934446 2195 leader_election.cc:290] T 0752fde3a30a4947a2a688c56ceadab5 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587), 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
I20250113 02:51:35.934190 1195 log.cc:1198] T a92f9b1c9c1a4cbdbf0cc7b2a3f2275b P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/a92f9b1c9c1a4cbdbf0cc7b2a3f2275b
I20250113 02:51:35.934937 1194 tablet_bootstrap.cc:654] T c508724edea147e5bf6e20bca2f50397 P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:35.936228 1195 ts_tablet_manager.cc:1939] T a92f9b1c9c1a4cbdbf0cc7b2a3f2275b P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:35.936394 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0752fde3a30a4947a2a688c56ceadab5" candidate_uuid: "df69f48656b44bb890f76587ee69cc0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
W20250113 02:51:35.938804 448 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c04be975c8c049b892b16129ed47a142 consensus state has no leader
I20250113 02:51:35.941499 2072 raft_consensus.cc:491] T f1e32e09a8244957a99bdfea5812ff31 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:35.941632 2072 raft_consensus.cc:513] T f1e32e09a8244957a99bdfea5812ff31 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:35.942241 2072 leader_election.cc:290] T f1e32e09a8244957a99bdfea5812ff31 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:35.943375 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1e32e09a8244957a99bdfea5812ff31" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:35.943890 711 leader_election.cc:343] T f1e32e09a8244957a99bdfea5812ff31 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:35.935057 1062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0752fde3a30a4947a2a688c56ceadab5" candidate_uuid: "df69f48656b44bb890f76587ee69cc0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:35.943014 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1e32e09a8244957a99bdfea5812ff31" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:35.950335 852 leader_election.cc:304] T 0752fde3a30a4947a2a688c56ceadab5 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df69f48656b44bb890f76587ee69cc0f; no voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca
W20250113 02:51:35.950484 709 leader_election.cc:343] T f1e32e09a8244957a99bdfea5812ff31 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:35.950598 709 leader_election.cc:304] T f1e32e09a8244957a99bdfea5812ff31 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:35.950498 1198 log.cc:1198] T 51d9483ecacb413796d0fd9e7560eac1 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/51d9483ecacb413796d0fd9e7560eac1
I20250113 02:51:35.950871 2195 raft_consensus.cc:2743] T 0752fde3a30a4947a2a688c56ceadab5 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:35.950948 2072 raft_consensus.cc:2743] T f1e32e09a8244957a99bdfea5812ff31 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:35.951838 1192 tablet_bootstrap.cc:492] T a92f9b1c9c1a4cbdbf0cc7b2a3f2275b P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:35.952040 1192 ts_tablet_manager.cc:1397] T a92f9b1c9c1a4cbdbf0cc7b2a3f2275b P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.212s user 0.023s sys 0.007s
I20250113 02:51:35.952483 1198 ts_tablet_manager.cc:1939] T 51d9483ecacb413796d0fd9e7560eac1 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:35.952850 1192 raft_consensus.cc:357] T a92f9b1c9c1a4cbdbf0cc7b2a3f2275b P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:35.942237 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 997) took 1041 ms. Trace:
I20250113 02:51:35.953065 1192 raft_consensus.cc:383] T a92f9b1c9c1a4cbdbf0cc7b2a3f2275b P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:35.953120 1192 raft_consensus.cc:738] T a92f9b1c9c1a4cbdbf0cc7b2a3f2275b P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:35.953171 1195 rpcz_store.cc:276] 0113 02:51:34.900760 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:34.900807 (+ 47us) service_pool.cc:225] Handling call
0113 02:51:35.822127 (+921320us) ts_tablet_manager.cc:1145] Deleting tablet a92f9b1c9c1a4cbdbf0cc7b2a3f2275b
0113 02:51:35.822135 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:35.838317 (+ 16182us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:35.931683 (+ 93366us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:35.942218 (+ 10535us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":921105}
I20250113 02:51:35.953330 1192 consensus_queue.cc:260] T a92f9b1c9c1a4cbdbf0cc7b2a3f2275b P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:35.953764 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet a92f9b1c9c1a4cbdbf0cc7b2a3f2275b (table default.Table_6185ed1a90a44576bc65ed60294c2537 [id=5634c185da26460f93cde9c65fbdb24a]) successfully deleted
I20250113 02:51:35.953876 1192 ts_tablet_manager.cc:1428] T a92f9b1c9c1a4cbdbf0cc7b2a3f2275b P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250113 02:51:35.954432 1192 tablet_bootstrap.cc:492] T eebad045e7bb43fa9ca9c9af8b3b7ea5 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:35.956305 1192 tablet_bootstrap.cc:654] T eebad045e7bb43fa9ca9c9af8b3b7ea5 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:35.957710 1191 tablet_bootstrap.cc:492] T dd566a49d3ff4d3889b80d8d2ccbee0e P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:35.957850 1191 ts_tablet_manager.cc:1397] T dd566a49d3ff4d3889b80d8d2ccbee0e P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.064s user 0.000s sys 0.004s
I20250113 02:51:35.958564 1191 raft_consensus.cc:357] T dd566a49d3ff4d3889b80d8d2ccbee0e P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:35.958748 1191 raft_consensus.cc:383] T dd566a49d3ff4d3889b80d8d2ccbee0e P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:35.958823 1191 raft_consensus.cc:738] T dd566a49d3ff4d3889b80d8d2ccbee0e P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:35.959080 1191 consensus_queue.cc:260] T dd566a49d3ff4d3889b80d8d2ccbee0e P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:35.960484 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet a92f9b1c9c1a4cbdbf0cc7b2a3f2275b with delete_type TABLET_DATA_DELETED (replaced by tablet bc6a7322c0c34adb85457750526d8955) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:35.961526 1191 ts_tablet_manager.cc:1428] T dd566a49d3ff4d3889b80d8d2ccbee0e P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20250113 02:51:35.979264 1195 tablet_replica.cc:331] T 7213e3284b4d4cb581d9fb946a9f87e9 P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:35.979516 1195 raft_consensus.cc:2238] T 7213e3284b4d4cb581d9fb946a9f87e9 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:35.979688 1195 raft_consensus.cc:2267] T 7213e3284b4d4cb581d9fb946a9f87e9 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:35.980520 2072 raft_consensus.cc:491] T 9e736211eb3c4536b9fd1d90d2d83838 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:35.980650 2072 raft_consensus.cc:513] T 9e736211eb3c4536b9fd1d90d2d83838 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:35.981395 2072 leader_election.cc:290] T 9e736211eb3c4536b9fd1d90d2d83838 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:35.980360 1195 ts_tablet_manager.cc:1905] T 7213e3284b4d4cb581d9fb946a9f87e9 P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:35.981786 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e736211eb3c4536b9fd1d90d2d83838" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:35.982000 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e736211eb3c4536b9fd1d90d2d83838" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
W20250113 02:51:35.982336 711 leader_election.cc:343] T 9e736211eb3c4536b9fd1d90d2d83838 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
W20250113 02:51:35.982477 709 leader_election.cc:343] T 9e736211eb3c4536b9fd1d90d2d83838 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:35.982609 709 leader_election.cc:304] T 9e736211eb3c4536b9fd1d90d2d83838 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:35.982949 2072 raft_consensus.cc:2743] T 9e736211eb3c4536b9fd1d90d2d83838 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:35.984470 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 937) took 2047 ms. Trace:
I20250113 02:51:35.984802 1198 rpcz_store.cc:276] 0113 02:51:33.937160 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:33.939944 (+ 2784us) service_pool.cc:225] Handling call
0113 02:51:35.754421 (+1814477us) ts_tablet_manager.cc:1145] Deleting tablet 51d9483ecacb413796d0fd9e7560eac1
0113 02:51:35.754661 (+ 240us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:35.900313 (+145652us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:35.929986 (+ 29673us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:35.984290 (+ 54304us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1802386}
I20250113 02:51:35.985675 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet 51d9483ecacb413796d0fd9e7560eac1 (table default.Table_9417f16fb6384fde8e191ca02808aa78 [id=b49a91d3b86d44e1824a987db3de13b6]) successfully deleted
I20250113 02:51:35.986485 1198 tablet_replica.cc:331] T 6666922d677a4c678f80d7c6c47fd2a5 P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:35.987471 1198 raft_consensus.cc:2238] T 6666922d677a4c678f80d7c6c47fd2a5 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:35.988237 1198 raft_consensus.cc:2267] T 6666922d677a4c678f80d7c6c47fd2a5 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:35.990139 450 master_service.cc:946] ReplaceTablet: received request to replace tablet ee289c3236834dc48d922062041be3b5 from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:35.991441 1191 tablet_bootstrap.cc:492] T 44d2c7d5334c476fac60fdd57e2c02b6 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:35.993889 1191 tablet_bootstrap.cc:654] T 44d2c7d5334c476fac60fdd57e2c02b6 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:36.002696 1194 tablet_bootstrap.cc:492] T c508724edea147e5bf6e20bca2f50397 P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:36.003335 1194 ts_tablet_manager.cc:1397] T c508724edea147e5bf6e20bca2f50397 P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.071s user 0.005s sys 0.003s
I20250113 02:51:36.004101 1198 ts_tablet_manager.cc:1905] T 6666922d677a4c678f80d7c6c47fd2a5 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:36.006978 1195 ts_tablet_manager.cc:1918] T 7213e3284b4d4cb581d9fb946a9f87e9 P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:36.011222 1192 tablet_bootstrap.cc:492] T eebad045e7bb43fa9ca9c9af8b3b7ea5 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:36.011374 1192 ts_tablet_manager.cc:1397] T eebad045e7bb43fa9ca9c9af8b3b7ea5 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.057s user 0.005s sys 0.004s
I20250113 02:51:36.012264 1192 raft_consensus.cc:357] T eebad045e7bb43fa9ca9c9af8b3b7ea5 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:36.012459 1192 raft_consensus.cc:383] T eebad045e7bb43fa9ca9c9af8b3b7ea5 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:36.012534 1192 raft_consensus.cc:738] T eebad045e7bb43fa9ca9c9af8b3b7ea5 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:36.012773 1192 consensus_queue.cc:260] T eebad045e7bb43fa9ca9c9af8b3b7ea5 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:36.016050 2112 raft_consensus.cc:491] T 56b022c94847411785a7e9db3757a2d9 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:36.016189 2112 raft_consensus.cc:513] T 56b022c94847411785a7e9db3757a2d9 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:36.016765 2112 leader_election.cc:290] T 56b022c94847411785a7e9db3757a2d9 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:36.016680 1194 raft_consensus.cc:357] T c508724edea147e5bf6e20bca2f50397 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:36.017335 1194 raft_consensus.cc:383] T c508724edea147e5bf6e20bca2f50397 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:36.018141 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56b022c94847411785a7e9db3757a2d9" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:36.018246 448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_d2b687ff8f614fa3ae9aeb8ca9a54531" } new_table_name: "default.Table_4ccbe1b1eddd4eaf83073751a60c7ba3" modify_external_catalogs: true
W20250113 02:51:36.019752 998 leader_election.cc:343] T 56b022c94847411785a7e9db3757a2d9 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:36.018129 1194 raft_consensus.cc:738] T c508724edea147e5bf6e20bca2f50397 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:36.022403 1194 consensus_queue.cc:260] T c508724edea147e5bf6e20bca2f50397 P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:36.022974 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_e8c6d21d2c8d4abeaca3e0f79c0fc63d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:36.025246 449 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_66000626bf85477d9623d439bad71554" } modify_external_catalogs: true
I20250113 02:51:36.025416 449 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_66000626bf85477d9623d439bad71554" } modify_external_catalogs: true
W20250113 02:51:36.026010 998 consensus_peers.cc:487] T 32cba8f3efc54e78b4df41c15df82eea P 4216a7f853d642b7bc3e5cff458f3aca -> Peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Couldn't send request to peer df69f48656b44bb890f76587ee69cc0f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 32cba8f3efc54e78b4df41c15df82eea. This is attempt 1: this message will repeat every 5th retry.
W20250113 02:51:36.027011 448 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9dc9ba26628441dab7256176bf6a0c3f has no consensus state
I20250113 02:51:36.026970 455 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_635a7ba43392414d91ff119440a6cdc7" } modify_external_catalogs: true
I20250113 02:51:36.027136 455 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_635a7ba43392414d91ff119440a6cdc7" } modify_external_catalogs: true
I20250113 02:51:36.029844 1192 ts_tablet_manager.cc:1428] T eebad045e7bb43fa9ca9c9af8b3b7ea5 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.018s user 0.003s sys 0.000s
W20250113 02:51:36.030314 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e8c6d21d2c8d4abeaca3e0f79c0fc63d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:36.032752 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet eebad045e7bb43fa9ca9c9af8b3b7ea5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:21 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:36.034797 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_923d4e514de74882b7cb333bee6c14b5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:36.035571 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_923d4e514de74882b7cb333bee6c14b5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:36.036927 1192 tablet_bootstrap.cc:492] T e70658c90555463498d3fc2e2f13ea7f P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:36.017758 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56b022c94847411785a7e9db3757a2d9" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
I20250113 02:51:36.038028 1195 log.cc:1198] T 7213e3284b4d4cb581d9fb946a9f87e9 P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/7213e3284b4d4cb581d9fb946a9f87e9
I20250113 02:51:36.038093 774 raft_consensus.cc:2463] T 56b022c94847411785a7e9db3757a2d9 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 0.
I20250113 02:51:36.023643 2072 raft_consensus.cc:491] T dd566a49d3ff4d3889b80d8d2ccbee0e P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:36.029879 820 heartbeater.cc:346] Connected to a master server at 127.0.104.60:42067
I20250113 02:51:36.038501 2072 raft_consensus.cc:513] T dd566a49d3ff4d3889b80d8d2ccbee0e P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:36.039112 2072 leader_election.cc:290] T dd566a49d3ff4d3889b80d8d2ccbee0e P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:36.037899 1191 tablet_bootstrap.cc:492] T 44d2c7d5334c476fac60fdd57e2c02b6 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:36.039964 1191 ts_tablet_manager.cc:1397] T 44d2c7d5334c476fac60fdd57e2c02b6 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.049s user 0.004s sys 0.000s
I20250113 02:51:36.040403 995 leader_election.cc:304] T 56b022c94847411785a7e9db3757a2d9 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters: df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:36.040763 1191 raft_consensus.cc:357] T 44d2c7d5334c476fac60fdd57e2c02b6 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:36.040951 1191 raft_consensus.cc:383] T 44d2c7d5334c476fac60fdd57e2c02b6 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:36.040935 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd566a49d3ff4d3889b80d8d2ccbee0e" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:36.041519 711 leader_election.cc:343] T dd566a49d3ff4d3889b80d8d2ccbee0e P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:36.041002 1191 raft_consensus.cc:738] T 44d2c7d5334c476fac60fdd57e2c02b6 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:36.041761 2112 raft_consensus.cc:2798] T 56b022c94847411785a7e9db3757a2d9 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 02:51:36.041839 2112 raft_consensus.cc:491] T 56b022c94847411785a7e9db3757a2d9 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 02:51:36.041901 2112 raft_consensus.cc:3054] T 56b022c94847411785a7e9db3757a2d9 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:36.041806 1191 consensus_queue.cc:260] T 44d2c7d5334c476fac60fdd57e2c02b6 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
W20250113 02:51:36.045890 998 consensus_peers.cc:487] T d5bd09a1ede644c59e400e49c8125793 P 4216a7f853d642b7bc3e5cff458f3aca -> Peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Couldn't send request to peer df69f48656b44bb890f76587ee69cc0f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d5bd09a1ede644c59e400e49c8125793. This is attempt 1: this message will repeat every 5th retry.
I20250113 02:51:36.046525 1196 ts_tablet_manager.cc:1918] T bffec36d567544c5a1c87e64a7e9730b P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:36.047513 1196 log.cc:1198] T bffec36d567544c5a1c87e64a7e9730b P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/bffec36d567544c5a1c87e64a7e9730b
I20250113 02:51:36.052201 1196 ts_tablet_manager.cc:1939] T bffec36d567544c5a1c87e64a7e9730b P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:36.040565 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd566a49d3ff4d3889b80d8d2ccbee0e" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:36.055860 1191 ts_tablet_manager.cc:1428] T 44d2c7d5334c476fac60fdd57e2c02b6 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.016s user 0.002s sys 0.000s
I20250113 02:51:36.056200 1191 tablet_bootstrap.cc:492] T 1014e51f1c184d729552a5b34292f232 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
W20250113 02:51:36.056268 709 leader_election.cc:343] T dd566a49d3ff4d3889b80d8d2ccbee0e P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:36.056403 709 leader_election.cc:304] T dd566a49d3ff4d3889b80d8d2ccbee0e P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:36.057345 2072 raft_consensus.cc:2743] T dd566a49d3ff4d3889b80d8d2ccbee0e P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:36.058759 1191 tablet_bootstrap.cc:654] T 1014e51f1c184d729552a5b34292f232 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:36.059865 2112 raft_consensus.cc:513] T 56b022c94847411785a7e9db3757a2d9 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:36.061669 2112 leader_election.cc:290] T 56b022c94847411785a7e9db3757a2d9 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Requested vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:36.062446 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet bffec36d567544c5a1c87e64a7e9730b (table default.Table_c79704f27bf14cba8a0afc2dd36a143f [id=f26e79c7d9164cc89d184f1ffbd21036]) successfully deleted
I20250113 02:51:36.063267 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56b022c94847411785a7e9db3757a2d9" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f"
W20250113 02:51:36.065074 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 29c17fd90d774c029a35e77888358067 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 29c17fd90d774c029a35e77888358067 already in progress: creating tablet
I20250113 02:51:36.066095 450 master_service.cc:946] ReplaceTablet: received request to replace tablet cf7b8bd6801743f49e9b9473747447c5 from {username='slave'} at 127.0.0.1:49258
W20250113 02:51:36.066195 998 leader_election.cc:343] T 56b022c94847411785a7e9db3757a2d9 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:36.061036 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56b022c94847411785a7e9db3757a2d9" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196"
I20250113 02:51:36.066511 774 raft_consensus.cc:3054] T 56b022c94847411785a7e9db3757a2d9 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:36.038403 1195 ts_tablet_manager.cc:1939] T 7213e3284b4d4cb581d9fb946a9f87e9 P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:36.067209 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_bccd061fec0d4a9e91a3b126aec48c92"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:36.067736 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bccd061fec0d4a9e91a3b126aec48c92 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:36.067960 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 998) took 1165 ms. Trace:
I20250113 02:51:36.068054 1195 rpcz_store.cc:276] 0113 02:51:34.902079 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:34.902132 (+ 53us) service_pool.cc:225] Handling call
0113 02:51:35.954072 (+1051940us) ts_tablet_manager.cc:1145] Deleting tablet 7213e3284b4d4cb581d9fb946a9f87e9
0113 02:51:35.979195 (+ 25123us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:35.999341 (+ 20146us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:36.006883 (+ 7542us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:36.067939 (+ 61056us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1051788}
I20250113 02:51:36.068529 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet 7213e3284b4d4cb581d9fb946a9f87e9 (table default.Table_ff6df4afcf4d4f9a964d079ee24f02b5 [id=8a5901fc685345579a254a0f3af32051]) successfully deleted
I20250113 02:51:36.069096 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_73f84e2870c44b50b38f55848ef2bbbe" } new_table_name: "default.Table_b0d80c74581c4398a6887b4cd3dcf798" modify_external_catalogs: true
I20250113 02:51:36.070508 774 raft_consensus.cc:2463] T 56b022c94847411785a7e9db3757a2d9 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 1.
I20250113 02:51:36.071264 995 leader_election.cc:304] T 56b022c94847411785a7e9db3757a2d9 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters: df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:36.071909 2113 raft_consensus.cc:2798] T 56b022c94847411785a7e9db3757a2d9 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Leader election won for term 1
I20250113 02:51:36.072084 2113 raft_consensus.cc:695] T 56b022c94847411785a7e9db3757a2d9 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: Becoming Leader. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Running, Role: LEADER
I20250113 02:51:36.072974 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_de89db4ee495473993f7cb9709fcd706"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:36.073477 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_de89db4ee495473993f7cb9709fcd706 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250113 02:51:36.075496 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet b2a23259f2c448cc8a904a13a291d6ca because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b2a23259f2c448cc8a904a13a291d6ca already in progress: creating tablet
W20250113 02:51:36.077443 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet d8e81a6fef794f26bdb017e16af8b30c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d8e81a6fef794f26bdb017e16af8b30c already in progress: creating tablet
I20250113 02:51:36.080781 2113 consensus_queue.cc:237] T 56b022c94847411785a7e9db3757a2d9 P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
W20250113 02:51:36.081817 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 62b71866acc941e8b116ec4192a8e1e7 consensus state has no leader
I20250113 02:51:36.085933 1196 tablet_replica.cc:331] T b0b267b0d9c54ba496c89f6cb287dffd P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:36.086230 1196 raft_consensus.cc:2238] T b0b267b0d9c54ba496c89f6cb287dffd P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:36.088191 1195 tablet_replica.cc:331] T 3025935f9743442eb3e6f2db22eb52ad P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:36.088241 446 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_bc51f0b8366d47bcb5f9e7069597036b" } modify_external_catalogs: true
I20250113 02:51:36.088413 1195 raft_consensus.cc:2238] T 3025935f9743442eb3e6f2db22eb52ad P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:36.088441 446 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_bc51f0b8366d47bcb5f9e7069597036b" } modify_external_catalogs: true
I20250113 02:51:36.088564 1195 raft_consensus.cc:2267] T 3025935f9743442eb3e6f2db22eb52ad P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:36.089150 1195 ts_tablet_manager.cc:1905] T 3025935f9743442eb3e6f2db22eb52ad P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:36.089872 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_e7e2f8efbe58458c968a5a649f789847"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:36.090497 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e7e2f8efbe58458c968a5a649f789847 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:36.092774 453 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_fc6121581e0a4a8c85474adbe1bdbcf8" } modify_external_catalogs: true
I20250113 02:51:36.092942 453 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_fc6121581e0a4a8c85474adbe1bdbcf8" } modify_external_catalogs: true
I20250113 02:51:36.094452 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_9ad8a6a024ee4ff7a664add48a47c662" } new_table_name: "default.Table_4eb24a1814a44375bbb62a777f73f3f8" modify_external_catalogs: true
I20250113 02:51:36.097129 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_67d9e82495c24463ba855129a5e7b1c2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:36.097214 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_3196fc458c3740f2a1af55de808edb06" } new_table_name: "default.Table_375ed1b6d9b640418d69f18ec32cee16" modify_external_catalogs: true
W20250113 02:51:36.097666 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_67d9e82495c24463ba855129a5e7b1c2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:36.100081 452 master_service.cc:946] ReplaceTablet: received request to replace tablet 6770a0cb45fb420d9fbf24d9a5fe4256 from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:36.105327 1192 tablet_bootstrap.cc:654] T e70658c90555463498d3fc2e2f13ea7f P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:36.106247 447 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_69d7dc3f1d7d4161b0a63c85367de424" } modify_external_catalogs: true
I20250113 02:51:36.106447 447 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_69d7dc3f1d7d4161b0a63c85367de424" } modify_external_catalogs: true
W20250113 02:51:36.107223 451 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 70ffabffb8c34677bb484ae3e4fb0394 consensus state has no leader
W20250113 02:51:36.107714 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b7276593af4d48b38e4f5da412fec2b6 consensus state has no leader
I20250113 02:51:36.107615 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_ac317bba2577464095cac3d5fe1374eb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:36.108179 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ac317bba2577464095cac3d5fe1374eb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:36.108268 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_e253836ed32b43b58cf1f7938dadf3dc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:36.108705 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e253836ed32b43b58cf1f7938dadf3dc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:36.113334 1198 ts_tablet_manager.cc:1918] T 6666922d677a4c678f80d7c6c47fd2a5 P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:36.114673 1198 log.cc:1198] T 6666922d677a4c678f80d7c6c47fd2a5 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/6666922d677a4c678f80d7c6c47fd2a5
I20250113 02:51:36.118013 1198 ts_tablet_manager.cc:1939] T 6666922d677a4c678f80d7c6c47fd2a5 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:36.121855 1196 raft_consensus.cc:2267] T b0b267b0d9c54ba496c89f6cb287dffd P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:36.129602 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 958) took 2001 ms. Trace:
W20250113 02:51:36.130396 995 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.1:40095, user_credentials={real_user=slave}} blocked reactor thread for 55712.1us
I20250113 02:51:36.124614 1196 ts_tablet_manager.cc:1905] T b0b267b0d9c54ba496c89f6cb287dffd P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:36.137086 1191 tablet_bootstrap.cc:492] T 1014e51f1c184d729552a5b34292f232 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:36.137413 1191 ts_tablet_manager.cc:1397] T 1014e51f1c184d729552a5b34292f232 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.081s user 0.003s sys 0.000s
I20250113 02:51:36.130630 1198 rpcz_store.cc:276] 0113 02:51:34.128096 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:34.128369 (+ 273us) service_pool.cc:225] Handling call
0113 02:51:35.986205 (+1857836us) ts_tablet_manager.cc:1145] Deleting tablet 6666922d677a4c678f80d7c6c47fd2a5
0113 02:51:35.986300 (+ 95us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:36.062532 (+ 76232us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:36.112743 (+ 50211us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:36.129416 (+ 16673us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1851325}
I20250113 02:51:36.138329 1191 raft_consensus.cc:357] T 1014e51f1c184d729552a5b34292f232 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:36.138613 1191 raft_consensus.cc:383] T 1014e51f1c184d729552a5b34292f232 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:36.138645 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet 6666922d677a4c678f80d7c6c47fd2a5 (table default.Table_03b7491a47674f7187f4952feb9b44e2 [id=6e014400813c4402891b32bcbe3555e7]) successfully deleted
I20250113 02:51:36.138727 1191 raft_consensus.cc:738] T 1014e51f1c184d729552a5b34292f232 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:36.138962 1191 consensus_queue.cc:260] T 1014e51f1c184d729552a5b34292f232 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:36.139542 1191 ts_tablet_manager.cc:1428] T 1014e51f1c184d729552a5b34292f232 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250113 02:51:36.140220 1191 tablet_bootstrap.cc:492] T bbc7832c50114d6f890fec45a3d994e9 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:36.141672 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_4f3d23a03f104bd7acd23395be7568c1" } new_table_name: "default.Table_ba3f5ae52be04b5e88f140ae075c1ee1" modify_external_catalogs: true
I20250113 02:51:36.141750 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_e975ec9af8ab463f911e916f9aec089b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:36.142007 1191 tablet_bootstrap.cc:654] T bbc7832c50114d6f890fec45a3d994e9 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
W20250113 02:51:36.142361 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e975ec9af8ab463f911e916f9aec089b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:36.143513 2112 raft_consensus.cc:491] T eebad045e7bb43fa9ca9c9af8b3b7ea5 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:36.143643 2112 raft_consensus.cc:513] T eebad045e7bb43fa9ca9c9af8b3b7ea5 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:36.144542 2112 leader_election.cc:290] T eebad045e7bb43fa9ca9c9af8b3b7ea5 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:36.144904 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_8d91c57d1c1941d4bf5ea4e798361591"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:36.145529 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8d91c57d1c1941d4bf5ea4e798361591 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:36.146004 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eebad045e7bb43fa9ca9c9af8b3b7ea5" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:36.146517 998 leader_election.cc:343] T eebad045e7bb43fa9ca9c9af8b3b7ea5 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:36.146553 1195 ts_tablet_manager.cc:1918] T 3025935f9743442eb3e6f2db22eb52ad P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:36.146641 1195 log.cc:1198] T 3025935f9743442eb3e6f2db22eb52ad P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/3025935f9743442eb3e6f2db22eb52ad
W20250113 02:51:36.151880 995 leader_election.cc:343] T eebad045e7bb43fa9ca9c9af8b3b7ea5 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Not found: Tablet not found: eebad045e7bb43fa9ca9c9af8b3b7ea5
I20250113 02:51:36.152093 995 leader_election.cc:304] T eebad045e7bb43fa9ca9c9af8b3b7ea5 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4216a7f853d642b7bc3e5cff458f3aca; no voters: 1186fa869eb14812832fa6562531f196, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:36.152673 2112 raft_consensus.cc:2743] T eebad045e7bb43fa9ca9c9af8b3b7ea5 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:36.153334 1198 tablet_replica.cc:331] T f56230a2c75d4763a54671c79358f4d9 P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:36.153646 1198 raft_consensus.cc:2238] T f56230a2c75d4763a54671c79358f4d9 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: Raft consensus shutting down.
W20250113 02:51:36.155192 451 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 43c78b3da74748158f5534203bdd0d33 has no consensus state
I20250113 02:51:36.156584 1198 pending_rounds.cc:62] T f56230a2c75d4763a54671c79358f4d9 P 4216a7f853d642b7bc3e5cff458f3aca: Trying to abort 1 pending ops.
I20250113 02:51:36.156944 1198 pending_rounds.cc:69] T f56230a2c75d4763a54671c79358f4d9 P 4216a7f853d642b7bc3e5cff458f3aca: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113673499593441280 op_type: NO_OP noop_request { }
I20250113 02:51:36.157650 1198 raft_consensus.cc:2883] T f56230a2c75d4763a54671c79358f4d9 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250113 02:51:36.158000 1198 raft_consensus.cc:2267] T f56230a2c75d4763a54671c79358f4d9 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:36.159408 1198 ts_tablet_manager.cc:1905] T f56230a2c75d4763a54671c79358f4d9 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:36.161375 1192 tablet_bootstrap.cc:492] T e70658c90555463498d3fc2e2f13ea7f P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:36.161656 1192 ts_tablet_manager.cc:1397] T e70658c90555463498d3fc2e2f13ea7f P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.125s user 0.006s sys 0.001s
I20250113 02:51:36.163031 1196 ts_tablet_manager.cc:1918] T b0b267b0d9c54ba496c89f6cb287dffd P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:36.163213 1196 log.cc:1198] T b0b267b0d9c54ba496c89f6cb287dffd P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/b0b267b0d9c54ba496c89f6cb287dffd
I20250113 02:51:36.163558 1196 ts_tablet_manager.cc:1939] T b0b267b0d9c54ba496c89f6cb287dffd P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:36.163118 1192 raft_consensus.cc:357] T e70658c90555463498d3fc2e2f13ea7f P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:36.163789 1192 raft_consensus.cc:383] T e70658c90555463498d3fc2e2f13ea7f P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:36.164083 1192 raft_consensus.cc:738] T e70658c90555463498d3fc2e2f13ea7f P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:36.164263 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet b0b267b0d9c54ba496c89f6cb287dffd (table default.Table_eaa71690c1224509a1b527415d09f340 [id=dae9b614d621428ab168fe6a978799a0]) successfully deleted
W20250113 02:51:36.164933 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet cb897004cbfb4a0283b42d06509a9ee5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cb897004cbfb4a0283b42d06509a9ee5 already in progress: creating tablet
I20250113 02:51:36.165251 1196 tablet_replica.cc:331] T 4c615b0b83d74fcfae0c51047dbfbbf4 P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:36.164743 1192 consensus_queue.cc:260] T e70658c90555463498d3fc2e2f13ea7f P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:36.165518 1196 raft_consensus.cc:2238] T 4c615b0b83d74fcfae0c51047dbfbbf4 P df69f48656b44bb890f76587ee69cc0f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:36.165726 1196 raft_consensus.cc:2267] T 4c615b0b83d74fcfae0c51047dbfbbf4 P df69f48656b44bb890f76587ee69cc0f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:36.166805 1196 ts_tablet_manager.cc:1905] T 4c615b0b83d74fcfae0c51047dbfbbf4 P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:36.168269 1195 ts_tablet_manager.cc:1939] T 3025935f9743442eb3e6f2db22eb52ad P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:36.169049 1192 ts_tablet_manager.cc:1428] T e70658c90555463498d3fc2e2f13ea7f P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20250113 02:51:36.170594 1192 tablet_bootstrap.cc:492] T 3025935f9743442eb3e6f2db22eb52ad P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:36.173456 1191 tablet_bootstrap.cc:492] T bbc7832c50114d6f890fec45a3d994e9 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:36.173650 1191 ts_tablet_manager.cc:1397] T bbc7832c50114d6f890fec45a3d994e9 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.033s user 0.003s sys 0.000s
I20250113 02:51:36.174949 1191 raft_consensus.cc:357] T bbc7832c50114d6f890fec45a3d994e9 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:36.175351 1191 raft_consensus.cc:383] T bbc7832c50114d6f890fec45a3d994e9 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:36.175459 1191 raft_consensus.cc:738] T bbc7832c50114d6f890fec45a3d994e9 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:36.175717 1191 consensus_queue.cc:260] T bbc7832c50114d6f890fec45a3d994e9 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:36.176293 1191 ts_tablet_manager.cc:1428] T bbc7832c50114d6f890fec45a3d994e9 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250113 02:51:36.184931 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 999) took 1282 ms. Trace:
I20250113 02:51:36.185077 1195 rpcz_store.cc:276] 0113 02:51:34.902569 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:34.902616 (+ 47us) service_pool.cc:225] Handling call
0113 02:51:36.088138 (+1185522us) ts_tablet_manager.cc:1145] Deleting tablet 3025935f9743442eb3e6f2db22eb52ad
0113 02:51:36.088150 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:36.143490 (+ 55340us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:36.146472 (+ 2982us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:36.184914 (+ 38442us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1185371}
I20250113 02:51:36.185568 1192 tablet_bootstrap.cc:654] T 3025935f9743442eb3e6f2db22eb52ad P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:36.187834 1196 ts_tablet_manager.cc:1918] T 4c615b0b83d74fcfae0c51047dbfbbf4 P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250113 02:51:36.188017 1196 log.cc:1198] T 4c615b0b83d74fcfae0c51047dbfbbf4 P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/4c615b0b83d74fcfae0c51047dbfbbf4
I20250113 02:51:36.188391 1196 ts_tablet_manager.cc:1939] T 4c615b0b83d74fcfae0c51047dbfbbf4 P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:36.189082 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet 4c615b0b83d74fcfae0c51047dbfbbf4 (table default.Table_bf689f8eb8884e88a72e9b50598fc389 [id=a956f95f6dd94b578f6bd3cb3b97f9da]) successfully deleted
I20250113 02:51:36.193583 1191 tablet_bootstrap.cc:492] T bc6a7322c0c34adb85457750526d8955 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:36.195005 1191 tablet_bootstrap.cc:654] T bc6a7322c0c34adb85457750526d8955 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
W20250113 02:51:36.199299 995 consensus_peers.cc:487] T 377ebf66bbef40988043f6200ae0893c P 4216a7f853d642b7bc3e5cff458f3aca -> Peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Couldn't send request to peer 1186fa869eb14812832fa6562531f196. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 377ebf66bbef40988043f6200ae0893c. This is attempt 1: this message will repeat every 5th retry.
W20250113 02:51:36.209522 998 consensus_peers.cc:487] T 4c615b0b83d74fcfae0c51047dbfbbf4 P 4216a7f853d642b7bc3e5cff458f3aca -> Peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Couldn't send request to peer df69f48656b44bb890f76587ee69cc0f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4c615b0b83d74fcfae0c51047dbfbbf4. This is attempt 1: this message will repeat every 5th retry.
I20250113 02:51:36.211620 1194 ts_tablet_manager.cc:1428] T c508724edea147e5bf6e20bca2f50397 P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.208s user 0.016s sys 0.003s
I20250113 02:51:36.217271 454 catalog_manager.cc:5618] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467) does not have the latest schema for tablet c508724edea147e5bf6e20bca2f50397 (table default.Table_810d62357c794aeaa18da712725a338f [id=08db71e435a24e36bdb20f9a5f1cc096]). Expected version 3 got 0
I20250113 02:51:36.217432 1191 tablet_bootstrap.cc:492] T bc6a7322c0c34adb85457750526d8955 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:36.217686 1191 ts_tablet_manager.cc:1397] T bc6a7322c0c34adb85457750526d8955 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.024s user 0.003s sys 0.000s
I20250113 02:51:36.218482 1191 raft_consensus.cc:357] T bc6a7322c0c34adb85457750526d8955 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:36.218729 1191 raft_consensus.cc:383] T bc6a7322c0c34adb85457750526d8955 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:36.218818 1191 raft_consensus.cc:738] T bc6a7322c0c34adb85457750526d8955 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:36.219064 1191 consensus_queue.cc:260] T bc6a7322c0c34adb85457750526d8955 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:36.219586 1191 ts_tablet_manager.cc:1428] T bc6a7322c0c34adb85457750526d8955 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250113 02:51:36.219916 1191 tablet_bootstrap.cc:492] T 80d31d5ead804ddaa032969e401a8f7e P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:36.220850 451 catalog_manager.cc:5526] T 56b022c94847411785a7e9db3757a2d9 P 4216a7f853d642b7bc3e5cff458f3aca reported cstate change: term changed from 0 to 1, leader changed from <none> to 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3). New cstate: current_term: 1 leader_uuid: "4216a7f853d642b7bc3e5cff458f3aca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } health_report { overall_health: HEALTHY } } }
I20250113 02:51:36.221261 1191 tablet_bootstrap.cc:654] T 80d31d5ead804ddaa032969e401a8f7e P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:36.221391 451 catalog_manager.cc:5618] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587) does not have the latest schema for tablet 56b022c94847411785a7e9db3757a2d9 (table default.Table_6461b062100240d59a5d91a87a74510c [id=40ca872c045b4349b63ba9046e31ed13]). Expected version 2 got 0
I20250113 02:51:36.231570 742 tablet_service.cc:1514] Processing DeleteTablet for tablet 1014e51f1c184d729552a5b34292f232 with delete_type TABLET_DATA_DELETED (replaced by tablet ec8196505dab4e05bb6487b566cf3144) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:36.233937 2072 raft_consensus.cc:491] T f21eb3fa0f414816a29e3d4560cdc0fe P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:36.234074 2072 raft_consensus.cc:513] T f21eb3fa0f414816a29e3d4560cdc0fe P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:36.234632 2072 leader_election.cc:290] T f21eb3fa0f414816a29e3d4560cdc0fe P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:36.235282 454 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_4e3ba1ddebc74043b121c831b4abacd8" } new_table_name: "default.Table_ea862ea580c043c5bd2f56522b97897d" modify_external_catalogs: true
I20250113 02:51:36.236128 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f21eb3fa0f414816a29e3d4560cdc0fe" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
W20250113 02:51:36.236913 709 leader_election.cc:343] T f21eb3fa0f414816a29e3d4560cdc0fe P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:36.240517 450 master_service.cc:946] ReplaceTablet: received request to replace tablet 9fa6a1c4f6ef4f98b4de114210af08c5 from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:36.224828 2195 raft_consensus.cc:491] T 8337f3c82d654ecdac2cbc1ffa88ef05 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:36.241279 2195 raft_consensus.cc:513] T 8337f3c82d654ecdac2cbc1ffa88ef05 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:36.241950 2195 leader_election.cc:290] T 8337f3c82d654ecdac2cbc1ffa88ef05 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587), 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
I20250113 02:51:36.244499 1191 tablet_bootstrap.cc:492] T 80d31d5ead804ddaa032969e401a8f7e P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:36.244613 1191 ts_tablet_manager.cc:1397] T 80d31d5ead804ddaa032969e401a8f7e P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.025s user 0.001s sys 0.001s
I20250113 02:51:36.245285 1191 raft_consensus.cc:357] T 80d31d5ead804ddaa032969e401a8f7e P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:36.245446 1191 raft_consensus.cc:383] T 80d31d5ead804ddaa032969e401a8f7e P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:36.245486 1191 raft_consensus.cc:738] T 80d31d5ead804ddaa032969e401a8f7e P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:36.245687 1191 consensus_queue.cc:260] T 80d31d5ead804ddaa032969e401a8f7e P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:36.246142 1191 ts_tablet_manager.cc:1428] T 80d31d5ead804ddaa032969e401a8f7e P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20250113 02:51:36.246655 742 tablet_service.cc:1514] Processing DeleteTablet for tablet bbc7832c50114d6f890fec45a3d994e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:22 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:36.248150 1191 tablet_bootstrap.cc:492] T c7842d6f2c074e3b9d1a3378db9856be P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:36.248587 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8337f3c82d654ecdac2cbc1ffa88ef05" candidate_uuid: "df69f48656b44bb890f76587ee69cc0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:36.251396 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 39a9c90677964441acb1e59d16bd1505
I20250113 02:51:36.252838 881 tablet_service.cc:1514] Processing DeleteTablet for tablet 39a9c90677964441acb1e59d16bd1505 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:36 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:36.253029 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet 39a9c90677964441acb1e59d16bd1505 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:36 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:36.253161 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 9938f5866a6149e3a1456b03389064b2
I20250113 02:51:36.253896 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet 9938f5866a6149e3a1456b03389064b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:36 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:36.253964 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_a8de05b02a0342898b051d8c7078f835"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:36.254546 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a8de05b02a0342898b051d8c7078f835 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250113 02:51:36.255895 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 39a9c90677964441acb1e59d16bd1505 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 39a9c90677964441acb1e59d16bd1505 already in progress: creating tablet
I20250113 02:51:36.256407 881 tablet_service.cc:1514] Processing DeleteTablet for tablet 9938f5866a6149e3a1456b03389064b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:36 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:36.256958 1191 tablet_bootstrap.cc:654] T c7842d6f2c074e3b9d1a3378db9856be P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:36.257434 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 3178924445d64fea89a9721be7d241e9
I20250113 02:51:36.258208 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet 3178924445d64fea89a9721be7d241e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:36 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:36.258932 736 tablet_service.cc:1514] Processing DeleteTablet for tablet 9938f5866a6149e3a1456b03389064b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:36 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:36.259145 746 tablet_service.cc:1514] Processing DeleteTablet for tablet 3178924445d64fea89a9721be7d241e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:36 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:36.259308 742 tablet_service.cc:1514] Processing DeleteTablet for tablet 39a9c90677964441acb1e59d16bd1505 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:36 UTC) from {username='slave'} at 127.0.0.1:55398
W20250113 02:51:36.260263 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 9938f5866a6149e3a1456b03389064b2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9938f5866a6149e3a1456b03389064b2 already in progress: creating tablet
I20250113 02:51:36.260659 881 tablet_service.cc:1514] Processing DeleteTablet for tablet 3178924445d64fea89a9721be7d241e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:36 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:36.261709 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_3525402d375e4d5fa1b2c0f61e27a40e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:36.262257 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3525402d375e4d5fa1b2c0f61e27a40e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250113 02:51:36.263437 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 3178924445d64fea89a9721be7d241e9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3178924445d64fea89a9721be7d241e9 already in progress: creating tablet
I20250113 02:51:36.263938 1194 tablet_bootstrap.cc:492] T 42fad91fe1b64a21a2c8412a2585af47 P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:36.266908 881 tablet_service.cc:1467] Processing CreateTablet for tablet 50c4d4783e6d4a55bd3bd355613e1ea0 (DEFAULT_TABLE table=default.Table_86e441512733438dae0893de5acd5c92 [id=41f97448164d4137896c83b0246dc272]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:36.267284 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 50c4d4783e6d4a55bd3bd355613e1ea0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:36.270128 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet f3771f0fe7ff4d54b433640c76bc07ae
I20250113 02:51:36.270903 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet ddf0295c286340bcaea28a1468f3c557
I20250113 02:51:36.271517 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 6770a0cb45fb420d9fbf24d9a5fe4256
I20250113 02:51:36.271857 452 catalog_manager.cc:6596] ReplaceTablet: tablet 6770a0cb45fb420d9fbf24d9a5fe4256 deleted and replaced by tablet 189dd9665f754a5d858dc9ffc227440d
I20250113 02:51:36.272770 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet ddf0295c286340bcaea28a1468f3c557 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:36 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:36.273018 1041 tablet_service.cc:1514] Processing DeleteTablet for tablet 6770a0cb45fb420d9fbf24d9a5fe4256 with delete_type TABLET_DATA_DELETED (replaced by tablet 189dd9665f754a5d858dc9ffc227440d) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:36.273185 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet f3771f0fe7ff4d54b433640c76bc07ae with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:36 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:36.273785 746 tablet_service.cc:1514] Processing DeleteTablet for tablet ddf0295c286340bcaea28a1468f3c557 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:36 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:36.274004 736 tablet_service.cc:1514] Processing DeleteTablet for tablet 6770a0cb45fb420d9fbf24d9a5fe4256 with delete_type TABLET_DATA_DELETED (replaced by tablet 189dd9665f754a5d858dc9ffc227440d) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:36.274132 742 tablet_service.cc:1514] Processing DeleteTablet for tablet f3771f0fe7ff4d54b433640c76bc07ae with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:36 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:36.275621 455 catalog_manager.cc:5618] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467) does not have the latest schema for tablet c508724edea147e5bf6e20bca2f50397 (table default.Table_810d62357c794aeaa18da712725a338f [id=08db71e435a24e36bdb20f9a5f1cc096]). Expected version 3 got 0
I20250113 02:51:36.277004 885 tablet_service.cc:1514] Processing DeleteTablet for tablet f3771f0fe7ff4d54b433640c76bc07ae with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:36 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:36.277243 887 tablet_service.cc:1514] Processing DeleteTablet for tablet ddf0295c286340bcaea28a1468f3c557 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:36 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:36.277360 917 raft_consensus.cc:3054] T c508724edea147e5bf6e20bca2f50397 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:36.279878 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet 3025935f9743442eb3e6f2db22eb52ad (table default.Table_669ef67f4e884a939ff3b3a0bc409ce3 [id=c0b628d230fd4eae83a20530cd998993]) successfully deleted
I20250113 02:51:36.277379 892 tablet_service.cc:1514] Processing DeleteTablet for tablet 6770a0cb45fb420d9fbf24d9a5fe4256 with delete_type TABLET_DATA_DELETED (replaced by tablet 189dd9665f754a5d858dc9ffc227440d) from {username='slave'} at 127.0.0.1:37366
W20250113 02:51:36.282159 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet f3771f0fe7ff4d54b433640c76bc07ae because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f3771f0fe7ff4d54b433640c76bc07ae already in progress: creating tablet
W20250113 02:51:36.282486 442 catalog_manager.cc:4839] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet ddf0295c286340bcaea28a1468f3c557 because the tablet was not found. No further retry: Not found: Tablet not found: ddf0295c286340bcaea28a1468f3c557
W20250113 02:51:36.282815 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 6770a0cb45fb420d9fbf24d9a5fe4256 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6770a0cb45fb420d9fbf24d9a5fe4256 already in progress: creating tablet
I20250113 02:51:36.283308 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_33d55ba79fc14c53be88aac64de3b16a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:36.283851 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_33d55ba79fc14c53be88aac64de3b16a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250113 02:51:36.285441 441 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet c04be975c8c049b892b16129ed47a142 (table default.Table_f98f15e1cb6a43348c4438096a7f9ea4 [id=36184fc3b73745c487d1d7a9a69e85ef], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet c04be975c8c049b892b16129ed47a142 consensus state has no leader
I20250113 02:51:36.286302 742 tablet_service.cc:1467] Processing CreateTablet for tablet 50c4d4783e6d4a55bd3bd355613e1ea0 (DEFAULT_TABLE table=default.Table_86e441512733438dae0893de5acd5c92 [id=41f97448164d4137896c83b0246dc272]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:36.286720 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 50c4d4783e6d4a55bd3bd355613e1ea0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:36.288832 1026 tablet_service.cc:1467] Processing CreateTablet for tablet 50c4d4783e6d4a55bd3bd355613e1ea0 (DEFAULT_TABLE table=default.Table_86e441512733438dae0893de5acd5c92 [id=41f97448164d4137896c83b0246dc272]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:36.289207 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 50c4d4783e6d4a55bd3bd355613e1ea0. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:36.292416 454 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f748397a164d42e3a7fda3585673e633 consensus state has no leader
I20250113 02:51:36.293620 1191 tablet_bootstrap.cc:492] T c7842d6f2c074e3b9d1a3378db9856be P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:36.293829 1191 ts_tablet_manager.cc:1397] T c7842d6f2c074e3b9d1a3378db9856be P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.046s user 0.006s sys 0.000s
I20250113 02:51:36.293959 1195 tablet_replica.cc:331] T d5bd09a1ede644c59e400e49c8125793 P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:36.294170 1195 raft_consensus.cc:2238] T d5bd09a1ede644c59e400e49c8125793 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:36.294396 1195 raft_consensus.cc:2267] T d5bd09a1ede644c59e400e49c8125793 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:36.294972 1195 ts_tablet_manager.cc:1905] T d5bd09a1ede644c59e400e49c8125793 P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:36.294860 1191 raft_consensus.cc:357] T c7842d6f2c074e3b9d1a3378db9856be P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:36.295176 1191 raft_consensus.cc:383] T c7842d6f2c074e3b9d1a3378db9856be P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:36.295171 1041 tablet_service.cc:1514] Processing DeleteTablet for tablet e70658c90555463498d3fc2e2f13ea7f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:23 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:36.295266 1191 raft_consensus.cc:738] T c7842d6f2c074e3b9d1a3378db9856be P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:36.295280 452 master_service.cc:946] ReplaceTablet: received request to replace tablet d7193e2799a54950902c4ceac65b663b from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:36.295560 1191 consensus_queue.cc:260] T c7842d6f2c074e3b9d1a3378db9856be P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:36.296206 1191 ts_tablet_manager.cc:1428] T c7842d6f2c074e3b9d1a3378db9856be P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250113 02:51:36.296509 1191 tablet_bootstrap.cc:492] T 655b63ff61ff4f4aa08e0449df39352a P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:36.297320 1194 tablet_bootstrap.cc:654] T 42fad91fe1b64a21a2c8412a2585af47 P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:36.297739 1191 tablet_bootstrap.cc:654] T 655b63ff61ff4f4aa08e0449df39352a P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:36.298812 736 tablet_service.cc:1467] Processing CreateTablet for tablet 184ea5b63c63467e816dac8bd2a51f80 (DEFAULT_TABLE table=default.Table_d0763569196645ab8a5c97710bb3600d [id=2e9f8e28d41a4553bf11e8397d45d4d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:36.299167 736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 184ea5b63c63467e816dac8bd2a51f80. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:36.299382 454 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_d382199622cd467494ab3cab7c491b25" } new_table_name: "default.Table_0874a35f3d5a44d2ac045a85227b148d" modify_external_catalogs: true
I20250113 02:51:36.303767 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_f5b4d64c86db49829574aea1b88e3968"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:36.303997 1026 tablet_service.cc:1467] Processing CreateTablet for tablet 184ea5b63c63467e816dac8bd2a51f80 (DEFAULT_TABLE table=default.Table_d0763569196645ab8a5c97710bb3600d [id=2e9f8e28d41a4553bf11e8397d45d4d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:36.304435 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 184ea5b63c63467e816dac8bd2a51f80. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:36.304507 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f5b4d64c86db49829574aea1b88e3968 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:36.309494 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_5d54936883fa4a068a24a069ad10dafb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:36.310139 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5d54936883fa4a068a24a069ad10dafb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:36.301811 892 tablet_service.cc:1467] Processing CreateTablet for tablet 184ea5b63c63467e816dac8bd2a51f80 (DEFAULT_TABLE table=default.Table_d0763569196645ab8a5c97710bb3600d [id=2e9f8e28d41a4553bf11e8397d45d4d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:36.313517 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 184ea5b63c63467e816dac8bd2a51f80. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:36.315368 746 tablet_service.cc:1467] Processing CreateTablet for tablet 72604b4925b44943a74a98a38ac66ea6 (DEFAULT_TABLE table=default.Table_2607e08cf8a043f7aaa54cc016d105b2 [id=6933ba93d9b34e94bd8f843ca7878a8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:36.315724 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 72604b4925b44943a74a98a38ac66ea6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:36.316552 1195 ts_tablet_manager.cc:1918] T d5bd09a1ede644c59e400e49c8125793 P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250113 02:51:36.317495 1195 log.cc:1198] T d5bd09a1ede644c59e400e49c8125793 P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/d5bd09a1ede644c59e400e49c8125793
I20250113 02:51:36.324020 1195 ts_tablet_manager.cc:1939] T d5bd09a1ede644c59e400e49c8125793 P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:36.327435 1041 tablet_service.cc:1467] Processing CreateTablet for tablet 72604b4925b44943a74a98a38ac66ea6 (DEFAULT_TABLE table=default.Table_2607e08cf8a043f7aaa54cc016d105b2 [id=6933ba93d9b34e94bd8f843ca7878a8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:36.327831 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 72604b4925b44943a74a98a38ac66ea6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:36.330971 1191 tablet_bootstrap.cc:492] T 655b63ff61ff4f4aa08e0449df39352a P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:36.331189 1191 ts_tablet_manager.cc:1397] T 655b63ff61ff4f4aa08e0449df39352a P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.035s user 0.006s sys 0.007s
I20250113 02:51:36.331947 1191 raft_consensus.cc:357] T 655b63ff61ff4f4aa08e0449df39352a P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:36.332227 1191 raft_consensus.cc:383] T 655b63ff61ff4f4aa08e0449df39352a P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:36.332314 1191 raft_consensus.cc:738] T 655b63ff61ff4f4aa08e0449df39352a P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:36.332547 1191 consensus_queue.cc:260] T 655b63ff61ff4f4aa08e0449df39352a P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:36.327059 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_367f2846b81e40eb82c5dd077f36017c" } new_table_name: "default.Table_6daec99b05744d0caed86e51e86f24c2" modify_external_catalogs: true
I20250113 02:51:36.336269 881 tablet_service.cc:1467] Processing CreateTablet for tablet 72604b4925b44943a74a98a38ac66ea6 (DEFAULT_TABLE table=default.Table_2607e08cf8a043f7aaa54cc016d105b2 [id=6933ba93d9b34e94bd8f843ca7878a8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:36.336676 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 72604b4925b44943a74a98a38ac66ea6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:36.337019 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1000) took 1255 ms. Trace:
I20250113 02:51:36.337289 1195 rpcz_store.cc:276] 0113 02:51:35.081616 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:35.082042 (+ 426us) service_pool.cc:225] Handling call
0113 02:51:36.293912 (+1211870us) ts_tablet_manager.cc:1145] Deleting tablet d5bd09a1ede644c59e400e49c8125793
0113 02:51:36.293922 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:36.303702 (+ 9780us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:36.316443 (+ 12741us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:36.336999 (+ 20556us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1211674}
I20250113 02:51:36.337903 1195 tablet_replica.cc:331] T 32cba8f3efc54e78b4df41c15df82eea P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:36.338204 1195 raft_consensus.cc:2238] T 32cba8f3efc54e78b4df41c15df82eea P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:36.338467 1195 raft_consensus.cc:2267] T 32cba8f3efc54e78b4df41c15df82eea P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:36.339056 1195 ts_tablet_manager.cc:1905] T 32cba8f3efc54e78b4df41c15df82eea P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:36.340879 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet d5bd09a1ede644c59e400e49c8125793 (table default.Table_bee5bcef1f534c6c812e6075291ff90d [id=e0ced6f3dc0049eaa0e76974916ff63c]) successfully deleted
I20250113 02:51:36.344934 742 tablet_service.cc:1514] Processing DeleteTablet for tablet c7842d6f2c074e3b9d1a3378db9856be with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:24 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:36.349992 746 tablet_service.cc:1514] Processing DeleteTablet for tablet 80d31d5ead804ddaa032969e401a8f7e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:22 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:36.350821 1191 ts_tablet_manager.cc:1428] T 655b63ff61ff4f4aa08e0449df39352a P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.020s user 0.004s sys 0.004s
I20250113 02:51:36.351286 1191 tablet_bootstrap.cc:492] T c518f791b4d74c54ab0c0364d29b4a64 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:36.352893 1195 ts_tablet_manager.cc:1918] T 32cba8f3efc54e78b4df41c15df82eea P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20250113 02:51:36.352950 1191 tablet_bootstrap.cc:654] T c518f791b4d74c54ab0c0364d29b4a64 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:36.353461 1195 log.cc:1198] T 32cba8f3efc54e78b4df41c15df82eea P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/32cba8f3efc54e78b4df41c15df82eea
I20250113 02:51:36.353906 1195 ts_tablet_manager.cc:1939] T 32cba8f3efc54e78b4df41c15df82eea P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:36.356890 448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_402477cbe689499490baedf0ede9910f" } new_table_name: "default.Table_c90f3a5c35c842f6ad2cb74cf29dac13" modify_external_catalogs: true
I20250113 02:51:36.361114 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 9fa6a1c4f6ef4f98b4de114210af08c5
I20250113 02:51:36.361653 450 catalog_manager.cc:6596] ReplaceTablet: tablet 9fa6a1c4f6ef4f98b4de114210af08c5 deleted and replaced by tablet 7da6f8a4b89647968609f73f9ad5ba14
I20250113 02:51:36.364141 736 tablet_service.cc:1514] Processing DeleteTablet for tablet 9fa6a1c4f6ef4f98b4de114210af08c5 with delete_type TABLET_DATA_DELETED (replaced by tablet 7da6f8a4b89647968609f73f9ad5ba14) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:36.364558 892 tablet_service.cc:1514] Processing DeleteTablet for tablet 9fa6a1c4f6ef4f98b4de114210af08c5 with delete_type TABLET_DATA_DELETED (replaced by tablet 7da6f8a4b89647968609f73f9ad5ba14) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:36.365204 1191 tablet_bootstrap.cc:492] T c518f791b4d74c54ab0c0364d29b4a64 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
W20250113 02:51:36.365327 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 9fa6a1c4f6ef4f98b4de114210af08c5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9fa6a1c4f6ef4f98b4de114210af08c5 already in progress: creating tablet
I20250113 02:51:36.365397 1191 ts_tablet_manager.cc:1397] T c518f791b4d74c54ab0c0364d29b4a64 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.002s
I20250113 02:51:36.366138 1191 raft_consensus.cc:357] T c518f791b4d74c54ab0c0364d29b4a64 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:36.366380 1191 raft_consensus.cc:383] T c518f791b4d74c54ab0c0364d29b4a64 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:36.366477 1191 raft_consensus.cc:738] T c518f791b4d74c54ab0c0364d29b4a64 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:36.366700 1191 consensus_queue.cc:260] T c518f791b4d74c54ab0c0364d29b4a64 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:36.367229 1191 ts_tablet_manager.cc:1428] T c518f791b4d74c54ab0c0364d29b4a64 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250113 02:51:36.368072 1191 tablet_bootstrap.cc:492] T ee240d838d4a4bbba97f8ed61f69de78 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:36.372288 1191 tablet_bootstrap.cc:654] T ee240d838d4a4bbba97f8ed61f69de78 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:36.373613 1041 tablet_service.cc:1514] Processing DeleteTablet for tablet 9fa6a1c4f6ef4f98b4de114210af08c5 with delete_type TABLET_DATA_DELETED (replaced by tablet 7da6f8a4b89647968609f73f9ad5ba14) from {username='slave'} at 127.0.0.1:60230
W20250113 02:51:36.381042 711 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.2:38467, user_credentials={real_user=slave}} blocked reactor thread for 148419us
I20250113 02:51:36.381757 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f21eb3fa0f414816a29e3d4560cdc0fe" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:36.382683 711 leader_election.cc:343] T f21eb3fa0f414816a29e3d4560cdc0fe P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:36.383811 711 leader_election.cc:304] T f21eb3fa0f414816a29e3d4560cdc0fe P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:36.384301 2072 raft_consensus.cc:2743] T f21eb3fa0f414816a29e3d4560cdc0fe P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:36.385196 1198 ts_tablet_manager.cc:1918] T f56230a2c75d4763a54671c79358f4d9 P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250113 02:51:36.385475 1198 log.cc:1198] T f56230a2c75d4763a54671c79358f4d9 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/f56230a2c75d4763a54671c79358f4d9
I20250113 02:51:36.386518 1198 ts_tablet_manager.cc:1939] T f56230a2c75d4763a54671c79358f4d9 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:36.386713 1191 tablet_bootstrap.cc:492] T ee240d838d4a4bbba97f8ed61f69de78 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:36.386910 1191 ts_tablet_manager.cc:1397] T ee240d838d4a4bbba97f8ed61f69de78 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.001s
I20250113 02:51:36.387138 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 956) took 2259 ms. Trace:
I20250113 02:51:36.387368 1198 rpcz_store.cc:276] 0113 02:51:34.127941 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:34.145776 (+ 17835us) service_pool.cc:225] Handling call
0113 02:51:36.138699 (+1992923us) ts_tablet_manager.cc:1145] Deleting tablet f56230a2c75d4763a54671c79358f4d9
0113 02:51:36.153272 (+ 14573us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:36.188527 (+ 35255us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:36.207492 (+ 18965us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:36.387115 (+179623us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1992646}
I20250113 02:51:36.387756 1191 raft_consensus.cc:357] T ee240d838d4a4bbba97f8ed61f69de78 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:36.388046 1191 raft_consensus.cc:383] T ee240d838d4a4bbba97f8ed61f69de78 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:36.388156 1191 raft_consensus.cc:738] T ee240d838d4a4bbba97f8ed61f69de78 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:36.388181 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet f56230a2c75d4763a54671c79358f4d9 (table default.Table_fb08fe0ed5cd4aaebce3169c264441cb [id=34b84bee300d4b30b1d426b9f26e6e4a]) successfully deleted
I20250113 02:51:36.388391 1191 consensus_queue.cc:260] T ee240d838d4a4bbba97f8ed61f69de78 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:36.389046 1198 tablet_replica.cc:331] T f129bb3fbbfd492d98822be0a6626e44 P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:36.389065 1191 ts_tablet_manager.cc:1428] T ee240d838d4a4bbba97f8ed61f69de78 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250113 02:51:36.389321 1191 tablet_bootstrap.cc:492] T b5c1882125b5491ea79a29a6dac6771e P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:36.389405 1198 raft_consensus.cc:2238] T f129bb3fbbfd492d98822be0a6626e44 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:36.393400 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_24bc600591b2484591266b7da6c66ce0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:36.394030 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_24bc600591b2484591266b7da6c66ce0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:36.389870 1198 raft_consensus.cc:2267] T f129bb3fbbfd492d98822be0a6626e44 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:36.395560 1192 tablet_bootstrap.cc:492] T 3025935f9743442eb3e6f2db22eb52ad P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:36.398196 1192 ts_tablet_manager.cc:1397] T 3025935f9743442eb3e6f2db22eb52ad P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.228s user 0.008s sys 0.000s
I20250113 02:51:36.399402 1191 tablet_bootstrap.cc:654] T b5c1882125b5491ea79a29a6dac6771e P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:36.401616 1192 raft_consensus.cc:357] T 3025935f9743442eb3e6f2db22eb52ad P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:36.402243 1192 raft_consensus.cc:383] T 3025935f9743442eb3e6f2db22eb52ad P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:36.402722 1192 raft_consensus.cc:738] T 3025935f9743442eb3e6f2db22eb52ad P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:36.408512 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_d795516e6fbc46f5b2fe2b81042b1d62"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:36.409063 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d795516e6fbc46f5b2fe2b81042b1d62 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:36.409665 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1002) took 1214 ms. Trace:
I20250113 02:51:36.409750 1195 rpcz_store.cc:276] 0113 02:51:35.195291 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:35.195346 (+ 55us) service_pool.cc:225] Handling call
0113 02:51:36.337857 (+1142511us) ts_tablet_manager.cc:1145] Deleting tablet 32cba8f3efc54e78b4df41c15df82eea
0113 02:51:36.337867 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:36.349893 (+ 12026us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:36.352812 (+ 2919us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:36.409642 (+ 56830us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1142328}
I20250113 02:51:36.410312 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet 32cba8f3efc54e78b4df41c15df82eea (table default.Table_3d65267002364c8a9175ccb2223cac44 [id=8aff13d6d3c5483eadb596b705fe5151]) successfully deleted
I20250113 02:51:36.410413 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_b638c7b0c29a41748e55d806b0924138"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:36.410666 1195 tablet_replica.cc:331] T d8bb60fdd06946739b75b255303d9d24 P 1186fa869eb14812832fa6562531f196: stopping tablet replica
W20250113 02:51:36.410826 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b638c7b0c29a41748e55d806b0924138 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:36.410851 1195 raft_consensus.cc:2238] T d8bb60fdd06946739b75b255303d9d24 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:36.411000 1195 raft_consensus.cc:2267] T d8bb60fdd06946739b75b255303d9d24 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:36.411535 1195 ts_tablet_manager.cc:1905] T d8bb60fdd06946739b75b255303d9d24 P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:36.411870 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet d7193e2799a54950902c4ceac65b663b
I20250113 02:51:36.412384 452 catalog_manager.cc:6596] ReplaceTablet: tablet d7193e2799a54950902c4ceac65b663b deleted and replaced by tablet 104d491605f64657a936174f5a889ffd
I20250113 02:51:36.412662 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_6fffe7a4aeca4314b182f3f0a0cd31e0" } new_table_name: "default.Table_ba4715b9d1f540d78da8b5c3982d9a2b" modify_external_catalogs: true
I20250113 02:51:36.413064 1041 tablet_service.cc:1514] Processing DeleteTablet for tablet d7193e2799a54950902c4ceac65b663b with delete_type TABLET_DATA_DELETED (replaced by tablet 104d491605f64657a936174f5a889ffd) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:36.416450 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_ffa9bfdfd58f4159b17a5185a1d2192e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:36.418617 1041 tablet_service.cc:1467] Processing CreateTablet for tablet 471cd6055e454540bbb479d44daf586a (DEFAULT_TABLE table=default.Table_b0050f363a17412aa670daf91361c7bb [id=7022391b63be4e2ebcad99fcc2106d91]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:36.419035 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 471cd6055e454540bbb479d44daf586a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:36.419847 887 tablet_service.cc:1467] Processing CreateTablet for tablet 471cd6055e454540bbb479d44daf586a (DEFAULT_TABLE table=default.Table_b0050f363a17412aa670daf91361c7bb [id=7022391b63be4e2ebcad99fcc2106d91]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:36.420253 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 471cd6055e454540bbb479d44daf586a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:36.420827 751 tablet_service.cc:1514] Processing DeleteTablet for tablet c518f791b4d74c54ab0c0364d29b4a64 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:23 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:36.421236 744 tablet_service.cc:1467] Processing CreateTablet for tablet 471cd6055e454540bbb479d44daf586a (DEFAULT_TABLE table=default.Table_b0050f363a17412aa670daf91361c7bb [id=7022391b63be4e2ebcad99fcc2106d91]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:36.421574 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 471cd6055e454540bbb479d44daf586a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:36.419184 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_fc1ec6582de6419e8333bb327867f4a0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:36.425024 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fc1ec6582de6419e8333bb327867f4a0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250113 02:51:36.426813 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 44d2c7d5334c476fac60fdd57e2c02b6 consensus state has no leader
W20250113 02:51:36.427553 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ffa9bfdfd58f4159b17a5185a1d2192e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250113 02:51:36.427964 454 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet cfa963a6df094ce0b9c54408f75ca6b2 consensus state has no leader
I20250113 02:51:36.430512 746 tablet_service.cc:1514] Processing DeleteTablet for tablet 655b63ff61ff4f4aa08e0449df39352a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:24 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:36.430769 736 tablet_service.cc:1514] Processing DeleteTablet for tablet d7193e2799a54950902c4ceac65b663b with delete_type TABLET_DATA_DELETED (replaced by tablet 104d491605f64657a936174f5a889ffd) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:36.432756 892 tablet_service.cc:1514] Processing DeleteTablet for tablet d7193e2799a54950902c4ceac65b663b with delete_type TABLET_DATA_DELETED (replaced by tablet 104d491605f64657a936174f5a889ffd) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:36.433111 742 tablet_service.cc:1514] Processing DeleteTablet for tablet ee240d838d4a4bbba97f8ed61f69de78 with delete_type TABLET_DATA_DELETED (replaced by tablet 95e47db8a4d74615800ba6c049ec7131) from {username='slave'} at 127.0.0.1:55398
W20250113 02:51:36.433554 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet d7193e2799a54950902c4ceac65b663b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d7193e2799a54950902c4ceac65b663b already in progress: creating tablet
I20250113 02:51:36.433661 1194 tablet_bootstrap.cc:492] T 42fad91fe1b64a21a2c8412a2585af47 P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:36.433789 1194 ts_tablet_manager.cc:1397] T 42fad91fe1b64a21a2c8412a2585af47 P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.170s user 0.003s sys 0.000s
I20250113 02:51:36.434501 1194 raft_consensus.cc:357] T 42fad91fe1b64a21a2c8412a2585af47 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:36.434679 1191 tablet_bootstrap.cc:492] T b5c1882125b5491ea79a29a6dac6771e P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:36.434801 1191 ts_tablet_manager.cc:1397] T b5c1882125b5491ea79a29a6dac6771e P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.046s user 0.006s sys 0.006s
I20250113 02:51:36.434785 1194 raft_consensus.cc:383] T 42fad91fe1b64a21a2c8412a2585af47 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:36.434798 455 catalog_manager.cc:5618] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587) does not have the latest schema for tablet 1cfbcd682b944c89847b40fe69f0a75e (table default.Table_32e88abaae96492fbcce39fb2cd7c221 [id=2b84deb07e5c4ab8b550fe42945d7b71]). Expected version 2 got 0
I20250113 02:51:36.434851 1194 raft_consensus.cc:738] T 42fad91fe1b64a21a2c8412a2585af47 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:36.435048 1194 consensus_queue.cc:260] T 42fad91fe1b64a21a2c8412a2585af47 P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:36.435470 1191 raft_consensus.cc:357] T b5c1882125b5491ea79a29a6dac6771e P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:36.435638 1191 raft_consensus.cc:383] T b5c1882125b5491ea79a29a6dac6771e P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:36.435683 1191 raft_consensus.cc:738] T b5c1882125b5491ea79a29a6dac6771e P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:36.435842 1191 consensus_queue.cc:260] T b5c1882125b5491ea79a29a6dac6771e P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:36.436308 1191 ts_tablet_manager.cc:1428] T b5c1882125b5491ea79a29a6dac6771e P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250113 02:51:36.436578 1191 tablet_bootstrap.cc:492] T d9df314ffaf042ec80180868d7e7bdf7 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:36.441185 455 catalog_manager.cc:5618] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095) does not have the latest schema for tablet b5c1882125b5491ea79a29a6dac6771e (table default.Table_99031e0341fd4ad8a3bb04a6e222a1d0 [id=3237c1be7e1c48918ff095162d191dc1]). Expected version 3 got 0
I20250113 02:51:36.442091 892 tablet_service.cc:1467] Processing CreateTablet for tablet ad2901e485814fab86506fdf4a41b45a (DEFAULT_TABLE table=default.Table_7c44942328c24dc9818e392f0b40a975 [id=ad0f2e53bacb4a9e959f9851ba212133]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:36.442546 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ad2901e485814fab86506fdf4a41b45a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:36.443842 885 tablet_service.cc:1467] Processing CreateTablet for tablet 97159fccfa154465a33ba90a27576f0a (DEFAULT_TABLE table=default.Table_38d715a0346f4b3db579e626066b5d49 [id=94b50da8277d483f9b62204fb6acf059]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:36.444233 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 97159fccfa154465a33ba90a27576f0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:36.445971 454 catalog_manager.cc:5618] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095) does not have the latest schema for tablet b5c1882125b5491ea79a29a6dac6771e (table default.Table_99031e0341fd4ad8a3bb04a6e222a1d0 [id=3237c1be7e1c48918ff095162d191dc1]). Expected version 3 got 0
I20250113 02:51:36.446692 1191 tablet_bootstrap.cc:654] T d9df314ffaf042ec80180868d7e7bdf7 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:36.450721 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8337f3c82d654ecdac2cbc1ffa88ef05" candidate_uuid: "df69f48656b44bb890f76587ee69cc0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
I20250113 02:51:36.451299 851 leader_election.cc:304] T 8337f3c82d654ecdac2cbc1ffa88ef05 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df69f48656b44bb890f76587ee69cc0f; no voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca
I20250113 02:51:36.451597 2195 raft_consensus.cc:2743] T 8337f3c82d654ecdac2cbc1ffa88ef05 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:36.452114 1026 tablet_service.cc:1467] Processing CreateTablet for tablet ad2901e485814fab86506fdf4a41b45a (DEFAULT_TABLE table=default.Table_7c44942328c24dc9818e392f0b40a975 [id=ad0f2e53bacb4a9e959f9851ba212133]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:36.452582 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ad2901e485814fab86506fdf4a41b45a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:36.453357 454 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_632d3a63935c4ff2ba48037a311708c4" } modify_external_catalogs: true
I20250113 02:51:36.453541 454 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_632d3a63935c4ff2ba48037a311708c4" } modify_external_catalogs: true
W20250113 02:51:36.454625 448 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 68c4a7e8c0fb4feb85498709b00d3258 consensus state has no leader
I20250113 02:51:36.455425 744 tablet_service.cc:1467] Processing CreateTablet for tablet ad2901e485814fab86506fdf4a41b45a (DEFAULT_TABLE table=default.Table_7c44942328c24dc9818e392f0b40a975 [id=ad0f2e53bacb4a9e959f9851ba212133]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:36.455837 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ad2901e485814fab86506fdf4a41b45a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:36.458020 448 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_459ff6639996470cb94e898c9ef4880e" } modify_external_catalogs: true
I20250113 02:51:36.458194 448 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_459ff6639996470cb94e898c9ef4880e" } modify_external_catalogs: true
I20250113 02:51:36.462615 742 tablet_service.cc:1467] Processing CreateTablet for tablet 97159fccfa154465a33ba90a27576f0a (DEFAULT_TABLE table=default.Table_38d715a0346f4b3db579e626066b5d49 [id=94b50da8277d483f9b62204fb6acf059]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:36.462661 455 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_ab3c82460efc4216a97966ab66413b5a" } modify_external_catalogs: true
I20250113 02:51:36.462848 455 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_ab3c82460efc4216a97966ab66413b5a" } modify_external_catalogs: true
I20250113 02:51:36.463101 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 97159fccfa154465a33ba90a27576f0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:36.464015 1191 tablet_bootstrap.cc:492] T d9df314ffaf042ec80180868d7e7bdf7 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:36.464200 1191 ts_tablet_manager.cc:1397] T d9df314ffaf042ec80180868d7e7bdf7 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.028s user 0.002s sys 0.008s
I20250113 02:51:36.465090 1191 raft_consensus.cc:357] T d9df314ffaf042ec80180868d7e7bdf7 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:36.465348 1191 raft_consensus.cc:383] T d9df314ffaf042ec80180868d7e7bdf7 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:36.465435 1191 raft_consensus.cc:738] T d9df314ffaf042ec80180868d7e7bdf7 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:36.465655 1191 consensus_queue.cc:260] T d9df314ffaf042ec80180868d7e7bdf7 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:36.466149 1191 ts_tablet_manager.cc:1428] T d9df314ffaf042ec80180868d7e7bdf7 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250113 02:51:36.403713 1192 consensus_queue.cc:260] T 3025935f9743442eb3e6f2db22eb52ad P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:36.468657 1191 tablet_bootstrap.cc:492] T 0644f5600c5b47e29f3fa73060ed96e3 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:36.472329 1191 tablet_bootstrap.cc:654] T 0644f5600c5b47e29f3fa73060ed96e3 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:36.473052 1198 ts_tablet_manager.cc:1905] T f129bb3fbbfd492d98822be0a6626e44 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:36.474226 1192 ts_tablet_manager.cc:1428] T 3025935f9743442eb3e6f2db22eb52ad P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.076s user 0.002s sys 0.000s
I20250113 02:51:36.476441 1192 tablet_bootstrap.cc:492] T 33c665e149dd48e8868acd935de581bf P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:36.479650 2227 raft_consensus.cc:491] T b5c1882125b5491ea79a29a6dac6771e P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:36.480041 2227 raft_consensus.cc:513] T b5c1882125b5491ea79a29a6dac6771e P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:36.480994 736 tablet_service.cc:1514] Processing DeleteTablet for tablet d9df314ffaf042ec80180868d7e7bdf7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:24 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:36.486860 447 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_8a9caf2b686f4017948ebdd7f3fe8f58" } modify_external_catalogs: true
I20250113 02:51:36.487076 447 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_8a9caf2b686f4017948ebdd7f3fe8f58" } modify_external_catalogs: true
I20250113 02:51:36.487722 1191 tablet_bootstrap.cc:492] T 0644f5600c5b47e29f3fa73060ed96e3 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:36.487849 1191 ts_tablet_manager.cc:1397] T 0644f5600c5b47e29f3fa73060ed96e3 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.001s
I20250113 02:51:36.488658 1191 raft_consensus.cc:357] T 0644f5600c5b47e29f3fa73060ed96e3 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:36.488917 1191 raft_consensus.cc:383] T 0644f5600c5b47e29f3fa73060ed96e3 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:36.489014 1191 raft_consensus.cc:738] T 0644f5600c5b47e29f3fa73060ed96e3 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:36.489252 1191 consensus_queue.cc:260] T 0644f5600c5b47e29f3fa73060ed96e3 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:36.489463 2227 leader_election.cc:290] T b5c1882125b5491ea79a29a6dac6771e P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:36.489840 1191 ts_tablet_manager.cc:1428] T 0644f5600c5b47e29f3fa73060ed96e3 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250113 02:51:36.490696 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5c1882125b5491ea79a29a6dac6771e" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
W20250113 02:51:36.491242 709 leader_election.cc:343] T b5c1882125b5491ea79a29a6dac6771e P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:36.491446 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5c1882125b5491ea79a29a6dac6771e" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:36.491957 711 leader_election.cc:343] T b5c1882125b5491ea79a29a6dac6771e P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:36.492133 711 leader_election.cc:304] T b5c1882125b5491ea79a29a6dac6771e P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:36.492623 2227 raft_consensus.cc:2743] T b5c1882125b5491ea79a29a6dac6771e P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:36.496163 1041 tablet_service.cc:1514] Processing DeleteTablet for tablet 3025935f9743442eb3e6f2db22eb52ad with delete_type TABLET_DATA_DELETED (replaced by tablet 5d2d14548b734b1fbe57d68c0ea3e2f8) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:36.497967 881 tablet_service.cc:1467] Processing CreateTablet for tablet 13f9711b16ae4e36be00db07d9bde7fc (DEFAULT_TABLE table=default.Table_4b9ceec9d6854b32894186e0c19dfe0f [id=27c5918d1ea349118760bba995fc3e7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:36.498471 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 13f9711b16ae4e36be00db07d9bde7fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:36.498436 736 tablet_service.cc:1467] Processing CreateTablet for tablet 13f9711b16ae4e36be00db07d9bde7fc (DEFAULT_TABLE table=default.Table_4b9ceec9d6854b32894186e0c19dfe0f [id=27c5918d1ea349118760bba995fc3e7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:36.499423 1023 tablet_service.cc:1467] Processing CreateTablet for tablet 13f9711b16ae4e36be00db07d9bde7fc (DEFAULT_TABLE table=default.Table_4b9ceec9d6854b32894186e0c19dfe0f [id=27c5918d1ea349118760bba995fc3e7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:36.499764 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 13f9711b16ae4e36be00db07d9bde7fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:36.502718 1041 tablet_service.cc:1467] Processing CreateTablet for tablet 97159fccfa154465a33ba90a27576f0a (DEFAULT_TABLE table=default.Table_38d715a0346f4b3db579e626066b5d49 [id=94b50da8277d483f9b62204fb6acf059]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:36.503119 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 97159fccfa154465a33ba90a27576f0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:36.507220 1192 tablet_bootstrap.cc:654] T 33c665e149dd48e8868acd935de581bf P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:36.507562 1194 ts_tablet_manager.cc:1428] T 42fad91fe1b64a21a2c8412a2585af47 P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.074s user 0.001s sys 0.001s
I20250113 02:51:36.507921 1194 tablet_bootstrap.cc:492] T 77cf8b0c824f401aba1880eb7fd1807b P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
W20250113 02:51:36.508960 995 consensus_peers.cc:487] T d5bd09a1ede644c59e400e49c8125793 P 4216a7f853d642b7bc3e5cff458f3aca -> Peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Couldn't send request to peer 1186fa869eb14812832fa6562531f196. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d5bd09a1ede644c59e400e49c8125793. This is attempt 1: this message will repeat every 5th retry.
I20250113 02:51:36.510223 1194 tablet_bootstrap.cc:654] T 77cf8b0c824f401aba1880eb7fd1807b P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:36.511058 736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 13f9711b16ae4e36be00db07d9bde7fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:36.511430 1107 heartbeater.cc:346] Connected to a master server at 127.0.104.60:42067
I20250113 02:51:36.516125 1191 tablet_bootstrap.cc:492] T bc533518faa24dffa3306ff092e8d507 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:36.518177 1191 tablet_bootstrap.cc:654] T bc533518faa24dffa3306ff092e8d507 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:36.518406 1194 tablet_bootstrap.cc:492] T 77cf8b0c824f401aba1880eb7fd1807b P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:36.518543 1194 ts_tablet_manager.cc:1397] T 77cf8b0c824f401aba1880eb7fd1807b P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.002s
I20250113 02:51:36.519284 1194 raft_consensus.cc:357] T 77cf8b0c824f401aba1880eb7fd1807b P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:36.527644 1195 ts_tablet_manager.cc:1918] T d8bb60fdd06946739b75b255303d9d24 P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:36.527760 1195 log.cc:1198] T d8bb60fdd06946739b75b255303d9d24 P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/d8bb60fdd06946739b75b255303d9d24
I20250113 02:51:36.527766 1194 raft_consensus.cc:383] T 77cf8b0c824f401aba1880eb7fd1807b P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:36.527906 1194 raft_consensus.cc:738] T 77cf8b0c824f401aba1880eb7fd1807b P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:36.528735 1198 ts_tablet_manager.cc:1918] T f129bb3fbbfd492d98822be0a6626e44 P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:36.529305 1198 log.cc:1198] T f129bb3fbbfd492d98822be0a6626e44 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/f129bb3fbbfd492d98822be0a6626e44
I20250113 02:51:36.530532 1198 ts_tablet_manager.cc:1939] T f129bb3fbbfd492d98822be0a6626e44 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:36.534655 1194 consensus_queue.cc:260] T 77cf8b0c824f401aba1880eb7fd1807b P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:36.535454 1194 ts_tablet_manager.cc:1428] T 77cf8b0c824f401aba1880eb7fd1807b P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.017s user 0.000s sys 0.002s
I20250113 02:51:36.535892 1194 tablet_bootstrap.cc:492] T 056bf1e67be04d179eace3d93282b145 P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:36.537688 1194 tablet_bootstrap.cc:654] T 056bf1e67be04d179eace3d93282b145 P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:36.539497 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 960) took 2398 ms. Trace:
I20250113 02:51:36.540108 1198 rpcz_store.cc:276] 0113 02:51:34.140935 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:34.140995 (+ 60us) service_pool.cc:225] Handling call
0113 02:51:36.388038 (+2247043us) ts_tablet_manager.cc:1145] Deleting tablet f129bb3fbbfd492d98822be0a6626e44
0113 02:51:36.388968 (+ 930us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:36.522372 (+133404us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:36.528456 (+ 6084us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:36.539377 (+ 10921us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2232189}
I20250113 02:51:36.541378 1191 tablet_bootstrap.cc:492] T bc533518faa24dffa3306ff092e8d507 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:36.541532 1191 ts_tablet_manager.cc:1397] T bc533518faa24dffa3306ff092e8d507 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.025s user 0.006s sys 0.004s
I20250113 02:51:36.542280 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet f129bb3fbbfd492d98822be0a6626e44 (table default.Table_a7b637f0a6a244fdb2172e2015816fce [id=6666a614063942c99a78b95d35add037]) successfully deleted
I20250113 02:51:36.542223 1191 raft_consensus.cc:357] T bc533518faa24dffa3306ff092e8d507 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:36.542389 1191 raft_consensus.cc:383] T bc533518faa24dffa3306ff092e8d507 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:36.542430 1191 raft_consensus.cc:738] T bc533518faa24dffa3306ff092e8d507 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:36.542603 1191 consensus_queue.cc:260] T bc533518faa24dffa3306ff092e8d507 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:36.543130 1191 ts_tablet_manager.cc:1428] T bc533518faa24dffa3306ff092e8d507 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250113 02:51:36.544302 1191 tablet_bootstrap.cc:492] T eceed0f52c9f46bbaa78ca773a7afde5 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:36.547629 1194 tablet_bootstrap.cc:492] T 056bf1e67be04d179eace3d93282b145 P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:36.547791 1194 ts_tablet_manager.cc:1397] T 056bf1e67be04d179eace3d93282b145 P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.002s
I20250113 02:51:36.548525 1194 raft_consensus.cc:357] T 056bf1e67be04d179eace3d93282b145 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:36.548735 1194 raft_consensus.cc:383] T 056bf1e67be04d179eace3d93282b145 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:36.548848 1194 raft_consensus.cc:738] T 056bf1e67be04d179eace3d93282b145 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:36.549089 1194 consensus_queue.cc:260] T 056bf1e67be04d179eace3d93282b145 P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:36.549615 1194 ts_tablet_manager.cc:1428] T 056bf1e67be04d179eace3d93282b145 P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250113 02:51:36.549880 1194 tablet_bootstrap.cc:492] T 6e1284c8aaba44b5810b0d06d36757eb P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:36.551386 1194 tablet_bootstrap.cc:654] T 6e1284c8aaba44b5810b0d06d36757eb P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:36.552424 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_ce5d67ac15bc4d89a173d58c766a259c" } modify_external_catalogs: true
I20250113 02:51:36.552608 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_ce5d67ac15bc4d89a173d58c766a259c" } modify_external_catalogs: true
W20250113 02:51:36.553970 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 032002b007424c028b4c671e1b0fb2ac consensus state has no leader
I20250113 02:51:36.554252 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_4b9ceec9d6854b32894186e0c19dfe0f" } new_table_name: "default.Table_892ad2448c3846d08d3e8fffd46084b4" modify_external_catalogs: true
I20250113 02:51:36.554999 881 tablet_service.cc:1467] Processing CreateTablet for tablet f3e00342156b4b988dfb5d7d5bcc28a9 (DEFAULT_TABLE table=default.Table_ea97918d1c344fdfb9491dbe8439ac69 [id=a30afa8fec694c0b97ae1806bcc0d910]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:36.555383 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3e00342156b4b988dfb5d7d5bcc28a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:36.555943 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet c4b7196ef5d54459921977e81a2432db
I20250113 02:51:36.556572 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet ff5e40047f9e43fbb980b5fedfb36ade
I20250113 02:51:36.557240 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet f426887763ed48e3bc87f59d80986672
I20250113 02:51:36.559098 736 tablet_service.cc:1514] Processing DeleteTablet for tablet c4b7196ef5d54459921977e81a2432db with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:36 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:36.559288 746 tablet_service.cc:1514] Processing DeleteTablet for tablet f426887763ed48e3bc87f59d80986672 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:36 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:36.560354 744 tablet_service.cc:1467] Processing CreateTablet for tablet f3e00342156b4b988dfb5d7d5bcc28a9 (DEFAULT_TABLE table=default.Table_ea97918d1c344fdfb9491dbe8439ac69 [id=a30afa8fec694c0b97ae1806bcc0d910]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:36.560714 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3e00342156b4b988dfb5d7d5bcc28a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:36.563989 1194 tablet_bootstrap.cc:492] T 6e1284c8aaba44b5810b0d06d36757eb P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:36.564152 1194 ts_tablet_manager.cc:1397] T 6e1284c8aaba44b5810b0d06d36757eb P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.002s
I20250113 02:51:36.564986 1194 raft_consensus.cc:357] T 6e1284c8aaba44b5810b0d06d36757eb P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:36.565198 1194 raft_consensus.cc:383] T 6e1284c8aaba44b5810b0d06d36757eb P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:36.565263 1194 raft_consensus.cc:738] T 6e1284c8aaba44b5810b0d06d36757eb P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:36.565486 1194 consensus_queue.cc:260] T 6e1284c8aaba44b5810b0d06d36757eb P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:36.565959 1194 ts_tablet_manager.cc:1428] T 6e1284c8aaba44b5810b0d06d36757eb P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250113 02:51:36.566221 1194 tablet_bootstrap.cc:492] T c1fb2b36c0504cb4ba32671ae4c620d1 P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:36.566437 887 tablet_service.cc:1514] Processing DeleteTablet for tablet f426887763ed48e3bc87f59d80986672 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:36 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:36.566481 885 tablet_service.cc:1514] Processing DeleteTablet for tablet ff5e40047f9e43fbb980b5fedfb36ade with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:36 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:36.566442 892 tablet_service.cc:1514] Processing DeleteTablet for tablet c4b7196ef5d54459921977e81a2432db with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:36 UTC) from {username='slave'} at 127.0.0.1:37366
W20250113 02:51:36.567482 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet f426887763ed48e3bc87f59d80986672 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f426887763ed48e3bc87f59d80986672 already in progress: creating tablet
W20250113 02:51:36.567692 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet ff5e40047f9e43fbb980b5fedfb36ade because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ff5e40047f9e43fbb980b5fedfb36ade already in progress: creating tablet
W20250113 02:51:36.567847 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet c4b7196ef5d54459921977e81a2432db because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c4b7196ef5d54459921977e81a2432db already in progress: creating tablet
I20250113 02:51:36.570132 1194 tablet_bootstrap.cc:654] T c1fb2b36c0504cb4ba32671ae4c620d1 P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:36.573781 746 tablet_service.cc:1467] Processing CreateTablet for tablet 8ee0b2e06dc74faab48912dd16fe1537 (DEFAULT_TABLE table=default.Table_035c14601d094e78bc0f136d22321b4d [id=1262f4cbebf8469ba8ea16d71e29b20e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:36.574210 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ee0b2e06dc74faab48912dd16fe1537. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:36.574848 736 tablet_service.cc:1467] Processing CreateTablet for tablet 39c5dfc3441a4bb4adb69564e613ff0a (DEFAULT_TABLE table=default.Table_a7cd30c14bce41f0aa51afc99452fddd [id=2b1c46a6041f49af8ef8f692f5c6138d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:36.575140 736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 39c5dfc3441a4bb4adb69564e613ff0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:36.578114 885 tablet_service.cc:1467] Processing CreateTablet for tablet 39c5dfc3441a4bb4adb69564e613ff0a (DEFAULT_TABLE table=default.Table_a7cd30c14bce41f0aa51afc99452fddd [id=2b1c46a6041f49af8ef8f692f5c6138d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:36.578442 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 39c5dfc3441a4bb4adb69564e613ff0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:36.572278 2242 tablet.cc:1721] T c508724edea147e5bf6e20bca2f50397 P df69f48656b44bb890f76587ee69cc0f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250113 02:51:36.573344 892 tablet_service.cc:1467] Processing CreateTablet for tablet 8ee0b2e06dc74faab48912dd16fe1537 (DEFAULT_TABLE table=default.Table_035c14601d094e78bc0f136d22321b4d [id=1262f4cbebf8469ba8ea16d71e29b20e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:36.582906 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ee0b2e06dc74faab48912dd16fe1537. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:36.585322 1194 tablet_bootstrap.cc:492] T c1fb2b36c0504cb4ba32671ae4c620d1 P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:36.585449 1194 ts_tablet_manager.cc:1397] T c1fb2b36c0504cb4ba32671ae4c620d1 P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.019s user 0.006s sys 0.000s
I20250113 02:51:36.586218 1194 raft_consensus.cc:357] T c1fb2b36c0504cb4ba32671ae4c620d1 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:36.586400 1194 raft_consensus.cc:383] T c1fb2b36c0504cb4ba32671ae4c620d1 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:36.586453 1194 raft_consensus.cc:738] T c1fb2b36c0504cb4ba32671ae4c620d1 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:36.586647 1194 consensus_queue.cc:260] T c1fb2b36c0504cb4ba32671ae4c620d1 P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:36.587163 1194 ts_tablet_manager.cc:1428] T c1fb2b36c0504cb4ba32671ae4c620d1 P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250113 02:51:36.587456 1194 tablet_bootstrap.cc:492] T a97efc70946c4af487e33a85abf6b90c P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:36.589150 1194 tablet_bootstrap.cc:654] T a97efc70946c4af487e33a85abf6b90c P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:36.592492 751 tablet_service.cc:1514] Processing DeleteTablet for tablet bc533518faa24dffa3306ff092e8d507 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:24 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:36.596312 2227 raft_consensus.cc:491] T 0644f5600c5b47e29f3fa73060ed96e3 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:36.596444 2227 raft_consensus.cc:513] T 0644f5600c5b47e29f3fa73060ed96e3 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:36.597036 2227 leader_election.cc:290] T 0644f5600c5b47e29f3fa73060ed96e3 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:36.597632 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0644f5600c5b47e29f3fa73060ed96e3" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:36.598210 711 leader_election.cc:343] T 0644f5600c5b47e29f3fa73060ed96e3 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:36.600811 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet fe8f075d4c9a4f9abe372bfdb81a829c
I20250113 02:51:36.601720 751 tablet_service.cc:1514] Processing DeleteTablet for tablet fe8f075d4c9a4f9abe372bfdb81a829c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:36 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:36.602245 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet ba8980ad3a364c888a825affbcbc9b1f
I20250113 02:51:36.602437 736 tablet_service.cc:1514] Processing DeleteTablet for tablet 0644f5600c5b47e29f3fa73060ed96e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:22 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:36.603037 736 tablet_service.cc:1514] Processing DeleteTablet for tablet ba8980ad3a364c888a825affbcbc9b1f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:36 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:36.604050 2227 raft_consensus.cc:491] T c7842d6f2c074e3b9d1a3378db9856be P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:36.604204 1194 tablet_bootstrap.cc:492] T a97efc70946c4af487e33a85abf6b90c P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:36.604178 2227 raft_consensus.cc:513] T c7842d6f2c074e3b9d1a3378db9856be P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:36.604322 1194 ts_tablet_manager.cc:1397] T a97efc70946c4af487e33a85abf6b90c P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.017s user 0.001s sys 0.005s
I20250113 02:51:36.604727 2227 leader_election.cc:290] T c7842d6f2c074e3b9d1a3378db9856be P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:36.605077 1194 raft_consensus.cc:357] T a97efc70946c4af487e33a85abf6b90c P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:36.605293 1194 raft_consensus.cc:383] T a97efc70946c4af487e33a85abf6b90c P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:36.605374 1194 raft_consensus.cc:738] T a97efc70946c4af487e33a85abf6b90c P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:36.605336 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7842d6f2c074e3b9d1a3378db9856be" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:36.605618 1194 consensus_queue.cc:260] T a97efc70946c4af487e33a85abf6b90c P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
W20250113 02:51:36.605852 711 leader_election.cc:343] T c7842d6f2c074e3b9d1a3378db9856be P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:36.606134 1194 ts_tablet_manager.cc:1428] T a97efc70946c4af487e33a85abf6b90c P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250113 02:51:36.606386 1194 tablet_bootstrap.cc:492] T c826a0b8327f44f9a00e5afe3a36cd8a P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:36.607843 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_cb9f6aa12bff4666bd309bc132674898"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:36.608398 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cb9f6aa12bff4666bd309bc132674898 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:36.609558 887 tablet_service.cc:1514] Processing DeleteTablet for tablet ba8980ad3a364c888a825affbcbc9b1f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:36 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:36.610440 1194 tablet_bootstrap.cc:654] T c826a0b8327f44f9a00e5afe3a36cd8a P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
W20250113 02:51:36.610690 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 13f9711b16ae4e36be00db07d9bde7fc consensus state has no leader
W20250113 02:51:36.611431 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet ba8980ad3a364c888a825affbcbc9b1f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ba8980ad3a364c888a825affbcbc9b1f already in progress: creating tablet
I20250113 02:51:36.611716 892 tablet_service.cc:1514] Processing DeleteTablet for tablet fe8f075d4c9a4f9abe372bfdb81a829c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:36 UTC) from {username='slave'} at 127.0.0.1:37366
W20250113 02:51:36.612286 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet fe8f075d4c9a4f9abe372bfdb81a829c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fe8f075d4c9a4f9abe372bfdb81a829c already in progress: creating tablet
I20250113 02:51:36.616232 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_bb52873ae0724253b10583549f8f3596"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:36.616773 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bb52873ae0724253b10583549f8f3596 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:36.616797 892 tablet_service.cc:1467] Processing CreateTablet for tablet e9767ab120c04e2694c913b951020a13 (DEFAULT_TABLE table=default.Table_2d36ffc5be5043fcb20b054d5283120d [id=1eb2ab09aec549c5baa5994d043a6898]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:36.617154 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9767ab120c04e2694c913b951020a13. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:36.618237 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_84a7c1edc9204358b6a5c3e818d46321"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:36.618661 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_84a7c1edc9204358b6a5c3e818d46321 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:36.620661 885 tablet_service.cc:1467] Processing CreateTablet for tablet 9cbdd7b3be19411482a4e17eb0d6b6fb (DEFAULT_TABLE table=default.Table_521ea6e222cb450797a9d20bc8112797 [id=92b59ea35f014ec883f6c1a6446ea71d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:36.621078 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9cbdd7b3be19411482a4e17eb0d6b6fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:36.622138 746 tablet_service.cc:1467] Processing CreateTablet for tablet 9cbdd7b3be19411482a4e17eb0d6b6fb (DEFAULT_TABLE table=default.Table_521ea6e222cb450797a9d20bc8112797 [id=92b59ea35f014ec883f6c1a6446ea71d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:36.622750 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9cbdd7b3be19411482a4e17eb0d6b6fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:36.626358 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_96b4ae94f08f485695f19b9d92f92bec"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:36.627157 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_96b4ae94f08f485695f19b9d92f92bec in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:36.628154 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_bf1d5374f8a54f8abe3314c2c9615fdf" } new_table_name: "default.Table_34138ec336f245abafeb5e3ba18706e8" modify_external_catalogs: true
I20250113 02:51:36.628309 1194 tablet_bootstrap.cc:492] T c826a0b8327f44f9a00e5afe3a36cd8a P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:36.628440 1194 ts_tablet_manager.cc:1397] T c826a0b8327f44f9a00e5afe3a36cd8a P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.001s
I20250113 02:51:36.628458 2227 raft_consensus.cc:491] T b0113b16110e4787ac1c4676b2c8bd1e P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:36.628587 2227 raft_consensus.cc:513] T b0113b16110e4787ac1c4676b2c8bd1e P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:36.629217 2227 leader_election.cc:290] T b0113b16110e4787ac1c4676b2c8bd1e P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:36.629207 1194 raft_consensus.cc:357] T c826a0b8327f44f9a00e5afe3a36cd8a P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:36.629380 1194 raft_consensus.cc:383] T c826a0b8327f44f9a00e5afe3a36cd8a P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:36.629431 1194 raft_consensus.cc:738] T c826a0b8327f44f9a00e5afe3a36cd8a P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:36.629619 1194 consensus_queue.cc:260] T c826a0b8327f44f9a00e5afe3a36cd8a P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:36.629748 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0113b16110e4787ac1c4676b2c8bd1e" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:36.630100 1194 ts_tablet_manager.cc:1428] T c826a0b8327f44f9a00e5afe3a36cd8a P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250113 02:51:36.630347 1194 tablet_bootstrap.cc:492] T 3b1910496e0849b193e390a5ef1a6b2b P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
W20250113 02:51:36.630347 711 leader_election.cc:343] T b0113b16110e4787ac1c4676b2c8bd1e P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:36.630474 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_022b76ef1f0a4679b59c414d4c77b135" } new_table_name: "default.Table_183e07b4f9aa438e8a1c37ef385f57d6" modify_external_catalogs: true
I20250113 02:51:36.631424 744 tablet_service.cc:1467] Processing CreateTablet for tablet e9767ab120c04e2694c913b951020a13 (DEFAULT_TABLE table=default.Table_2d36ffc5be5043fcb20b054d5283120d [id=1eb2ab09aec549c5baa5994d043a6898]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:36.631886 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9767ab120c04e2694c913b951020a13. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:36.632457 1194 tablet_bootstrap.cc:654] T 3b1910496e0849b193e390a5ef1a6b2b P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:36.633689 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_9b1d5bafaf7b49599008dc137ff247b9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:36.634161 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9b1d5bafaf7b49599008dc137ff247b9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:36.650539 454 catalog_manager.cc:5618] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467) does not have the latest schema for tablet c508724edea147e5bf6e20bca2f50397 (table default.Table_810d62357c794aeaa18da712725a338f [id=08db71e435a24e36bdb20f9a5f1cc096]). Expected version 3 got 0
I20250113 02:51:36.656153 454 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_d7b990d07d59419da5ad4a9bc9dcc215" } new_table_name: "default.Table_f6f2b94540f64e45a00fa44fdab358eb" modify_external_catalogs: true
I20250113 02:51:36.656951 736 tablet_service.cc:1467] Processing CreateTablet for tablet 49c2e648a49f4034a521f38e7d98edc4 (DEFAULT_TABLE table=default.Table_64430a2da8d04ce88ae813a23452ecb6 [id=0354002efedc4902902ec83180ebad87]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:36.657406 736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49c2e648a49f4034a521f38e7d98edc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:36.658694 448 catalog_manager.cc:5618] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467) does not have the latest schema for tablet 6e1284c8aaba44b5810b0d06d36757eb (table default.Table_bfc5031f68f64006aceefe2ee986223a [id=aa4679fc0d7148e3bef21342b3a0ea22]). Expected version 2 got 0
I20250113 02:51:36.666496 1195 ts_tablet_manager.cc:1939] T d8bb60fdd06946739b75b255303d9d24 P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:36.666602 1191 tablet_bootstrap.cc:654] T eceed0f52c9f46bbaa78ca773a7afde5 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:36.671028 2195 raft_consensus.cc:491] T a97efc70946c4af487e33a85abf6b90c P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:36.671167 2195 raft_consensus.cc:513] T a97efc70946c4af487e33a85abf6b90c P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:36.671789 2195 leader_election.cc:290] T a97efc70946c4af487e33a85abf6b90c P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
W20250113 02:51:36.672673 851 leader_election.cc:343] T a97efc70946c4af487e33a85abf6b90c P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Not found: Tablet not found: a97efc70946c4af487e33a85abf6b90c
W20250113 02:51:36.673765 852 leader_election.cc:343] T a97efc70946c4af487e33a85abf6b90c P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Not found: Tablet not found: a97efc70946c4af487e33a85abf6b90c
I20250113 02:51:36.673862 852 leader_election.cc:304] T a97efc70946c4af487e33a85abf6b90c P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df69f48656b44bb890f76587ee69cc0f; no voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca
I20250113 02:51:36.674250 2195 raft_consensus.cc:2743] T a97efc70946c4af487e33a85abf6b90c P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:36.674628 751 tablet_service.cc:1467] Processing CreateTablet for tablet de806919192f4691bf9ee088ec77963c (DEFAULT_TABLE table=default.Table_c7728cb38c5b4dcc8cd76e68b6a24a20 [id=7dc192681389446cae9d375802224be9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:36.675026 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de806919192f4691bf9ee088ec77963c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:36.675843 739 tablet_service.cc:1467] Processing CreateTablet for tablet 51e44b812fb84b8cb9e5ebe98fb187e4 (DEFAULT_TABLE table=default.Table_2f511897104f42e3a51b770d16d7073d [id=1dc2e444755c4cac814bd27d5216d9ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:36.676149 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 51e44b812fb84b8cb9e5ebe98fb187e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:36.685456 887 tablet_service.cc:1514] Processing DeleteTablet for tablet 42fad91fe1b64a21a2c8412a2585af47 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:30 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:36.685732 882 tablet_service.cc:1514] Processing DeleteTablet for tablet c1fb2b36c0504cb4ba32671ae4c620d1 with delete_type TABLET_DATA_DELETED (replaced by tablet fb2458e2044d49029ed2d6851db9de25) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:36.685794 1196 tablet_replica.cc:331] T 42fad91fe1b64a21a2c8412a2585af47 P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:36.685890 884 tablet_service.cc:1514] Processing DeleteTablet for tablet a97efc70946c4af487e33a85abf6b90c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:20 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:36.685982 1196 raft_consensus.cc:2238] T 42fad91fe1b64a21a2c8412a2585af47 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:36.686069 894 tablet_service.cc:1514] Processing DeleteTablet for tablet c826a0b8327f44f9a00e5afe3a36cd8a with delete_type TABLET_DATA_DELETED (replaced by tablet bc533518faa24dffa3306ff092e8d507) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:36.686117 1196 raft_consensus.cc:2267] T 42fad91fe1b64a21a2c8412a2585af47 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:36.686223 889 tablet_service.cc:1514] Processing DeleteTablet for tablet 77cf8b0c824f401aba1880eb7fd1807b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:25 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:36.685887 887 tablet_service.cc:1514] Processing DeleteTablet for tablet 056bf1e67be04d179eace3d93282b145 with delete_type TABLET_DATA_DELETED (replaced by tablet dfea918c04c1408a8b86a44bb0911a7c) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:36.687031 1196 ts_tablet_manager.cc:1905] T 42fad91fe1b64a21a2c8412a2585af47 P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250113 02:51:36.688364 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6b7716e67cab449ab3ed4e58eae20c9b consensus state has no leader
W20250113 02:51:36.689216 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0269663b35f148c4a2a93cfa6132d0f8 consensus state has no leader
I20250113 02:51:36.690657 2227 raft_consensus.cc:491] T 44d2c7d5334c476fac60fdd57e2c02b6 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:36.690786 2227 raft_consensus.cc:513] T 44d2c7d5334c476fac60fdd57e2c02b6 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:36.691485 2227 leader_election.cc:290] T 44d2c7d5334c476fac60fdd57e2c02b6 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:36.691641 1992 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45e9d43ecac0401fb2c18fe5eaf089b5: Bootstrap replayed 1/1 log segments. Stats: ops{read=567 overwritten=0 applied=567 ignored=0} inserts{seen=544 ignored=0} mutations{seen=644 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250113 02:51:36.692395 1992 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45e9d43ecac0401fb2c18fe5eaf089b5: Bootstrap complete.
I20250113 02:51:36.556749 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet c4b7196ef5d54459921977e81a2432db with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:36 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:36.558235 1042 tablet_service.cc:1514] Processing DeleteTablet for tablet ff5e40047f9e43fbb980b5fedfb36ade with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:36 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:36.558276 1040 tablet_service.cc:1514] Processing DeleteTablet for tablet f426887763ed48e3bc87f59d80986672 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:36 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:36.576279 1038 tablet_service.cc:1467] Processing CreateTablet for tablet 8ee0b2e06dc74faab48912dd16fe1537 (DEFAULT_TABLE table=default.Table_035c14601d094e78bc0f136d22321b4d [id=1262f4cbebf8469ba8ea16d71e29b20e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:36.693827 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ee0b2e06dc74faab48912dd16fe1537. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:36.693789 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44d2c7d5334c476fac60fdd57e2c02b6" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:36.694424 711 leader_election.cc:343] T 44d2c7d5334c476fac60fdd57e2c02b6 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:36.598838 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0644f5600c5b47e29f3fa73060ed96e3" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:36.602319 1036 tablet_service.cc:1514] Processing DeleteTablet for tablet fe8f075d4c9a4f9abe372bfdb81a829c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:36 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:36.603698 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet ba8980ad3a364c888a825affbcbc9b1f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:36 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:36.606568 1062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7842d6f2c074e3b9d1a3378db9856be" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:36.696945 1196 ts_tablet_manager.cc:1918] T 42fad91fe1b64a21a2c8412a2585af47 P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:36.697041 1196 log.cc:1198] T 42fad91fe1b64a21a2c8412a2585af47 P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/42fad91fe1b64a21a2c8412a2585af47
I20250113 02:51:36.697741 1196 ts_tablet_manager.cc:1939] T 42fad91fe1b64a21a2c8412a2585af47 P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:36.698246 1196 tablet_replica.cc:331] T c1fb2b36c0504cb4ba32671ae4c620d1 P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:36.698493 1196 raft_consensus.cc:2238] T c1fb2b36c0504cb4ba32671ae4c620d1 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:36.698693 1196 raft_consensus.cc:2267] T c1fb2b36c0504cb4ba32671ae4c620d1 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:36.699707 892 tablet_service.cc:1467] Processing CreateTablet for tablet 51e44b812fb84b8cb9e5ebe98fb187e4 (DEFAULT_TABLE table=default.Table_2f511897104f42e3a51b770d16d7073d [id=1dc2e444755c4cac814bd27d5216d9ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:36.700196 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 51e44b812fb84b8cb9e5ebe98fb187e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:36.700250 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet 42fad91fe1b64a21a2c8412a2585af47 (table default.Table_639e38f37be1466e9c2ce92fa1c2d890 [id=290b815241d648b6b62b8a70d2efd1c9]) successfully deleted
I20250113 02:51:36.614455 1192 tablet_bootstrap.cc:492] T 33c665e149dd48e8868acd935de581bf P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:36.615433 1034 tablet_service.cc:1467] Processing CreateTablet for tablet 9cbdd7b3be19411482a4e17eb0d6b6fb (DEFAULT_TABLE table=default.Table_521ea6e222cb450797a9d20bc8112797 [id=92b59ea35f014ec883f6c1a6446ea71d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:36.701728 1192 ts_tablet_manager.cc:1397] T 33c665e149dd48e8868acd935de581bf P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.226s user 0.007s sys 0.001s
I20250113 02:51:36.701851 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9cbdd7b3be19411482a4e17eb0d6b6fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:36.702527 1192 raft_consensus.cc:357] T 33c665e149dd48e8868acd935de581bf P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:36.702760 1192 raft_consensus.cc:383] T 33c665e149dd48e8868acd935de581bf P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:36.702849 1192 raft_consensus.cc:738] T 33c665e149dd48e8868acd935de581bf P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:36.703080 1192 consensus_queue.cc:260] T 33c665e149dd48e8868acd935de581bf P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:36.703526 1192 ts_tablet_manager.cc:1428] T 33c665e149dd48e8868acd935de581bf P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250113 02:51:36.703828 1192 tablet_bootstrap.cc:492] T fb18c16fb3ba44ec8a190b88d3bbd1b1 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:36.630930 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0113b16110e4787ac1c4676b2c8bd1e" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:36.704533 1196 ts_tablet_manager.cc:1905] T c1fb2b36c0504cb4ba32671ae4c620d1 P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250113 02:51:36.661518 998 consensus_peers.cc:487] T aa5c172a69bf4ca79999f4b58df08de8 P 4216a7f853d642b7bc3e5cff458f3aca -> Peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Couldn't send request to peer df69f48656b44bb890f76587ee69cc0f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250113 02:51:36.705034 998 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.2:38467, user_credentials={real_user=slave}} blocked reactor thread for 43556.5us
I20250113 02:51:36.706022 2218 raft_consensus.cc:491] T e70658c90555463498d3fc2e2f13ea7f P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:36.706156 2218 raft_consensus.cc:513] T e70658c90555463498d3fc2e2f13ea7f P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:36.706748 2218 leader_election.cc:290] T e70658c90555463498d3fc2e2f13ea7f P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:36.706841 1192 tablet_bootstrap.cc:654] T fb18c16fb3ba44ec8a190b88d3bbd1b1 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
W20250113 02:51:36.707005 709 leader_election.cc:343] T 0644f5600c5b47e29f3fa73060ed96e3 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:36.707122 709 leader_election.cc:304] T 0644f5600c5b47e29f3fa73060ed96e3 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
W20250113 02:51:36.707540 709 leader_election.cc:343] T c7842d6f2c074e3b9d1a3378db9856be P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:36.707636 709 leader_election.cc:304] T c7842d6f2c074e3b9d1a3378db9856be P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
W20250113 02:51:36.707948 709 leader_election.cc:343] T b0113b16110e4787ac1c4676b2c8bd1e P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:36.708041 709 leader_election.cc:304] T b0113b16110e4787ac1c4676b2c8bd1e P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:36.708267 2228 raft_consensus.cc:2743] T 0644f5600c5b47e29f3fa73060ed96e3 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:36.708451 2228 raft_consensus.cc:2743] T c7842d6f2c074e3b9d1a3378db9856be P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:36.708619 2228 raft_consensus.cc:2743] T b0113b16110e4787ac1c4676b2c8bd1e P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:36.542476 1198 tablet_replica.cc:331] T 3b1910496e0849b193e390a5ef1a6b2b P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:36.709600 742 tablet_service.cc:1514] Processing DeleteTablet for tablet ff5e40047f9e43fbb980b5fedfb36ade with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:36 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:36.709975 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44d2c7d5334c476fac60fdd57e2c02b6" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:36.714387 1992 raft_consensus.cc:357] T 00000000000000000000000000000000 P 45e9d43ecac0401fb2c18fe5eaf089b5 [term 4 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e13f5bcdcee34225a46c6d2651e5ede9" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44217 } } peers { permanent_uuid: "6bbc3541604a455584bb32dfc2cc4efd" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 35271 } } peers { permanent_uuid: "45e9d43ecac0401fb2c18fe5eaf089b5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42067 } }
I20250113 02:51:36.715476 1992 raft_consensus.cc:738] T 00000000000000000000000000000000 P 45e9d43ecac0401fb2c18fe5eaf089b5 [term 4 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45e9d43ecac0401fb2c18fe5eaf089b5, State: Initialized, Role: FOLLOWER
W20250113 02:51:36.715863 995 leader_election.cc:343] T e70658c90555463498d3fc2e2f13ea7f P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Not found: Tablet not found: e70658c90555463498d3fc2e2f13ea7f
I20250113 02:51:36.716790 1198 raft_consensus.cc:2238] T 3b1910496e0849b193e390a5ef1a6b2b P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:36.716956 1198 raft_consensus.cc:2267] T 3b1910496e0849b193e390a5ef1a6b2b P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:36.717455 1198 ts_tablet_manager.cc:1905] T 3b1910496e0849b193e390a5ef1a6b2b P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250113 02:51:36.717697 709 leader_election.cc:343] T 44d2c7d5334c476fac60fdd57e2c02b6 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:36.717795 709 leader_election.cc:304] T 44d2c7d5334c476fac60fdd57e2c02b6 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:36.718106 2227 raft_consensus.cc:2743] T 44d2c7d5334c476fac60fdd57e2c02b6 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:36.719172 1992 consensus_queue.cc:260] T 00000000000000000000000000000000 P 45e9d43ecac0401fb2c18fe5eaf089b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 567, Last appended: 4.567, Last appended by leader: 567, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e13f5bcdcee34225a46c6d2651e5ede9" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44217 } } peers { permanent_uuid: "6bbc3541604a455584bb32dfc2cc4efd" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 35271 } } peers { permanent_uuid: "45e9d43ecac0401fb2c18fe5eaf089b5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42067 } }
I20250113 02:51:36.720078 1038 tablet_service.cc:1514] Processing DeleteTablet for tablet 33c665e149dd48e8868acd935de581bf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:20 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:36.720410 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e70658c90555463498d3fc2e2f13ea7f" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:36.720945 998 leader_election.cc:343] T e70658c90555463498d3fc2e2f13ea7f P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:36.721043 998 leader_election.cc:304] T e70658c90555463498d3fc2e2f13ea7f P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4216a7f853d642b7bc3e5cff458f3aca; no voters: 1186fa869eb14812832fa6562531f196, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:36.721422 2221 raft_consensus.cc:2743] T e70658c90555463498d3fc2e2f13ea7f P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:36.724270 885 tablet_service.cc:1467] Processing CreateTablet for tablet 49c2e648a49f4034a521f38e7d98edc4 (DEFAULT_TABLE table=default.Table_64430a2da8d04ce88ae813a23452ecb6 [id=0354002efedc4902902ec83180ebad87]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:36.724978 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49c2e648a49f4034a521f38e7d98edc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:36.729709 1191 tablet_bootstrap.cc:492] T eceed0f52c9f46bbaa78ca773a7afde5 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:36.729956 1191 ts_tablet_manager.cc:1397] T eceed0f52c9f46bbaa78ca773a7afde5 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.186s user 0.014s sys 0.022s
I20250113 02:51:36.730775 1191 raft_consensus.cc:357] T eceed0f52c9f46bbaa78ca773a7afde5 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:36.731022 1191 raft_consensus.cc:383] T eceed0f52c9f46bbaa78ca773a7afde5 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:36.731112 1191 raft_consensus.cc:738] T eceed0f52c9f46bbaa78ca773a7afde5 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:36.669237 1039 tablet_service.cc:1467] Processing CreateTablet for tablet 49c2e648a49f4034a521f38e7d98edc4 (DEFAULT_TABLE table=default.Table_64430a2da8d04ce88ae813a23452ecb6 [id=0354002efedc4902902ec83180ebad87]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:36.731380 1191 consensus_queue.cc:260] T eceed0f52c9f46bbaa78ca773a7afde5 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:36.731782 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49c2e648a49f4034a521f38e7d98edc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:36.731763 1196 ts_tablet_manager.cc:1918] T c1fb2b36c0504cb4ba32671ae4c620d1 P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:36.731874 1196 log.cc:1198] T c1fb2b36c0504cb4ba32671ae4c620d1 P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/c1fb2b36c0504cb4ba32671ae4c620d1
I20250113 02:51:36.732184 1191 ts_tablet_manager.cc:1428] T eceed0f52c9f46bbaa78ca773a7afde5 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250113 02:51:36.732261 1196 ts_tablet_manager.cc:1939] T c1fb2b36c0504cb4ba32671ae4c620d1 P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:36.734653 1191 tablet_bootstrap.cc:492] T 788ab82626424b6e833e31761a15b9b2 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:36.735580 2221 raft_consensus.cc:491] T 33c665e149dd48e8868acd935de581bf P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:36.735719 2221 raft_consensus.cc:513] T 33c665e149dd48e8868acd935de581bf P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:36.736366 2221 leader_election.cc:290] T 33c665e149dd48e8868acd935de581bf P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:36.737041 1191 tablet_bootstrap.cc:654] T 788ab82626424b6e833e31761a15b9b2 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:36.674961 1024 tablet_service.cc:1467] Processing CreateTablet for tablet de806919192f4691bf9ee088ec77963c (DEFAULT_TABLE table=default.Table_c7728cb38c5b4dcc8cd76e68b6a24a20 [id=7dc192681389446cae9d375802224be9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:36.738355 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de806919192f4691bf9ee088ec77963c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:36.738510 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33c665e149dd48e8868acd935de581bf" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:36.739243 998 leader_election.cc:343] T 33c665e149dd48e8868acd935de581bf P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:36.569445 1030 tablet_service.cc:1467] Processing CreateTablet for tablet f3e00342156b4b988dfb5d7d5bcc28a9 (DEFAULT_TABLE table=default.Table_ea97918d1c344fdfb9491dbe8439ac69 [id=a30afa8fec694c0b97ae1806bcc0d910]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:36.613616 1029 tablet_service.cc:1467] Processing CreateTablet for tablet 39c5dfc3441a4bb4adb69564e613ff0a (DEFAULT_TABLE table=default.Table_a7cd30c14bce41f0aa51afc99452fddd [id=2b1c46a6041f49af8ef8f692f5c6138d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:36.745322 820 heartbeater.cc:346] Connected to a master server at 127.0.104.60:42067
W20250113 02:51:36.746815 995 leader_election.cc:343] T 33c665e149dd48e8868acd935de581bf P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Not found: Tablet not found: 33c665e149dd48e8868acd935de581bf
I20250113 02:51:36.746923 995 leader_election.cc:304] T 33c665e149dd48e8868acd935de581bf P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4216a7f853d642b7bc3e5cff458f3aca; no voters: 1186fa869eb14812832fa6562531f196, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:36.747480 2221 raft_consensus.cc:2743] T 33c665e149dd48e8868acd935de581bf P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:36.667925 1026 tablet_service.cc:1467] Processing CreateTablet for tablet e9767ab120c04e2694c913b951020a13 (DEFAULT_TABLE table=default.Table_2d36ffc5be5043fcb20b054d5283120d [id=1eb2ab09aec549c5baa5994d043a6898]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:36.749989 446 catalog_manager.cc:5618] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095) does not have the latest schema for tablet eceed0f52c9f46bbaa78ca773a7afde5 (table default.Table_6a0195b054a043269e133f6ed8dd5979 [id=4b2bd999368c4b628c70cb3ee8c2b1f3]). Expected version 1 got 0
W20250113 02:51:36.750720 446 catalog_manager.cc:5718] Failed to send AlterTable RPC for tablet eceed0f52c9f46bbaa78ca773a7afde5 (table default.Table_6a0195b054a043269e133f6ed8dd5979 [id=4b2bd999368c4b628c70cb3ee8c2b1f3], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet eceed0f52c9f46bbaa78ca773a7afde5 consensus state has no leader
I20250113 02:51:36.755102 447 catalog_manager.cc:5618] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095) does not have the latest schema for tablet eceed0f52c9f46bbaa78ca773a7afde5 (table default.Table_6a0195b054a043269e133f6ed8dd5979 [id=4b2bd999368c4b628c70cb3ee8c2b1f3]). Expected version 1 got 0
I20250113 02:51:36.760005 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_6d41dae8ad174b7fa4364a4edb1fc307"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:36.760803 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6d41dae8ad174b7fa4364a4edb1fc307 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:36.761885 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1003) took 1539 ms. Trace:
I20250113 02:51:36.761976 1195 rpcz_store.cc:276] 0113 02:51:35.221964 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:35.222018 (+ 54us) service_pool.cc:225] Handling call
0113 02:51:36.410616 (+1188598us) ts_tablet_manager.cc:1145] Deleting tablet d8bb60fdd06946739b75b255303d9d24
0113 02:51:36.410627 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:36.524707 (+114080us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:36.527547 (+ 2840us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:36.761866 (+234319us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1188404}
I20250113 02:51:36.762360 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet d8bb60fdd06946739b75b255303d9d24 (table default.Table_ba98ed25e93a45a288784d4e5bdee029 [id=7c84304f6d1e4b4ea99f0bc38215d74c]) successfully deleted
I20250113 02:51:36.763450 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1004) took 1535 ms. Trace:
I20250113 02:51:36.763530 1195 rpcz_store.cc:276] 0113 02:51:35.227610 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:35.227662 (+ 52us) service_pool.cc:225] Handling call
0113 02:51:36.762633 (+1534971us) ts_tablet_manager.cc:1145] Deleting tablet 2c49f1eefd2041b4ae617a4cb35d6f15
0113 02:51:36.763312 (+ 679us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:36.763419 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1534797}
W20250113 02:51:36.763903 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 2c49f1eefd2041b4ae617a4cb35d6f15 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2c49f1eefd2041b4ae617a4cb35d6f15 already in progress: creating tablet
I20250113 02:51:36.770538 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 39c5dfc3441a4bb4adb69564e613ff0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:36.771140 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3e00342156b4b988dfb5d7d5bcc28a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:36.773406 455 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_ae01ec20166046f3918f5d29086e4274" } modify_external_catalogs: true
I20250113 02:51:36.773595 455 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_ae01ec20166046f3918f5d29086e4274" } modify_external_catalogs: true
I20250113 02:51:36.775063 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_f85400f8b51646018d553e67a5fdcffb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:36.775596 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f85400f8b51646018d553e67a5fdcffb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:36.775769 1192 tablet_bootstrap.cc:492] T fb18c16fb3ba44ec8a190b88d3bbd1b1 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:36.775900 1192 ts_tablet_manager.cc:1397] T fb18c16fb3ba44ec8a190b88d3bbd1b1 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.072s user 0.004s sys 0.000s
I20250113 02:51:36.776705 1192 raft_consensus.cc:357] T fb18c16fb3ba44ec8a190b88d3bbd1b1 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:36.776897 1192 raft_consensus.cc:383] T fb18c16fb3ba44ec8a190b88d3bbd1b1 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:36.776959 1192 raft_consensus.cc:738] T fb18c16fb3ba44ec8a190b88d3bbd1b1 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:36.777253 1192 consensus_queue.cc:260] T fb18c16fb3ba44ec8a190b88d3bbd1b1 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:36.777374 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_5030a9f5589a4f85865ee2ba6562ee1d" } modify_external_catalogs: true
I20250113 02:51:36.777503 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_5030a9f5589a4f85865ee2ba6562ee1d" } modify_external_catalogs: true
I20250113 02:51:36.777799 1192 ts_tablet_manager.cc:1428] T fb18c16fb3ba44ec8a190b88d3bbd1b1 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250113 02:51:36.779072 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_1796b3c54f624e3aa374222d74b931b5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:36.779597 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1796b3c54f624e3aa374222d74b931b5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:36.779733 1191 tablet_bootstrap.cc:492] T 788ab82626424b6e833e31761a15b9b2 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:36.779847 1191 ts_tablet_manager.cc:1397] T 788ab82626424b6e833e31761a15b9b2 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.045s user 0.004s sys 0.000s
I20250113 02:51:36.780730 1191 raft_consensus.cc:357] T 788ab82626424b6e833e31761a15b9b2 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:36.780934 1191 raft_consensus.cc:383] T 788ab82626424b6e833e31761a15b9b2 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:36.780978 1191 raft_consensus.cc:738] T 788ab82626424b6e833e31761a15b9b2 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:36.781167 1191 consensus_queue.cc:260] T 788ab82626424b6e833e31761a15b9b2 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:36.781519 448 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_5255ab9ad2034d32aef64cbb9c9a59c8" } modify_external_catalogs: true
I20250113 02:51:36.781610 1191 ts_tablet_manager.cc:1428] T 788ab82626424b6e833e31761a15b9b2 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250113 02:51:36.781666 448 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_5255ab9ad2034d32aef64cbb9c9a59c8" } modify_external_catalogs: true
I20250113 02:51:36.781857 1191 tablet_bootstrap.cc:492] T f8a6eebb6b6d4b63903e68750e181707 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:36.783864 1191 tablet_bootstrap.cc:654] T f8a6eebb6b6d4b63903e68750e181707 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:36.791229 1192 tablet_bootstrap.cc:492] T 7527e6e0b65e4a43b7f48899a34cdf7f P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:36.792682 912 raft_consensus.cc:3054] T 8337f3c82d654ecdac2cbc1ffa88ef05 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:36.794808 449 master_service.cc:946] ReplaceTablet: received request to replace tablet 93c0ce9f457e431193a2ceb0ad4d4af6 from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:36.795773 739 tablet_service.cc:1514] Processing DeleteTablet for tablet 788ab82626424b6e833e31761a15b9b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:26 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:36.803120 2221 raft_consensus.cc:491] T a92f9b1c9c1a4cbdbf0cc7b2a3f2275b P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:36.803315 2221 raft_consensus.cc:513] T a92f9b1c9c1a4cbdbf0cc7b2a3f2275b P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:36.803987 2221 leader_election.cc:290] T a92f9b1c9c1a4cbdbf0cc7b2a3f2275b P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:36.804692 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a92f9b1c9c1a4cbdbf0cc7b2a3f2275b" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:36.804986 1192 tablet_bootstrap.cc:654] T 7527e6e0b65e4a43b7f48899a34cdf7f P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
W20250113 02:51:36.805996 998 leader_election.cc:343] T a92f9b1c9c1a4cbdbf0cc7b2a3f2275b P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
W20250113 02:51:36.806195 995 leader_election.cc:343] T a92f9b1c9c1a4cbdbf0cc7b2a3f2275b P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Not found: Tablet not found: a92f9b1c9c1a4cbdbf0cc7b2a3f2275b
I20250113 02:51:36.806263 995 leader_election.cc:304] T a92f9b1c9c1a4cbdbf0cc7b2a3f2275b P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4216a7f853d642b7bc3e5cff458f3aca; no voters: 1186fa869eb14812832fa6562531f196, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:36.806608 2227 raft_consensus.cc:491] T bbc7832c50114d6f890fec45a3d994e9 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:36.806732 2227 raft_consensus.cc:513] T bbc7832c50114d6f890fec45a3d994e9 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:36.807333 2227 leader_election.cc:290] T bbc7832c50114d6f890fec45a3d994e9 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:36.808233 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbc7832c50114d6f890fec45a3d994e9" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
W20250113 02:51:36.808701 709 leader_election.cc:343] T bbc7832c50114d6f890fec45a3d994e9 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:36.809577 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbc7832c50114d6f890fec45a3d994e9" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:36.810099 711 leader_election.cc:343] T bbc7832c50114d6f890fec45a3d994e9 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:36.810202 711 leader_election.cc:304] T bbc7832c50114d6f890fec45a3d994e9 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:36.810629 2227 raft_consensus.cc:2743] T bbc7832c50114d6f890fec45a3d994e9 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:36.810881 2221 raft_consensus.cc:2743] T a92f9b1c9c1a4cbdbf0cc7b2a3f2275b P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:36.813673 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_d1195c2276654128b31755efa960e72c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:36.814319 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d1195c2276654128b31755efa960e72c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:36.828310 2221 consensus_queue.cc:1035] T aa5c172a69bf4ca79999f4b58df08de8 P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 02:51:36.830454 917 raft_consensus.cc:3054] T 0752fde3a30a4947a2a688c56ceadab5 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:36.856051 2227 raft_consensus.cc:491] T bc533518faa24dffa3306ff092e8d507 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:36.856251 2227 raft_consensus.cc:513] T bc533518faa24dffa3306ff092e8d507 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:36.856911 2227 leader_election.cc:290] T bc533518faa24dffa3306ff092e8d507 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:36.857625 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc533518faa24dffa3306ff092e8d507" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
W20250113 02:51:36.858157 709 leader_election.cc:343] T bc533518faa24dffa3306ff092e8d507 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:36.859320 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc533518faa24dffa3306ff092e8d507" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:36.859930 711 leader_election.cc:343] T bc533518faa24dffa3306ff092e8d507 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:36.860055 711 leader_election.cc:304] T bc533518faa24dffa3306ff092e8d507 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:36.860563 2227 raft_consensus.cc:2743] T bc533518faa24dffa3306ff092e8d507 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:36.882383 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9767ab120c04e2694c913b951020a13. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:36.883373 881 tablet_service.cc:1467] Processing CreateTablet for tablet de806919192f4691bf9ee088ec77963c (DEFAULT_TABLE table=default.Table_c7728cb38c5b4dcc8cd76e68b6a24a20 [id=7dc192681389446cae9d375802224be9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:36.883881 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de806919192f4691bf9ee088ec77963c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:36.885890 2227 raft_consensus.cc:491] T bc6a7322c0c34adb85457750526d8955 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:36.886047 2227 raft_consensus.cc:513] T bc6a7322c0c34adb85457750526d8955 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:36.887280 2227 leader_election.cc:290] T bc6a7322c0c34adb85457750526d8955 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:36.887954 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc6a7322c0c34adb85457750526d8955" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:36.889662 1198 ts_tablet_manager.cc:1918] T 3b1910496e0849b193e390a5ef1a6b2b P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:36.889794 1198 log.cc:1198] T 3b1910496e0849b193e390a5ef1a6b2b P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/3b1910496e0849b193e390a5ef1a6b2b
I20250113 02:51:36.890112 1198 ts_tablet_manager.cc:1939] T 3b1910496e0849b193e390a5ef1a6b2b P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
W20250113 02:51:36.895378 711 leader_election.cc:343] T bc6a7322c0c34adb85457750526d8955 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:36.895722 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc6a7322c0c34adb85457750526d8955" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
W20250113 02:51:36.896368 709 leader_election.cc:343] T bc6a7322c0c34adb85457750526d8955 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:36.896479 709 leader_election.cc:304] T bc6a7322c0c34adb85457750526d8955 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:36.897583 2227 raft_consensus.cc:2743] T bc6a7322c0c34adb85457750526d8955 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:36.898437 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 959) took 2765 ms. Trace:
I20250113 02:51:36.898523 1198 rpcz_store.cc:276] 0113 02:51:34.133395 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:34.133454 (+ 59us) service_pool.cc:225] Handling call
0113 02:51:36.542335 (+2408881us) ts_tablet_manager.cc:1145] Deleting tablet 3b1910496e0849b193e390a5ef1a6b2b
0113 02:51:36.542345 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:36.770238 (+227893us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:36.889544 (+119306us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:36.898416 (+ 8872us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":7582,"tablet-delete.queue_time_us":2349160}
I20250113 02:51:36.898955 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 964) took 2731 ms. Trace:
I20250113 02:51:36.899032 1198 rpcz_store.cc:276] 0113 02:51:34.167939 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:34.192685 (+ 24746us) service_pool.cc:225] Handling call
0113 02:51:36.898827 (+2706142us) ts_tablet_manager.cc:1145] Deleting tablet 9b2d0aa6f0e9456d95f4e48c360ab6e8
0113 02:51:36.898835 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:36.898925 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2704970}
I20250113 02:51:36.899369 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 966) took 2706 ms. Trace:
I20250113 02:51:36.899443 1198 rpcz_store.cc:276] 0113 02:51:34.193355 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:34.193400 (+ 45us) service_pool.cc:225] Handling call
0113 02:51:36.899274 (+2705874us) ts_tablet_manager.cc:1145] Deleting tablet ec06c9f6a3bf46a9bcb90a85a82b6ff8
0113 02:51:36.899282 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:36.899355 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2698227}
I20250113 02:51:36.899502 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet 3b1910496e0849b193e390a5ef1a6b2b (table default.Table_2f3720d8d897466388c3f9dde157d701 [id=8af6fc6e393d4cd3a5eb0671346df592]) successfully deleted
I20250113 02:51:36.899722 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 965) took 2731 ms. Trace:
W20250113 02:51:36.899763 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 9b2d0aa6f0e9456d95f4e48c360ab6e8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9b2d0aa6f0e9456d95f4e48c360ab6e8 already in progress: creating tablet
I20250113 02:51:36.899783 1198 rpcz_store.cc:276] 0113 02:51:34.168149 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:34.168667 (+ 518us) service_pool.cc:225] Handling call
0113 02:51:36.899635 (+2730968us) ts_tablet_manager.cc:1145] Deleting tablet d2699bb529bf4f0f9d7f2668ffe4c9ff
0113 02:51:36.899642 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:36.899708 (+ 66us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2687000}
I20250113 02:51:36.900043 1198 tablet_replica.cc:331] T b8b6ffe690fc459db78b2b3bc12671dd P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:36.900229 1198 raft_consensus.cc:2238] T b8b6ffe690fc459db78b2b3bc12671dd P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:36.900374 1198 raft_consensus.cc:2267] T b8b6ffe690fc459db78b2b3bc12671dd P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus is shut down!
W20250113 02:51:36.900521 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet ec06c9f6a3bf46a9bcb90a85a82b6ff8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ec06c9f6a3bf46a9bcb90a85a82b6ff8 already in progress: creating tablet
W20250113 02:51:36.900718 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet d2699bb529bf4f0f9d7f2668ffe4c9ff because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d2699bb529bf4f0f9d7f2668ffe4c9ff already in progress: creating tablet
I20250113 02:51:36.900938 1198 ts_tablet_manager.cc:1905] T b8b6ffe690fc459db78b2b3bc12671dd P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:36.906070 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_b016b8047ff84e33959561a580efd594" } new_table_name: "default.Table_626d5cb595994368aa592eec7021b889" modify_external_catalogs: true
I20250113 02:51:36.765410 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1006) took 1517 ms. Trace:
W20250113 02:51:36.912390 998 consensus_peers.cc:487] T 56b022c94847411785a7e9db3757a2d9 P 4216a7f853d642b7bc3e5cff458f3aca -> Peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Couldn't send request to peer df69f48656b44bb890f76587ee69cc0f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250113 02:51:36.914019 1030 tablet_service.cc:1467] Processing CreateTablet for tablet 51e44b812fb84b8cb9e5ebe98fb187e4 (DEFAULT_TABLE table=default.Table_2f511897104f42e3a51b770d16d7073d [id=1dc2e444755c4cac814bd27d5216d9ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:36.910974 1195 rpcz_store.cc:276] 0113 02:51:35.248093 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:35.248156 (+ 63us) service_pool.cc:225] Handling call
0113 02:51:36.765306 (+1517150us) ts_tablet_manager.cc:1145] Deleting tablet 7fb282707196444690352dea3eb1387f
0113 02:51:36.765315 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:36.765394 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1516977}
W20250113 02:51:36.914884 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 7fb282707196444690352dea3eb1387f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7fb282707196444690352dea3eb1387f already in progress: creating tablet
I20250113 02:51:36.915426 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1008) took 1619 ms. Trace:
I20250113 02:51:36.915566 1195 rpcz_store.cc:276] 0113 02:51:35.295655 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:35.295699 (+ 44us) service_pool.cc:225] Handling call
0113 02:51:36.915276 (+1619577us) ts_tablet_manager.cc:1145] Deleting tablet f9269967dea94fc3a337546ad0842e11
0113 02:51:36.915290 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:36.915403 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1619390}
W20250113 02:51:36.916082 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet f9269967dea94fc3a337546ad0842e11 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f9269967dea94fc3a337546ad0842e11 already in progress: creating tablet
I20250113 02:51:36.916519 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1007) took 1621 ms. Trace:
I20250113 02:51:36.916647 1195 rpcz_store.cc:276] 0113 02:51:35.295550 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:35.302700 (+ 7150us) service_pool.cc:225] Handling call
0113 02:51:36.916398 (+1613698us) ts_tablet_manager.cc:1145] Deleting tablet f105f8b32a654e4ca154e8fc0cb96091
0113 02:51:36.916408 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:36.916499 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1613270}
W20250113 02:51:36.917102 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet f105f8b32a654e4ca154e8fc0cb96091 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f105f8b32a654e4ca154e8fc0cb96091 already in progress: creating tablet
I20250113 02:51:36.917526 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1009) took 1602 ms. Trace:
I20250113 02:51:36.917652 1195 rpcz_store.cc:276] 0113 02:51:35.314682 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:35.314738 (+ 56us) service_pool.cc:225] Handling call
0113 02:51:36.917420 (+1602682us) ts_tablet_manager.cc:1145] Deleting tablet f24cda32b6e947c7983ae2f1f9a64c23
0113 02:51:36.917429 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:36.917505 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1602439}
W20250113 02:51:36.918098 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet f24cda32b6e947c7983ae2f1f9a64c23 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f24cda32b6e947c7983ae2f1f9a64c23 already in progress: creating tablet
I20250113 02:51:36.918484 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1013) took 1519 ms. Trace:
I20250113 02:51:36.918603 1195 rpcz_store.cc:276] 0113 02:51:35.399373 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:35.399426 (+ 53us) service_pool.cc:225] Handling call
0113 02:51:36.918376 (+1518950us) ts_tablet_manager.cc:1145] Deleting tablet fb2458e2044d49029ed2d6851db9de25
0113 02:51:36.918388 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:36.918469 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1516568}
W20250113 02:51:36.919152 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet fb2458e2044d49029ed2d6851db9de25 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fb2458e2044d49029ed2d6851db9de25 already in progress: creating tablet
I20250113 02:51:36.919548 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1017) took 1476 ms. Trace:
I20250113 02:51:36.927875 885 tablet_service.cc:1467] Processing CreateTablet for tablet 9fad58cccf1744b0a476523e0af0036e (DEFAULT_TABLE table=default.Table_ba66360c176446ee945a533bd8358296 [id=74975c535a3341db98ff8da041656e10]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:36.928364 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9fad58cccf1744b0a476523e0af0036e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:36.937922 1194 tablet_bootstrap.cc:492] T 3b1910496e0849b193e390a5ef1a6b2b P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:36.938138 1194 ts_tablet_manager.cc:1397] T 3b1910496e0849b193e390a5ef1a6b2b P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.308s user 0.040s sys 0.041s
I20250113 02:51:36.938896 1194 raft_consensus.cc:357] T 3b1910496e0849b193e390a5ef1a6b2b P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:36.939198 1194 raft_consensus.cc:383] T 3b1910496e0849b193e390a5ef1a6b2b P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:36.939308 1194 raft_consensus.cc:738] T 3b1910496e0849b193e390a5ef1a6b2b P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:36.939604 1194 consensus_queue.cc:260] T 3b1910496e0849b193e390a5ef1a6b2b P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:36.940270 1194 ts_tablet_manager.cc:1428] T 3b1910496e0849b193e390a5ef1a6b2b P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250113 02:51:36.940570 1194 tablet_bootstrap.cc:492] T 1a63f485caf54068abd6b50d7de3d14a P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:36.941088 1039 tablet_service.cc:1467] Processing CreateTablet for tablet 9fad58cccf1744b0a476523e0af0036e (DEFAULT_TABLE table=default.Table_ba66360c176446ee945a533bd8358296 [id=74975c535a3341db98ff8da041656e10]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:36.942132 1194 tablet_bootstrap.cc:654] T 1a63f485caf54068abd6b50d7de3d14a P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
W20250113 02:51:36.946379 454 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f29e16fea2534c7dbe57639c7c6b37d6 consensus state has no leader
I20250113 02:51:36.949904 885 tablet_service.cc:1467] Processing CreateTablet for tablet 34245259b47743a59da080379b33e719 (DEFAULT_TABLE table=default.Table_1cfa6c2edbe9497ba84ffc25ce7b1573 [id=44d7c51d0fa64c54a03c0a54c0fcd726]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:36.950554 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34245259b47743a59da080379b33e719. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:36.956874 1029 tablet_service.cc:1467] Processing CreateTablet for tablet 34245259b47743a59da080379b33e719 (DEFAULT_TABLE table=default.Table_1cfa6c2edbe9497ba84ffc25ce7b1573 [id=44d7c51d0fa64c54a03c0a54c0fcd726]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:36.958791 1194 tablet_bootstrap.cc:492] T 1a63f485caf54068abd6b50d7de3d14a P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:36.959000 1194 ts_tablet_manager.cc:1397] T 1a63f485caf54068abd6b50d7de3d14a P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.000s
I20250113 02:51:36.959833 1194 raft_consensus.cc:357] T 1a63f485caf54068abd6b50d7de3d14a P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:36.960141 1194 raft_consensus.cc:383] T 1a63f485caf54068abd6b50d7de3d14a P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:36.960237 1194 raft_consensus.cc:738] T 1a63f485caf54068abd6b50d7de3d14a P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:36.960485 1194 consensus_queue.cc:260] T 1a63f485caf54068abd6b50d7de3d14a P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:36.961027 1194 ts_tablet_manager.cc:1428] T 1a63f485caf54068abd6b50d7de3d14a P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250113 02:51:36.961319 1194 tablet_bootstrap.cc:492] T 05bf5ab0df874dd5b6559f6d82069184 P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:36.961534 2242 tablet.cc:1721] T 0752fde3a30a4947a2a688c56ceadab5 P df69f48656b44bb890f76587ee69cc0f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250113 02:51:36.962965 1194 tablet_bootstrap.cc:654] T 05bf5ab0df874dd5b6559f6d82069184 P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:36.974622 1194 tablet_bootstrap.cc:492] T 05bf5ab0df874dd5b6559f6d82069184 P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:36.974866 1194 ts_tablet_manager.cc:1397] T 05bf5ab0df874dd5b6559f6d82069184 P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.001s
I20250113 02:51:36.975328 1192 tablet_bootstrap.cc:492] T 7527e6e0b65e4a43b7f48899a34cdf7f P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:36.975459 1192 ts_tablet_manager.cc:1397] T 7527e6e0b65e4a43b7f48899a34cdf7f P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.185s user 0.012s sys 0.011s
I20250113 02:51:36.976171 1192 raft_consensus.cc:357] T 7527e6e0b65e4a43b7f48899a34cdf7f P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:36.975699 1194 raft_consensus.cc:357] T 05bf5ab0df874dd5b6559f6d82069184 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:36.976544 1194 raft_consensus.cc:383] T 05bf5ab0df874dd5b6559f6d82069184 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:36.976646 1194 raft_consensus.cc:738] T 05bf5ab0df874dd5b6559f6d82069184 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:36.976891 1194 consensus_queue.cc:260] T 05bf5ab0df874dd5b6559f6d82069184 P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:36.977439 1194 ts_tablet_manager.cc:1428] T 05bf5ab0df874dd5b6559f6d82069184 P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250113 02:51:36.977802 1194 tablet_bootstrap.cc:492] T 0a7e7713bc8648378b668d8b769ac927 P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:36.979449 1192 raft_consensus.cc:383] T 7527e6e0b65e4a43b7f48899a34cdf7f P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:36.979529 1192 raft_consensus.cc:738] T 7527e6e0b65e4a43b7f48899a34cdf7f P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:36.979802 1194 tablet_bootstrap.cc:654] T 0a7e7713bc8648378b668d8b769ac927 P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:36.979723 1192 consensus_queue.cc:260] T 7527e6e0b65e4a43b7f48899a34cdf7f P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:36.980223 1192 ts_tablet_manager.cc:1428] T 7527e6e0b65e4a43b7f48899a34cdf7f P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250113 02:51:36.980502 1192 tablet_bootstrap.cc:492] T df87e25053a74990959f85288a85217b P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:36.982746 1192 tablet_bootstrap.cc:654] T df87e25053a74990959f85288a85217b P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:36.983090 1198 ts_tablet_manager.cc:1918] T b8b6ffe690fc459db78b2b3bc12671dd P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:36.983255 1198 log.cc:1198] T b8b6ffe690fc459db78b2b3bc12671dd P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/b8b6ffe690fc459db78b2b3bc12671dd
I20250113 02:51:36.983585 1198 ts_tablet_manager.cc:1939] T b8b6ffe690fc459db78b2b3bc12671dd P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:36.971954 454 catalog_manager.cc:5618] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587) does not have the latest schema for tablet fb18c16fb3ba44ec8a190b88d3bbd1b1 (table default.Table_7a4628807cc44ebdadf927ff709de3a2 [id=75ca7cbd7b9548d784f9a77d0f0cb64f]). Expected version 1 got 0
I20250113 02:51:36.991047 1192 tablet_bootstrap.cc:492] T df87e25053a74990959f85288a85217b P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:36.991210 1192 ts_tablet_manager.cc:1397] T df87e25053a74990959f85288a85217b P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.000s
I20250113 02:51:36.992075 1192 raft_consensus.cc:357] T df87e25053a74990959f85288a85217b P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:36.992288 1192 raft_consensus.cc:383] T df87e25053a74990959f85288a85217b P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:36.992344 1192 raft_consensus.cc:738] T df87e25053a74990959f85288a85217b P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:36.992548 1192 consensus_queue.cc:260] T df87e25053a74990959f85288a85217b P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:36.992722 1194 tablet_bootstrap.cc:492] T 0a7e7713bc8648378b668d8b769ac927 P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:36.992892 1194 ts_tablet_manager.cc:1397] T 0a7e7713bc8648378b668d8b769ac927 P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.003s
I20250113 02:51:36.993049 1192 ts_tablet_manager.cc:1428] T df87e25053a74990959f85288a85217b P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250113 02:51:36.993306 1192 tablet_bootstrap.cc:492] T 7213e3284b4d4cb581d9fb946a9f87e9 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:36.993709 1194 raft_consensus.cc:357] T 0a7e7713bc8648378b668d8b769ac927 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:36.993976 1194 raft_consensus.cc:383] T 0a7e7713bc8648378b668d8b769ac927 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:36.994084 1194 raft_consensus.cc:738] T 0a7e7713bc8648378b668d8b769ac927 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:36.994402 1194 consensus_queue.cc:260] T 0a7e7713bc8648378b668d8b769ac927 P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:36.994939 1192 tablet_bootstrap.cc:654] T 7213e3284b4d4cb581d9fb946a9f87e9 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:36.996660 1194 ts_tablet_manager.cc:1428] T 0a7e7713bc8648378b668d8b769ac927 P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250113 02:51:36.997006 1194 tablet_bootstrap.cc:492] T 6666922d677a4c678f80d7c6c47fd2a5 P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:36.999001 1194 tablet_bootstrap.cc:654] T 6666922d677a4c678f80d7c6c47fd2a5 P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:37.000634 2195 raft_consensus.cc:491] T 6e1284c8aaba44b5810b0d06d36757eb P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:37.000761 2195 raft_consensus.cc:513] T 6e1284c8aaba44b5810b0d06d36757eb P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:37.001082 1192 tablet_bootstrap.cc:492] T 7213e3284b4d4cb581d9fb946a9f87e9 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:37.001211 1192 ts_tablet_manager.cc:1397] T 7213e3284b4d4cb581d9fb946a9f87e9 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.003s
I20250113 02:51:37.001358 2195 leader_election.cc:290] T 6e1284c8aaba44b5810b0d06d36757eb P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:37.002660 1192 raft_consensus.cc:357] T 7213e3284b4d4cb581d9fb946a9f87e9 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:37.002888 1192 raft_consensus.cc:383] T 7213e3284b4d4cb581d9fb946a9f87e9 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:37.002848 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e1284c8aaba44b5810b0d06d36757eb" candidate_uuid: "df69f48656b44bb890f76587ee69cc0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:37.002957 1192 raft_consensus.cc:738] T 7213e3284b4d4cb581d9fb946a9f87e9 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:37.003173 1192 consensus_queue.cc:260] T 7213e3284b4d4cb581d9fb946a9f87e9 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:37.003747 1192 ts_tablet_manager.cc:1428] T 7213e3284b4d4cb581d9fb946a9f87e9 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250113 02:51:37.005928 2195 raft_consensus.cc:491] T 3b1910496e0849b193e390a5ef1a6b2b P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:37.006072 2195 raft_consensus.cc:513] T 3b1910496e0849b193e390a5ef1a6b2b P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:37.006551 1194 tablet_bootstrap.cc:492] T 6666922d677a4c678f80d7c6c47fd2a5 P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:37.006690 1194 ts_tablet_manager.cc:1397] T 6666922d677a4c678f80d7c6c47fd2a5 P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.003s
I20250113 02:51:37.006757 2195 leader_election.cc:290] T 3b1910496e0849b193e390a5ef1a6b2b P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:37.007530 1194 raft_consensus.cc:357] T 6666922d677a4c678f80d7c6c47fd2a5 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:37.007748 1194 raft_consensus.cc:383] T 6666922d677a4c678f80d7c6c47fd2a5 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:37.007807 1194 raft_consensus.cc:738] T 6666922d677a4c678f80d7c6c47fd2a5 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:37.008047 1194 consensus_queue.cc:260] T 6666922d677a4c678f80d7c6c47fd2a5 P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
W20250113 02:51:37.008294 851 leader_election.cc:343] T 3b1910496e0849b193e390a5ef1a6b2b P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Not found: Tablet not found: 3b1910496e0849b193e390a5ef1a6b2b
I20250113 02:51:37.008670 1194 ts_tablet_manager.cc:1428] T 6666922d677a4c678f80d7c6c47fd2a5 P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250113 02:51:37.008994 1194 tablet_bootstrap.cc:492] T f56230a2c75d4763a54671c79358f4d9 P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:37.010313 1192 tablet_bootstrap.cc:492] T f21eb3fa0f414816a29e3d4560cdc0fe P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
W20250113 02:51:37.010421 852 leader_election.cc:343] T 3b1910496e0849b193e390a5ef1a6b2b P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Not found: Tablet not found: 3b1910496e0849b193e390a5ef1a6b2b
I20250113 02:51:37.010528 852 leader_election.cc:304] T 3b1910496e0849b193e390a5ef1a6b2b P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df69f48656b44bb890f76587ee69cc0f; no voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca
I20250113 02:51:37.011001 2195 raft_consensus.cc:2743] T 3b1910496e0849b193e390a5ef1a6b2b P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:37.012374 1192 tablet_bootstrap.cc:654] T f21eb3fa0f414816a29e3d4560cdc0fe P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:36.926023 2227 raft_consensus.cc:491] T ee240d838d4a4bbba97f8ed61f69de78 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:37.020669 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 967) took 2826 ms. Trace:
I20250113 02:51:37.020716 2227 raft_consensus.cc:513] T ee240d838d4a4bbba97f8ed61f69de78 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:37.020768 1198 rpcz_store.cc:276] 0113 02:51:34.194081 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:34.194964 (+ 883us) service_pool.cc:225] Handling call
0113 02:51:36.899999 (+2705035us) ts_tablet_manager.cc:1145] Deleting tablet b8b6ffe690fc459db78b2b3bc12671dd
0113 02:51:36.900008 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:36.976531 (+ 76523us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:36.982995 (+ 6464us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:37.020645 (+ 37650us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2671016}
I20250113 02:51:37.020998 1194 tablet_bootstrap.cc:654] T f56230a2c75d4763a54671c79358f4d9 P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:36.937479 744 tablet_service.cc:1467] Processing CreateTablet for tablet 9fad58cccf1744b0a476523e0af0036e (DEFAULT_TABLE table=default.Table_ba66360c176446ee945a533bd8358296 [id=74975c535a3341db98ff8da041656e10]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:37.021476 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9fad58cccf1744b0a476523e0af0036e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:36.949362 739 tablet_service.cc:1467] Processing CreateTablet for tablet 34245259b47743a59da080379b33e719 (DEFAULT_TABLE table=default.Table_1cfa6c2edbe9497ba84ffc25ce7b1573 [id=44d7c51d0fa64c54a03c0a54c0fcd726]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:37.024504 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34245259b47743a59da080379b33e719. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:37.029520 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee240d838d4a4bbba97f8ed61f69de78" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:37.029956 2227 leader_election.cc:290] T ee240d838d4a4bbba97f8ed61f69de78 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
W20250113 02:51:37.030364 709 leader_election.cc:343] T ee240d838d4a4bbba97f8ed61f69de78 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:37.002228 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e1284c8aaba44b5810b0d06d36757eb" candidate_uuid: "df69f48656b44bb890f76587ee69cc0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
I20250113 02:51:37.032727 851 leader_election.cc:304] T 6e1284c8aaba44b5810b0d06d36757eb P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df69f48656b44bb890f76587ee69cc0f; no voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca
I20250113 02:51:37.035588 2195 raft_consensus.cc:2743] T 6e1284c8aaba44b5810b0d06d36757eb P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:37.036912 1192 tablet_bootstrap.cc:492] T f21eb3fa0f414816a29e3d4560cdc0fe P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:37.036929 912 raft_consensus.cc:3054] T 6e1284c8aaba44b5810b0d06d36757eb P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:37.037074 1192 ts_tablet_manager.cc:1397] T f21eb3fa0f414816a29e3d4560cdc0fe P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.033s user 0.006s sys 0.001s
I20250113 02:51:37.037767 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee240d838d4a4bbba97f8ed61f69de78" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:37.038411 711 leader_election.cc:343] T ee240d838d4a4bbba97f8ed61f69de78 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:37.038306 1192 raft_consensus.cc:357] T f21eb3fa0f414816a29e3d4560cdc0fe P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:37.038554 1192 raft_consensus.cc:383] T f21eb3fa0f414816a29e3d4560cdc0fe P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:37.038539 711 leader_election.cc:304] T ee240d838d4a4bbba97f8ed61f69de78 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:37.038612 1192 raft_consensus.cc:738] T f21eb3fa0f414816a29e3d4560cdc0fe P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:37.038857 1192 consensus_queue.cc:260] T f21eb3fa0f414816a29e3d4560cdc0fe P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:37.039542 1192 ts_tablet_manager.cc:1428] T f21eb3fa0f414816a29e3d4560cdc0fe P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250113 02:51:37.039837 2227 raft_consensus.cc:2743] T ee240d838d4a4bbba97f8ed61f69de78 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:37.040349 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet b8b6ffe690fc459db78b2b3bc12671dd (table default.Table_b77c072e465c474db9349f1a3b31c323 [id=d7fe00879d3143c1bf0ff603f9284750]) successfully deleted
W20250113 02:51:37.051236 995 consensus_peers.cc:487] T 32cba8f3efc54e78b4df41c15df82eea P 4216a7f853d642b7bc3e5cff458f3aca -> Peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Couldn't send request to peer 1186fa869eb14812832fa6562531f196. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 32cba8f3efc54e78b4df41c15df82eea. This is attempt 1: this message will repeat every 5th retry.
I20250113 02:51:37.051368 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 970) took 2729 ms. Trace:
I20250113 02:51:37.051450 1198 rpcz_store.cc:276] 0113 02:51:34.321892 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:34.321936 (+ 44us) service_pool.cc:225] Handling call
0113 02:51:37.021072 (+2699136us) ts_tablet_manager.cc:1145] Deleting tablet 6e208f83f3384a78a959909b4ac690c0
0113 02:51:37.051229 (+ 30157us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:37.051351 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2698937}
I20250113 02:51:37.051823 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 971) took 2729 ms. Trace:
I20250113 02:51:37.051905 1198 rpcz_store.cc:276] 0113 02:51:34.322293 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:34.322326 (+ 33us) service_pool.cc:225] Handling call
0113 02:51:37.051722 (+2729396us) ts_tablet_manager.cc:1145] Deleting tablet db486978b33a45df90025988b084aecd
0113 02:51:37.051732 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:37.051807 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2729233}
I20250113 02:51:37.052241 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 972) took 2729 ms. Trace:
I20250113 02:51:37.052306 1198 rpcz_store.cc:276] 0113 02:51:34.322719 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:34.322840 (+ 121us) service_pool.cc:225] Handling call
0113 02:51:37.052142 (+2729302us) ts_tablet_manager.cc:1145] Deleting tablet 1c77033ca393469e9324d2406a65a5e9
0113 02:51:37.052150 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:37.052228 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2729159}
I20250113 02:51:37.052569 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 977) took 2621 ms. Trace:
I20250113 02:51:37.052654 1198 rpcz_store.cc:276] 0113 02:51:34.431438 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:34.431921 (+ 483us) service_pool.cc:225] Handling call
0113 02:51:37.052476 (+2620555us) ts_tablet_manager.cc:1145] Deleting tablet 4a3ba2cd77b8416db72d894b09cc1100
0113 02:51:37.052483 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:37.052555 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2620309}
I20250113 02:51:37.052665 1195 rpcz_store.cc:276] 0113 02:51:35.442633 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:35.442679 (+ 46us) service_pool.cc:225] Handling call
0113 02:51:36.919444 (+1476765us) ts_tablet_manager.cc:1145] Deleting tablet fa9052eea538439a9cb4c527e97897a4
0113 02:51:36.919453 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:36.919527 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1476375}
I20250113 02:51:37.052887 1198 tablet_replica.cc:331] T 377ebf66bbef40988043f6200ae0893c P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:37.053066 1198 raft_consensus.cc:2238] T 377ebf66bbef40988043f6200ae0893c P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: Raft consensus shutting down.
I20250113 02:51:37.053352 1198 pending_rounds.cc:62] T 377ebf66bbef40988043f6200ae0893c P 4216a7f853d642b7bc3e5cff458f3aca: Trying to abort 1 pending ops.
I20250113 02:51:37.053401 1198 pending_rounds.cc:69] T 377ebf66bbef40988043f6200ae0893c P 4216a7f853d642b7bc3e5cff458f3aca: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113673502840918016 op_type: NO_OP noop_request { }
I20250113 02:51:37.053499 1198 raft_consensus.cc:2883] T 377ebf66bbef40988043f6200ae0893c P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250113 02:51:37.053592 1198 raft_consensus.cc:2267] T 377ebf66bbef40988043f6200ae0893c P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Raft consensus is shut down!
W20250113 02:51:37.053707 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet fa9052eea538439a9cb4c527e97897a4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fa9052eea538439a9cb4c527e97897a4 already in progress: creating tablet
I20250113 02:51:37.053892 739 tablet_service.cc:1467] Processing CreateTablet for tablet 837bd7c330034bccbd45a08ca6beae1e (DEFAULT_TABLE table=default.Table_fd1e5342502a4cd9b3f8078ae35cbd52 [id=b4fe077a7bd74054898502244a6c5958]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:37.054131 1198 ts_tablet_manager.cc:1905] T 377ebf66bbef40988043f6200ae0893c P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:37.054307 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 837bd7c330034bccbd45a08ca6beae1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:37.056362 1192 tablet_bootstrap.cc:492] T 0b119704bc774ca9b4155c9619d4765b P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:37.057262 887 tablet_service.cc:1467] Processing CreateTablet for tablet 837bd7c330034bccbd45a08ca6beae1e (DEFAULT_TABLE table=default.Table_fd1e5342502a4cd9b3f8078ae35cbd52 [id=b4fe077a7bd74054898502244a6c5958]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:37.058048 1041 tablet_service.cc:1467] Processing CreateTablet for tablet 837bd7c330034bccbd45a08ca6beae1e (DEFAULT_TABLE table=default.Table_fd1e5342502a4cd9b3f8078ae35cbd52 [id=b4fe077a7bd74054898502244a6c5958]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:37.058414 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 837bd7c330034bccbd45a08ca6beae1e. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:37.064155 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 6e208f83f3384a78a959909b4ac690c0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6e208f83f3384a78a959909b4ac690c0 already in progress: creating tablet
I20250113 02:51:37.065272 1194 tablet_bootstrap.cc:492] T f56230a2c75d4763a54671c79358f4d9 P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:37.065412 1194 ts_tablet_manager.cc:1397] T f56230a2c75d4763a54671c79358f4d9 P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.056s user 0.002s sys 0.005s
W20250113 02:51:37.065632 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet db486978b33a45df90025988b084aecd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet db486978b33a45df90025988b084aecd already in progress: creating tablet
W20250113 02:51:37.065989 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 1c77033ca393469e9324d2406a65a5e9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1c77033ca393469e9324d2406a65a5e9 already in progress: creating tablet
I20250113 02:51:37.066089 1192 tablet_bootstrap.cc:654] T 0b119704bc774ca9b4155c9619d4765b P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
W20250113 02:51:37.066334 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 4a3ba2cd77b8416db72d894b09cc1100 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4a3ba2cd77b8416db72d894b09cc1100 already in progress: creating tablet
I20250113 02:51:37.068831 1194 raft_consensus.cc:357] T f56230a2c75d4763a54671c79358f4d9 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:37.069095 1194 raft_consensus.cc:383] T f56230a2c75d4763a54671c79358f4d9 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:37.069160 1194 raft_consensus.cc:738] T f56230a2c75d4763a54671c79358f4d9 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:37.069438 1194 consensus_queue.cc:260] T f56230a2c75d4763a54671c79358f4d9 P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:37.072366 2195 raft_consensus.cc:491] T c826a0b8327f44f9a00e5afe3a36cd8a P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:37.072515 2195 raft_consensus.cc:513] T c826a0b8327f44f9a00e5afe3a36cd8a P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:37.074215 1192 tablet_bootstrap.cc:492] T 0b119704bc774ca9b4155c9619d4765b P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
W20250113 02:51:37.074357 852 leader_election.cc:343] T c826a0b8327f44f9a00e5afe3a36cd8a P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Not found: Tablet not found: c826a0b8327f44f9a00e5afe3a36cd8a
I20250113 02:51:37.074406 2195 leader_election.cc:290] T c826a0b8327f44f9a00e5afe3a36cd8a P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587), 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
I20250113 02:51:37.074757 1192 ts_tablet_manager.cc:1397] T 0b119704bc774ca9b4155c9619d4765b P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.002s
I20250113 02:51:37.075002 1194 ts_tablet_manager.cc:1428] T f56230a2c75d4763a54671c79358f4d9 P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
W20250113 02:51:37.075279 851 leader_election.cc:343] T c826a0b8327f44f9a00e5afe3a36cd8a P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Not found: Tablet not found: c826a0b8327f44f9a00e5afe3a36cd8a
I20250113 02:51:37.075390 851 leader_election.cc:304] T c826a0b8327f44f9a00e5afe3a36cd8a P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df69f48656b44bb890f76587ee69cc0f; no voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca
I20250113 02:51:37.075331 1194 tablet_bootstrap.cc:492] T 6d4d8b5cc8ad4c9e935d8afaf5766e66 P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:37.076643 1195 tablet_replica.cc:331] T dd566a49d3ff4d3889b80d8d2ccbee0e P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:37.076881 1195 raft_consensus.cc:2238] T dd566a49d3ff4d3889b80d8d2ccbee0e P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:37.077013 1195 raft_consensus.cc:2267] T dd566a49d3ff4d3889b80d8d2ccbee0e P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:37.076970 1192 raft_consensus.cc:357] T 0b119704bc774ca9b4155c9619d4765b P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:37.077253 1192 raft_consensus.cc:383] T 0b119704bc774ca9b4155c9619d4765b P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:37.077363 1192 raft_consensus.cc:738] T 0b119704bc774ca9b4155c9619d4765b P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:37.077478 1195 ts_tablet_manager.cc:1905] T dd566a49d3ff4d3889b80d8d2ccbee0e P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:37.077678 1192 consensus_queue.cc:260] T 0b119704bc774ca9b4155c9619d4765b P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:37.078510 1192 ts_tablet_manager.cc:1428] T 0b119704bc774ca9b4155c9619d4765b P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250113 02:51:37.078897 1192 tablet_bootstrap.cc:492] T 9438472464304da9a0c7658d4f0736b6 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:37.082404 454 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_6b6db917d3974c9eab95631eb01b7a6e" } modify_external_catalogs: true
I20250113 02:51:37.082585 454 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_6b6db917d3974c9eab95631eb01b7a6e" } modify_external_catalogs: true
I20250113 02:51:37.084041 2195 raft_consensus.cc:2743] T c826a0b8327f44f9a00e5afe3a36cd8a P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:37.087287 889 tablet_service.cc:1467] Processing CreateTablet for tablet e51b750440a24570b734bb51e7a6f923 (DEFAULT_TABLE table=default.Table_6b6db917d3974c9eab95631eb01b7a6e [id=d1560d8fab564f3a8073fd394ee2f6ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:37.087659 894 tablet_service.cc:1467] Processing CreateTablet for tablet 9dc9ba26628441dab7256176bf6a0c3f (DEFAULT_TABLE table=default.Table_4ccbe1b1eddd4eaf83073751a60c7ba3 [id=1fac9e6d89064fa6a0f1b1f82826f81e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:37.088199 916 raft_consensus.cc:3054] T 1a63f485caf54068abd6b50d7de3d14a P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:37.088402 736 tablet_service.cc:1467] Processing CreateTablet for tablet 305e7a5304804580b7d302d82fdb5d6f (DEFAULT_TABLE table=default.Table_93aa0e80539449feacb8085adce4b4ea [id=17a95463aa1a455e92c152b416e0e698]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:37.088579 751 tablet_service.cc:1467] Processing CreateTablet for tablet 9dc9ba26628441dab7256176bf6a0c3f (DEFAULT_TABLE table=default.Table_4ccbe1b1eddd4eaf83073751a60c7ba3 [id=1fac9e6d89064fa6a0f1b1f82826f81e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:37.088613 2277 tablet.cc:1721] T 1cfbcd682b944c89847b40fe69f0a75e P 4216a7f853d642b7bc3e5cff458f3aca: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250113 02:51:37.088824 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 837bd7c330034bccbd45a08ca6beae1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:37.089021 746 tablet_service.cc:1467] Processing CreateTablet for tablet cff2d65491ac47cea9aacd87e3a1f00d (DEFAULT_TABLE table=default.Table_ac79159d146b433ebdea04d6746fb7d4 [id=00da4e53785343eaa164830e553f87ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:37.089321 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9dc9ba26628441dab7256176bf6a0c3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:37.089432 744 tablet_service.cc:1467] Processing CreateTablet for tablet e51b750440a24570b734bb51e7a6f923 (DEFAULT_TABLE table=default.Table_6b6db917d3974c9eab95631eb01b7a6e [id=d1560d8fab564f3a8073fd394ee2f6ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:37.091100 1034 tablet_service.cc:1467] Processing CreateTablet for tablet 9dc9ba26628441dab7256176bf6a0c3f (DEFAULT_TABLE table=default.Table_4ccbe1b1eddd4eaf83073751a60c7ba3 [id=1fac9e6d89064fa6a0f1b1f82826f81e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:37.091234 1038 tablet_service.cc:1467] Processing CreateTablet for tablet cff2d65491ac47cea9aacd87e3a1f00d (DEFAULT_TABLE table=default.Table_ac79159d146b433ebdea04d6746fb7d4 [id=00da4e53785343eaa164830e553f87ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:37.091648 1035 tablet_service.cc:1467] Processing CreateTablet for tablet 305e7a5304804580b7d302d82fdb5d6f (DEFAULT_TABLE table=default.Table_93aa0e80539449feacb8085adce4b4ea [id=17a95463aa1a455e92c152b416e0e698]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:37.094313 882 tablet_service.cc:1467] Processing CreateTablet for tablet 305e7a5304804580b7d302d82fdb5d6f (DEFAULT_TABLE table=default.Table_93aa0e80539449feacb8085adce4b4ea [id=17a95463aa1a455e92c152b416e0e698]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:37.094403 884 tablet_service.cc:1467] Processing CreateTablet for tablet cff2d65491ac47cea9aacd87e3a1f00d (DEFAULT_TABLE table=default.Table_ac79159d146b433ebdea04d6746fb7d4 [id=00da4e53785343eaa164830e553f87ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:37.094698 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 305e7a5304804580b7d302d82fdb5d6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:37.094980 891 tablet_service.cc:1467] Processing CreateTablet for tablet ddf0295c286340bcaea28a1468f3c557 (DEFAULT_TABLE table=default.Table_66000626bf85477d9623d439bad71554 [id=67631710ddb749f7af85fe8008b53791]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:37.095376 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ddf0295c286340bcaea28a1468f3c557. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:37.097415 1041 tablet_service.cc:1467] Processing CreateTablet for tablet e51b750440a24570b734bb51e7a6f923 (DEFAULT_TABLE table=default.Table_6b6db917d3974c9eab95631eb01b7a6e [id=d1560d8fab564f3a8073fd394ee2f6ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:37.098181 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cff2d65491ac47cea9aacd87e3a1f00d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:37.100574 2227 raft_consensus.cc:491] T 1014e51f1c184d729552a5b34292f232 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:37.100719 2227 raft_consensus.cc:513] T 1014e51f1c184d729552a5b34292f232 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:37.101467 2227 leader_election.cc:290] T 1014e51f1c184d729552a5b34292f232 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:37.101665 1192 tablet_bootstrap.cc:654] T 9438472464304da9a0c7658d4f0736b6 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:37.102463 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1014e51f1c184d729552a5b34292f232" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
W20250113 02:51:37.103152 709 leader_election.cc:343] T 1014e51f1c184d729552a5b34292f232 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:37.104703 2254 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45e9d43ecac0401fb2c18fe5eaf089b5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 4 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e13f5bcdcee34225a46c6d2651e5ede9" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44217 } } peers { permanent_uuid: "6bbc3541604a455584bb32dfc2cc4efd" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 35271 } } peers { permanent_uuid: "45e9d43ecac0401fb2c18fe5eaf089b5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42067 } } }
I20250113 02:51:37.106988 742 tablet_service.cc:1467] Processing CreateTablet for tablet ddf0295c286340bcaea28a1468f3c557 (DEFAULT_TABLE table=default.Table_66000626bf85477d9623d439bad71554 [id=67631710ddb749f7af85fe8008b53791]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:37.107206 1992 sys_catalog.cc:564] T 00000000000000000000000000000000 P 45e9d43ecac0401fb2c18fe5eaf089b5 [sys.catalog]: configured and running, proceeding with master startup.
I20250113 02:51:37.109798 2254 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45e9d43ecac0401fb2c18fe5eaf089b5 [sys.catalog]: This master's current role is: FOLLOWER
I20250113 02:51:37.110936 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet cb141fb2e9894757b3ae423242ad93d2
I20250113 02:51:37.112812 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 60fac96c00d64ec193b0c706ec64bfaf
I20250113 02:51:37.114722 890 tablet_service.cc:1467] Processing CreateTablet for tablet ab5209b86b1d4078b28bf1517dce8136 (DEFAULT_TABLE table=default.Table_fa2b5d3af3ce4b3fae8bbaa614a7d6c0 [id=622c64099f5a4976b219a17acf0bb6d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:37.115199 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab5209b86b1d4078b28bf1517dce8136. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:37.120059 1037 tablet_service.cc:1467] Processing CreateTablet for tablet 4aec0adca28d4da2bf09bc8966ed6886 (DEFAULT_TABLE table=default.Table_b2f523343ede4ce5950a27d55c16eac2 [id=0b6c39954268497e97619a15d4d7dc15]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:37.120030 1032 tablet_service.cc:1467] Processing CreateTablet for tablet 43c78b3da74748158f5534203bdd0d33 (DEFAULT_TABLE table=default.Table_ba3f5ae52be04b5e88f140ae075c1ee1 [id=8b934f50c8e04ad9910fd9467162d90d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:37.120138 1036 tablet_service.cc:1467] Processing CreateTablet for tablet ddf0295c286340bcaea28a1468f3c557 (DEFAULT_TABLE table=default.Table_66000626bf85477d9623d439bad71554 [id=67631710ddb749f7af85fe8008b53791]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:37.120527 1040 tablet_service.cc:1514] Processing DeleteTablet for tablet cb141fb2e9894757b3ae423242ad93d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:36 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:37.120734 1042 tablet_service.cc:1514] Processing DeleteTablet for tablet 60fac96c00d64ec193b0c706ec64bfaf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:36 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:37.121078 1023 tablet_service.cc:1467] Processing CreateTablet for tablet ab5209b86b1d4078b28bf1517dce8136 (DEFAULT_TABLE table=default.Table_fa2b5d3af3ce4b3fae8bbaa614a7d6c0 [id=622c64099f5a4976b219a17acf0bb6d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:37.121520 1031 tablet_service.cc:1467] Processing CreateTablet for tablet a6d0f9a0a7104b8fb3d22aac976b7d99 (DEFAULT_TABLE table=default.Table_ae7b078c404f4196ac5289edb6ac5b3f [id=8ff9f8850f374206826a9a8e1403bb72]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:37.122005 1191 tablet_bootstrap.cc:492] T f8a6eebb6b6d4b63903e68750e181707 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:37.122226 1191 ts_tablet_manager.cc:1397] T f8a6eebb6b6d4b63903e68750e181707 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.340s user 0.028s sys 0.070s
I20250113 02:51:37.122596 2195 raft_consensus.cc:491] T 77cf8b0c824f401aba1880eb7fd1807b P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:37.122750 2195 raft_consensus.cc:513] T 77cf8b0c824f401aba1880eb7fd1807b P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:37.123133 1191 raft_consensus.cc:357] T f8a6eebb6b6d4b63903e68750e181707 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:37.123438 1191 raft_consensus.cc:383] T f8a6eebb6b6d4b63903e68750e181707 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:37.123531 1191 raft_consensus.cc:738] T f8a6eebb6b6d4b63903e68750e181707 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:37.123868 1191 consensus_queue.cc:260] T f8a6eebb6b6d4b63903e68750e181707 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:37.124696 1042 tablet_service.cc:1467] Processing CreateTablet for tablet 32e3dc0b7c044415a6343368ffa08552 (DEFAULT_TABLE table=default.Table_53a85211f49241c4af531e479ecdc0cb [id=ff8b884b04954ff48a7c8a72df232e8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:37.125195 739 tablet_service.cc:1514] Processing DeleteTablet for tablet cb141fb2e9894757b3ae423242ad93d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:36 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:37.126639 754 tablet_service.cc:1467] Processing CreateTablet for tablet 43c78b3da74748158f5534203bdd0d33 (DEFAULT_TABLE table=default.Table_ba3f5ae52be04b5e88f140ae075c1ee1 [id=8b934f50c8e04ad9910fd9467162d90d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:37.126770 738 tablet_service.cc:1467] Processing CreateTablet for tablet 4aec0adca28d4da2bf09bc8966ed6886 (DEFAULT_TABLE table=default.Table_b2f523343ede4ce5950a27d55c16eac2 [id=0b6c39954268497e97619a15d4d7dc15]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:37.126932 743 tablet_service.cc:1467] Processing CreateTablet for tablet ab5209b86b1d4078b28bf1517dce8136 (DEFAULT_TABLE table=default.Table_fa2b5d3af3ce4b3fae8bbaa614a7d6c0 [id=622c64099f5a4976b219a17acf0bb6d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:37.127231 1191 ts_tablet_manager.cc:1428] T f8a6eebb6b6d4b63903e68750e181707 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250113 02:51:37.127312 752 tablet_service.cc:1467] Processing CreateTablet for tablet a6d0f9a0a7104b8fb3d22aac976b7d99 (DEFAULT_TABLE table=default.Table_ae7b078c404f4196ac5289edb6ac5b3f [id=8ff9f8850f374206826a9a8e1403bb72]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:37.127317 747 tablet_service.cc:1467] Processing CreateTablet for tablet 32e3dc0b7c044415a6343368ffa08552 (DEFAULT_TABLE table=default.Table_53a85211f49241c4af531e479ecdc0cb [id=ff8b884b04954ff48a7c8a72df232e8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:37.130089 741 tablet_service.cc:1467] Processing CreateTablet for tablet 221808fe5b3a4edbab5e7bdff971c032 (DEFAULT_TABLE table=default.Table_2ebcc2867e6e41d1aaf0f53f9582a11f [id=14bb96a3166b4952b143b16b7ae795fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:37.130089 1040 tablet_service.cc:1467] Processing CreateTablet for tablet 221808fe5b3a4edbab5e7bdff971c032 (DEFAULT_TABLE table=default.Table_2ebcc2867e6e41d1aaf0f53f9582a11f [id=14bb96a3166b4952b143b16b7ae795fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:37.131304 1191 tablet_bootstrap.cc:492] T 0c408921a09a463eb464e08edcf042b3 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:37.133412 748 tablet_service.cc:1467] Processing CreateTablet for tablet 038ff55bc0184ad1899bee4f96d67d6f (DEFAULT_TABLE table=default.Table_14183012578a4660ad28c7c483de871d [id=7e412e073b4d411fabf1728c023601c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:37.134347 1191 tablet_bootstrap.cc:654] T 0c408921a09a463eb464e08edcf042b3 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:37.142227 2195 leader_election.cc:290] T 77cf8b0c824f401aba1880eb7fd1807b P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587), 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
I20250113 02:51:37.142377 2195 raft_consensus.cc:491] T 0a7e7713bc8648378b668d8b769ac927 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:37.142490 2195 raft_consensus.cc:513] T 0a7e7713bc8648378b668d8b769ac927 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:37.143043 2195 leader_election.cc:290] T 0a7e7713bc8648378b668d8b769ac927 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587), 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
I20250113 02:51:37.143494 1191 tablet_bootstrap.cc:492] T 0c408921a09a463eb464e08edcf042b3 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:37.143471 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1014e51f1c184d729552a5b34292f232" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:37.143724 1191 ts_tablet_manager.cc:1397] T 0c408921a09a463eb464e08edcf042b3 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.002s
I20250113 02:51:37.144457 1191 raft_consensus.cc:357] T 0c408921a09a463eb464e08edcf042b3 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:37.144992 1191 raft_consensus.cc:383] T 0c408921a09a463eb464e08edcf042b3 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:37.145205 1191 raft_consensus.cc:738] T 0c408921a09a463eb464e08edcf042b3 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:37.145642 1191 consensus_queue.cc:260] T 0c408921a09a463eb464e08edcf042b3 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:37.146688 1192 tablet_bootstrap.cc:492] T 9438472464304da9a0c7658d4f0736b6 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:37.146821 1192 ts_tablet_manager.cc:1397] T 9438472464304da9a0c7658d4f0736b6 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.068s user 0.002s sys 0.005s
I20250113 02:51:37.147272 1191 ts_tablet_manager.cc:1428] T 0c408921a09a463eb464e08edcf042b3 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20250113 02:51:37.147491 1192 raft_consensus.cc:357] T 9438472464304da9a0c7658d4f0736b6 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:37.147678 1192 raft_consensus.cc:383] T 9438472464304da9a0c7658d4f0736b6 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:37.147694 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet c1fb2b36c0504cb4ba32671ae4c620d1 (table default.Table_53a85211f49241c4af531e479ecdc0cb [id=ff8b884b04954ff48a7c8a72df232e8a]) successfully deleted
I20250113 02:51:37.147744 1192 raft_consensus.cc:738] T 9438472464304da9a0c7658d4f0736b6 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:37.148144 2221 raft_consensus.cc:491] T fb18c16fb3ba44ec8a190b88d3bbd1b1 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:37.146086 895 tablet_service.cc:1467] Processing CreateTablet for tablet 221808fe5b3a4edbab5e7bdff971c032 (DEFAULT_TABLE table=default.Table_2ebcc2867e6e41d1aaf0f53f9582a11f [id=14bb96a3166b4952b143b16b7ae795fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:37.148257 2221 raft_consensus.cc:513] T fb18c16fb3ba44ec8a190b88d3bbd1b1 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:37.148804 2221 leader_election.cc:290] T fb18c16fb3ba44ec8a190b88d3bbd1b1 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:37.146378 896 tablet_service.cc:1514] Processing DeleteTablet for tablet cb141fb2e9894757b3ae423242ad93d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:36 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:37.146488 880 tablet_service.cc:1514] Processing DeleteTablet for tablet 60fac96c00d64ec193b0c706ec64bfaf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:36 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:37.147011 899 tablet_service.cc:1467] Processing CreateTablet for tablet 32e3dc0b7c044415a6343368ffa08552 (DEFAULT_TABLE table=default.Table_53a85211f49241c4af531e479ecdc0cb [id=ff8b884b04954ff48a7c8a72df232e8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:37.147336 898 tablet_service.cc:1467] Processing CreateTablet for tablet 038ff55bc0184ad1899bee4f96d67d6f (DEFAULT_TABLE table=default.Table_14183012578a4660ad28c7c483de871d [id=7e412e073b4d411fabf1728c023601c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:37.149827 1191 tablet_bootstrap.cc:492] T dc680b5680b84abbab219296bc7ee707 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
W20250113 02:51:37.149996 852 leader_election.cc:343] T 77cf8b0c824f401aba1880eb7fd1807b P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Not found: Tablet not found: 77cf8b0c824f401aba1880eb7fd1807b
I20250113 02:51:37.150045 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb18c16fb3ba44ec8a190b88d3bbd1b1" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
I20250113 02:51:37.148095 2072 raft_consensus.cc:491] T f8a6eebb6b6d4b63903e68750e181707 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250113 02:51:37.150283 711 leader_election.cc:343] T 1014e51f1c184d729552a5b34292f232 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:37.150427 711 leader_election.cc:304] T 1014e51f1c184d729552a5b34292f232 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:37.150403 2072 raft_consensus.cc:513] T f8a6eebb6b6d4b63903e68750e181707 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:37.150589 739 tablet_service.cc:1514] Processing DeleteTablet for tablet 60fac96c00d64ec193b0c706ec64bfaf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:36 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:37.151001 2072 leader_election.cc:290] T f8a6eebb6b6d4b63903e68750e181707 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:37.151134 2072 raft_consensus.cc:2743] T 1014e51f1c184d729552a5b34292f232 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:37.150290 775 raft_consensus.cc:2463] T fb18c16fb3ba44ec8a190b88d3bbd1b1 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 0.
I20250113 02:51:37.151835 995 leader_election.cc:304] T fb18c16fb3ba44ec8a190b88d3bbd1b1 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters:
I20250113 02:51:37.152343 2113 raft_consensus.cc:2798] T fb18c16fb3ba44ec8a190b88d3bbd1b1 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 02:51:37.152426 2113 raft_consensus.cc:491] T fb18c16fb3ba44ec8a190b88d3bbd1b1 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 02:51:37.152499 2113 raft_consensus.cc:3054] T fb18c16fb3ba44ec8a190b88d3bbd1b1 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:37.152547 1192 consensus_queue.cc:260] T 9438472464304da9a0c7658d4f0736b6 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:37.153090 1192 ts_tablet_manager.cc:1428] T 9438472464304da9a0c7658d4f0736b6 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20250113 02:51:37.153342 1192 tablet_bootstrap.cc:492] T d8bb60fdd06946739b75b255303d9d24 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:37.153494 1191 tablet_bootstrap.cc:654] T dc680b5680b84abbab219296bc7ee707 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:37.155119 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_c4e4c90fea79464ca6cdb5decd2c4a5c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:37.155586 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8a6eebb6b6d4b63903e68750e181707" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
W20250113 02:51:37.155705 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c4e4c90fea79464ca6cdb5decd2c4a5c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:37.155797 883 tablet_service.cc:1467] Processing CreateTablet for tablet 43c78b3da74748158f5534203bdd0d33 (DEFAULT_TABLE table=default.Table_ba3f5ae52be04b5e88f140ae075c1ee1 [id=8b934f50c8e04ad9910fd9467162d90d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:37.156054 888 tablet_service.cc:1467] Processing CreateTablet for tablet 4aec0adca28d4da2bf09bc8966ed6886 (DEFAULT_TABLE table=default.Table_b2f523343ede4ce5950a27d55c16eac2 [id=0b6c39954268497e97619a15d4d7dc15]), partition=RANGE (key) PARTITION UNBOUNDED
W20250113 02:51:37.156513 709 leader_election.cc:343] T f8a6eebb6b6d4b63903e68750e181707 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:37.159056 446 catalog_manager.cc:5618] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587) does not have the latest schema for tablet aa5c172a69bf4ca79999f4b58df08de8 (table default.Table_3f1e7146fed34d53896f138e43a4d26e [id=5dceb76e1dbe422ab98d4da5bf6c45a0]). Expected version 1 got 0
I20250113 02:51:37.159297 446 catalog_manager.cc:5618] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587) does not have the latest schema for tablet 7527e6e0b65e4a43b7f48899a34cdf7f (table default.Table_c8b93363989c4e109540af6bef426f73 [id=b7846a8ebee84005b1b5d259560a26bd]). Expected version 1 got 0
I20250113 02:51:37.159824 2227 raft_consensus.cc:491] T 80d31d5ead804ddaa032969e401a8f7e P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:37.159965 2227 raft_consensus.cc:513] T 80d31d5ead804ddaa032969e401a8f7e P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:37.161973 1191 tablet_bootstrap.cc:492] T dc680b5680b84abbab219296bc7ee707 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:37.162731 1191 ts_tablet_manager.cc:1397] T dc680b5680b84abbab219296bc7ee707 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.001s
I20250113 02:51:37.162951 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80d31d5ead804ddaa032969e401a8f7e" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:37.164215 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e51b750440a24570b734bb51e7a6f923. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:37.164335 446 master_service.cc:946] ReplaceTablet: received request to replace tablet 5d2d14548b734b1fbe57d68c0ea3e2f8 from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:37.165347 1191 raft_consensus.cc:357] T dc680b5680b84abbab219296bc7ee707 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:37.165524 1191 raft_consensus.cc:383] T dc680b5680b84abbab219296bc7ee707 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:37.165581 1191 raft_consensus.cc:738] T dc680b5680b84abbab219296bc7ee707 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:37.166288 2281 tablet.cc:1721] T 1cfbcd682b944c89847b40fe69f0a75e P 1186fa869eb14812832fa6562531f196: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250113 02:51:37.166975 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cff2d65491ac47cea9aacd87e3a1f00d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:37.167214 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 43c78b3da74748158f5534203bdd0d33. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:37.167385 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab5209b86b1d4078b28bf1517dce8136. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:37.170326 1036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ddf0295c286340bcaea28a1468f3c557. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:37.170572 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9fad58cccf1744b0a476523e0af0036e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:37.170704 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 305e7a5304804580b7d302d82fdb5d6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:37.171082 1040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 221808fe5b3a4edbab5e7bdff971c032. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:37.171607 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb18c16fb3ba44ec8a190b88d3bbd1b1" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:37.172586 998 leader_election.cc:343] T fb18c16fb3ba44ec8a190b88d3bbd1b1 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:37.172705 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e51b750440a24570b734bb51e7a6f923. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:37.176172 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 51e44b812fb84b8cb9e5ebe98fb187e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:37.176652 1037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4aec0adca28d4da2bf09bc8966ed6886. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:37.176856 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34245259b47743a59da080379b33e719. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:37.177322 1042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 32e3dc0b7c044415a6343368ffa08552. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:37.166661 709 leader_election.cc:343] T 80d31d5ead804ddaa032969e401a8f7e P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:37.183439 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6d0f9a0a7104b8fb3d22aac976b7d99. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:37.184394 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet fe3d35d9aaf64188aaa3c1ac38d279d8
I20250113 02:51:37.185863 886 tablet_service.cc:1467] Processing CreateTablet for tablet a6d0f9a0a7104b8fb3d22aac976b7d99 (DEFAULT_TABLE table=default.Table_ae7b078c404f4196ac5289edb6ac5b3f [id=8ff9f8850f374206826a9a8e1403bb72]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:37.186509 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9dc9ba26628441dab7256176bf6a0c3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:37.186658 1198 ts_tablet_manager.cc:1918] T 377ebf66bbef40988043f6200ae0893c P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250113 02:51:37.186754 1198 log.cc:1198] T 377ebf66bbef40988043f6200ae0893c P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/377ebf66bbef40988043f6200ae0893c
I20250113 02:51:37.187086 1198 ts_tablet_manager.cc:1939] T 377ebf66bbef40988043f6200ae0893c P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:37.187639 880 tablet_service.cc:1514] Processing DeleteTablet for tablet fe3d35d9aaf64188aaa3c1ac38d279d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:36 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:37.187844 739 tablet_service.cc:1514] Processing DeleteTablet for tablet fe3d35d9aaf64188aaa3c1ac38d279d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:36 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:37.187881 2113 raft_consensus.cc:513] T fb18c16fb3ba44ec8a190b88d3bbd1b1 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
W20250113 02:51:37.150966 852 leader_election.cc:343] T 0a7e7713bc8648378b668d8b769ac927 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Not found: Tablet not found: 0a7e7713bc8648378b668d8b769ac927
W20250113 02:51:37.198336 852 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.3:42587, user_credentials={real_user=slave}} blocked reactor thread for 47369.3us
I20250113 02:51:37.162400 2227 leader_election.cc:290] T 80d31d5ead804ddaa032969e401a8f7e P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:37.203272 2113 leader_election.cc:290] T fb18c16fb3ba44ec8a190b88d3bbd1b1 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Requested vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:37.207834 1196 tablet_replica.cc:331] T a97efc70946c4af487e33a85abf6b90c P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:37.208643 1033 tablet_service.cc:1514] Processing DeleteTablet for tablet fe3d35d9aaf64188aaa3c1ac38d279d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:36 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:37.207927 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80d31d5ead804ddaa032969e401a8f7e" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:37.209031 2228 raft_consensus.cc:491] T c518f791b4d74c54ab0c0364d29b4a64 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:37.208318 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 32e3dc0b7c044415a6343368ffa08552. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:37.209321 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6d0f9a0a7104b8fb3d22aac976b7d99. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:37.209251 2228 raft_consensus.cc:513] T c518f791b4d74c54ab0c0364d29b4a64 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:37.209991 2228 leader_election.cc:290] T c518f791b4d74c54ab0c0364d29b4a64 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:37.207665 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8a6eebb6b6d4b63903e68750e181707" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:37.211344 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4aec0adca28d4da2bf09bc8966ed6886. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:37.211385 2221 raft_consensus.cc:491] T 7527e6e0b65e4a43b7f48899a34cdf7f P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:37.211520 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 43c78b3da74748158f5534203bdd0d33. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:37.211544 2221 raft_consensus.cc:513] T 7527e6e0b65e4a43b7f48899a34cdf7f P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:37.211773 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb18c16fb3ba44ec8a190b88d3bbd1b1" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f"
I20250113 02:51:37.212085 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 221808fe5b3a4edbab5e7bdff971c032. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:37.212327 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 038ff55bc0184ad1899bee4f96d67d6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:37.213491 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c518f791b4d74c54ab0c0364d29b4a64" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:37.214018 711 leader_election.cc:343] T 80d31d5ead804ddaa032969e401a8f7e P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:37.214138 711 leader_election.cc:304] T 80d31d5ead804ddaa032969e401a8f7e P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:37.214583 2228 raft_consensus.cc:2743] T 80d31d5ead804ddaa032969e401a8f7e P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250113 02:51:37.215047 711 leader_election.cc:343] T c518f791b4d74c54ab0c0364d29b4a64 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:37.215382 1196 raft_consensus.cc:2238] T a97efc70946c4af487e33a85abf6b90c P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:37.166661 1191 consensus_queue.cc:260] T dc680b5680b84abbab219296bc7ee707 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:37.218308 1191 ts_tablet_manager.cc:1428] T dc680b5680b84abbab219296bc7ee707 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.054s user 0.001s sys 0.001s
I20250113 02:51:37.218674 1191 tablet_bootstrap.cc:492] T 153b8f90b5a847b88944f683d306f579 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:37.218935 2221 leader_election.cc:290] T 7527e6e0b65e4a43b7f48899a34cdf7f P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:37.221213 1191 tablet_bootstrap.cc:654] T 153b8f90b5a847b88944f683d306f579 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:37.224118 2228 raft_consensus.cc:491] T d9df314ffaf042ec80180868d7e7bdf7 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:37.224290 2228 raft_consensus.cc:513] T d9df314ffaf042ec80180868d7e7bdf7 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:37.224920 2228 leader_election.cc:290] T d9df314ffaf042ec80180868d7e7bdf7 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:37.230448 2221 raft_consensus.cc:491] T 9438472464304da9a0c7658d4f0736b6 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:37.230613 2221 raft_consensus.cc:513] T 9438472464304da9a0c7658d4f0736b6 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:37.232160 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 978) took 2779 ms. Trace:
I20250113 02:51:37.232259 1198 rpcz_store.cc:276] 0113 02:51:34.452495 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:34.476103 (+ 23608us) service_pool.cc:225] Handling call
0113 02:51:37.052849 (+2576746us) ts_tablet_manager.cc:1145] Deleting tablet 377ebf66bbef40988043f6200ae0893c
0113 02:51:37.052857 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:37.161461 (+108604us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:37.186581 (+ 25120us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:37.232136 (+ 45555us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2576445}
I20250113 02:51:37.232589 1198 tablet_replica.cc:331] T e468b652f71f44e3945f8d919ac0e1a8 P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:37.232772 1198 raft_consensus.cc:2238] T e468b652f71f44e3945f8d919ac0e1a8 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:37.232884 1198 raft_consensus.cc:2267] T e468b652f71f44e3945f8d919ac0e1a8 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:37.233366 1198 ts_tablet_manager.cc:1905] T e468b652f71f44e3945f8d919ac0e1a8 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250113 02:51:37.233443 711 leader_election.cc:343] T f8a6eebb6b6d4b63903e68750e181707 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:37.233624 711 leader_election.cc:304] T f8a6eebb6b6d4b63903e68750e181707 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:37.234170 2228 raft_consensus.cc:2743] T f8a6eebb6b6d4b63903e68750e181707 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250113 02:51:37.235373 998 leader_election.cc:343] T fb18c16fb3ba44ec8a190b88d3bbd1b1 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:37.235821 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c518f791b4d74c54ab0c0364d29b4a64" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:37.238173 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet 377ebf66bbef40988043f6200ae0893c (table default.Table_5ddfdff2cc8c4d12a1c4552e9a302027 [id=8b35e389de9f4917904e87b43618737e]) successfully deleted
I20250113 02:51:37.239934 1192 tablet_bootstrap.cc:654] T d8bb60fdd06946739b75b255303d9d24 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:37.236507 2221 leader_election.cc:290] T 9438472464304da9a0c7658d4f0736b6 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
W20250113 02:51:37.240959 709 leader_election.cc:343] T c518f791b4d74c54ab0c0364d29b4a64 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:37.241078 709 leader_election.cc:304] T c518f791b4d74c54ab0c0364d29b4a64 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:37.241642 2228 raft_consensus.cc:2743] T c518f791b4d74c54ab0c0364d29b4a64 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:37.241935 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9df314ffaf042ec80180868d7e7bdf7" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:37.242990 1191 tablet_bootstrap.cc:492] T 153b8f90b5a847b88944f683d306f579 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
W20250113 02:51:37.243441 709 leader_election.cc:343] T d9df314ffaf042ec80180868d7e7bdf7 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:37.244587 1191 ts_tablet_manager.cc:1397] T 153b8f90b5a847b88944f683d306f579 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.026s user 0.000s sys 0.007s
I20250113 02:51:37.246529 1191 raft_consensus.cc:357] T 153b8f90b5a847b88944f683d306f579 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:37.246937 1198 ts_tablet_manager.cc:1918] T e468b652f71f44e3945f8d919ac0e1a8 P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:37.247249 1191 raft_consensus.cc:383] T 153b8f90b5a847b88944f683d306f579 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:37.247505 1191 raft_consensus.cc:738] T 153b8f90b5a847b88944f683d306f579 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:37.248010 1191 consensus_queue.cc:260] T 153b8f90b5a847b88944f683d306f579 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:37.248448 1198 log.cc:1198] T e468b652f71f44e3945f8d919ac0e1a8 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/e468b652f71f44e3945f8d919ac0e1a8
I20250113 02:51:37.250351 2113 consensus_queue.cc:1035] T 56b022c94847411785a7e9db3757a2d9 P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 02:51:37.255023 1198 ts_tablet_manager.cc:1939] T e468b652f71f44e3945f8d919ac0e1a8 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:37.248617 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9438472464304da9a0c7658d4f0736b6" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
I20250113 02:51:37.248714 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb18c16fb3ba44ec8a190b88d3bbd1b1" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196"
I20250113 02:51:37.255614 775 raft_consensus.cc:3054] T fb18c16fb3ba44ec8a190b88d3bbd1b1 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:37.255648 769 raft_consensus.cc:2463] T 9438472464304da9a0c7658d4f0736b6 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 0.
I20250113 02:51:37.255851 1191 ts_tablet_manager.cc:1428] T 153b8f90b5a847b88944f683d306f579 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.011s user 0.000s sys 0.002s
I20250113 02:51:37.256201 1191 tablet_bootstrap.cc:492] T 438604e8859f422f9caccb61c5b5b125 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:37.256726 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 983) took 2733 ms. Trace:
I20250113 02:51:37.257606 1198 rpcz_store.cc:276] 0113 02:51:34.523497 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:34.529922 (+ 6425us) service_pool.cc:225] Handling call
0113 02:51:37.232537 (+2702615us) ts_tablet_manager.cc:1145] Deleting tablet e468b652f71f44e3945f8d919ac0e1a8
0113 02:51:37.232549 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:37.237656 (+ 5107us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:37.246838 (+ 9182us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:37.256708 (+ 9870us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2702392}
I20250113 02:51:37.249387 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7527e6e0b65e4a43b7f48899a34cdf7f" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
I20250113 02:51:37.259045 995 leader_election.cc:304] T 9438472464304da9a0c7658d4f0736b6 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters:
I20250113 02:51:37.259124 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet e468b652f71f44e3945f8d919ac0e1a8 (table default.Table_c52b68d0d3d74003bf10bc8ff205d25b [id=caf9972c69f14619be22fb3a9e563b05]) successfully deleted
I20250113 02:51:37.259524 1198 tablet_replica.cc:331] T 0fe5421dc3ea4be684cab084f13d8df0 P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:37.259907 1198 raft_consensus.cc:2238] T 0fe5421dc3ea4be684cab084f13d8df0 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:37.260201 775 raft_consensus.cc:2463] T fb18c16fb3ba44ec8a190b88d3bbd1b1 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 1.
I20250113 02:51:37.260396 771 raft_consensus.cc:2463] T 7527e6e0b65e4a43b7f48899a34cdf7f P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 0.
I20250113 02:51:37.261665 1191 tablet_bootstrap.cc:654] T 438604e8859f422f9caccb61c5b5b125 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:37.261324 995 leader_election.cc:304] T fb18c16fb3ba44ec8a190b88d3bbd1b1 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters: df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:37.263399 995 leader_election.cc:304] T 7527e6e0b65e4a43b7f48899a34cdf7f P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters:
I20250113 02:51:37.263734 2221 raft_consensus.cc:2798] T fb18c16fb3ba44ec8a190b88d3bbd1b1 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Leader election won for term 1
I20250113 02:51:37.259601 2113 raft_consensus.cc:2798] T 9438472464304da9a0c7658d4f0736b6 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 02:51:37.265575 2113 raft_consensus.cc:491] T 9438472464304da9a0c7658d4f0736b6 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 02:51:37.265838 2113 raft_consensus.cc:3054] T 9438472464304da9a0c7658d4f0736b6 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:37.270169 2113 raft_consensus.cc:513] T 9438472464304da9a0c7658d4f0736b6 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:37.271113 2228 raft_consensus.cc:491] T 655b63ff61ff4f4aa08e0449df39352a P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:37.271296 2228 raft_consensus.cc:513] T 655b63ff61ff4f4aa08e0449df39352a P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:37.272397 2228 leader_election.cc:290] T 655b63ff61ff4f4aa08e0449df39352a P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:37.234746 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9df314ffaf042ec80180868d7e7bdf7" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:37.274194 1196 raft_consensus.cc:2267] T a97efc70946c4af487e33a85abf6b90c P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:37.241202 1194 tablet_bootstrap.cc:654] T 6d4d8b5cc8ad4c9e935d8afaf5766e66 P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:37.242614 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9438472464304da9a0c7658d4f0736b6" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:37.236456 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7527e6e0b65e4a43b7f48899a34cdf7f" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:37.274963 1196 ts_tablet_manager.cc:1905] T a97efc70946c4af487e33a85abf6b90c P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:37.273078 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "655b63ff61ff4f4aa08e0449df39352a" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
W20250113 02:51:37.275749 709 leader_election.cc:343] T 655b63ff61ff4f4aa08e0449df39352a P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:37.277644 2113 leader_election.cc:290] T 9438472464304da9a0c7658d4f0736b6 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Requested vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:37.277889 2113 raft_consensus.cc:2798] T 7527e6e0b65e4a43b7f48899a34cdf7f P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 02:51:37.278378 2113 raft_consensus.cc:491] T 7527e6e0b65e4a43b7f48899a34cdf7f P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 02:51:37.278700 2113 raft_consensus.cc:3054] T 7527e6e0b65e4a43b7f48899a34cdf7f P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Advancing to term 1
W20250113 02:51:37.280565 998 leader_election.cc:343] T 9438472464304da9a0c7658d4f0736b6 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:37.280548 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9438472464304da9a0c7658d4f0736b6" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f"
W20250113 02:51:37.281041 998 leader_election.cc:343] T 7527e6e0b65e4a43b7f48899a34cdf7f P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:37.279021 1198 raft_consensus.cc:2267] T 0fe5421dc3ea4be684cab084f13d8df0 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus is shut down!
W20250113 02:51:37.282040 711 leader_election.cc:343] T d9df314ffaf042ec80180868d7e7bdf7 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:37.282204 711 leader_election.cc:304] T d9df314ffaf042ec80180868d7e7bdf7 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:37.282385 1198 ts_tablet_manager.cc:1905] T 0fe5421dc3ea4be684cab084f13d8df0 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:37.283123 2228 raft_consensus.cc:2743] T d9df314ffaf042ec80180868d7e7bdf7 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:37.283123 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "655b63ff61ff4f4aa08e0449df39352a" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:37.288306 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9438472464304da9a0c7658d4f0736b6" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196"
I20250113 02:51:37.288708 774 raft_consensus.cc:3054] T 9438472464304da9a0c7658d4f0736b6 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:37.290515 2113 raft_consensus.cc:513] T 7527e6e0b65e4a43b7f48899a34cdf7f P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
W20250113 02:51:37.291801 711 leader_election.cc:343] T 655b63ff61ff4f4aa08e0449df39352a P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:37.291949 711 leader_election.cc:304] T 655b63ff61ff4f4aa08e0449df39352a P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:37.292050 1194 tablet_bootstrap.cc:492] T 6d4d8b5cc8ad4c9e935d8afaf5766e66 P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:37.292230 1194 ts_tablet_manager.cc:1397] T 6d4d8b5cc8ad4c9e935d8afaf5766e66 P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.217s user 0.004s sys 0.001s
I20250113 02:51:37.292421 2228 raft_consensus.cc:2743] T 655b63ff61ff4f4aa08e0449df39352a P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:37.293146 1194 raft_consensus.cc:357] T 6d4d8b5cc8ad4c9e935d8afaf5766e66 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:37.293361 1194 raft_consensus.cc:383] T 6d4d8b5cc8ad4c9e935d8afaf5766e66 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:37.293426 1194 raft_consensus.cc:738] T 6d4d8b5cc8ad4c9e935d8afaf5766e66 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:37.293653 1194 consensus_queue.cc:260] T 6d4d8b5cc8ad4c9e935d8afaf5766e66 P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:37.294293 1194 ts_tablet_manager.cc:1428] T 6d4d8b5cc8ad4c9e935d8afaf5766e66 P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250113 02:51:37.294610 1194 tablet_bootstrap.cc:492] T c7243d27a1c642269a3d0e67632e458c P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:37.295754 2228 raft_consensus.cc:491] T eceed0f52c9f46bbaa78ca773a7afde5 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:37.296113 736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 305e7a5304804580b7d302d82fdb5d6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:37.296301 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ddf0295c286340bcaea28a1468f3c557. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:37.296465 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 038ff55bc0184ad1899bee4f96d67d6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:37.296670 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cff2d65491ac47cea9aacd87e3a1f00d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:37.296844 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 221808fe5b3a4edbab5e7bdff971c032. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:37.299345 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab5209b86b1d4078b28bf1517dce8136. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:37.300083 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 43c78b3da74748158f5534203bdd0d33. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:37.303092 441 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet f29e16fea2534c7dbe57639c7c6b37d6 (table default.Table_f6f2b94540f64e45a00fa44fdab358eb [id=5f903ae1c50e4e7487a3b70555f4bb1b], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet f29e16fea2534c7dbe57639c7c6b37d6 consensus state has no leader
I20250113 02:51:37.303742 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 32e3dc0b7c044415a6343368ffa08552. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:37.304066 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6d0f9a0a7104b8fb3d22aac976b7d99. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:37.308076 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9dc9ba26628441dab7256176bf6a0c3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:37.309222 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4aec0adca28d4da2bf09bc8966ed6886. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:37.314493 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e51b750440a24570b734bb51e7a6f923. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:37.291317 998 leader_election.cc:343] T 9438472464304da9a0c7658d4f0736b6 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:37.330293 2306 raft_consensus.cc:491] T 3025935f9743442eb3e6f2db22eb52ad P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:37.333457 2306 raft_consensus.cc:513] T 3025935f9743442eb3e6f2db22eb52ad P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:37.334817 2306 leader_election.cc:290] T 3025935f9743442eb3e6f2db22eb52ad P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:37.337023 1192 tablet_bootstrap.cc:492] T d8bb60fdd06946739b75b255303d9d24 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:37.337224 1192 ts_tablet_manager.cc:1397] T d8bb60fdd06946739b75b255303d9d24 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.184s user 0.003s sys 0.011s
I20250113 02:51:37.338007 2306 raft_consensus.cc:491] T 0b119704bc774ca9b4155c9619d4765b P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:37.339367 2306 raft_consensus.cc:513] T 0b119704bc774ca9b4155c9619d4765b P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:37.339155 1192 raft_consensus.cc:357] T d8bb60fdd06946739b75b255303d9d24 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:37.340179 1192 raft_consensus.cc:383] T d8bb60fdd06946739b75b255303d9d24 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:37.342535 1192 raft_consensus.cc:738] T d8bb60fdd06946739b75b255303d9d24 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:37.295877 2228 raft_consensus.cc:513] T eceed0f52c9f46bbaa78ca773a7afde5 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:37.343355 1192 consensus_queue.cc:260] T d8bb60fdd06946739b75b255303d9d24 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:37.343856 1192 ts_tablet_manager.cc:1428] T d8bb60fdd06946739b75b255303d9d24 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.007s user 0.000s sys 0.001s
I20250113 02:51:37.342451 2306 leader_election.cc:290] T 0b119704bc774ca9b4155c9619d4765b P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:37.338690 2113 leader_election.cc:290] T 7527e6e0b65e4a43b7f48899a34cdf7f P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Requested vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:37.349344 1192 tablet_bootstrap.cc:492] T 4eb5f1944cb44fd5a40ca92a6a3d7214 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:37.349263 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7527e6e0b65e4a43b7f48899a34cdf7f" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f"
I20250113 02:51:37.349879 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b119704bc774ca9b4155c9619d4765b" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:37.350800 2242 tablet.cc:1721] T 1cfbcd682b944c89847b40fe69f0a75e P df69f48656b44bb890f76587ee69cc0f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20250113 02:51:37.351061 998 leader_election.cc:343] T 7527e6e0b65e4a43b7f48899a34cdf7f P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
W20250113 02:51:37.351230 998 leader_election.cc:343] T 0b119704bc774ca9b4155c9619d4765b P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:37.351449 2228 leader_election.cc:290] T eceed0f52c9f46bbaa78ca773a7afde5 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:37.351642 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eceed0f52c9f46bbaa78ca773a7afde5" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
W20250113 02:51:37.352290 709 leader_election.cc:343] T eceed0f52c9f46bbaa78ca773a7afde5 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:37.353348 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7527e6e0b65e4a43b7f48899a34cdf7f" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196"
I20250113 02:51:37.353505 771 raft_consensus.cc:3054] T 7527e6e0b65e4a43b7f48899a34cdf7f P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:37.354257 1198 ts_tablet_manager.cc:1918] T 0fe5421dc3ea4be684cab084f13d8df0 P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:37.354349 1198 log.cc:1198] T 0fe5421dc3ea4be684cab084f13d8df0 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/0fe5421dc3ea4be684cab084f13d8df0
I20250113 02:51:37.354615 1198 ts_tablet_manager.cc:1939] T 0fe5421dc3ea4be684cab084f13d8df0 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:37.355315 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 986) took 2663 ms. Trace:
I20250113 02:51:37.355391 1198 rpcz_store.cc:276] 0113 02:51:34.692016 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:34.692075 (+ 59us) service_pool.cc:225] Handling call
0113 02:51:37.259482 (+2567407us) ts_tablet_manager.cc:1145] Deleting tablet 0fe5421dc3ea4be684cab084f13d8df0
0113 02:51:37.259491 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:37.340496 (+ 81005us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:37.354164 (+ 13668us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:37.355298 (+ 1134us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2567211}
I20250113 02:51:37.355732 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 988) took 2596 ms. Trace:
I20250113 02:51:37.355790 1198 rpcz_store.cc:276] 0113 02:51:34.759679 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:34.759760 (+ 81us) service_pool.cc:225] Handling call
0113 02:51:37.355633 (+2595873us) ts_tablet_manager.cc:1145] Deleting tablet 370c53f565df4e6f915e54a2fd2bf2ce
0113 02:51:37.355642 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:37.355720 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2595677}
I20250113 02:51:37.355932 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3025935f9743442eb3e6f2db22eb52ad" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:37.356317 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eceed0f52c9f46bbaa78ca773a7afde5" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:37.356431 998 leader_election.cc:343] T 3025935f9743442eb3e6f2db22eb52ad P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
W20250113 02:51:37.356786 711 leader_election.cc:343] T eceed0f52c9f46bbaa78ca773a7afde5 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:37.356902 771 raft_consensus.cc:2463] T 7527e6e0b65e4a43b7f48899a34cdf7f P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 1.
I20250113 02:51:37.357061 1192 tablet_bootstrap.cc:654] T 4eb5f1944cb44fd5a40ca92a6a3d7214 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:37.356976 711 leader_election.cc:304] T eceed0f52c9f46bbaa78ca773a7afde5 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:37.358417 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b119704bc774ca9b4155c9619d4765b" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
I20250113 02:51:37.358625 771 raft_consensus.cc:2463] T 0b119704bc774ca9b4155c9619d4765b P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 0.
W20250113 02:51:37.359030 995 leader_election.cc:343] T 3025935f9743442eb3e6f2db22eb52ad P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Not found: Tablet not found: 3025935f9743442eb3e6f2db22eb52ad
I20250113 02:51:37.359191 995 leader_election.cc:304] T 3025935f9743442eb3e6f2db22eb52ad P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4216a7f853d642b7bc3e5cff458f3aca; no voters: 1186fa869eb14812832fa6562531f196, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:37.359601 995 leader_election.cc:304] T 7527e6e0b65e4a43b7f48899a34cdf7f P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters: df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:37.359804 2228 raft_consensus.cc:2743] T eceed0f52c9f46bbaa78ca773a7afde5 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:37.360546 995 leader_election.cc:304] T 0b119704bc774ca9b4155c9619d4765b P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters: df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:37.360724 1195 ts_tablet_manager.cc:1918] T dd566a49d3ff4d3889b80d8d2ccbee0e P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:37.360896 1195 log.cc:1198] T dd566a49d3ff4d3889b80d8d2ccbee0e P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/dd566a49d3ff4d3889b80d8d2ccbee0e
I20250113 02:51:37.361335 1195 ts_tablet_manager.cc:1939] T dd566a49d3ff4d3889b80d8d2ccbee0e P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:37.361716 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1019) took 1899 ms. Trace:
I20250113 02:51:37.361864 1195 rpcz_store.cc:276] 0113 02:51:35.465522 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:35.465680 (+ 158us) service_pool.cc:225] Handling call
0113 02:51:37.076578 (+1610898us) ts_tablet_manager.cc:1145] Deleting tablet dd566a49d3ff4d3889b80d8d2ccbee0e
0113 02:51:37.076592 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:37.307381 (+230789us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:37.360619 (+ 53238us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:37.361699 (+ 1080us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1610729}
I20250113 02:51:37.364409 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet dd566a49d3ff4d3889b80d8d2ccbee0e (table default.Table_34e676c9d2fb4f7ba8a20c74bc14eccf [id=0e2055968e5a492492305d4513f0edba]) successfully deleted
I20250113 02:51:37.364840 2113 raft_consensus.cc:2743] T 3025935f9743442eb3e6f2db22eb52ad P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:37.365083 2113 raft_consensus.cc:2798] T 7527e6e0b65e4a43b7f48899a34cdf7f P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Leader election won for term 1
I20250113 02:51:37.365182 2113 raft_consensus.cc:695] T 7527e6e0b65e4a43b7f48899a34cdf7f P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: Becoming Leader. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Running, Role: LEADER
I20250113 02:51:37.365410 2228 raft_consensus.cc:491] T 788ab82626424b6e833e31761a15b9b2 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:37.365406 2113 consensus_queue.cc:237] T 7527e6e0b65e4a43b7f48899a34cdf7f P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:37.365562 2228 raft_consensus.cc:513] T 788ab82626424b6e833e31761a15b9b2 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:37.366263 2228 leader_election.cc:290] T 788ab82626424b6e833e31761a15b9b2 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:37.366971 2306 raft_consensus.cc:2798] T 0b119704bc774ca9b4155c9619d4765b P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 02:51:37.367040 2306 raft_consensus.cc:491] T 0b119704bc774ca9b4155c9619d4765b P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 02:51:37.367095 2306 raft_consensus.cc:3054] T 0b119704bc774ca9b4155c9619d4765b P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:37.368330 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "788ab82626424b6e833e31761a15b9b2" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:37.368894 711 leader_election.cc:343] T 788ab82626424b6e833e31761a15b9b2 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:37.369334 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "788ab82626424b6e833e31761a15b9b2" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:37.369553 2306 raft_consensus.cc:513] T 0b119704bc774ca9b4155c9619d4765b P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
W20250113 02:51:37.369796 709 leader_election.cc:343] T 788ab82626424b6e833e31761a15b9b2 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:37.369887 709 leader_election.cc:304] T 788ab82626424b6e833e31761a15b9b2 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:37.370088 2228 raft_consensus.cc:2743] T 788ab82626424b6e833e31761a15b9b2 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:37.372609 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_4f5f9d0a5adb402598009f489797d7a1" } new_table_name: "default.Table_25657b231b2d43b096996e80f8a35274" modify_external_catalogs: true
I20250113 02:51:37.374038 453 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_da3e49fa1e654145a05858466ec2260f" } modify_external_catalogs: true
I20250113 02:51:37.374210 453 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_da3e49fa1e654145a05858466ec2260f" } modify_external_catalogs: true
I20250113 02:51:37.378609 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_737a54941ce54064a9191c33fe730efb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:37.379417 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_737a54941ce54064a9191c33fe730efb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:37.380180 455 catalog_manager.cc:5618] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095) does not have the latest schema for tablet aa5c172a69bf4ca79999f4b58df08de8 (table default.Table_3f1e7146fed34d53896f138e43a4d26e [id=5dceb76e1dbe422ab98d4da5bf6c45a0]). Expected version 1 got 0
I20250113 02:51:37.380841 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_adea383eb53e46f7a4dc11af1e7f2fc7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:37.381243 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_adea383eb53e46f7a4dc11af1e7f2fc7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:37.382547 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 93c0ce9f457e431193a2ceb0ad4d4af6
I20250113 02:51:37.382927 449 catalog_manager.cc:6596] ReplaceTablet: tablet 93c0ce9f457e431193a2ceb0ad4d4af6 deleted and replaced by tablet becee3bc8d824a6d94e58730416aa076
I20250113 02:51:37.383276 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_695dfeac867647bab36791ab4dac6bdc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:37.383687 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_695dfeac867647bab36791ab4dac6bdc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:37.384953 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_ecba1621e21641e98ab0246b22af7417"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:37.385368 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ecba1621e21641e98ab0246b22af7417 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:37.386627 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet e51b750440a24570b734bb51e7a6f923
I20250113 02:51:37.387243 454 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_9ad6dc8c7c204c7ea01374d6d8c5844d" } new_table_name: "default.Table_c03ce30fb76741269f2c95d0cdd3941c" modify_external_catalogs: true
I20250113 02:51:37.388581 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_ccae53f816ca40deb917dce54de4698a" } new_table_name: "default.Table_2d947c38688144d9a7534584d059e5b8" modify_external_catalogs: true
W20250113 02:51:37.391795 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9b9e7deda7334ff4a3429f70d9b3a7fc consensus state has no leader
I20250113 02:51:37.392271 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_cdfc4b389e7c4bd984d3dd2c3b33b1c8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:37.392817 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cdfc4b389e7c4bd984d3dd2c3b33b1c8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:37.392818 448 catalog_manager.cc:5618] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587) does not have the latest schema for tablet 9438472464304da9a0c7658d4f0736b6 (table default.Table_acdc5be1660742d7a356474b86e8b35f [id=aaf6558a15124f2b8eb9f01d692b845d]). Expected version 2 got 0
I20250113 02:51:37.394022 448 catalog_manager.cc:5618] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587) does not have the latest schema for tablet 0b119704bc774ca9b4155c9619d4765b (table default.Table_b95a9b1d12284c9aac701a8ef9c93482 [id=843aabcff7074acd98e5801b21282faa]). Expected version 1 got 0
I20250113 02:51:37.395386 774 raft_consensus.cc:2463] T 9438472464304da9a0c7658d4f0736b6 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 1.
I20250113 02:51:37.395625 754 tablet_service.cc:1514] Processing DeleteTablet for tablet e51b750440a24570b734bb51e7a6f923 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:37 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:37.396200 448 master_service.cc:946] ReplaceTablet: received request to replace tablet a1d6cbfb61564e929efab798d4681409 from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:37.395625 751 tablet_service.cc:1514] Processing DeleteTablet for tablet 93c0ce9f457e431193a2ceb0ad4d4af6 with delete_type TABLET_DATA_DELETED (replaced by tablet becee3bc8d824a6d94e58730416aa076) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:37.398419 995 leader_election.cc:304] T 9438472464304da9a0c7658d4f0736b6 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters: df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:37.398466 899 tablet_service.cc:1514] Processing DeleteTablet for tablet e51b750440a24570b734bb51e7a6f923 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:37 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:37.398838 881 tablet_service.cc:1514] Processing DeleteTablet for tablet 93c0ce9f457e431193a2ceb0ad4d4af6 with delete_type TABLET_DATA_DELETED (replaced by tablet becee3bc8d824a6d94e58730416aa076) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:37.398869 2221 raft_consensus.cc:695] T fb18c16fb3ba44ec8a190b88d3bbd1b1 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: Becoming Leader. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Running, Role: LEADER
I20250113 02:51:37.399230 2221 consensus_queue.cc:237] T fb18c16fb3ba44ec8a190b88d3bbd1b1 P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:37.399608 2307 raft_consensus.cc:2798] T 9438472464304da9a0c7658d4f0736b6 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Leader election won for term 1
I20250113 02:51:37.399716 2307 raft_consensus.cc:695] T 9438472464304da9a0c7658d4f0736b6 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: Becoming Leader. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Running, Role: LEADER
I20250113 02:51:37.399924 2307 consensus_queue.cc:237] T 9438472464304da9a0c7658d4f0736b6 P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:37.401017 881 tablet_service.cc:1467] Processing CreateTablet for tablet ed9e6ba0729b483fa6599dd6dde016e7 (DEFAULT_TABLE table=default.Table_b030b1ebb66e4b1482e9a6d2efdb7c8a [id=e5d078de0535400aa33c8607b71c7e92]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:37.401512 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ed9e6ba0729b483fa6599dd6dde016e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:37.405023 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 0752fde3a30a4947a2a688c56ceadab5
I20250113 02:51:37.405328 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 5d2d14548b734b1fbe57d68c0ea3e2f8
I20250113 02:51:37.405591 453 master_service.cc:946] ReplaceTablet: received request to replace tablet 726e68108f5b4de4a51ffdf778ef825d from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:37.405846 446 catalog_manager.cc:6596] ReplaceTablet: tablet 5d2d14548b734b1fbe57d68c0ea3e2f8 deleted and replaced by tablet 6c4868fe133043169238e2d67ea7276b
I20250113 02:51:37.406101 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_669ef67f4e884a939ff3b3a0bc409ce3" } new_table_name: "default.Table_854e525352dd47c889aaabfc312e33c7" modify_external_catalogs: true
I20250113 02:51:37.406231 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_1b352c8cf2224b2e9b9f4e1322ee8b37"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:37.406812 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1b352c8cf2224b2e9b9f4e1322ee8b37 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:37.407085 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b119704bc774ca9b4155c9619d4765b" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196"
I20250113 02:51:37.407256 775 raft_consensus.cc:3054] T 0b119704bc774ca9b4155c9619d4765b P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:37.408928 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1021) took 1915 ms. Trace:
I20250113 02:51:37.409062 1195 rpcz_store.cc:276] 0113 02:51:35.493139 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:35.493216 (+ 77us) service_pool.cc:225] Handling call
0113 02:51:37.363394 (+1870178us) ts_tablet_manager.cc:1145] Deleting tablet 2002c69206ec468aad1b44f1b5b0d3b2
0113 02:51:37.405325 (+ 41931us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:37.408908 (+ 3583us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1869128}
I20250113 02:51:37.409446 2286 raft_consensus.cc:491] T 056bf1e67be04d179eace3d93282b145 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:37.409613 2286 raft_consensus.cc:513] T 056bf1e67be04d179eace3d93282b145 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:37.410074 888 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d2d14548b734b1fbe57d68c0ea3e2f8 with delete_type TABLET_DATA_DELETED (replaced by tablet 6c4868fe133043169238e2d67ea7276b) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:37.410276 899 tablet_service.cc:1514] Processing DeleteTablet for tablet 0752fde3a30a4947a2a688c56ceadab5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:37 UTC) from {username='slave'} at 127.0.0.1:37366
W20250113 02:51:37.410715 851 leader_election.cc:343] T 77cf8b0c824f401aba1880eb7fd1807b P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Not found: Tablet not found: 77cf8b0c824f401aba1880eb7fd1807b
W20250113 02:51:37.410827 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 2002c69206ec468aad1b44f1b5b0d3b2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2002c69206ec468aad1b44f1b5b0d3b2 already in progress: creating tablet
I20250113 02:51:37.410812 851 leader_election.cc:304] T 77cf8b0c824f401aba1880eb7fd1807b P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df69f48656b44bb890f76587ee69cc0f; no voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca
I20250113 02:51:37.411252 2286 leader_election.cc:290] T 056bf1e67be04d179eace3d93282b145 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587), 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
I20250113 02:51:37.411383 2286 raft_consensus.cc:2743] T 77cf8b0c824f401aba1880eb7fd1807b P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:37.411528 751 tablet_service.cc:1514] Processing DeleteTablet for tablet 0752fde3a30a4947a2a688c56ceadab5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:37 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:37.411528 754 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d2d14548b734b1fbe57d68c0ea3e2f8 with delete_type TABLET_DATA_DELETED (replaced by tablet 6c4868fe133043169238e2d67ea7276b) from {username='slave'} at 127.0.0.1:55398
W20250113 02:51:37.412150 851 leader_election.cc:343] T 0a7e7713bc8648378b668d8b769ac927 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Not found: Tablet not found: 0a7e7713bc8648378b668d8b769ac927
I20250113 02:51:37.412248 851 leader_election.cc:304] T 0a7e7713bc8648378b668d8b769ac927 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df69f48656b44bb890f76587ee69cc0f; no voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca
I20250113 02:51:37.412498 2195 raft_consensus.cc:2743] T 0a7e7713bc8648378b668d8b769ac927 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250113 02:51:37.412564 851 leader_election.cc:343] T 056bf1e67be04d179eace3d93282b145 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Not found: Tablet not found: 056bf1e67be04d179eace3d93282b145
I20250113 02:51:37.413460 775 raft_consensus.cc:2463] T 0b119704bc774ca9b4155c9619d4765b P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 1.
I20250113 02:51:37.422247 450 catalog_manager.cc:5618] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095) does not have the latest schema for tablet 56b022c94847411785a7e9db3757a2d9 (table default.Table_6461b062100240d59a5d91a87a74510c [id=40ca872c045b4349b63ba9046e31ed13]). Expected version 2 got 0
I20250113 02:51:37.426769 820 heartbeater.cc:346] Connected to a master server at 127.0.104.60:42067
I20250113 02:51:37.429271 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b119704bc774ca9b4155c9619d4765b" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f"
I20250113 02:51:37.433315 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1024) took 1902 ms. Trace:
I20250113 02:51:37.433344 736 tablet_service.cc:1467] Processing CreateTablet for tablet ed9e6ba0729b483fa6599dd6dde016e7 (DEFAULT_TABLE table=default.Table_b030b1ebb66e4b1482e9a6d2efdb7c8a [id=e5d078de0535400aa33c8607b71c7e92]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:37.433463 1195 rpcz_store.cc:276] 0113 02:51:35.531271 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:35.531318 (+ 47us) service_pool.cc:225] Handling call
0113 02:51:37.409386 (+1878068us) ts_tablet_manager.cc:1145] Deleting tablet c548a79f394d4ab88a309ca6400a05a2
0113 02:51:37.433190 (+ 23804us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:37.433296 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1877900}
I20250113 02:51:37.433748 736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ed9e6ba0729b483fa6599dd6dde016e7. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:37.434056 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet c548a79f394d4ab88a309ca6400a05a2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c548a79f394d4ab88a309ca6400a05a2 already in progress: creating tablet
I20250113 02:51:37.434376 1195 tablet_replica.cc:331] T f21eb3fa0f414816a29e3d4560cdc0fe P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:37.434561 1195 raft_consensus.cc:2238] T f21eb3fa0f414816a29e3d4560cdc0fe P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:37.434707 1195 raft_consensus.cc:2267] T f21eb3fa0f414816a29e3d4560cdc0fe P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:37.435232 1195 ts_tablet_manager.cc:1905] T f21eb3fa0f414816a29e3d4560cdc0fe P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:37.402575 1192 tablet_bootstrap.cc:492] T 4eb5f1944cb44fd5a40ca92a6a3d7214 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:37.440281 1192 ts_tablet_manager.cc:1397] T 4eb5f1944cb44fd5a40ca92a6a3d7214 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.091s user 0.003s sys 0.000s
I20250113 02:51:37.440891 2242 tablet.cc:1721] T 1a63f485caf54068abd6b50d7de3d14a P df69f48656b44bb890f76587ee69cc0f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250113 02:51:37.441056 1192 raft_consensus.cc:357] T 4eb5f1944cb44fd5a40ca92a6a3d7214 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:37.441270 1192 raft_consensus.cc:383] T 4eb5f1944cb44fd5a40ca92a6a3d7214 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:37.441330 1192 raft_consensus.cc:738] T 4eb5f1944cb44fd5a40ca92a6a3d7214 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:37.441512 1192 consensus_queue.cc:260] T 4eb5f1944cb44fd5a40ca92a6a3d7214 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:37.442003 1192 ts_tablet_manager.cc:1428] T 4eb5f1944cb44fd5a40ca92a6a3d7214 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250113 02:51:37.402776 2306 leader_election.cc:290] T 0b119704bc774ca9b4155c9619d4765b P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Requested vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:37.445111 1194 tablet_bootstrap.cc:654] T c7243d27a1c642269a3d0e67632e458c P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:37.404107 1198 tablet_replica.cc:331] T df87e25053a74990959f85288a85217b P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:37.445771 1198 raft_consensus.cc:2238] T df87e25053a74990959f85288a85217b P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:37.445994 1198 raft_consensus.cc:2267] T df87e25053a74990959f85288a85217b P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:37.420547 995 leader_election.cc:304] T 0b119704bc774ca9b4155c9619d4765b P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters:
W20250113 02:51:37.446934 852 leader_election.cc:343] T 056bf1e67be04d179eace3d93282b145 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Not found: Tablet not found: 056bf1e67be04d179eace3d93282b145
I20250113 02:51:37.447045 852 leader_election.cc:304] T 056bf1e67be04d179eace3d93282b145 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df69f48656b44bb890f76587ee69cc0f; no voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca
I20250113 02:51:37.447430 2306 raft_consensus.cc:2798] T 0b119704bc774ca9b4155c9619d4765b P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Leader election won for term 1
I20250113 02:51:37.447427 2286 raft_consensus.cc:2743] T 056bf1e67be04d179eace3d93282b145 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:37.447543 2306 raft_consensus.cc:695] T 0b119704bc774ca9b4155c9619d4765b P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: Becoming Leader. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Running, Role: LEADER
I20250113 02:51:37.447790 2306 consensus_queue.cc:237] T 0b119704bc774ca9b4155c9619d4765b P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:37.449424 2221 raft_consensus.cc:491] T f21eb3fa0f414816a29e3d4560cdc0fe P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:37.449532 2221 raft_consensus.cc:513] T f21eb3fa0f414816a29e3d4560cdc0fe P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:37.450078 2221 leader_election.cc:290] T f21eb3fa0f414816a29e3d4560cdc0fe P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:37.450982 1198 ts_tablet_manager.cc:1905] T df87e25053a74990959f85288a85217b P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:37.451071 964 heartbeater.cc:346] Connected to a master server at 127.0.104.60:42067
I20250113 02:51:37.451567 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f21eb3fa0f414816a29e3d4560cdc0fe" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
I20250113 02:51:37.451934 2281 tablet.cc:1721] T 56b022c94847411785a7e9db3757a2d9 P 1186fa869eb14812832fa6562531f196: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250113 02:51:37.453152 1192 tablet_bootstrap.cc:492] T d9df314ffaf042ec80180868d7e7bdf7 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:37.453665 2277 tablet.cc:1721] T 56b022c94847411785a7e9db3757a2d9 P 4216a7f853d642b7bc3e5cff458f3aca: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250113 02:51:37.455013 1192 tablet_bootstrap.cc:654] T d9df314ffaf042ec80180868d7e7bdf7 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
W20250113 02:51:37.454555 998 leader_election.cc:343] T 0b119704bc774ca9b4155c9619d4765b P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:37.457388 455 catalog_manager.cc:5618] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467) does not have the latest schema for tablet 1a63f485caf54068abd6b50d7de3d14a (table default.Table_f8a6812c7fb94ea69cd9bf775b914ea6 [id=71508871e2d5469a91aaf0e19b02b638]). Expected version 1 got 0
I20250113 02:51:37.459820 455 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_f9117b338db445fba012e659c3b51f66" } modify_external_catalogs: true
I20250113 02:51:37.461680 886 tablet_service.cc:1514] Processing DeleteTablet for tablet 05bf5ab0df874dd5b6559f6d82069184 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:21 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:37.462520 886 tablet_service.cc:1514] Processing DeleteTablet for tablet f56230a2c75d4763a54671c79358f4d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:22 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:37.463110 886 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b1910496e0849b193e390a5ef1a6b2b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:26 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:37.463745 888 tablet_service.cc:1514] Processing DeleteTablet for tablet 6666922d677a4c678f80d7c6c47fd2a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:28 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:37.463953 899 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a7e7713bc8648378b668d8b769ac927 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:21 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:37.464324 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f21eb3fa0f414816a29e3d4560cdc0fe" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:37.465086 998 leader_election.cc:343] T f21eb3fa0f414816a29e3d4560cdc0fe P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:37.467864 899 tablet_service.cc:1467] Processing CreateTablet for tablet 9c9d8a68451145f9964895c3fb2c0744 (DEFAULT_TABLE table=default.Table_18b709d327d2407ea3909a1ec7091d92 [id=6b8f0e69a9a442afac69ced2bd9e6c72]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:37.468431 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c9d8a68451145f9964895c3fb2c0744. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:37.469885 995 leader_election.cc:343] T f21eb3fa0f414816a29e3d4560cdc0fe P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Illegal state: Tablet not RUNNING: STOPPED
I20250113 02:51:37.470054 995 leader_election.cc:304] T f21eb3fa0f414816a29e3d4560cdc0fe P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4216a7f853d642b7bc3e5cff458f3aca; no voters: 1186fa869eb14812832fa6562531f196, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:37.477109 2307 raft_consensus.cc:2743] T f21eb3fa0f414816a29e3d4560cdc0fe P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:37.478062 1194 tablet_bootstrap.cc:492] T c7243d27a1c642269a3d0e67632e458c P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:37.478272 1194 ts_tablet_manager.cc:1397] T c7243d27a1c642269a3d0e67632e458c P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.184s user 0.002s sys 0.002s
I20250113 02:51:37.479107 1194 raft_consensus.cc:357] T c7243d27a1c642269a3d0e67632e458c P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:37.479714 1194 raft_consensus.cc:383] T c7243d27a1c642269a3d0e67632e458c P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:37.479950 1194 raft_consensus.cc:738] T c7243d27a1c642269a3d0e67632e458c P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:37.480387 455 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_f9117b338db445fba012e659c3b51f66" } modify_external_catalogs: true
I20250113 02:51:37.480437 1194 consensus_queue.cc:260] T c7243d27a1c642269a3d0e67632e458c P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:37.481242 2286 raft_consensus.cc:491] T 6666922d677a4c678f80d7c6c47fd2a5 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:37.481366 2286 raft_consensus.cc:513] T 6666922d677a4c678f80d7c6c47fd2a5 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:37.481806 1194 ts_tablet_manager.cc:1428] T c7243d27a1c642269a3d0e67632e458c P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250113 02:51:37.482231 1194 tablet_bootstrap.cc:492] T 51d9483ecacb413796d0fd9e7560eac1 P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
W20250113 02:51:37.483016 852 leader_election.cc:343] T 6666922d677a4c678f80d7c6c47fd2a5 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Not found: Tablet not found: 6666922d677a4c678f80d7c6c47fd2a5
I20250113 02:51:37.484117 1198 ts_tablet_manager.cc:1918] T df87e25053a74990959f85288a85217b P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:37.485811 2195 raft_consensus.cc:491] T 05bf5ab0df874dd5b6559f6d82069184 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:37.485944 2195 raft_consensus.cc:513] T 05bf5ab0df874dd5b6559f6d82069184 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:37.486524 2195 leader_election.cc:290] T 05bf5ab0df874dd5b6559f6d82069184 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:37.486997 1198 log.cc:1198] T df87e25053a74990959f85288a85217b P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/df87e25053a74990959f85288a85217b
I20250113 02:51:37.487389 1198 ts_tablet_manager.cc:1939] T df87e25053a74990959f85288a85217b P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
W20250113 02:51:37.487722 852 leader_election.cc:343] T 05bf5ab0df874dd5b6559f6d82069184 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Not found: Tablet not found: 05bf5ab0df874dd5b6559f6d82069184
I20250113 02:51:37.487769 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 989) took 2727 ms. Trace:
I20250113 02:51:37.487838 1198 rpcz_store.cc:276] 0113 02:51:34.759818 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:34.759868 (+ 50us) service_pool.cc:225] Handling call
0113 02:51:37.404060 (+2644192us) ts_tablet_manager.cc:1145] Deleting tablet df87e25053a74990959f85288a85217b
0113 02:51:37.404070 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:37.477490 (+ 73420us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:37.484017 (+ 6527us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:37.487754 (+ 3737us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2644014}
I20250113 02:51:37.488019 2286 leader_election.cc:290] T 6666922d677a4c678f80d7c6c47fd2a5 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:37.488518 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_026ff82c780e4fb5803728166c8952ce"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:37.488366 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 990) took 2616 ms. Trace:
W20250113 02:51:37.489019 851 leader_election.cc:343] T 6666922d677a4c678f80d7c6c47fd2a5 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Not found: Tablet not found: 6666922d677a4c678f80d7c6c47fd2a5
W20250113 02:51:37.489131 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_026ff82c780e4fb5803728166c8952ce in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:37.489078 1198 rpcz_store.cc:276] 0113 02:51:34.872181 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:34.872226 (+ 45us) service_pool.cc:225] Handling call
0113 02:51:37.488257 (+2616031us) ts_tablet_manager.cc:1145] Deleting tablet d565b77f90784d87b64996772c938085
0113 02:51:37.488267 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:37.488351 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2615861}
I20250113 02:51:37.489207 851 leader_election.cc:304] T 6666922d677a4c678f80d7c6c47fd2a5 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df69f48656b44bb890f76587ee69cc0f; no voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca
I20250113 02:51:37.489436 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 991) took 2593 ms. Trace:
W20250113 02:51:37.489826 851 leader_election.cc:343] T 05bf5ab0df874dd5b6559f6d82069184 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Not found: Tablet not found: 05bf5ab0df874dd5b6559f6d82069184
I20250113 02:51:37.489113 1194 tablet_bootstrap.cc:654] T 51d9483ecacb413796d0fd9e7560eac1 P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:37.490176 2286 raft_consensus.cc:2743] T 6666922d677a4c678f80d7c6c47fd2a5 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:37.490008 851 leader_election.cc:304] T 05bf5ab0df874dd5b6559f6d82069184 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df69f48656b44bb890f76587ee69cc0f; no voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca
W20250113 02:51:37.490434 454 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c87118901d7f4790be6f17600bd7c946 consensus state has no leader
I20250113 02:51:37.490769 2286 raft_consensus.cc:2743] T 05bf5ab0df874dd5b6559f6d82069184 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250113 02:51:37.490887 451 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 01a455eda0ee488fb3d45a6a09c43833 consensus state has no leader
I20250113 02:51:37.491305 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_3f1e7146fed34d53896f138e43a4d26e" } modify_external_catalogs: true
I20250113 02:51:37.491479 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_3f1e7146fed34d53896f138e43a4d26e" } modify_external_catalogs: true
I20250113 02:51:37.492237 1198 rpcz_store.cc:276] 0113 02:51:34.896237 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:34.896290 (+ 53us) service_pool.cc:225] Handling call
0113 02:51:37.489316 (+2593026us) ts_tablet_manager.cc:1145] Deleting tablet 726e68108f5b4de4a51ffdf778ef825d
0113 02:51:37.489323 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:37.489418 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2592831}
I20250113 02:51:37.492503 1198 tablet_replica.cc:331] T d5bd09a1ede644c59e400e49c8125793 P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:37.492233 736 tablet_service.cc:1467] Processing CreateTablet for tablet 9c9d8a68451145f9964895c3fb2c0744 (DEFAULT_TABLE table=default.Table_18b709d327d2407ea3909a1ec7091d92 [id=6b8f0e69a9a442afac69ced2bd9e6c72]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:37.492702 1198 raft_consensus.cc:2238] T d5bd09a1ede644c59e400e49c8125793 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: Raft consensus shutting down.
I20250113 02:51:37.492957 736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c9d8a68451145f9964895c3fb2c0744. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:37.493075 1198 raft_consensus.cc:2267] T d5bd09a1ede644c59e400e49c8125793 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:37.493005 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_2ab49714961e452d8adfa6f8acb9be86"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:37.493533 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2ab49714961e452d8adfa6f8acb9be86 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:37.493851 1198 ts_tablet_manager.cc:1905] T d5bd09a1ede644c59e400e49c8125793 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:37.496470 449 catalog_manager.cc:5618] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467) does not have the latest schema for tablet 6e1284c8aaba44b5810b0d06d36757eb (table default.Table_bfc5031f68f64006aceefe2ee986223a [id=aa4679fc0d7148e3bef21342b3a0ea22]). Expected version 2 got 0
I20250113 02:51:37.499327 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_28444afd72804ea5bc5099ae319b1321" } modify_external_catalogs: true
I20250113 02:51:37.499506 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_28444afd72804ea5bc5099ae319b1321" } modify_external_catalogs: true
I20250113 02:51:37.499702 1194 tablet_bootstrap.cc:492] T 51d9483ecacb413796d0fd9e7560eac1 P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:37.500293 1194 ts_tablet_manager.cc:1397] T 51d9483ecacb413796d0fd9e7560eac1 P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.002s
I20250113 02:51:37.501822 1194 raft_consensus.cc:357] T 51d9483ecacb413796d0fd9e7560eac1 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:37.503469 1194 raft_consensus.cc:383] T 51d9483ecacb413796d0fd9e7560eac1 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:37.503695 1194 raft_consensus.cc:738] T 51d9483ecacb413796d0fd9e7560eac1 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:37.504150 1194 consensus_queue.cc:260] T 51d9483ecacb413796d0fd9e7560eac1 P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:37.505015 1194 ts_tablet_manager.cc:1428] T 51d9483ecacb413796d0fd9e7560eac1 P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
W20250113 02:51:37.505471 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6c4868fe133043169238e2d67ea7276b has no consensus state
I20250113 02:51:37.505602 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet a1d6cbfb61564e929efab798d4681409
I20250113 02:51:37.505755 453 master_service.cc:946] ReplaceTablet: received request to replace tablet fe3d35d9aaf64188aaa3c1ac38d279d8 from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:37.505837 1194 tablet_bootstrap.cc:492] T e468b652f71f44e3945f8d919ac0e1a8 P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:37.506058 448 catalog_manager.cc:6596] ReplaceTablet: tablet a1d6cbfb61564e929efab798d4681409 deleted and replaced by tablet 1c82cf9373d24fedad88b5df7d5a8b64
I20250113 02:51:37.506635 754 tablet_service.cc:1514] Processing DeleteTablet for tablet a1d6cbfb61564e929efab798d4681409 with delete_type TABLET_DATA_DELETED (replaced by tablet 1c82cf9373d24fedad88b5df7d5a8b64) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:37.507925 1192 tablet_bootstrap.cc:492] T d9df314ffaf042ec80180868d7e7bdf7 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:37.508168 1192 ts_tablet_manager.cc:1397] T d9df314ffaf042ec80180868d7e7bdf7 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.055s user 0.003s sys 0.004s
I20250113 02:51:37.509025 1194 tablet_bootstrap.cc:654] T e468b652f71f44e3945f8d919ac0e1a8 P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:37.509034 1192 raft_consensus.cc:357] T d9df314ffaf042ec80180868d7e7bdf7 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:37.509227 1192 raft_consensus.cc:383] T d9df314ffaf042ec80180868d7e7bdf7 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:37.509290 1192 raft_consensus.cc:738] T d9df314ffaf042ec80180868d7e7bdf7 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:37.509496 1192 consensus_queue.cc:260] T d9df314ffaf042ec80180868d7e7bdf7 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:37.510201 754 tablet_service.cc:1467] Processing CreateTablet for tablet 65e09181ee4c4061a7b5a5353be440cf (DEFAULT_TABLE table=default.Table_3463038bce4545df8ec6c3834403a95d [id=505134ec091a46c9a1dc599ae2514a3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:37.510597 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65e09181ee4c4061a7b5a5353be440cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:37.510911 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet 0fe5421dc3ea4be684cab084f13d8df0 (table default.Table_ca04ad8d0d8b40d1ba6b147731579526 [id=beb83fe520f94241949a46d436e6374f]) successfully deleted
W20250113 02:51:37.511418 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 370c53f565df4e6f915e54a2fd2bf2ce because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 370c53f565df4e6f915e54a2fd2bf2ce already in progress: creating tablet
I20250113 02:51:37.503005 1196 ts_tablet_manager.cc:1918] T a97efc70946c4af487e33a85abf6b90c P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:37.512243 1196 log.cc:1198] T a97efc70946c4af487e33a85abf6b90c P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/a97efc70946c4af487e33a85abf6b90c
I20250113 02:51:37.512176 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet df87e25053a74990959f85288a85217b (table default.Table_2f1a7bb2c4cd4ead95b175e1d3545f20 [id=b2c8431e06ac46be9840edf78b6c19a3]) successfully deleted
W20250113 02:51:37.512580 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet d565b77f90784d87b64996772c938085 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d565b77f90784d87b64996772c938085 already in progress: creating tablet
I20250113 02:51:37.512660 1196 ts_tablet_manager.cc:1939] T a97efc70946c4af487e33a85abf6b90c P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
W20250113 02:51:37.512912 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 726e68108f5b4de4a51ffdf778ef825d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 726e68108f5b4de4a51ffdf778ef825d already in progress: creating tablet
I20250113 02:51:37.513388 1198 ts_tablet_manager.cc:1918] T d5bd09a1ede644c59e400e49c8125793 P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250113 02:51:37.513502 1198 log.cc:1198] T d5bd09a1ede644c59e400e49c8125793 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/d5bd09a1ede644c59e400e49c8125793
I20250113 02:51:37.513834 1198 ts_tablet_manager.cc:1939] T d5bd09a1ede644c59e400e49c8125793 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:37.514433 888 tablet_service.cc:1467] Processing CreateTablet for tablet 65e09181ee4c4061a7b5a5353be440cf (DEFAULT_TABLE table=default.Table_3463038bce4545df8ec6c3834403a95d [id=505134ec091a46c9a1dc599ae2514a3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:37.514947 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65e09181ee4c4061a7b5a5353be440cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:37.515055 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet a97efc70946c4af487e33a85abf6b90c (table default.Table_a5e21e7ff75b4d70a6944712fb3d1392 [id=d29a78ceccaa42808de9c2736d3eb4ac]) successfully deleted
I20250113 02:51:37.515404 1196 tablet_replica.cc:331] T c826a0b8327f44f9a00e5afe3a36cd8a P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:37.515801 1196 raft_consensus.cc:2238] T c826a0b8327f44f9a00e5afe3a36cd8a P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:37.516688 1196 raft_consensus.cc:2267] T c826a0b8327f44f9a00e5afe3a36cd8a P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:37.517230 1196 ts_tablet_manager.cc:1905] T c826a0b8327f44f9a00e5afe3a36cd8a P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:37.523829 453 catalog_manager.cc:5526] T 0b119704bc774ca9b4155c9619d4765b P 4216a7f853d642b7bc3e5cff458f3aca reported cstate change: term changed from 0 to 1, leader changed from <none> to 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3). New cstate: current_term: 1 leader_uuid: "4216a7f853d642b7bc3e5cff458f3aca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } health_report { overall_health: UNKNOWN } } }
I20250113 02:51:37.524654 453 catalog_manager.cc:5618] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587) does not have the latest schema for tablet 0b119704bc774ca9b4155c9619d4765b (table default.Table_b95a9b1d12284c9aac701a8ef9c93482 [id=843aabcff7074acd98e5801b21282faa]). Expected version 1 got 0
I20250113 02:51:37.525033 453 catalog_manager.cc:5526] T 9438472464304da9a0c7658d4f0736b6 P 4216a7f853d642b7bc3e5cff458f3aca reported cstate change: term changed from 0 to 1, leader changed from <none> to 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3). New cstate: current_term: 1 leader_uuid: "4216a7f853d642b7bc3e5cff458f3aca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } health_report { overall_health: UNKNOWN } } }
I20250113 02:51:37.525494 453 catalog_manager.cc:5618] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587) does not have the latest schema for tablet 9438472464304da9a0c7658d4f0736b6 (table default.Table_acdc5be1660742d7a356474b86e8b35f [id=aaf6558a15124f2b8eb9f01d692b845d]). Expected version 2 got 0
I20250113 02:51:37.525874 453 catalog_manager.cc:5526] T fb18c16fb3ba44ec8a190b88d3bbd1b1 P 4216a7f853d642b7bc3e5cff458f3aca reported cstate change: term changed from 0 to 1, leader changed from <none> to 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3). New cstate: current_term: 1 leader_uuid: "4216a7f853d642b7bc3e5cff458f3aca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } health_report { overall_health: HEALTHY } } }
I20250113 02:51:37.526310 453 catalog_manager.cc:5618] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587) does not have the latest schema for tablet fb18c16fb3ba44ec8a190b88d3bbd1b1 (table default.Table_7a4628807cc44ebdadf927ff709de3a2 [id=75ca7cbd7b9548d784f9a77d0f0cb64f]). Expected version 1 got 0
I20250113 02:51:37.526780 453 catalog_manager.cc:5526] T 7527e6e0b65e4a43b7f48899a34cdf7f P 4216a7f853d642b7bc3e5cff458f3aca reported cstate change: term changed from 0 to 1, leader changed from <none> to 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3). New cstate: current_term: 1 leader_uuid: "4216a7f853d642b7bc3e5cff458f3aca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } health_report { overall_health: HEALTHY } } }
I20250113 02:51:37.527230 453 catalog_manager.cc:5618] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587) does not have the latest schema for tablet 7527e6e0b65e4a43b7f48899a34cdf7f (table default.Table_c8b93363989c4e109540af6bef426f73 [id=b7846a8ebee84005b1b5d259560a26bd]). Expected version 1 got 0
I20250113 02:51:37.515256 1037 tablet_service.cc:1514] Processing DeleteTablet for tablet 93c0ce9f457e431193a2ceb0ad4d4af6 with delete_type TABLET_DATA_DELETED (replaced by tablet becee3bc8d824a6d94e58730416aa076) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:37.515584 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet e51b750440a24570b734bb51e7a6f923 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:37 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:37.530788 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 993) took 2446 ms. Trace:
I20250113 02:51:37.515924 1039 tablet_service.cc:1514] Processing DeleteTablet for tablet f21eb3fa0f414816a29e3d4560cdc0fe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:30 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:37.516291 1041 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d2d14548b734b1fbe57d68c0ea3e2f8 with delete_type TABLET_DATA_DELETED (replaced by tablet 6c4868fe133043169238e2d67ea7276b) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:37.516259 1031 tablet_service.cc:1467] Processing CreateTablet for tablet 038ff55bc0184ad1899bee4f96d67d6f (DEFAULT_TABLE table=default.Table_14183012578a4660ad28c7c483de871d [id=7e412e073b4d411fabf1728c023601c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:37.532567 450 catalog_manager.cc:6071] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: default.Table_32e88abaae96492fbcce39fb2cd7c221 [id=2b84deb07e5c4ab8b550fe42945d7b71] alter complete (version 2)
I20250113 02:51:37.532611 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 097cc4bd230046e2ac8551ca23d8f3e8
I20250113 02:51:37.532810 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 038ff55bc0184ad1899bee4f96d67d6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:37.532305 1198 rpcz_store.cc:276] 0113 02:51:35.084370 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:35.084422 (+ 52us) service_pool.cc:225] Handling call
0113 02:51:37.492461 (+2408039us) ts_tablet_manager.cc:1145] Deleting tablet d5bd09a1ede644c59e400e49c8125793
0113 02:51:37.492469 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:37.499951 (+ 7482us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:37.513294 (+ 13343us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:37.530769 (+ 17475us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2407853}
I20250113 02:51:37.533456 1198 tablet_replica.cc:331] T 42cf1e708644427f9d4dcd41ae2bcb37 P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:37.533800 1198 raft_consensus.cc:2238] T 42cf1e708644427f9d4dcd41ae2bcb37 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:37.533972 1198 raft_consensus.cc:2267] T 42cf1e708644427f9d4dcd41ae2bcb37 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:37.534555 1198 ts_tablet_manager.cc:1905] T 42cf1e708644427f9d4dcd41ae2bcb37 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:37.535169 744 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c408921a09a463eb464e08edcf042b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:24 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:37.535360 752 tablet_service.cc:1514] Processing DeleteTablet for tablet dc680b5680b84abbab219296bc7ee707 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:23 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:37.535383 751 tablet_service.cc:1514] Processing DeleteTablet for tablet 097cc4bd230046e2ac8551ca23d8f3e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:37 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:37.535490 741 tablet_service.cc:1514] Processing DeleteTablet for tablet 153b8f90b5a847b88944f683d306f579 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:24 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:37.535595 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet d5bd09a1ede644c59e400e49c8125793 (table default.Table_bee5bcef1f534c6c812e6075291ff90d [id=e0ced6f3dc0049eaa0e76974916ff63c]) successfully deleted
I20250113 02:51:37.535996 754 tablet_service.cc:1514] Processing DeleteTablet for tablet f8a6eebb6b6d4b63903e68750e181707 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:31 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:37.516420 1040 tablet_service.cc:1514] Processing DeleteTablet for tablet 0752fde3a30a4947a2a688c56ceadab5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:37 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:37.536463 881 tablet_service.cc:1514] Processing DeleteTablet for tablet a1d6cbfb61564e929efab798d4681409 with delete_type TABLET_DATA_DELETED (replaced by tablet 1c82cf9373d24fedad88b5df7d5a8b64) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:37.536751 1041 tablet_service.cc:1514] Processing DeleteTablet for tablet 097cc4bd230046e2ac8551ca23d8f3e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:37 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:37.536736 886 tablet_service.cc:1514] Processing DeleteTablet for tablet 097cc4bd230046e2ac8551ca23d8f3e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:37 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:37.536926 891 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d4d8b5cc8ad4c9e935d8afaf5766e66 with delete_type TABLET_DATA_DELETED (replaced by tablet e57ac440265e4646bfc5d5d9ca4c35d2) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:37.517050 1030 tablet_service.cc:1467] Processing CreateTablet for tablet ed9e6ba0729b483fa6599dd6dde016e7 (DEFAULT_TABLE table=default.Table_b030b1ebb66e4b1482e9a6d2efdb7c8a [id=e5d078de0535400aa33c8607b71c7e92]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:37.538959 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ed9e6ba0729b483fa6599dd6dde016e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:37.517496 1042 tablet_service.cc:1467] Processing CreateTablet for tablet 9c9d8a68451145f9964895c3fb2c0744 (DEFAULT_TABLE table=default.Table_18b709d327d2407ea3909a1ec7091d92 [id=6b8f0e69a9a442afac69ced2bd9e6c72]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:37.542661 1042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c9d8a68451145f9964895c3fb2c0744. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:37.543969 1196 ts_tablet_manager.cc:1918] T c826a0b8327f44f9a00e5afe3a36cd8a P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:37.544083 1196 log.cc:1198] T c826a0b8327f44f9a00e5afe3a36cd8a P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/c826a0b8327f44f9a00e5afe3a36cd8a
I20250113 02:51:37.544420 1196 ts_tablet_manager.cc:1939] T c826a0b8327f44f9a00e5afe3a36cd8a P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:37.517791 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet a1d6cbfb61564e929efab798d4681409 with delete_type TABLET_DATA_DELETED (replaced by tablet 1c82cf9373d24fedad88b5df7d5a8b64) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:37.549571 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet c826a0b8327f44f9a00e5afe3a36cd8a (table default.Table_03babf194cc448f296512a09b2e3fe3e [id=64b42698782f41a589577af20b33501e]) successfully deleted
I20250113 02:51:37.550211 891 tablet_service.cc:1514] Processing DeleteTablet for tablet 51d9483ecacb413796d0fd9e7560eac1 with delete_type TABLET_DATA_DELETED (replaced by tablet 8898e1e9efce453488fa4a4562a82127) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:37.551020 1196 tablet_replica.cc:331] T 77cf8b0c824f401aba1880eb7fd1807b P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:37.530056 1192 ts_tablet_manager.cc:1428] T d9df314ffaf042ec80180868d7e7bdf7 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.022s user 0.000s sys 0.001s
I20250113 02:51:37.515671 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet 7213e3284b4d4cb581d9fb946a9f87e9 with delete_type TABLET_DATA_DELETED (replaced by tablet 7c443454994a47f2bfa7e2ec7ac576e0) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:37.551301 1192 tablet_bootstrap.cc:492] T f8a6eebb6b6d4b63903e68750e181707 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:37.551822 1196 raft_consensus.cc:2238] T 77cf8b0c824f401aba1880eb7fd1807b P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:37.552560 1196 raft_consensus.cc:2267] T 77cf8b0c824f401aba1880eb7fd1807b P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:37.551107 888 tablet_service.cc:1514] Processing DeleteTablet for tablet c7243d27a1c642269a3d0e67632e458c with delete_type TABLET_DATA_DELETED (replaced by tablet ed68a2b7ebcf4841bef54798dd3ed80d) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:37.553068 1192 tablet_bootstrap.cc:654] T f8a6eebb6b6d4b63903e68750e181707 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:37.554303 1026 tablet_service.cc:1467] Processing CreateTablet for tablet 65e09181ee4c4061a7b5a5353be440cf (DEFAULT_TABLE table=default.Table_3463038bce4545df8ec6c3834403a95d [id=505134ec091a46c9a1dc599ae2514a3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:37.554495 1196 ts_tablet_manager.cc:1905] T 77cf8b0c824f401aba1880eb7fd1807b P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:37.554683 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65e09181ee4c4061a7b5a5353be440cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:37.556088 888 tablet_service.cc:1467] Processing CreateTablet for tablet d780ce78e8ee4726a9277174c9b58a03 (DEFAULT_TABLE table=default.Table_d9a1e87da55c409281a423318555570c [id=59ccea88a7ec43278a191534439277f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:37.556495 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d780ce78e8ee4726a9277174c9b58a03. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:37.559499 1191 tablet_bootstrap.cc:492] T 438604e8859f422f9caccb61c5b5b125 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:37.559630 1191 ts_tablet_manager.cc:1397] T 438604e8859f422f9caccb61c5b5b125 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.304s user 0.023s sys 0.077s
I20250113 02:51:37.560370 1191 raft_consensus.cc:357] T 438604e8859f422f9caccb61c5b5b125 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:37.560600 1191 raft_consensus.cc:383] T 438604e8859f422f9caccb61c5b5b125 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:37.560649 1195 ts_tablet_manager.cc:1918] T f21eb3fa0f414816a29e3d4560cdc0fe P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:37.560775 1195 log.cc:1198] T f21eb3fa0f414816a29e3d4560cdc0fe P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/f21eb3fa0f414816a29e3d4560cdc0fe
I20250113 02:51:37.561060 1195 ts_tablet_manager.cc:1939] T f21eb3fa0f414816a29e3d4560cdc0fe P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:37.561604 754 tablet_service.cc:1467] Processing CreateTablet for tablet d780ce78e8ee4726a9277174c9b58a03 (DEFAULT_TABLE table=default.Table_d9a1e87da55c409281a423318555570c [id=59ccea88a7ec43278a191534439277f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:37.561965 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d780ce78e8ee4726a9277174c9b58a03. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:37.565047 1026 tablet_service.cc:1467] Processing CreateTablet for tablet d780ce78e8ee4726a9277174c9b58a03 (DEFAULT_TABLE table=default.Table_d9a1e87da55c409281a423318555570c [id=59ccea88a7ec43278a191534439277f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:37.565465 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d780ce78e8ee4726a9277174c9b58a03. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:37.566074 741 tablet_service.cc:1467] Processing CreateTablet for tablet d97dedb1194b449b9346e916841ee1cb (DEFAULT_TABLE table=default.Table_603d7765bb3442899a83e29f0cc0cac3 [id=8a3d504a62e9413e867a09aec833a108]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:37.566183 1196 ts_tablet_manager.cc:1918] T 77cf8b0c824f401aba1880eb7fd1807b P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:37.566395 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d97dedb1194b449b9346e916841ee1cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:37.567646 1196 log.cc:1198] T 77cf8b0c824f401aba1880eb7fd1807b P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/77cf8b0c824f401aba1880eb7fd1807b
I20250113 02:51:37.567560 1194 tablet_bootstrap.cc:492] T e468b652f71f44e3945f8d919ac0e1a8 P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:37.567879 1194 ts_tablet_manager.cc:1397] T e468b652f71f44e3945f8d919ac0e1a8 P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.062s user 0.001s sys 0.011s
I20250113 02:51:37.568670 1194 raft_consensus.cc:357] T e468b652f71f44e3945f8d919ac0e1a8 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:37.568925 1194 raft_consensus.cc:383] T e468b652f71f44e3945f8d919ac0e1a8 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:37.569015 1194 raft_consensus.cc:738] T e468b652f71f44e3945f8d919ac0e1a8 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:37.569265 1194 consensus_queue.cc:260] T e468b652f71f44e3945f8d919ac0e1a8 P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:37.569835 1194 ts_tablet_manager.cc:1428] T e468b652f71f44e3945f8d919ac0e1a8 P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250113 02:51:37.566738 888 tablet_service.cc:1467] Processing CreateTablet for tablet d97dedb1194b449b9346e916841ee1cb (DEFAULT_TABLE table=default.Table_603d7765bb3442899a83e29f0cc0cac3 [id=8a3d504a62e9413e867a09aec833a108]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:37.570315 1196 ts_tablet_manager.cc:1939] T 77cf8b0c824f401aba1880eb7fd1807b P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:37.572131 899 tablet_service.cc:1467] Processing CreateTablet for tablet a9cbff492570449b8ded5b907d817889 (DEFAULT_TABLE table=default.Table_307a6dea2b2949119bdcfb0dd2c722c0 [id=9e6ea30a8c234a179197d583fe7cd59f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:37.572599 1024 tablet_service.cc:1467] Processing CreateTablet for tablet d97dedb1194b449b9346e916841ee1cb (DEFAULT_TABLE table=default.Table_603d7765bb3442899a83e29f0cc0cac3 [id=8a3d504a62e9413e867a09aec833a108]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:37.574657 1194 tablet_bootstrap.cc:492] T 377ebf66bbef40988043f6200ae0893c P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:37.576267 1194 tablet_bootstrap.cc:654] T 377ebf66bbef40988043f6200ae0893c P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:37.576617 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d97dedb1194b449b9346e916841ee1cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:37.583055 891 tablet_service.cc:1514] Processing DeleteTablet for tablet e468b652f71f44e3945f8d919ac0e1a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:29 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:37.583385 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1031) took 1958 ms. Trace:
I20250113 02:51:37.583444 1196 tablet_replica.cc:331] T 056bf1e67be04d179eace3d93282b145 P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:37.583492 1195 rpcz_store.cc:276] 0113 02:51:35.625210 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:35.637793 (+ 12583us) service_pool.cc:225] Handling call
0113 02:51:37.434333 (+1796540us) ts_tablet_manager.cc:1145] Deleting tablet f21eb3fa0f414816a29e3d4560cdc0fe
0113 02:51:37.434342 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:37.438181 (+ 3839us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:37.560562 (+122381us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:37.583357 (+ 22795us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1796322}
I20250113 02:51:37.583616 1196 raft_consensus.cc:2238] T 056bf1e67be04d179eace3d93282b145 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:37.583748 1196 raft_consensus.cc:2267] T 056bf1e67be04d179eace3d93282b145 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:37.583895 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1030) took 1958 ms. Trace:
I20250113 02:51:37.583966 1195 rpcz_store.cc:276] 0113 02:51:35.625120 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:35.638121 (+ 13001us) service_pool.cc:225] Handling call
0113 02:51:37.583772 (+1945651us) ts_tablet_manager.cc:1145] Deleting tablet 78bf9b924b144837b6e17b54086d7908
0113 02:51:37.583781 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:37.583878 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1945470}
I20250113 02:51:37.584137 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet f21eb3fa0f414816a29e3d4560cdc0fe (table default.Table_6a3d6ef326e74c4ab0a5ed1e1bec2ecc [id=02e7cb8431884f67989a39c2ffab28ad]) successfully deleted
I20250113 02:51:37.584304 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1029) took 1959 ms. Trace:
I20250113 02:51:37.584378 1195 rpcz_store.cc:276] 0113 02:51:35.625031 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:35.638358 (+ 13327us) service_pool.cc:225] Handling call
0113 02:51:37.584205 (+1945847us) ts_tablet_manager.cc:1145] Deleting tablet 51a7cb7e7e2c49949c3be328005ce468
0113 02:51:37.584215 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:37.584290 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1945686}
I20250113 02:51:37.584612 1195 tablet_replica.cc:331] T 30ca8a716bfc4ab789b216e17567785e P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:37.584781 1195 raft_consensus.cc:2238] T 30ca8a716bfc4ab789b216e17567785e P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus shutting down.
W20250113 02:51:37.584857 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 78bf9b924b144837b6e17b54086d7908 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 78bf9b924b144837b6e17b54086d7908 already in progress: creating tablet
I20250113 02:51:37.584939 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a9cbff492570449b8ded5b907d817889. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:37.584919 1195 raft_consensus.cc:2267] T 30ca8a716bfc4ab789b216e17567785e P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250113 02:51:37.585117 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 51a7cb7e7e2c49949c3be328005ce468 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 51a7cb7e7e2c49949c3be328005ce468 already in progress: creating tablet
I20250113 02:51:37.585402 1195 ts_tablet_manager.cc:1905] T 30ca8a716bfc4ab789b216e17567785e P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:37.585963 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet 77cf8b0c824f401aba1880eb7fd1807b (table default.Table_ea18ad8d07b34ee4b3ef07b8f0431cfd [id=6069bb344e624c3bb8ca97e6c7fe23cc]) successfully deleted
I20250113 02:51:37.587353 1196 ts_tablet_manager.cc:1905] T 056bf1e67be04d179eace3d93282b145 P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:37.592393 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet aa5c172a69bf4ca79999f4b58df08de8
I20250113 02:51:37.593439 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet abc49f968ea94bdbb0443fba6864a04a
I20250113 02:51:37.594662 741 tablet_service.cc:1514] Processing DeleteTablet for tablet abc49f968ea94bdbb0443fba6864a04a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:37 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:37.594931 754 tablet_service.cc:1514] Processing DeleteTablet for tablet aa5c172a69bf4ca79999f4b58df08de8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:37 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:37.560665 1191 raft_consensus.cc:738] T 438604e8859f422f9caccb61c5b5b125 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:37.595294 891 tablet_service.cc:1514] Processing DeleteTablet for tablet abc49f968ea94bdbb0443fba6864a04a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:37 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:37.595427 1191 consensus_queue.cc:260] T 438604e8859f422f9caccb61c5b5b125 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:37.599431 1191 ts_tablet_manager.cc:1428] T 438604e8859f422f9caccb61c5b5b125 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.040s user 0.003s sys 0.000s
I20250113 02:51:37.599761 1191 tablet_bootstrap.cc:492] T 64aa8390b1b1475c8743b6491d40674c P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:37.600418 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_2eb9374b5361463297e0aba5d4432877"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:37.600644 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_86c5f0e60c9347858bc135037c194bb7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:37.600991 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2eb9374b5361463297e0aba5d4432877 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250113 02:51:37.601226 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_86c5f0e60c9347858bc135037c194bb7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:37.602243 454 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_49dafeaa3df04cb789a3e54f469973e8" } modify_external_catalogs: true
I20250113 02:51:37.602419 454 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_49dafeaa3df04cb789a3e54f469973e8" } modify_external_catalogs: true
I20250113 02:51:37.614780 1191 tablet_bootstrap.cc:654] T 64aa8390b1b1475c8743b6491d40674c P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:37.618746 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_8174140b535e44008667efc3800186da" } modify_external_catalogs: true
I20250113 02:51:37.619007 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_8174140b535e44008667efc3800186da" } modify_external_catalogs: true
I20250113 02:51:37.625257 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 378d59d314a84ddb84cf2ab041c51cf1
I20250113 02:51:37.625562 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_aab0b6c811674e558515778d0dd658a1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:37.626133 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_aab0b6c811674e558515778d0dd658a1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:37.628074 890 tablet_service.cc:1514] Processing DeleteTablet for tablet 378d59d314a84ddb84cf2ab041c51cf1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:37 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:37.628458 754 tablet_service.cc:1514] Processing DeleteTablet for tablet 378d59d314a84ddb84cf2ab041c51cf1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:37 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:37.632306 453 catalog_manager.cc:6071] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: default.Table_6461b062100240d59a5d91a87a74510c [id=40ca872c045b4349b63ba9046e31ed13] alter complete (version 2)
I20250113 02:51:37.632730 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet dff74d7c093b45eba7a47d8eceb0c1d1
I20250113 02:51:37.633615 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_b0d80c74581c4398a6887b4cd3dcf798" } new_table_name: "default.Table_d22a5cddd8e14633b22fb984025285d2" modify_external_catalogs: true
I20250113 02:51:37.635419 754 tablet_service.cc:1514] Processing DeleteTablet for tablet dff74d7c093b45eba7a47d8eceb0c1d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:37 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:37.635859 1196 ts_tablet_manager.cc:1918] T 056bf1e67be04d179eace3d93282b145 P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:37.636551 1196 log.cc:1198] T 056bf1e67be04d179eace3d93282b145 P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/056bf1e67be04d179eace3d93282b145
I20250113 02:51:37.636893 1196 ts_tablet_manager.cc:1939] T 056bf1e67be04d179eace3d93282b145 P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:37.637894 1195 ts_tablet_manager.cc:1918] T 30ca8a716bfc4ab789b216e17567785e P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:37.640653 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet 056bf1e67be04d179eace3d93282b145 (table default.Table_f5f150e8da7a4848bfab67568975d66f [id=5a295e63ad43499f90b2993ac92a16cf]) successfully deleted
W20250113 02:51:37.640933 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet cb141fb2e9894757b3ae423242ad93d2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cb141fb2e9894757b3ae423242ad93d2 already in progress: creating tablet
I20250113 02:51:37.641795 899 tablet_service.cc:1514] Processing DeleteTablet for tablet dff74d7c093b45eba7a47d8eceb0c1d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:37 UTC) from {username='slave'} at 127.0.0.1:37366
W20250113 02:51:37.642524 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 60fac96c00d64ec193b0c706ec64bfaf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 60fac96c00d64ec193b0c706ec64bfaf already in progress: creating tablet
I20250113 02:51:37.642390 754 tablet_service.cc:1514] Processing DeleteTablet for tablet 438604e8859f422f9caccb61c5b5b125 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:24 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:37.643347 1195 log.cc:1198] T 30ca8a716bfc4ab789b216e17567785e P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/30ca8a716bfc4ab789b216e17567785e
I20250113 02:51:37.643836 1195 ts_tablet_manager.cc:1939] T 30ca8a716bfc4ab789b216e17567785e P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
W20250113 02:51:37.643851 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet fe3d35d9aaf64188aaa3c1ac38d279d8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fe3d35d9aaf64188aaa3c1ac38d279d8 already in progress: creating tablet
W20250113 02:51:37.645133 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet e51b750440a24570b734bb51e7a6f923 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e51b750440a24570b734bb51e7a6f923 already in progress: creating tablet
W20250113 02:51:37.645957 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 93c0ce9f457e431193a2ceb0ad4d4af6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 93c0ce9f457e431193a2ceb0ad4d4af6 already in progress: creating tablet
I20250113 02:51:37.646467 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1033) took 1835 ms. Trace:
I20250113 02:51:37.646952 1195 rpcz_store.cc:276] 0113 02:51:35.810712 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:35.810764 (+ 52us) service_pool.cc:225] Handling call
0113 02:51:37.584574 (+1773810us) ts_tablet_manager.cc:1145] Deleting tablet 30ca8a716bfc4ab789b216e17567785e
0113 02:51:37.584582 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:37.626125 (+ 41543us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:37.637800 (+ 11675us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:37.646451 (+ 8651us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1773634}
W20250113 02:51:37.647092 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 5d2d14548b734b1fbe57d68c0ea3e2f8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5d2d14548b734b1fbe57d68c0ea3e2f8 already in progress: creating tablet
I20250113 02:51:37.647625 1196 tablet_replica.cc:331] T 0752fde3a30a4947a2a688c56ceadab5 P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:37.648183 1196 raft_consensus.cc:2238] T 0752fde3a30a4947a2a688c56ceadab5 P df69f48656b44bb890f76587ee69cc0f [term 1 FOLLOWER]: Raft consensus shutting down.
W20250113 02:51:37.648316 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 62b71866acc941e8b116ec4192a8e1e7 consensus state has no leader
I20250113 02:51:37.604077 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_2f84c2362a2b4da7b088342232749760" } new_table_name: "default.Table_82c1ba1d2d8048c7b50167f1eb6ec54a" modify_external_catalogs: true
I20250113 02:51:37.648758 1196 raft_consensus.cc:2267] T 0752fde3a30a4947a2a688c56ceadab5 P df69f48656b44bb890f76587ee69cc0f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:37.649232 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet 30ca8a716bfc4ab789b216e17567785e (table default.Table_aac49fd87c1f4c3f9fc59bd8afe6e48f [id=9eaf97b701fd4e79847dce3a9d259beb]) successfully deleted
I20250113 02:51:37.649569 1195 tablet_replica.cc:331] T f1e32e09a8244957a99bdfea5812ff31 P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:37.649655 1196 ts_tablet_manager.cc:1905] T 0752fde3a30a4947a2a688c56ceadab5 P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:37.649752 1195 raft_consensus.cc:2238] T f1e32e09a8244957a99bdfea5812ff31 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:37.652217 1195 raft_consensus.cc:2267] T f1e32e09a8244957a99bdfea5812ff31 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:37.652974 1195 ts_tablet_manager.cc:1905] T f1e32e09a8244957a99bdfea5812ff31 P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250113 02:51:37.655661 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a58588b1befe48a8a1418b788a227519 consensus state has no leader
I20250113 02:51:37.580605 1192 tablet_bootstrap.cc:492] T f8a6eebb6b6d4b63903e68750e181707 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:37.657677 1192 ts_tablet_manager.cc:1397] T f8a6eebb6b6d4b63903e68750e181707 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.106s user 0.000s sys 0.003s
I20250113 02:51:37.657634 754 tablet_service.cc:1467] Processing CreateTablet for tablet a9cbff492570449b8ded5b907d817889 (DEFAULT_TABLE table=default.Table_307a6dea2b2949119bdcfb0dd2c722c0 [id=9e6ea30a8c234a179197d583fe7cd59f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:37.658031 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a9cbff492570449b8ded5b907d817889. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:37.658179 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d97dedb1194b449b9346e916841ee1cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:37.658594 1192 raft_consensus.cc:357] T f8a6eebb6b6d4b63903e68750e181707 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:37.658816 1192 raft_consensus.cc:383] T f8a6eebb6b6d4b63903e68750e181707 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:37.658883 1192 raft_consensus.cc:738] T f8a6eebb6b6d4b63903e68750e181707 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:37.659133 1192 consensus_queue.cc:260] T f8a6eebb6b6d4b63903e68750e181707 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:37.659632 1192 ts_tablet_manager.cc:1428] T f8a6eebb6b6d4b63903e68750e181707 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250113 02:51:37.659960 1192 tablet_bootstrap.cc:492] T 9e736211eb3c4536b9fd1d90d2d83838 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:37.590812 1198 ts_tablet_manager.cc:1918] T 42cf1e708644427f9d4dcd41ae2bcb37 P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:37.661336 1198 log.cc:1198] T 42cf1e708644427f9d4dcd41ae2bcb37 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/42cf1e708644427f9d4dcd41ae2bcb37
I20250113 02:51:37.661696 1198 ts_tablet_manager.cc:1939] T 42cf1e708644427f9d4dcd41ae2bcb37 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:37.593290 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet aa5c172a69bf4ca79999f4b58df08de8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:37 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:37.594309 1042 tablet_service.cc:1514] Processing DeleteTablet for tablet abc49f968ea94bdbb0443fba6864a04a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:37 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:37.627260 1030 tablet_service.cc:1514] Processing DeleteTablet for tablet 378d59d314a84ddb84cf2ab041c51cf1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:37 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:37.634295 1031 tablet_service.cc:1514] Processing DeleteTablet for tablet d9df314ffaf042ec80180868d7e7bdf7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:24 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:37.634471 1040 tablet_service.cc:1514] Processing DeleteTablet for tablet dff74d7c093b45eba7a47d8eceb0c1d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:37 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:37.638584 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet d8bb60fdd06946739b75b255303d9d24 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:26 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:37.638696 1041 tablet_service.cc:1514] Processing DeleteTablet for tablet d5bd09a1ede644c59e400e49c8125793 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:34 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:37.666133 1041 tablet_service.cc:1467] Processing CreateTablet for tablet a9cbff492570449b8ded5b907d817889 (DEFAULT_TABLE table=default.Table_307a6dea2b2949119bdcfb0dd2c722c0 [id=9e6ea30a8c234a179197d583fe7cd59f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:37.667398 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a9cbff492570449b8ded5b907d817889. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:37.667719 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 994) took 2482 ms. Trace:
I20250113 02:51:37.667834 1198 rpcz_store.cc:276] 0113 02:51:35.185159 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:35.185884 (+ 725us) service_pool.cc:225] Handling call
0113 02:51:37.533416 (+2347532us) ts_tablet_manager.cc:1145] Deleting tablet 42cf1e708644427f9d4dcd41ae2bcb37
0113 02:51:37.533425 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:37.574658 (+ 41233us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:37.590700 (+ 16042us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:37.667648 (+ 76948us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2336309}
I20250113 02:51:37.668283 1198 tablet_replica.cc:331] T 32cba8f3efc54e78b4df41c15df82eea P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:37.668437 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet 42cf1e708644427f9d4dcd41ae2bcb37 (table default.Table_54963a1a7ff24d5a99644441a755f5de [id=acf16114671d4a3eb7ed4a0ffc984fef]) successfully deleted
I20250113 02:51:37.668501 1198 raft_consensus.cc:2238] T 32cba8f3efc54e78b4df41c15df82eea P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: Raft consensus shutting down.
I20250113 02:51:37.668860 1198 raft_consensus.cc:2267] T 32cba8f3efc54e78b4df41c15df82eea P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:37.666246 1192 tablet_bootstrap.cc:654] T 9e736211eb3c4536b9fd1d90d2d83838 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:37.669425 1198 ts_tablet_manager.cc:1905] T 32cba8f3efc54e78b4df41c15df82eea P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:37.671442 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_b761dd2912d143b886fa84c37f0047a8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:37.672073 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b761dd2912d143b886fa84c37f0047a8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:37.675333 1198 ts_tablet_manager.cc:1918] T 32cba8f3efc54e78b4df41c15df82eea P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20250113 02:51:37.675452 1198 log.cc:1198] T 32cba8f3efc54e78b4df41c15df82eea P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/32cba8f3efc54e78b4df41c15df82eea
I20250113 02:51:37.676395 1198 ts_tablet_manager.cc:1939] T 32cba8f3efc54e78b4df41c15df82eea P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:37.677122 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 995) took 2478 ms. Trace:
I20250113 02:51:37.677248 1198 rpcz_store.cc:276] 0113 02:51:35.199047 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:35.199099 (+ 52us) service_pool.cc:225] Handling call
0113 02:51:37.668189 (+2469090us) ts_tablet_manager.cc:1145] Deleting tablet 32cba8f3efc54e78b4df41c15df82eea
0113 02:51:37.668198 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:37.672354 (+ 4156us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:37.675230 (+ 2876us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:37.677090 (+ 1860us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2468927}
I20250113 02:51:37.677592 1198 tablet_replica.cc:331] T 999fecb3fc65466cad21cc1ff5ef2d44 P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:37.677872 1198 raft_consensus.cc:2238] T 999fecb3fc65466cad21cc1ff5ef2d44 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:37.678047 1198 raft_consensus.cc:2267] T 999fecb3fc65466cad21cc1ff5ef2d44 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:37.678392 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet 32cba8f3efc54e78b4df41c15df82eea (table default.Table_3d65267002364c8a9175ccb2223cac44 [id=8aff13d6d3c5483eadb596b705fe5151]) successfully deleted
I20250113 02:51:37.678591 1198 ts_tablet_manager.cc:1905] T 999fecb3fc65466cad21cc1ff5ef2d44 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:37.679313 899 tablet_service.cc:1467] Processing CreateTablet for tablet 5fead010e60544d394e0114ee4afc00c (DEFAULT_TABLE table=default.Table_05b6401543ad445d8ba261889e75c3f8 [id=7dd041424ac843d6940f790b73d858e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:37.679530 888 tablet_service.cc:1514] Processing DeleteTablet for tablet aa5c172a69bf4ca79999f4b58df08de8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:37 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:37.679769 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5fead010e60544d394e0114ee4afc00c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:37.680950 1041 tablet_service.cc:1514] Processing DeleteTablet for tablet f8a6eebb6b6d4b63903e68750e181707 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:31 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:37.681349 754 tablet_service.cc:1467] Processing CreateTablet for tablet 5fead010e60544d394e0114ee4afc00c (DEFAULT_TABLE table=default.Table_05b6401543ad445d8ba261889e75c3f8 [id=7dd041424ac843d6940f790b73d858e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:37.681820 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5fead010e60544d394e0114ee4afc00c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:37.684731 1041 tablet_service.cc:1467] Processing CreateTablet for tablet 5fead010e60544d394e0114ee4afc00c (DEFAULT_TABLE table=default.Table_05b6401543ad445d8ba261889e75c3f8 [id=7dd041424ac843d6940f790b73d858e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:37.685129 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5fead010e60544d394e0114ee4afc00c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:37.697685 1192 tablet_bootstrap.cc:492] T 9e736211eb3c4536b9fd1d90d2d83838 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:37.697854 1192 ts_tablet_manager.cc:1397] T 9e736211eb3c4536b9fd1d90d2d83838 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.038s user 0.000s sys 0.012s
I20250113 02:51:37.698683 1195 ts_tablet_manager.cc:1918] T f1e32e09a8244957a99bdfea5812ff31 P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:37.698678 1192 raft_consensus.cc:357] T 9e736211eb3c4536b9fd1d90d2d83838 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:37.698957 1192 raft_consensus.cc:383] T 9e736211eb3c4536b9fd1d90d2d83838 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:37.699075 1192 raft_consensus.cc:738] T 9e736211eb3c4536b9fd1d90d2d83838 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:37.699374 1192 consensus_queue.cc:260] T 9e736211eb3c4536b9fd1d90d2d83838 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:37.700559 1195 log.cc:1198] T f1e32e09a8244957a99bdfea5812ff31 P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/f1e32e09a8244957a99bdfea5812ff31
I20250113 02:51:37.701141 1195 ts_tablet_manager.cc:1939] T f1e32e09a8244957a99bdfea5812ff31 P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:37.701839 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1034) took 1881 ms. Trace:
I20250113 02:51:37.702319 1195 rpcz_store.cc:276] 0113 02:51:35.820402 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:35.820458 (+ 56us) service_pool.cc:225] Handling call
0113 02:51:37.649520 (+1829062us) ts_tablet_manager.cc:1145] Deleting tablet f1e32e09a8244957a99bdfea5812ff31
0113 02:51:37.649532 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:37.668480 (+ 18948us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:37.698575 (+ 30095us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:37.701820 (+ 3245us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1828906}
I20250113 02:51:37.703116 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet f1e32e09a8244957a99bdfea5812ff31 (table default.Table_12a092a4543e4b04a466945f4b694c53 [id=c84b256e3b5744edbbf4de12eeaff0a3]) successfully deleted
I20250113 02:51:37.703652 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1035) took 1881 ms. Trace:
I20250113 02:51:37.703780 1196 ts_tablet_manager.cc:1918] T 0752fde3a30a4947a2a688c56ceadab5 P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250113 02:51:37.704062 1195 rpcz_store.cc:276] 0113 02:51:35.821831 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:35.821881 (+ 50us) service_pool.cc:225] Handling call
0113 02:51:37.703520 (+1881639us) ts_tablet_manager.cc:1145] Deleting tablet 29c17fd90d774c029a35e77888358067
0113 02:51:37.703531 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:37.703635 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1881483}
I20250113 02:51:37.706338 1192 ts_tablet_manager.cc:1428] T 9e736211eb3c4536b9fd1d90d2d83838 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20250113 02:51:37.706669 1192 tablet_bootstrap.cc:492] T b0113b16110e4787ac1c4676b2c8bd1e P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:37.707077 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e736211eb3c4536b9fd1d90d2d83838 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:24 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:37.708696 1192 tablet_bootstrap.cc:654] T b0113b16110e4787ac1c4676b2c8bd1e P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:37.709836 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_901b9f58fa9941bea7289bfcb72b22bb" } new_table_name: "default.Table_74819cd117f64b5c80c1f9c4011fe8f7" modify_external_catalogs: true
I20250113 02:51:37.711310 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_56ac841ab1804eafbd61ea8fa855824f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:37.711874 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_56ac841ab1804eafbd61ea8fa855824f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:37.712519 1035 tablet_service.cc:1467] Processing CreateTablet for tablet 6b7716e67cab449ab3ed4e58eae20c9b (DEFAULT_TABLE table=default.Table_183e07b4f9aa438e8a1c37ef385f57d6 [id=6c0fff9644b841aa81430ac18d524e22]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:37.712942 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b7716e67cab449ab3ed4e58eae20c9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:37.716296 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_25480b2b4ab44c05b96c1c77cec60f61" } new_table_name: "default.Table_022a7c61a6864393acd739f562893b7c" modify_external_catalogs: true
I20250113 02:51:37.716881 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_ccac11f0e0014fe7a1277b59017d92e4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:37.717423 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ccac11f0e0014fe7a1277b59017d92e4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250113 02:51:37.721366 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 29c17fd90d774c029a35e77888358067 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 29c17fd90d774c029a35e77888358067 already in progress: creating tablet
I20250113 02:51:37.722369 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_fc1ec6582de6419e8333bb327867f4a0" } new_table_name: "default.Table_f9e0115a922e4a7c825265fc1f7222f9" modify_external_catalogs: true
I20250113 02:51:37.724498 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_8c95a106e0684de587166acbc0de43e1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:37.724985 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8c95a106e0684de587166acbc0de43e1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:37.724913 736 tablet_service.cc:1467] Processing CreateTablet for tablet d1f30873a02f4b61ac6940e44c332ad3 (DEFAULT_TABLE table=default.Table_6ca7cb0180f844548bfb870bb68aaeb9 [id=d4b770fcdadb4a9cb77931f58a2c0609]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:37.725024 1198 ts_tablet_manager.cc:1918] T 999fecb3fc65466cad21cc1ff5ef2d44 P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:37.725184 1198 log.cc:1198] T 999fecb3fc65466cad21cc1ff5ef2d44 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/999fecb3fc65466cad21cc1ff5ef2d44
I20250113 02:51:37.725317 736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d1f30873a02f4b61ac6940e44c332ad3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:37.725507 1198 ts_tablet_manager.cc:1939] T 999fecb3fc65466cad21cc1ff5ef2d44 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:37.726272 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 997) took 2507 ms. Trace:
I20250113 02:51:37.726358 1198 rpcz_store.cc:276] 0113 02:51:35.219188 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:35.219237 (+ 49us) service_pool.cc:225] Handling call
0113 02:51:37.677519 (+2458282us) ts_tablet_manager.cc:1145] Deleting tablet 999fecb3fc65466cad21cc1ff5ef2d44
0113 02:51:37.677528 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:37.691113 (+ 13585us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:37.724900 (+ 33787us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:37.726218 (+ 1318us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2458111}
I20250113 02:51:37.726691 1198 tablet_replica.cc:331] T 4b07a985897641c3ac48021c2a4daedb P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:37.726856 1198 raft_consensus.cc:2238] T 4b07a985897641c3ac48021c2a4daedb P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:37.727005 1198 raft_consensus.cc:2267] T 4b07a985897641c3ac48021c2a4daedb P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:37.727058 754 tablet_service.cc:1467] Processing CreateTablet for tablet 6b7716e67cab449ab3ed4e58eae20c9b (DEFAULT_TABLE table=default.Table_183e07b4f9aa438e8a1c37ef385f57d6 [id=6c0fff9644b841aa81430ac18d524e22]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:37.727392 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b7716e67cab449ab3ed4e58eae20c9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:37.727512 1198 ts_tablet_manager.cc:1905] T 4b07a985897641c3ac48021c2a4daedb P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:37.729033 1196 log.cc:1198] T 0752fde3a30a4947a2a688c56ceadab5 P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/0752fde3a30a4947a2a688c56ceadab5
I20250113 02:51:37.729782 1196 ts_tablet_manager.cc:1939] T 0752fde3a30a4947a2a688c56ceadab5 P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:37.731243 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet 0752fde3a30a4947a2a688c56ceadab5 (table default.Table_da3e49fa1e654145a05858466ec2260f [id=94704429b49b46e38a48258a450d2705]) successfully deleted
I20250113 02:51:37.731200 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_458c7f623bc4476c918b4a71f74e3170"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:37.731745 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_458c7f623bc4476c918b4a71f74e3170 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:37.732360 1196 tablet_replica.cc:331] T 05bf5ab0df874dd5b6559f6d82069184 P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:37.709570 888 tablet_service.cc:1467] Processing CreateTablet for tablet 6b7716e67cab449ab3ed4e58eae20c9b (DEFAULT_TABLE table=default.Table_183e07b4f9aa438e8a1c37ef385f57d6 [id=6c0fff9644b841aa81430ac18d524e22]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:37.717844 899 tablet_service.cc:1467] Processing CreateTablet for tablet d1f30873a02f4b61ac6940e44c332ad3 (DEFAULT_TABLE table=default.Table_6ca7cb0180f844548bfb870bb68aaeb9 [id=d4b770fcdadb4a9cb77931f58a2c0609]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:37.724478 2286 raft_consensus.cc:491] T e468b652f71f44e3945f8d919ac0e1a8 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:37.732591 2286 raft_consensus.cc:513] T e468b652f71f44e3945f8d919ac0e1a8 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:37.732755 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b7716e67cab449ab3ed4e58eae20c9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:37.733408 2286 leader_election.cc:290] T e468b652f71f44e3945f8d919ac0e1a8 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
W20250113 02:51:37.733865 851 leader_election.cc:343] T e468b652f71f44e3945f8d919ac0e1a8 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Not found: Tablet not found: e468b652f71f44e3945f8d919ac0e1a8
I20250113 02:51:37.733839 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet 999fecb3fc65466cad21cc1ff5ef2d44 (table default.Table_991d8c72ce924153ad119a440b54fb1c [id=71e76dffaa6f4b0682422afed1749a64]) successfully deleted
I20250113 02:51:37.734062 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1036) took 1907 ms. Trace:
I20250113 02:51:37.734112 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d1f30873a02f4b61ac6940e44c332ad3. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:37.734458 852 leader_election.cc:343] T e468b652f71f44e3945f8d919ac0e1a8 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Not found: Tablet not found: e468b652f71f44e3945f8d919ac0e1a8
I20250113 02:51:37.734550 852 leader_election.cc:304] T e468b652f71f44e3945f8d919ac0e1a8 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df69f48656b44bb890f76587ee69cc0f; no voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca
I20250113 02:51:37.734911 2286 raft_consensus.cc:2743] T e468b652f71f44e3945f8d919ac0e1a8 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:37.734889 1195 rpcz_store.cc:276] 0113 02:51:35.826531 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:35.826582 (+ 51us) service_pool.cc:225] Handling call
0113 02:51:37.733951 (+1907369us) ts_tablet_manager.cc:1145] Deleting tablet b2a23259f2c448cc8a904a13a291d6ca
0113 02:51:37.733960 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:37.734043 (+ 83us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1907068}
W20250113 02:51:37.735394 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 03f24cc9d3bb413485b132228d73d272 consensus state has no leader
W20250113 02:51:37.735365 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet b2a23259f2c448cc8a904a13a291d6ca because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b2a23259f2c448cc8a904a13a291d6ca already in progress: creating tablet
I20250113 02:51:37.735863 1195 tablet_replica.cc:331] T b0113b16110e4787ac1c4676b2c8bd1e P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:37.736258 1195 raft_consensus.cc:2238] T b0113b16110e4787ac1c4676b2c8bd1e P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:37.737728 1195 raft_consensus.cc:2267] T b0113b16110e4787ac1c4676b2c8bd1e P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:37.738957 1195 ts_tablet_manager.cc:1905] T b0113b16110e4787ac1c4676b2c8bd1e P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250113 02:51:37.743175 451 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 019eb21c661547cfa0b619d119eb024e consensus state has no leader
I20250113 02:51:37.743187 1196 raft_consensus.cc:2238] T 05bf5ab0df874dd5b6559f6d82069184 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:37.743352 1196 raft_consensus.cc:2267] T 05bf5ab0df874dd5b6559f6d82069184 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
W20250113 02:51:37.744422 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4791a82b5e13429184d0c7192dd7aef6 has no consensus state
I20250113 02:51:37.745298 449 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_38d715a0346f4b3db579e626066b5d49" } modify_external_catalogs: true
I20250113 02:51:37.744437 1196 ts_tablet_manager.cc:1905] T 05bf5ab0df874dd5b6559f6d82069184 P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:37.745820 449 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_38d715a0346f4b3db579e626066b5d49" } modify_external_catalogs: true
I20250113 02:51:37.748867 2306 raft_consensus.cc:491] T 7213e3284b4d4cb581d9fb946a9f87e9 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:37.749042 2306 raft_consensus.cc:513] T 7213e3284b4d4cb581d9fb946a9f87e9 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:37.749682 2306 leader_election.cc:290] T 7213e3284b4d4cb581d9fb946a9f87e9 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
W20250113 02:51:37.750540 995 leader_election.cc:343] T 7213e3284b4d4cb581d9fb946a9f87e9 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Not found: Tablet not found: 7213e3284b4d4cb581d9fb946a9f87e9
I20250113 02:51:37.756335 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 97159fccfa154465a33ba90a27576f0a
I20250113 02:51:37.757843 1040 tablet_service.cc:1514] Processing DeleteTablet for tablet 97159fccfa154465a33ba90a27576f0a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:37 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:37.759156 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_e031078584204383b805a4e04f36395a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:37.759691 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e031078584204383b805a4e04f36395a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:37.761482 754 tablet_service.cc:1514] Processing DeleteTablet for tablet 97159fccfa154465a33ba90a27576f0a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:37 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:37.761718 899 tablet_service.cc:1514] Processing DeleteTablet for tablet 97159fccfa154465a33ba90a27576f0a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:37 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:37.764289 2306 raft_consensus.cc:491] T 4eb5f1944cb44fd5a40ca92a6a3d7214 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:37.764470 2306 raft_consensus.cc:513] T 4eb5f1944cb44fd5a40ca92a6a3d7214 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:37.765056 2306 leader_election.cc:290] T 4eb5f1944cb44fd5a40ca92a6a3d7214 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:37.765398 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4eb5f1944cb44fd5a40ca92a6a3d7214" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
I20250113 02:51:37.765586 774 raft_consensus.cc:2463] T 4eb5f1944cb44fd5a40ca92a6a3d7214 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 0.
I20250113 02:51:37.766129 995 leader_election.cc:304] T 4eb5f1944cb44fd5a40ca92a6a3d7214 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters:
I20250113 02:51:37.768085 2306 raft_consensus.cc:2798] T 4eb5f1944cb44fd5a40ca92a6a3d7214 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 02:51:37.768183 2306 raft_consensus.cc:491] T 4eb5f1944cb44fd5a40ca92a6a3d7214 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 02:51:37.768276 2306 raft_consensus.cc:3054] T 4eb5f1944cb44fd5a40ca92a6a3d7214 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:37.769863 1196 ts_tablet_manager.cc:1918] T 05bf5ab0df874dd5b6559f6d82069184 P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:37.769968 1196 log.cc:1198] T 05bf5ab0df874dd5b6559f6d82069184 P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/05bf5ab0df874dd5b6559f6d82069184
I20250113 02:51:37.770319 1196 ts_tablet_manager.cc:1939] T 05bf5ab0df874dd5b6559f6d82069184 P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:37.770691 2306 raft_consensus.cc:513] T 4eb5f1944cb44fd5a40ca92a6a3d7214 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:37.770869 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet 05bf5ab0df874dd5b6559f6d82069184 (table default.Table_5175e117e3f34778a5ef456bed6b5e28 [id=5c9d31a6ab2445ad949f45abffdf4cc7]) successfully deleted
I20250113 02:51:37.770953 1195 ts_tablet_manager.cc:1918] T b0113b16110e4787ac1c4676b2c8bd1e P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:37.771051 1195 log.cc:1198] T b0113b16110e4787ac1c4676b2c8bd1e P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/b0113b16110e4787ac1c4676b2c8bd1e
I20250113 02:51:37.771366 1195 ts_tablet_manager.cc:1939] T b0113b16110e4787ac1c4676b2c8bd1e P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:37.771404 2306 leader_election.cc:290] T 4eb5f1944cb44fd5a40ca92a6a3d7214 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Requested vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:37.771699 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4eb5f1944cb44fd5a40ca92a6a3d7214" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196"
I20250113 02:51:37.771878 774 raft_consensus.cc:3054] T 4eb5f1944cb44fd5a40ca92a6a3d7214 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:37.772128 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1037) took 1908 ms. Trace:
I20250113 02:51:37.772218 1195 rpcz_store.cc:276] 0113 02:51:35.863534 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:35.863706 (+ 172us) service_pool.cc:225] Handling call
0113 02:51:37.735820 (+1872114us) ts_tablet_manager.cc:1145] Deleting tablet b0113b16110e4787ac1c4676b2c8bd1e
0113 02:51:37.735829 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:37.751069 (+ 15240us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:37.770861 (+ 19792us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:37.772110 (+ 1249us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1871930}
I20250113 02:51:37.772588 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1038) took 1892 ms. Trace:
I20250113 02:51:37.772653 1195 rpcz_store.cc:276] 0113 02:51:35.879999 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:35.880076 (+ 77us) service_pool.cc:225] Handling call
0113 02:51:37.772475 (+1892399us) ts_tablet_manager.cc:1145] Deleting tablet d8e81a6fef794f26bdb017e16af8b30c
0113 02:51:37.772484 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:37.772572 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1892201}
I20250113 02:51:37.772939 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1039) took 1882 ms. Trace:
I20250113 02:51:37.773002 1195 rpcz_store.cc:276] 0113 02:51:35.890337 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:35.890388 (+ 51us) service_pool.cc:225] Handling call
0113 02:51:37.772843 (+1882455us) ts_tablet_manager.cc:1145] Deleting tablet cb897004cbfb4a0283b42d06509a9ee5
0113 02:51:37.772852 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:37.772924 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1882290}
I20250113 02:51:37.773229 1195 tablet_replica.cc:331] T 9e736211eb3c4536b9fd1d90d2d83838 P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:37.773387 1195 raft_consensus.cc:2238] T 9e736211eb3c4536b9fd1d90d2d83838 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:37.773507 1195 raft_consensus.cc:2267] T 9e736211eb3c4536b9fd1d90d2d83838 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:37.774305 774 raft_consensus.cc:2463] T 4eb5f1944cb44fd5a40ca92a6a3d7214 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 1.
I20250113 02:51:37.775205 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet b0113b16110e4787ac1c4676b2c8bd1e (table default.Table_d5c2b805dfbf4227bb1ac190525b7dee [id=bf855203fc20403294cc733cdbc593e5]) successfully deleted
I20250113 02:51:37.775347 754 tablet_service.cc:1467] Processing CreateTablet for tablet 4f5849587a004a0cb64fae4d9be94160 (DEFAULT_TABLE table=default.Table_f407d41e445c4422901d17d1a02a31b2 [id=2e6d666dab514d56bcc1dd28d93cf14b]), partition=RANGE (key) PARTITION UNBOUNDED
W20250113 02:51:37.775442 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet d8e81a6fef794f26bdb017e16af8b30c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d8e81a6fef794f26bdb017e16af8b30c already in progress: creating tablet
W20250113 02:51:37.775652 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet cb897004cbfb4a0283b42d06509a9ee5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cb897004cbfb4a0283b42d06509a9ee5 already in progress: creating tablet
I20250113 02:51:37.775683 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f5849587a004a0cb64fae4d9be94160. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:37.776114 995 leader_election.cc:304] T 4eb5f1944cb44fd5a40ca92a6a3d7214 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters:
I20250113 02:51:37.776480 2306 raft_consensus.cc:2798] T 4eb5f1944cb44fd5a40ca92a6a3d7214 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Leader election won for term 1
I20250113 02:51:37.776631 2306 raft_consensus.cc:695] T 4eb5f1944cb44fd5a40ca92a6a3d7214 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: Becoming Leader. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Running, Role: LEADER
I20250113 02:51:37.776604 2277 tablet.cc:1721] T 06f36d09bbb1443bafce906907795404 P 4216a7f853d642b7bc3e5cff458f3aca: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20250113 02:51:37.776865 2306 consensus_queue.cc:237] T 4eb5f1944cb44fd5a40ca92a6a3d7214 P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:37.778791 1040 tablet_service.cc:1467] Processing CreateTablet for tablet 4f5849587a004a0cb64fae4d9be94160 (DEFAULT_TABLE table=default.Table_f407d41e445c4422901d17d1a02a31b2 [id=2e6d666dab514d56bcc1dd28d93cf14b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:37.779173 1040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f5849587a004a0cb64fae4d9be94160. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:37.781281 899 tablet_service.cc:1467] Processing CreateTablet for tablet 4f5849587a004a0cb64fae4d9be94160 (DEFAULT_TABLE table=default.Table_f407d41e445c4422901d17d1a02a31b2 [id=2e6d666dab514d56bcc1dd28d93cf14b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:37.781675 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f5849587a004a0cb64fae4d9be94160. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:37.782924 2281 tablet.cc:1721] T 06f36d09bbb1443bafce906907795404 P 1186fa869eb14812832fa6562531f196: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20250113 02:51:37.785779 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4eb5f1944cb44fd5a40ca92a6a3d7214" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:37.786135 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4eb5f1944cb44fd5a40ca92a6a3d7214" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f"
I20250113 02:51:37.786496 2227 raft_consensus.cc:491] T 0c408921a09a463eb464e08edcf042b3 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:37.786626 2227 raft_consensus.cc:513] T 0c408921a09a463eb464e08edcf042b3 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:37.787159 2227 leader_election.cc:290] T 0c408921a09a463eb464e08edcf042b3 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:37.787319 2242 tablet.cc:1721] T 06f36d09bbb1443bafce906907795404 P df69f48656b44bb890f76587ee69cc0f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20250113 02:51:37.787700 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7213e3284b4d4cb581d9fb946a9f87e9" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:37.787823 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c408921a09a463eb464e08edcf042b3" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:37.788446 711 leader_election.cc:343] T 0c408921a09a463eb464e08edcf042b3 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:37.789484 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c408921a09a463eb464e08edcf042b3" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
W20250113 02:51:37.789981 998 leader_election.cc:343] T 4eb5f1944cb44fd5a40ca92a6a3d7214 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
W20250113 02:51:37.790419 998 leader_election.cc:343] T 4eb5f1944cb44fd5a40ca92a6a3d7214 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
W20250113 02:51:37.791533 998 leader_election.cc:343] T 7213e3284b4d4cb581d9fb946a9f87e9 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:37.791697 998 leader_election.cc:304] T 7213e3284b4d4cb581d9fb946a9f87e9 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4216a7f853d642b7bc3e5cff458f3aca; no voters: 1186fa869eb14812832fa6562531f196, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:37.794061 2300 raft_consensus.cc:2743] T 7213e3284b4d4cb581d9fb946a9f87e9 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:37.795334 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_511c88ce360749d0a2bbcab80bb64629" } modify_external_catalogs: true
I20250113 02:51:37.795580 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_511c88ce360749d0a2bbcab80bb64629" } modify_external_catalogs: true
I20250113 02:51:37.797034 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_2bcd0f2a378c4d2aaefbbbfa2c144431"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:37.797641 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2bcd0f2a378c4d2aaefbbbfa2c144431 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:37.800873 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_ef9c0cf86b0943aab6bbb6a3fe74ba85" } modify_external_catalogs: true
I20250113 02:51:37.801566 2286 raft_consensus.cc:491] T f56230a2c75d4763a54671c79358f4d9 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:37.801693 2286 raft_consensus.cc:513] T f56230a2c75d4763a54671c79358f4d9 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:37.802274 2286 leader_election.cc:290] T f56230a2c75d4763a54671c79358f4d9 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
W20250113 02:51:37.803025 851 leader_election.cc:343] T f56230a2c75d4763a54671c79358f4d9 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Not found: Tablet not found: f56230a2c75d4763a54671c79358f4d9
W20250113 02:51:37.803474 852 leader_election.cc:343] T f56230a2c75d4763a54671c79358f4d9 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Not found: Tablet not found: f56230a2c75d4763a54671c79358f4d9
I20250113 02:51:37.803570 852 leader_election.cc:304] T f56230a2c75d4763a54671c79358f4d9 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df69f48656b44bb890f76587ee69cc0f; no voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca
I20250113 02:51:37.803686 1194 tablet_bootstrap.cc:492] T 377ebf66bbef40988043f6200ae0893c P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:37.803792 1194 ts_tablet_manager.cc:1397] T 377ebf66bbef40988043f6200ae0893c P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.229s user 0.014s sys 0.013s
I20250113 02:51:37.804040 2286 raft_consensus.cc:2743] T f56230a2c75d4763a54671c79358f4d9 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:37.804481 1194 raft_consensus.cc:357] T 377ebf66bbef40988043f6200ae0893c P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:37.804679 1194 raft_consensus.cc:383] T 377ebf66bbef40988043f6200ae0893c P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:37.804718 1191 tablet_bootstrap.cc:492] T 64aa8390b1b1475c8743b6491d40674c P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:37.804745 1194 raft_consensus.cc:738] T 377ebf66bbef40988043f6200ae0893c P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:37.804848 1191 ts_tablet_manager.cc:1397] T 64aa8390b1b1475c8743b6491d40674c P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.205s user 0.013s sys 0.011s
I20250113 02:51:37.805018 1194 consensus_queue.cc:260] T 377ebf66bbef40988043f6200ae0893c P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:37.805207 1041 tablet_service.cc:1467] Processing CreateTablet for tablet d1f30873a02f4b61ac6940e44c332ad3 (DEFAULT_TABLE table=default.Table_6ca7cb0180f844548bfb870bb68aaeb9 [id=d4b770fcdadb4a9cb77931f58a2c0609]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:37.805606 1194 ts_tablet_manager.cc:1428] T 377ebf66bbef40988043f6200ae0893c P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250113 02:51:37.805640 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d1f30873a02f4b61ac6940e44c332ad3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:37.805549 1191 raft_consensus.cc:357] T 64aa8390b1b1475c8743b6491d40674c P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:37.805725 1191 raft_consensus.cc:383] T 64aa8390b1b1475c8743b6491d40674c P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:37.805780 1191 raft_consensus.cc:738] T 64aa8390b1b1475c8743b6491d40674c P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:37.806061 1194 tablet_bootstrap.cc:492] T aa5c172a69bf4ca79999f4b58df08de8 P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:37.805977 1191 consensus_queue.cc:260] T 64aa8390b1b1475c8743b6491d40674c P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:37.806427 1191 ts_tablet_manager.cc:1428] T 64aa8390b1b1475c8743b6491d40674c P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250113 02:51:37.808460 1194 tablet_bootstrap.cc:654] T aa5c172a69bf4ca79999f4b58df08de8 P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:37.809770 1192 tablet_bootstrap.cc:492] T b0113b16110e4787ac1c4676b2c8bd1e P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:37.810065 1192 ts_tablet_manager.cc:1397] T b0113b16110e4787ac1c4676b2c8bd1e P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.103s user 0.000s sys 0.004s
I20250113 02:51:37.811159 1192 raft_consensus.cc:357] T b0113b16110e4787ac1c4676b2c8bd1e P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:37.811429 1192 raft_consensus.cc:383] T b0113b16110e4787ac1c4676b2c8bd1e P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:37.811553 1192 raft_consensus.cc:738] T b0113b16110e4787ac1c4676b2c8bd1e P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:37.812436 1191 tablet_bootstrap.cc:492] T 1d2fdc75d00f4e749d322ac6cd5fc410 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:37.814268 1191 tablet_bootstrap.cc:654] T 1d2fdc75d00f4e749d322ac6cd5fc410 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:37.820135 1107 heartbeater.cc:346] Connected to a master server at 127.0.104.60:42067
I20250113 02:51:37.820956 1198 ts_tablet_manager.cc:1918] T 4b07a985897641c3ac48021c2a4daedb P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:37.821076 1198 log.cc:1198] T 4b07a985897641c3ac48021c2a4daedb P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/4b07a985897641c3ac48021c2a4daedb
I20250113 02:51:37.821463 1198 ts_tablet_manager.cc:1939] T 4b07a985897641c3ac48021c2a4daedb P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:37.821777 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 996) took 2602 ms. Trace:
I20250113 02:51:37.821903 1198 rpcz_store.cc:276] 0113 02:51:35.219074 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:35.219435 (+ 361us) service_pool.cc:225] Handling call
0113 02:51:37.726618 (+2507183us) ts_tablet_manager.cc:1145] Deleting tablet 4b07a985897641c3ac48021c2a4daedb
0113 02:51:37.726627 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:37.762238 (+ 35611us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:37.820805 (+ 58567us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:37.821735 (+ 930us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2507064}
I20250113 02:51:37.822343 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 998) took 2585 ms. Trace:
I20250113 02:51:37.822453 1198 rpcz_store.cc:276] 0113 02:51:35.237201 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:35.237250 (+ 49us) service_pool.cc:225] Handling call
0113 02:51:37.822206 (+2584956us) ts_tablet_manager.cc:1145] Deleting tablet 2c49f1eefd2041b4ae617a4cb35d6f15
0113 02:51:37.822215 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:37.822299 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2584769}
I20250113 02:51:37.824642 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_593fc8a8736e4d4aaeda0d72bbb30337"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:37.825281 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_593fc8a8736e4d4aaeda0d72bbb30337 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:37.825621 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_9de04b3793d248e6be61bb3b1e7feb77"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:37.826050 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9de04b3793d248e6be61bb3b1e7feb77 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:37.828325 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1000) took 2577 ms. Trace:
I20250113 02:51:37.830112 455 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_ed883bfc96094be9bb6e07bc510b6042" } modify_external_catalogs: true
I20250113 02:51:37.817950 1195 ts_tablet_manager.cc:1905] T 9e736211eb3c4536b9fd1d90d2d83838 P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:37.819065 2227 raft_consensus.cc:491] T 153b8f90b5a847b88944f683d306f579 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:37.833367 2227 raft_consensus.cc:513] T 153b8f90b5a847b88944f683d306f579 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:37.833987 2227 leader_election.cc:290] T 153b8f90b5a847b88944f683d306f579 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:37.835134 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "153b8f90b5a847b88944f683d306f579" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:37.835742 711 leader_election.cc:343] T 153b8f90b5a847b88944f683d306f579 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:37.841693 449 catalog_manager.cc:6071] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: default.Table_626d5cb595994368aa592eec7021b889 [id=423021f304964246bd29e4daaf470154] alter complete (version 4)
I20250113 02:51:37.842868 1191 tablet_bootstrap.cc:492] T 1d2fdc75d00f4e749d322ac6cd5fc410 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:37.843086 1191 ts_tablet_manager.cc:1397] T 1d2fdc75d00f4e749d322ac6cd5fc410 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.031s user 0.001s sys 0.002s
I20250113 02:51:37.843163 455 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_ed883bfc96094be9bb6e07bc510b6042" } modify_external_catalogs: true
I20250113 02:51:37.843734 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 524b4bd640a04123b4abbd716e7d40f2
I20250113 02:51:37.843935 1191 raft_consensus.cc:357] T 1d2fdc75d00f4e749d322ac6cd5fc410 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:37.844131 1191 raft_consensus.cc:383] T 1d2fdc75d00f4e749d322ac6cd5fc410 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:37.844184 1191 raft_consensus.cc:738] T 1d2fdc75d00f4e749d322ac6cd5fc410 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:37.844396 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_ef9c0cf86b0943aab6bbb6a3fe74ba85" } modify_external_catalogs: true
I20250113 02:51:37.844398 1191 consensus_queue.cc:260] T 1d2fdc75d00f4e749d322ac6cd5fc410 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:37.844857 1191 ts_tablet_manager.cc:1428] T 1d2fdc75d00f4e749d322ac6cd5fc410 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250113 02:51:37.845060 899 tablet_service.cc:1514] Processing DeleteTablet for tablet 524b4bd640a04123b4abbd716e7d40f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:37 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:37.846552 899 tablet_service.cc:1514] Processing DeleteTablet for tablet 377ebf66bbef40988043f6200ae0893c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:34 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:37.846958 454 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_c7728cb38c5b4dcc8cd76e68b6a24a20" } modify_external_catalogs: true
I20250113 02:51:37.847143 454 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_c7728cb38c5b4dcc8cd76e68b6a24a20" } modify_external_catalogs: true
I20250113 02:51:37.847208 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_0a0352107b6c4b01a7abdce96fbb3c48"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:37.847782 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0a0352107b6c4b01a7abdce96fbb3c48 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:37.849850 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_1c82125f87e542e0932f372cd1d99dab" } new_table_name: "default.Table_38228ee6b228417c8785773ba970e800" modify_external_catalogs: true
I20250113 02:51:37.851258 1191 tablet_bootstrap.cc:492] T 82505f1389184d0ca12ec0e2d6084d22 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:37.852703 736 tablet_service.cc:1514] Processing DeleteTablet for tablet 524b4bd640a04123b4abbd716e7d40f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:37 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:37.853541 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet 4b07a985897641c3ac48021c2a4daedb (table default.Table_066f136aea8141fa91862f7285fbaac2 [id=fa29f9bbea034fafad86ad86aa5775eb]) successfully deleted
W20250113 02:51:37.853869 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 2c49f1eefd2041b4ae617a4cb35d6f15 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2c49f1eefd2041b4ae617a4cb35d6f15 already in progress: creating tablet
I20250113 02:51:37.855060 448 catalog_manager.cc:5526] T 4eb5f1944cb44fd5a40ca92a6a3d7214 P 4216a7f853d642b7bc3e5cff458f3aca reported cstate change: term changed from 0 to 1, leader changed from <none> to 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3). New cstate: current_term: 1 leader_uuid: "4216a7f853d642b7bc3e5cff458f3aca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } health_report { overall_health: UNKNOWN } } }
I20250113 02:51:37.856340 1196 tablet_replica.cc:331] T f56230a2c75d4763a54671c79358f4d9 P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:37.856518 1196 raft_consensus.cc:2238] T f56230a2c75d4763a54671c79358f4d9 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:37.856652 1196 raft_consensus.cc:2267] T f56230a2c75d4763a54671c79358f4d9 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:37.857290 1196 ts_tablet_manager.cc:1905] T f56230a2c75d4763a54671c79358f4d9 P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:37.858579 1191 tablet_bootstrap.cc:654] T 82505f1389184d0ca12ec0e2d6084d22 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:37.860239 1195 ts_tablet_manager.cc:1918] T 9e736211eb3c4536b9fd1d90d2d83838 P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:37.860345 1195 log.cc:1198] T 9e736211eb3c4536b9fd1d90d2d83838 P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/9e736211eb3c4536b9fd1d90d2d83838
I20250113 02:51:37.860626 1195 ts_tablet_manager.cc:1939] T 9e736211eb3c4536b9fd1d90d2d83838 P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:37.860931 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1040) took 1961 ms. Trace:
I20250113 02:51:37.861009 1195 rpcz_store.cc:276] 0113 02:51:35.899829 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:35.899881 (+ 52us) service_pool.cc:225] Handling call
0113 02:51:37.773191 (+1873310us) ts_tablet_manager.cc:1145] Deleting tablet 9e736211eb3c4536b9fd1d90d2d83838
0113 02:51:37.773199 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:37.836322 (+ 63123us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:37.860127 (+ 23805us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:37.860910 (+ 783us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1873089}
I20250113 02:51:37.861336 1195 tablet_replica.cc:331] T 4c615b0b83d74fcfae0c51047dbfbbf4 P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:37.861522 1195 raft_consensus.cc:2238] T 4c615b0b83d74fcfae0c51047dbfbbf4 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:37.861631 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet 9e736211eb3c4536b9fd1d90d2d83838 (table default.Table_f8af5d3e0dc64d3fae054f82a7b33ccf [id=4b739228564344b8b394a4718750dbb0]) successfully deleted
I20250113 02:51:37.861701 1195 raft_consensus.cc:2267] T 4c615b0b83d74fcfae0c51047dbfbbf4 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:37.862185 1195 ts_tablet_manager.cc:1905] T 4c615b0b83d74fcfae0c51047dbfbbf4 P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:37.869815 754 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d2fdc75d00f4e749d322ac6cd5fc410 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:27 UTC) from {username='slave'} at 127.0.0.1:55398
W20250113 02:51:37.838472 998 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:38467, user_credentials={real_user=slave}} blocked reactor thread for 47159.6us
W20250113 02:51:37.872267 709 leader_election.cc:343] T 0c408921a09a463eb464e08edcf042b3 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:37.872399 709 leader_election.cc:304] T 0c408921a09a463eb464e08edcf042b3 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:37.872889 2227 raft_consensus.cc:2743] T 0c408921a09a463eb464e08edcf042b3 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:37.874279 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet a8aedbc8d7bb4183bb1ab9902f07bea7
I20250113 02:51:37.874621 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet b1cd18ee2e2145ba9273f500d468343d
I20250113 02:51:37.856181 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet 524b4bd640a04123b4abbd716e7d40f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:37 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:37.875613 741 tablet_service.cc:1514] Processing DeleteTablet for tablet b1cd18ee2e2145ba9273f500d468343d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:37 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:37.875500 754 tablet_service.cc:1514] Processing DeleteTablet for tablet a8aedbc8d7bb4183bb1ab9902f07bea7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:37 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:37.876127 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet de806919192f4691bf9ee088ec77963c
I20250113 02:51:37.877199 754 tablet_service.cc:1514] Processing DeleteTablet for tablet de806919192f4691bf9ee088ec77963c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:37 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:37.877830 890 tablet_service.cc:1514] Processing DeleteTablet for tablet de806919192f4691bf9ee088ec77963c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:37 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:37.878849 1040 tablet_service.cc:1514] Processing DeleteTablet for tablet de806919192f4691bf9ee088ec77963c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:37 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:37.879065 899 tablet_service.cc:1514] Processing DeleteTablet for tablet a8aedbc8d7bb4183bb1ab9902f07bea7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:37 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:37.879540 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_eefebb7e9073472daecb078bef21b08d" } new_table_name: "default.Table_8a3f2660341f4712aa264ffc3e3a9a0d" modify_external_catalogs: true
I20250113 02:51:37.881738 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_fa2ea67525994579bd81a459396f4281"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:37.882323 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fa2ea67525994579bd81a459396f4281 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250113 02:51:37.895254 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3a12785c43294c39b4c2aa0dc66b5b4c consensus state has no leader
I20250113 02:51:37.896204 736 tablet_service.cc:1514] Processing DeleteTablet for tablet 64aa8390b1b1475c8743b6491d40674c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:29 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:37.896575 1191 tablet_bootstrap.cc:492] T 82505f1389184d0ca12ec0e2d6084d22 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:37.896548 888 tablet_service.cc:1514] Processing DeleteTablet for tablet b1cd18ee2e2145ba9273f500d468343d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:37 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:37.896785 1191 ts_tablet_manager.cc:1397] T 82505f1389184d0ca12ec0e2d6084d22 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.046s user 0.006s sys 0.000s
I20250113 02:51:37.897543 1191 raft_consensus.cc:357] T 82505f1389184d0ca12ec0e2d6084d22 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:37.897622 1192 consensus_queue.cc:260] T b0113b16110e4787ac1c4676b2c8bd1e P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:37.897787 1191 raft_consensus.cc:383] T 82505f1389184d0ca12ec0e2d6084d22 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:37.897935 1191 raft_consensus.cc:738] T 82505f1389184d0ca12ec0e2d6084d22 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:37.898137 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "153b8f90b5a847b88944f683d306f579" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:37.898402 1192 ts_tablet_manager.cc:1428] T b0113b16110e4787ac1c4676b2c8bd1e P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.088s user 0.002s sys 0.000s
W20250113 02:51:37.898713 709 leader_election.cc:343] T 153b8f90b5a847b88944f683d306f579 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:37.898828 709 leader_election.cc:304] T 153b8f90b5a847b88944f683d306f579 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:37.899010 1191 consensus_queue.cc:260] T 82505f1389184d0ca12ec0e2d6084d22 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:37.899183 2227 raft_consensus.cc:2743] T 153b8f90b5a847b88944f683d306f579 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:37.899490 1191 ts_tablet_manager.cc:1428] T 82505f1389184d0ca12ec0e2d6084d22 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250113 02:51:37.901261 2300 raft_consensus.cc:491] T 9e736211eb3c4536b9fd1d90d2d83838 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:37.901511 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet b1cd18ee2e2145ba9273f500d468343d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:37 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:37.901675 2306 raft_consensus.cc:491] T d8bb60fdd06946739b75b255303d9d24 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:37.901774 2306 raft_consensus.cc:513] T d8bb60fdd06946739b75b255303d9d24 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:37.902420 2306 leader_election.cc:290] T d8bb60fdd06946739b75b255303d9d24 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:37.906322 1191 tablet_bootstrap.cc:492] T b815003a27f64809baaa2d43d6293cef P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
W20250113 02:51:37.909624 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0e2d8726a2ab489cae80fb1685d6951a has no consensus state
I20250113 02:51:37.909649 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8bb60fdd06946739b75b255303d9d24" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:37.909883 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet a8aedbc8d7bb4183bb1ab9902f07bea7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:37 UTC) from {username='slave'} at 127.0.0.1:60230
W20250113 02:51:37.910647 998 leader_election.cc:343] T d8bb60fdd06946739b75b255303d9d24 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:37.898743 1198 rpcz_store.cc:276] 0113 02:51:35.251044 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:35.251099 (+ 55us) service_pool.cc:225] Handling call
0113 02:51:37.828191 (+2577092us) ts_tablet_manager.cc:1145] Deleting tablet 7fb282707196444690352dea3eb1387f
0113 02:51:37.828201 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:37.828291 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2576882}
I20250113 02:51:37.912268 1195 ts_tablet_manager.cc:1918] T 4c615b0b83d74fcfae0c51047dbfbbf4 P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250113 02:51:37.912365 1198 tablet_replica.cc:331] T 1d15c7a01f854f0aaabcd30ec3166e31 P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:37.912387 1195 log.cc:1198] T 4c615b0b83d74fcfae0c51047dbfbbf4 P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/4c615b0b83d74fcfae0c51047dbfbbf4
I20250113 02:51:37.912555 1198 raft_consensus.cc:2238] T 1d15c7a01f854f0aaabcd30ec3166e31 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:37.912696 1198 raft_consensus.cc:2267] T 1d15c7a01f854f0aaabcd30ec3166e31 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:37.912724 1195 ts_tablet_manager.cc:1939] T 4c615b0b83d74fcfae0c51047dbfbbf4 P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:37.912961 736 tablet_service.cc:1514] Processing DeleteTablet for tablet 82505f1389184d0ca12ec0e2d6084d22 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:27 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:37.913064 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1041) took 2009 ms. Trace:
I20250113 02:51:37.913133 1195 rpcz_store.cc:276] 0113 02:51:35.903685 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:35.903732 (+ 47us) service_pool.cc:225] Handling call
0113 02:51:37.861294 (+1957562us) ts_tablet_manager.cc:1145] Deleting tablet 4c615b0b83d74fcfae0c51047dbfbbf4
0113 02:51:37.861302 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:37.901147 (+ 39845us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:37.912169 (+ 11022us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:37.913050 (+ 881us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1957405}
W20250113 02:51:37.913305 995 leader_election.cc:343] T d8bb60fdd06946739b75b255303d9d24 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Not found: Tablet not found: d8bb60fdd06946739b75b255303d9d24
I20250113 02:51:37.913416 1195 tablet_replica.cc:331] T 1014e51f1c184d729552a5b34292f232 P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:37.913424 995 leader_election.cc:304] T d8bb60fdd06946739b75b255303d9d24 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4216a7f853d642b7bc3e5cff458f3aca; no voters: 1186fa869eb14812832fa6562531f196, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:37.907949 1191 tablet_bootstrap.cc:654] T b815003a27f64809baaa2d43d6293cef P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:37.913928 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet 4c615b0b83d74fcfae0c51047dbfbbf4 (table default.Table_bf689f8eb8884e88a72e9b50598fc389 [id=a956f95f6dd94b578f6bd3cb3b97f9da]) successfully deleted
I20250113 02:51:37.914184 2306 raft_consensus.cc:2743] T d8bb60fdd06946739b75b255303d9d24 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:37.913590 1195 raft_consensus.cc:2238] T 1014e51f1c184d729552a5b34292f232 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:37.921751 1195 raft_consensus.cc:2267] T 1014e51f1c184d729552a5b34292f232 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:37.922305 1195 ts_tablet_manager.cc:1905] T 1014e51f1c184d729552a5b34292f232 P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:37.923372 1191 tablet_bootstrap.cc:492] T b815003a27f64809baaa2d43d6293cef P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:37.924376 1191 ts_tablet_manager.cc:1397] T b815003a27f64809baaa2d43d6293cef P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.000s
I20250113 02:51:37.925273 1191 raft_consensus.cc:357] T b815003a27f64809baaa2d43d6293cef P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:37.927814 1191 raft_consensus.cc:383] T b815003a27f64809baaa2d43d6293cef P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:37.928028 1191 raft_consensus.cc:738] T b815003a27f64809baaa2d43d6293cef P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:37.928422 1191 consensus_queue.cc:260] T b815003a27f64809baaa2d43d6293cef P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:37.929441 1191 ts_tablet_manager.cc:1428] T b815003a27f64809baaa2d43d6293cef P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20250113 02:51:37.929755 1191 tablet_bootstrap.cc:492] T 3d85539cf0074a03b167d3279e940079 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:37.931710 1191 tablet_bootstrap.cc:654] T 3d85539cf0074a03b167d3279e940079 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:37.940424 446 catalog_manager.cc:5618] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095) does not have the latest schema for tablet b815003a27f64809baaa2d43d6293cef (table default.Table_c2e8fd7d2cc14fa39c0abfece783c6c9 [id=42e19138c08f422a92ba9621e9609957]). Expected version 1 got 0
I20250113 02:51:37.940639 2300 raft_consensus.cc:513] T 9e736211eb3c4536b9fd1d90d2d83838 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:37.941349 2300 leader_election.cc:290] T 9e736211eb3c4536b9fd1d90d2d83838 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:37.942104 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e736211eb3c4536b9fd1d90d2d83838" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:37.942734 998 leader_election.cc:343] T 9e736211eb3c4536b9fd1d90d2d83838 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
W20250113 02:51:37.943698 995 leader_election.cc:343] T 9e736211eb3c4536b9fd1d90d2d83838 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Not found: Tablet not found: 9e736211eb3c4536b9fd1d90d2d83838
I20250113 02:51:37.943796 995 leader_election.cc:304] T 9e736211eb3c4536b9fd1d90d2d83838 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4216a7f853d642b7bc3e5cff458f3aca; no voters: 1186fa869eb14812832fa6562531f196, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:37.944237 2300 raft_consensus.cc:2743] T 9e736211eb3c4536b9fd1d90d2d83838 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:37.944743 1192 tablet_bootstrap.cc:492] T eceed0f52c9f46bbaa78ca773a7afde5 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:37.913305 1198 ts_tablet_manager.cc:1905] T 1d15c7a01f854f0aaabcd30ec3166e31 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:37.947127 1192 tablet_bootstrap.cc:654] T eceed0f52c9f46bbaa78ca773a7afde5 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:37.944610 448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_b22d94712c2746ac9a3f7f71248fce80" } new_table_name: "default.Table_6a1167a25a9d452e9b7c477aadd97279" modify_external_catalogs: true
I20250113 02:51:37.947256 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_e9f6c43232944cb090b686edb4607e6a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:37.954696 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e9f6c43232944cb090b686edb4607e6a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:37.947405 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_f547905e49744f99810317090e10ba97" } new_table_name: "default.Table_90d1fd99656d4ad89beb35946d8ad9d4" modify_external_catalogs: true
I20250113 02:51:37.947598 454 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_a30fe05769f5429cb4d5f680984dae16" } new_table_name: "default.Table_d6a8490ab19e4b60acdc6f7f05331ec3" modify_external_catalogs: true
I20250113 02:51:37.947753 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_5dd36060f1944c05a472dc98020c553f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:37.958473 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5dd36060f1944c05a472dc98020c553f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:37.947906 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_cd22304dd0d64c369da3a7566f56bc16"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:37.949435 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_6dc54733221e4e7b8d4b5752f841c953"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:37.959810 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cd22304dd0d64c369da3a7566f56bc16 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250113 02:51:37.959956 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6dc54733221e4e7b8d4b5752f841c953 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:37.961122 455 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_0874a35f3d5a44d2ac045a85227b148d" } modify_external_catalogs: true
I20250113 02:51:37.961337 455 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_0874a35f3d5a44d2ac045a85227b148d" } modify_external_catalogs: true
I20250113 02:51:37.965867 1194 tablet_bootstrap.cc:492] T aa5c172a69bf4ca79999f4b58df08de8 P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:37.966087 1194 ts_tablet_manager.cc:1397] T aa5c172a69bf4ca79999f4b58df08de8 P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.160s user 0.005s sys 0.003s
I20250113 02:51:37.966928 1194 raft_consensus.cc:357] T aa5c172a69bf4ca79999f4b58df08de8 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:37.967208 1194 raft_consensus.cc:383] T aa5c172a69bf4ca79999f4b58df08de8 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:37.967723 1194 raft_consensus.cc:738] T aa5c172a69bf4ca79999f4b58df08de8 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
W20250113 02:51:37.967854 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 67fe2f52d98048dab9f37dd820b7f712 consensus state has no leader
I20250113 02:51:37.968065 1194 consensus_queue.cc:260] T aa5c172a69bf4ca79999f4b58df08de8 P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:37.968966 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_16ae427ef10542e28949285e970acb17"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:37.969465 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_16ae427ef10542e28949285e970acb17 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:37.953117 1198 ts_tablet_manager.cc:1918] T 1d15c7a01f854f0aaabcd30ec3166e31 P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:37.970348 1198 log.cc:1198] T 1d15c7a01f854f0aaabcd30ec3166e31 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/1d15c7a01f854f0aaabcd30ec3166e31
I20250113 02:51:37.971532 1198 ts_tablet_manager.cc:1939] T 1d15c7a01f854f0aaabcd30ec3166e31 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
W20250113 02:51:37.973310 448 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a04cec5ed9f34a16a80e112e5c9671e6 consensus state has no leader
W20250113 02:51:37.974059 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 7fb282707196444690352dea3eb1387f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7fb282707196444690352dea3eb1387f already in progress: creating tablet
W20250113 02:51:37.982411 454 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0741955d74f64f0393444fa64cfe1c90 consensus state has no leader
I20250113 02:51:37.983188 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet cfa963a6df094ce0b9c54408f75ca6b2
I20250113 02:51:37.984558 736 tablet_service.cc:1514] Processing DeleteTablet for tablet cfa963a6df094ce0b9c54408f75ca6b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:37 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:37.986042 888 tablet_service.cc:1514] Processing DeleteTablet for tablet cfa963a6df094ce0b9c54408f75ca6b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:37 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:37.992162 1041 tablet_service.cc:1514] Processing DeleteTablet for tablet cfa963a6df094ce0b9c54408f75ca6b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:37 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:37.994419 1192 tablet_bootstrap.cc:492] T eceed0f52c9f46bbaa78ca773a7afde5 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:37.994561 1192 ts_tablet_manager.cc:1397] T eceed0f52c9f46bbaa78ca773a7afde5 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.095s user 0.014s sys 0.004s
I20250113 02:51:37.995276 1192 raft_consensus.cc:357] T eceed0f52c9f46bbaa78ca773a7afde5 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:37.995479 1192 raft_consensus.cc:383] T eceed0f52c9f46bbaa78ca773a7afde5 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:37.995571 1192 raft_consensus.cc:738] T eceed0f52c9f46bbaa78ca773a7afde5 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:37.995795 1192 consensus_queue.cc:260] T eceed0f52c9f46bbaa78ca773a7afde5 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:37.996522 1192 ts_tablet_manager.cc:1428] T eceed0f52c9f46bbaa78ca773a7afde5 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250113 02:51:37.997090 1196 ts_tablet_manager.cc:1918] T f56230a2c75d4763a54671c79358f4d9 P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250113 02:51:37.997974 421 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.104.2:38467, user_credentials={real_user=slave}} blocked reactor thread for 62625.7us
I20250113 02:51:37.997249 1196 log.cc:1198] T f56230a2c75d4763a54671c79358f4d9 P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/f56230a2c75d4763a54671c79358f4d9
I20250113 02:51:38.000478 1196 ts_tablet_manager.cc:1939] T f56230a2c75d4763a54671c79358f4d9 P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:38.001103 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet f56230a2c75d4763a54671c79358f4d9 (table default.Table_fb08fe0ed5cd4aaebce3169c264441cb [id=34b84bee300d4b30b1d426b9f26e6e4a]) successfully deleted
I20250113 02:51:38.001466 1196 tablet_replica.cc:331] T 3b1910496e0849b193e390a5ef1a6b2b P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:38.004611 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 999) took 2753 ms. Trace:
I20250113 02:51:38.004702 1198 rpcz_store.cc:276] 0113 02:51:35.250849 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:35.251338 (+ 489us) service_pool.cc:225] Handling call
0113 02:51:37.912298 (+2660960us) ts_tablet_manager.cc:1145] Deleting tablet 1d15c7a01f854f0aaabcd30ec3166e31
0113 02:51:37.912310 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:37.950144 (+ 37834us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:37.953035 (+ 2891us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:38.004588 (+ 51553us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2660806}
I20250113 02:51:38.005069 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1002) took 2732 ms. Trace:
I20250113 02:51:38.005132 1198 rpcz_store.cc:276] 0113 02:51:35.272762 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:35.272830 (+ 68us) service_pool.cc:225] Handling call
0113 02:51:38.004956 (+2732126us) ts_tablet_manager.cc:1145] Deleting tablet f9269967dea94fc3a337546ad0842e11
0113 02:51:38.004965 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:38.005054 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2731890}
I20250113 02:51:38.005429 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1001) took 2732 ms. Trace:
I20250113 02:51:38.005491 1198 rpcz_store.cc:276] 0113 02:51:35.272628 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:35.273226 (+ 598us) service_pool.cc:225] Handling call
0113 02:51:38.005330 (+2732104us) ts_tablet_manager.cc:1145] Deleting tablet f105f8b32a654e4ca154e8fc0cb96091
0113 02:51:38.005337 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:38.005413 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2731947}
I20250113 02:51:38.005746 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1004) took 2690 ms. Trace:
I20250113 02:51:38.005807 1198 rpcz_store.cc:276] 0113 02:51:35.314911 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:35.314966 (+ 55us) service_pool.cc:225] Handling call
0113 02:51:38.005669 (+2690703us) ts_tablet_manager.cc:1145] Deleting tablet f24cda32b6e947c7983ae2f1f9a64c23
0113 02:51:38.005677 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:38.005735 (+ 58us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2690517}
I20250113 02:51:38.006076 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1011) took 2606 ms. Trace:
I20250113 02:51:38.006135 1198 rpcz_store.cc:276] 0113 02:51:35.399647 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:35.404096 (+ 4449us) service_pool.cc:225] Handling call
0113 02:51:38.005995 (+2601899us) ts_tablet_manager.cc:1145] Deleting tablet fb2458e2044d49029ed2d6851db9de25
0113 02:51:38.006004 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:38.006064 (+ 60us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2601661}
I20250113 02:51:38.006412 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1015) took 2567 ms. Trace:
I20250113 02:51:38.006466 1198 rpcz_store.cc:276] 0113 02:51:35.439338 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:35.439392 (+ 54us) service_pool.cc:225] Handling call
0113 02:51:38.006328 (+2566936us) ts_tablet_manager.cc:1145] Deleting tablet fa9052eea538439a9cb4c527e97897a4
0113 02:51:38.006338 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:38.006397 (+ 59us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2566736}
I20250113 02:51:38.006752 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1016) took 2547 ms. Trace:
I20250113 02:51:38.006815 1198 rpcz_store.cc:276] 0113 02:51:35.459634 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:35.459686 (+ 52us) service_pool.cc:225] Handling call
0113 02:51:38.006639 (+2546953us) ts_tablet_manager.cc:1145] Deleting tablet dd566a49d3ff4d3889b80d8d2ccbee0e
0113 02:51:38.006648 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:38.006737 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2546765}
I20250113 02:51:38.007066 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1019) took 2511 ms. Trace:
I20250113 02:51:38.007141 1198 rpcz_store.cc:276] 0113 02:51:35.495898 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:35.495947 (+ 49us) service_pool.cc:225] Handling call
0113 02:51:38.006983 (+2511036us) ts_tablet_manager.cc:1145] Deleting tablet 2002c69206ec468aad1b44f1b5b0d3b2
0113 02:51:38.006990 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:38.007053 (+ 63us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2510832}
I20250113 02:51:38.007426 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1022) took 2470 ms. Trace:
I20250113 02:51:38.007498 1198 rpcz_store.cc:276] 0113 02:51:35.536568 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:35.536934 (+ 366us) service_pool.cc:225] Handling call
0113 02:51:38.007341 (+2470407us) ts_tablet_manager.cc:1145] Deleting tablet c548a79f394d4ab88a309ca6400a05a2
0113 02:51:38.007349 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:38.007411 (+ 62us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2470247}
I20250113 02:51:38.007774 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1027) took 2387 ms. Trace:
I20250113 02:51:38.009294 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet 1d15c7a01f854f0aaabcd30ec3166e31 (table default.Table_21648ca05a3344848aeac2ad300f7400 [id=fe31ebf2077b43a285155bef4c983d75]) successfully deleted
W20250113 02:51:38.009564 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet f9269967dea94fc3a337546ad0842e11 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f9269967dea94fc3a337546ad0842e11 already in progress: creating tablet
W20250113 02:51:38.009795 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet f105f8b32a654e4ca154e8fc0cb96091 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f105f8b32a654e4ca154e8fc0cb96091 already in progress: creating tablet
W20250113 02:51:38.009986 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet f24cda32b6e947c7983ae2f1f9a64c23 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f24cda32b6e947c7983ae2f1f9a64c23 already in progress: creating tablet
W20250113 02:51:38.010190 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet fb2458e2044d49029ed2d6851db9de25 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fb2458e2044d49029ed2d6851db9de25 already in progress: creating tablet
W20250113 02:51:38.010390 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet fa9052eea538439a9cb4c527e97897a4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fa9052eea538439a9cb4c527e97897a4 already in progress: creating tablet
I20250113 02:51:38.007844 1198 rpcz_store.cc:276] 0113 02:51:35.620663 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:35.625883 (+ 5220us) service_pool.cc:225] Handling call
0113 02:51:38.007691 (+2381808us) ts_tablet_manager.cc:1145] Deleting tablet 78bf9b924b144837b6e17b54086d7908
0113 02:51:38.007699 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:38.007761 (+ 62us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2381630}
W20250113 02:51:38.010618 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet dd566a49d3ff4d3889b80d8d2ccbee0e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dd566a49d3ff4d3889b80d8d2ccbee0e already in progress: creating tablet
I20250113 02:51:38.010643 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1026) took 2390 ms. Trace:
W20250113 02:51:38.010807 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 2002c69206ec468aad1b44f1b5b0d3b2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2002c69206ec468aad1b44f1b5b0d3b2 already in progress: creating tablet
I20250113 02:51:38.010725 1198 rpcz_store.cc:276] 0113 02:51:35.620577 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:35.632863 (+ 12286us) service_pool.cc:225] Handling call
0113 02:51:38.010549 (+2377686us) ts_tablet_manager.cc:1145] Deleting tablet 51a7cb7e7e2c49949c3be328005ce468
0113 02:51:38.010557 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:38.010627 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2377233}
W20250113 02:51:38.011013 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet c548a79f394d4ab88a309ca6400a05a2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c548a79f394d4ab88a309ca6400a05a2 already in progress: creating tablet
I20250113 02:51:38.011304 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1031) took 2183 ms. Trace:
I20250113 02:51:38.011394 1194 ts_tablet_manager.cc:1428] T aa5c172a69bf4ca79999f4b58df08de8 P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.045s user 0.002s sys 0.000s
I20250113 02:51:38.011387 1198 rpcz_store.cc:276] 0113 02:51:35.827456 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:35.827520 (+ 64us) service_pool.cc:225] Handling call
0113 02:51:38.011078 (+2183558us) ts_tablet_manager.cc:1145] Deleting tablet b2a23259f2c448cc8a904a13a291d6ca
0113 02:51:38.011202 (+ 124us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:38.011287 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2183375}
W20250113 02:51:38.012025 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 78bf9b924b144837b6e17b54086d7908 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 78bf9b924b144837b6e17b54086d7908 already in progress: creating tablet
W20250113 02:51:38.012250 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 51a7cb7e7e2c49949c3be328005ce468 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 51a7cb7e7e2c49949c3be328005ce468 already in progress: creating tablet
W20250113 02:51:38.012936 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet b2a23259f2c448cc8a904a13a291d6ca because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b2a23259f2c448cc8a904a13a291d6ca already in progress: creating tablet
I20250113 02:51:38.013962 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1030) took 2186 ms. Trace:
I20250113 02:51:38.014032 1198 rpcz_store.cc:276] 0113 02:51:35.827332 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:35.827559 (+ 227us) service_pool.cc:225] Handling call
0113 02:51:38.011588 (+2184029us) ts_tablet_manager.cc:1145] Deleting tablet 29c17fd90d774c029a35e77888358067
0113 02:51:38.013850 (+ 2262us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:38.013936 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2183871}
W20250113 02:51:38.015594 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 29c17fd90d774c029a35e77888358067 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 29c17fd90d774c029a35e77888358067 already in progress: creating tablet
I20250113 02:51:38.017601 1194 tablet_bootstrap.cc:492] T 0fe5421dc3ea4be684cab084f13d8df0 P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:38.019429 1194 tablet_bootstrap.cc:654] T 0fe5421dc3ea4be684cab084f13d8df0 P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:38.019845 1198 tablet_replica.cc:331] T 0916b41c171a419791797559fbc2807d P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:38.019882 1041 tablet_service.cc:1514] Processing DeleteTablet for tablet b0113b16110e4787ac1c4676b2c8bd1e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:22 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:38.020092 1198 raft_consensus.cc:2238] T 0916b41c171a419791797559fbc2807d P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:38.020164 1196 raft_consensus.cc:2238] T 3b1910496e0849b193e390a5ef1a6b2b P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:38.020248 1198 raft_consensus.cc:2267] T 0916b41c171a419791797559fbc2807d P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:38.020315 1196 raft_consensus.cc:2267] T 3b1910496e0849b193e390a5ef1a6b2b P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:38.020757 1198 ts_tablet_manager.cc:1905] T 0916b41c171a419791797559fbc2807d P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:38.020972 1196 ts_tablet_manager.cc:1905] T 3b1910496e0849b193e390a5ef1a6b2b P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:38.023362 449 catalog_manager.cc:5618] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587) does not have the latest schema for tablet eceed0f52c9f46bbaa78ca773a7afde5 (table default.Table_6a0195b054a043269e133f6ed8dd5979 [id=4b2bd999368c4b628c70cb3ee8c2b1f3]). Expected version 1 got 0
I20250113 02:51:38.023886 1192 tablet_bootstrap.cc:492] T 0644f5600c5b47e29f3fa73060ed96e3 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:38.025838 449 master_service.cc:946] ReplaceTablet: received request to replace tablet 71befed8b5c74b41b199919b82325891 from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:38.027846 455 catalog_manager.cc:5618] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587) does not have the latest schema for tablet eceed0f52c9f46bbaa78ca773a7afde5 (table default.Table_6a0195b054a043269e133f6ed8dd5979 [id=4b2bd999368c4b628c70cb3ee8c2b1f3]). Expected version 1 got 0
W20250113 02:51:38.029853 998 consensus_peers.cc:487] T 0752fde3a30a4947a2a688c56ceadab5 P 4216a7f853d642b7bc3e5cff458f3aca -> Peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Couldn't send request to peer df69f48656b44bb890f76587ee69cc0f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0752fde3a30a4947a2a688c56ceadab5. This is attempt 1: this message will repeat every 5th retry.
I20250113 02:51:38.030359 1198 ts_tablet_manager.cc:1918] T 0916b41c171a419791797559fbc2807d P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:38.030467 1198 log.cc:1198] T 0916b41c171a419791797559fbc2807d P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/0916b41c171a419791797559fbc2807d
I20250113 02:51:38.030746 1198 ts_tablet_manager.cc:1939] T 0916b41c171a419791797559fbc2807d P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:38.031443 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1032) took 2203 ms. Trace:
I20250113 02:51:38.031533 1198 rpcz_store.cc:276] 0113 02:51:35.827600 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:35.827646 (+ 46us) service_pool.cc:225] Handling call
0113 02:51:38.014230 (+2186584us) ts_tablet_manager.cc:1145] Deleting tablet 0916b41c171a419791797559fbc2807d
0113 02:51:38.019785 (+ 5555us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:38.027035 (+ 7250us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:38.030276 (+ 3241us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:38.031425 (+ 1149us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2186478}
I20250113 02:51:38.033246 1192 tablet_bootstrap.cc:654] T 0644f5600c5b47e29f3fa73060ed96e3 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:38.032299 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet 0916b41c171a419791797559fbc2807d (table default.Table_8be7ebd7e1984b979fce6e7aac1f576e [id=8231428f86ba4a6388c55c3c06f6b194]) successfully deleted
I20250113 02:51:38.044970 1191 tablet_bootstrap.cc:492] T 3d85539cf0074a03b167d3279e940079 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:38.045114 1191 ts_tablet_manager.cc:1397] T 3d85539cf0074a03b167d3279e940079 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.115s user 0.008s sys 0.012s
I20250113 02:51:38.046018 1195 ts_tablet_manager.cc:1918] T 1014e51f1c184d729552a5b34292f232 P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:38.045758 1191 raft_consensus.cc:357] T 3d85539cf0074a03b167d3279e940079 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:38.046115 1195 log.cc:1198] T 1014e51f1c184d729552a5b34292f232 P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/1014e51f1c184d729552a5b34292f232
I20250113 02:51:38.046133 1191 raft_consensus.cc:383] T 3d85539cf0074a03b167d3279e940079 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:38.046185 1191 raft_consensus.cc:738] T 3d85539cf0074a03b167d3279e940079 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:38.046406 1195 ts_tablet_manager.cc:1939] T 1014e51f1c184d729552a5b34292f232 P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:38.046442 1191 consensus_queue.cc:260] T 3d85539cf0074a03b167d3279e940079 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:38.046932 1191 ts_tablet_manager.cc:1428] T 3d85539cf0074a03b167d3279e940079 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250113 02:51:38.047113 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1042) took 1815 ms. Trace:
I20250113 02:51:38.047180 1195 rpcz_store.cc:276] 0113 02:51:36.231462 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:36.231517 (+ 55us) service_pool.cc:225] Handling call
0113 02:51:37.913376 (+1681859us) ts_tablet_manager.cc:1145] Deleting tablet 1014e51f1c184d729552a5b34292f232
0113 02:51:37.913385 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:38.004918 (+ 91533us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:38.045909 (+ 40991us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:38.047097 (+ 1188us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1681649}
I20250113 02:51:38.047957 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet 1014e51f1c184d729552a5b34292f232 (table default.Table_2abef959c5c34459a8dfa3034e207995 [id=f7e15a7c1fe848d591ba3232281c45ed]) successfully deleted
I20250113 02:51:38.048552 455 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_64430a2da8d04ce88ae813a23452ecb6" } modify_external_catalogs: true
I20250113 02:51:38.048732 455 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_64430a2da8d04ce88ae813a23452ecb6" } modify_external_catalogs: true
I20250113 02:51:38.050520 1191 tablet_bootstrap.cc:492] T 42dbd37e20fa47318bc6497423dbe565 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:38.055898 736 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d85539cf0074a03b167d3279e940079 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:29 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:38.056372 1191 tablet_bootstrap.cc:654] T 42dbd37e20fa47318bc6497423dbe565 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:38.060925 2325 raft_consensus.cc:491] T 00000000000000000000000000000000 P 45e9d43ecac0401fb2c18fe5eaf089b5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader e13f5bcdcee34225a46c6d2651e5ede9)
I20250113 02:51:38.061025 2286 raft_consensus.cc:491] T c7243d27a1c642269a3d0e67632e458c P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:38.061175 2286 raft_consensus.cc:513] T c7243d27a1c642269a3d0e67632e458c P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:38.061204 2325 raft_consensus.cc:513] T 00000000000000000000000000000000 P 45e9d43ecac0401fb2c18fe5eaf089b5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e13f5bcdcee34225a46c6d2651e5ede9" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44217 } } peers { permanent_uuid: "6bbc3541604a455584bb32dfc2cc4efd" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 35271 } } peers { permanent_uuid: "45e9d43ecac0401fb2c18fe5eaf089b5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42067 } }
I20250113 02:51:38.061815 2286 leader_election.cc:290] T c7243d27a1c642269a3d0e67632e458c P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:38.062765 2254 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45e9d43ecac0401fb2c18fe5eaf089b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e13f5bcdcee34225a46c6d2651e5ede9. Latest consensus state: current_term: 4 leader_uuid: "e13f5bcdcee34225a46c6d2651e5ede9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e13f5bcdcee34225a46c6d2651e5ede9" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44217 } } peers { permanent_uuid: "6bbc3541604a455584bb32dfc2cc4efd" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 35271 } } peers { permanent_uuid: "45e9d43ecac0401fb2c18fe5eaf089b5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42067 } } }
I20250113 02:51:38.063050 2254 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45e9d43ecac0401fb2c18fe5eaf089b5 [sys.catalog]: This master's current role is: FOLLOWER
W20250113 02:51:38.063781 852 leader_election.cc:343] T c7243d27a1c642269a3d0e67632e458c P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Not found: Tablet not found: c7243d27a1c642269a3d0e67632e458c
W20250113 02:51:38.065040 851 leader_election.cc:343] T c7243d27a1c642269a3d0e67632e458c P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Not found: Tablet not found: c7243d27a1c642269a3d0e67632e458c
I20250113 02:51:38.065151 851 leader_election.cc:304] T c7243d27a1c642269a3d0e67632e458c P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df69f48656b44bb890f76587ee69cc0f; no voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca
I20250113 02:51:38.065764 2325 leader_election.cc:290] T 00000000000000000000000000000000 P 45e9d43ecac0401fb2c18fe5eaf089b5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers e13f5bcdcee34225a46c6d2651e5ede9 (127.0.104.62:44217), 6bbc3541604a455584bb32dfc2cc4efd (127.0.104.61:35271)
I20250113 02:51:38.065842 2286 raft_consensus.cc:2743] T c7243d27a1c642269a3d0e67632e458c P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:38.066372 1195 tablet_replica.cc:331] T bbc7832c50114d6f890fec45a3d994e9 P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:38.066644 1195 raft_consensus.cc:2238] T bbc7832c50114d6f890fec45a3d994e9 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:38.066850 1195 raft_consensus.cc:2267] T bbc7832c50114d6f890fec45a3d994e9 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:38.067510 1195 ts_tablet_manager.cc:1905] T bbc7832c50114d6f890fec45a3d994e9 P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:38.069721 1191 tablet_bootstrap.cc:492] T 42dbd37e20fa47318bc6497423dbe565 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:38.069936 1191 ts_tablet_manager.cc:1397] T 42dbd37e20fa47318bc6497423dbe565 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.000s
I20250113 02:51:38.071130 1191 raft_consensus.cc:357] T 42dbd37e20fa47318bc6497423dbe565 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:38.071475 1191 raft_consensus.cc:383] T 42dbd37e20fa47318bc6497423dbe565 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:38.071610 1191 raft_consensus.cc:738] T 42dbd37e20fa47318bc6497423dbe565 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:38.071939 1191 consensus_queue.cc:260] T 42dbd37e20fa47318bc6497423dbe565 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:38.072670 1191 ts_tablet_manager.cc:1428] T 42dbd37e20fa47318bc6497423dbe565 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250113 02:51:38.076154 1191 tablet_bootstrap.cc:492] T b33025a462e74f2aa26cc3779f36615b P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:38.084642 1191 tablet_bootstrap.cc:654] T b33025a462e74f2aa26cc3779f36615b P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:38.084932 1192 tablet_bootstrap.cc:492] T 0644f5600c5b47e29f3fa73060ed96e3 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:38.085088 1192 ts_tablet_manager.cc:1397] T 0644f5600c5b47e29f3fa73060ed96e3 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.061s user 0.003s sys 0.000s
I20250113 02:51:38.085882 1192 raft_consensus.cc:357] T 0644f5600c5b47e29f3fa73060ed96e3 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:38.086084 1192 raft_consensus.cc:383] T 0644f5600c5b47e29f3fa73060ed96e3 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:38.086131 1192 raft_consensus.cc:738] T 0644f5600c5b47e29f3fa73060ed96e3 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:38.086316 1192 consensus_queue.cc:260] T 0644f5600c5b47e29f3fa73060ed96e3 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:38.088564 736 tablet_service.cc:1514] Processing DeleteTablet for tablet 42dbd37e20fa47318bc6497423dbe565 with delete_type TABLET_DATA_DELETED (replaced by tablet 9174df876fe64adc805fba0a36fb4d38) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:38.088757 1192 ts_tablet_manager.cc:1428] T 0644f5600c5b47e29f3fa73060ed96e3 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250113 02:51:38.089056 1192 tablet_bootstrap.cc:492] T dd566a49d3ff4d3889b80d8d2ccbee0e P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:38.090791 1192 tablet_bootstrap.cc:654] T dd566a49d3ff4d3889b80d8d2ccbee0e P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:38.096071 1041 tablet_service.cc:1514] Processing DeleteTablet for tablet 0644f5600c5b47e29f3fa73060ed96e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:22 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:38.099037 1911 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48446 (request call id 0) took 6744 ms. Trace:
I20250113 02:51:38.105094 2228 raft_consensus.cc:491] T dc680b5680b84abbab219296bc7ee707 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:38.105242 2228 raft_consensus.cc:513] T dc680b5680b84abbab219296bc7ee707 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:38.105829 2228 leader_election.cc:290] T dc680b5680b84abbab219296bc7ee707 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:38.106547 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc680b5680b84abbab219296bc7ee707" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:38.107362 711 leader_election.cc:343] T dc680b5680b84abbab219296bc7ee707 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:38.107503 1195 ts_tablet_manager.cc:1918] T bbc7832c50114d6f890fec45a3d994e9 P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:38.107590 1195 log.cc:1198] T bbc7832c50114d6f890fec45a3d994e9 P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/bbc7832c50114d6f890fec45a3d994e9
I20250113 02:51:38.107877 1195 ts_tablet_manager.cc:1939] T bbc7832c50114d6f890fec45a3d994e9 P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:38.108292 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc680b5680b84abbab219296bc7ee707" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:38.108736 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1043) took 1862 ms. Trace:
W20250113 02:51:38.108871 709 leader_election.cc:343] T dc680b5680b84abbab219296bc7ee707 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:38.108944 709 leader_election.cc:304] T dc680b5680b84abbab219296bc7ee707 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:38.108845 1195 rpcz_store.cc:276] 0113 02:51:36.246521 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:36.246603 (+ 82us) service_pool.cc:225] Handling call
0113 02:51:38.047424 (+1800821us) ts_tablet_manager.cc:1145] Deleting tablet bbc7832c50114d6f890fec45a3d994e9
0113 02:51:38.058843 (+ 11419us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:38.103840 (+ 44997us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:38.107412 (+ 3572us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:38.108719 (+ 1307us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1800648}
I20250113 02:51:38.109205 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1045) took 1850 ms. Trace:
I20250113 02:51:38.109285 1195 rpcz_store.cc:276] 0113 02:51:36.258720 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:36.258886 (+ 166us) service_pool.cc:225] Handling call
0113 02:51:38.109106 (+1850220us) ts_tablet_manager.cc:1145] Deleting tablet 9938f5866a6149e3a1456b03389064b2
0113 02:51:38.109115 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:38.109192 (+ 77us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1850043}
I20250113 02:51:38.109596 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1046) took 1850 ms. Trace:
I20250113 02:51:38.109663 1195 rpcz_store.cc:276] 0113 02:51:36.258835 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:36.259103 (+ 268us) service_pool.cc:225] Handling call
0113 02:51:38.109496 (+1850393us) ts_tablet_manager.cc:1145] Deleting tablet 3178924445d64fea89a9721be7d241e9
0113 02:51:38.109505 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:38.109583 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1850243}
I20250113 02:51:38.109736 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet bbc7832c50114d6f890fec45a3d994e9 (table default.Table_3c637eb181574498b44ecb4fc41f0117 [id=10c0207c68d8463e91f09972031ab8d6]) successfully deleted
I20250113 02:51:38.109997 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1044) took 1851 ms. Trace:
W20250113 02:51:38.110029 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 9938f5866a6149e3a1456b03389064b2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9938f5866a6149e3a1456b03389064b2 already in progress: creating tablet
I20250113 02:51:38.110067 1195 rpcz_store.cc:276] 0113 02:51:36.258622 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:36.259277 (+ 655us) service_pool.cc:225] Handling call
0113 02:51:38.109879 (+1850602us) ts_tablet_manager.cc:1145] Deleting tablet 39a9c90677964441acb1e59d16bd1505
0113 02:51:38.109886 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:38.109982 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1850488}
I20250113 02:51:38.110337 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1048) took 1836 ms. Trace:
I20250113 02:51:38.110404 1195 rpcz_store.cc:276] 0113 02:51:36.273575 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:36.273736 (+ 161us) service_pool.cc:225] Handling call
0113 02:51:38.110251 (+1836515us) ts_tablet_manager.cc:1145] Deleting tablet ddf0295c286340bcaea28a1468f3c557
0113 02:51:38.110258 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:38.110322 (+ 64us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1836324}
I20250113 02:51:38.110531 2342 catalog_manager.cc:796] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20250113 02:51:38.110680 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1049) took 1837 ms. Trace:
I20250113 02:51:38.110740 1195 rpcz_store.cc:276] 0113 02:51:36.273673 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:36.273975 (+ 302us) service_pool.cc:225] Handling call
0113 02:51:38.110589 (+1836614us) ts_tablet_manager.cc:1145] Deleting tablet 6770a0cb45fb420d9fbf24d9a5fe4256
0113 02:51:38.110596 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:38.110667 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1836510}
I20250113 02:51:38.111043 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1047) took 1837 ms. Trace:
I20250113 02:51:38.111115 1195 rpcz_store.cc:276] 0113 02:51:36.273470 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:36.274108 (+ 638us) service_pool.cc:225] Handling call
0113 02:51:38.110934 (+1836826us) ts_tablet_manager.cc:1145] Deleting tablet f3771f0fe7ff4d54b433640c76bc07ae
0113 02:51:38.110942 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:38.111027 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1836713}
I20250113 02:51:38.111342 1195 tablet_replica.cc:331] T c7842d6f2c074e3b9d1a3378db9856be P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:38.111531 1195 raft_consensus.cc:2238] T c7842d6f2c074e3b9d1a3378db9856be P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus shutting down.
W20250113 02:51:38.111647 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 3178924445d64fea89a9721be7d241e9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3178924445d64fea89a9721be7d241e9 already in progress: creating tablet
I20250113 02:51:38.111675 1195 raft_consensus.cc:2267] T c7842d6f2c074e3b9d1a3378db9856be P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250113 02:51:38.111865 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 39a9c90677964441acb1e59d16bd1505 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 39a9c90677964441acb1e59d16bd1505 already in progress: creating tablet
I20250113 02:51:38.111944 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_f82ef9176e6043d28f8d6a337ac4bce3" } modify_external_catalogs: true
W20250113 02:51:38.112115 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet ddf0295c286340bcaea28a1468f3c557 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ddf0295c286340bcaea28a1468f3c557 already in progress: creating tablet
I20250113 02:51:38.112197 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_f82ef9176e6043d28f8d6a337ac4bce3" } modify_external_catalogs: true
W20250113 02:51:38.112354 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 6770a0cb45fb420d9fbf24d9a5fe4256 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6770a0cb45fb420d9fbf24d9a5fe4256 already in progress: creating tablet
W20250113 02:51:38.112604 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet f3771f0fe7ff4d54b433640c76bc07ae because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f3771f0fe7ff4d54b433640c76bc07ae already in progress: creating tablet
I20250113 02:51:38.113024 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_79688f9ecca44fccb291229417508b84"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:38.113250 446 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_ba66360c176446ee945a533bd8358296" } modify_external_catalogs: true
I20250113 02:51:38.113425 446 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_ba66360c176446ee945a533bd8358296" } modify_external_catalogs: true
W20250113 02:51:38.113754 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_79688f9ecca44fccb291229417508b84 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:38.114113 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_5e1a59c4550b4cc5b58b5b624b5d71b5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:38.114635 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5e1a59c4550b4cc5b58b5b624b5d71b5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:38.115150 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_b7f2f1b2cac147dd9ee0a9af3186aaf9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:38.115568 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b7f2f1b2cac147dd9ee0a9af3186aaf9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:38.113046 453 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_b030b1ebb66e4b1482e9a6d2efdb7c8a" } modify_external_catalogs: true
I20250113 02:51:38.119402 453 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_b030b1ebb66e4b1482e9a6d2efdb7c8a" } modify_external_catalogs: true
W20250113 02:51:38.114953 2331 log.cc:926] Time spent T 00000000000000000000000000000000 P 45e9d43ecac0401fb2c18fe5eaf089b5: Append to log took a long time: real 0.055s user 0.001s sys 0.000s
I20250113 02:51:38.126673 447 master_service.cc:946] ReplaceTablet: received request to replace tablet acd83024592e431898e9675c17f50d4f from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:38.127331 1196 ts_tablet_manager.cc:1918] T 3b1910496e0849b193e390a5ef1a6b2b P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:38.131457 2228 raft_consensus.cc:2743] T dc680b5680b84abbab219296bc7ee707 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:38.135008 454 master_service.cc:946] ReplaceTablet: received request to replace tablet a6d0f9a0a7104b8fb3d22aac976b7d99 from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:38.135866 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_ba9a270799304a45a458f11c1c84cf0c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:38.137071 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ba9a270799304a45a458f11c1c84cf0c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:38.139636 1195 ts_tablet_manager.cc:1905] T c7842d6f2c074e3b9d1a3378db9856be P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:38.148121 1891 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" instance_seqno: 1736736675135870) as {username='slave'} at 127.0.104.3:35511; Asking this server to re-register.
I20250113 02:51:38.150506 1107 heartbeater.cc:463] Registering TS with master...
I20250113 02:51:38.031904 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1033) took 2143 ms. Trace:
I20250113 02:51:38.150903 1198 rpcz_store.cc:276] 0113 02:51:35.888483 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:35.888705 (+ 222us) service_pool.cc:225] Handling call
0113 02:51:38.031796 (+2143091us) ts_tablet_manager.cc:1145] Deleting tablet d8e81a6fef794f26bdb017e16af8b30c
0113 02:51:38.031805 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:38.031890 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2142901}
I20250113 02:51:38.154781 1890 ts_manager.cc:194] Registered new tserver with Master: 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
W20250113 02:51:38.160092 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet d8e81a6fef794f26bdb017e16af8b30c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d8e81a6fef794f26bdb017e16af8b30c already in progress: creating tablet
I20250113 02:51:38.160835 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1034) took 2269 ms. Trace:
I20250113 02:51:38.144609 1194 tablet_bootstrap.cc:492] T 0fe5421dc3ea4be684cab084f13d8df0 P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:38.161365 1194 ts_tablet_manager.cc:1397] T 0fe5421dc3ea4be684cab084f13d8df0 P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.144s user 0.004s sys 0.000s
I20250113 02:51:38.161332 1198 rpcz_store.cc:276] 0113 02:51:35.891843 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:35.891895 (+ 52us) service_pool.cc:225] Handling call
0113 02:51:38.160701 (+2268806us) ts_tablet_manager.cc:1145] Deleting tablet cb897004cbfb4a0283b42d06509a9ee5
0113 02:51:38.160712 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:38.160815 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2268605}
W20250113 02:51:38.162017 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet cb897004cbfb4a0283b42d06509a9ee5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cb897004cbfb4a0283b42d06509a9ee5 already in progress: creating tablet
I20250113 02:51:38.162482 1198 tablet_replica.cc:331] T 4c615b0b83d74fcfae0c51047dbfbbf4 P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:38.162369 1194 raft_consensus.cc:357] T 0fe5421dc3ea4be684cab084f13d8df0 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:38.162544 1194 raft_consensus.cc:383] T 0fe5421dc3ea4be684cab084f13d8df0 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:38.162652 1194 raft_consensus.cc:738] T 0fe5421dc3ea4be684cab084f13d8df0 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:38.162751 1198 raft_consensus.cc:2238] T 4c615b0b83d74fcfae0c51047dbfbbf4 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: Raft consensus shutting down.
I20250113 02:51:38.162955 1194 consensus_queue.cc:260] T 0fe5421dc3ea4be684cab084f13d8df0 P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:38.163113 1198 raft_consensus.cc:2267] T 4c615b0b83d74fcfae0c51047dbfbbf4 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:38.163725 1198 ts_tablet_manager.cc:1905] T 4c615b0b83d74fcfae0c51047dbfbbf4 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:38.161722 1192 tablet_bootstrap.cc:492] T dd566a49d3ff4d3889b80d8d2ccbee0e P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:38.167526 1192 ts_tablet_manager.cc:1397] T dd566a49d3ff4d3889b80d8d2ccbee0e P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.079s user 0.001s sys 0.008s
I20250113 02:51:38.168336 1192 raft_consensus.cc:357] T dd566a49d3ff4d3889b80d8d2ccbee0e P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:38.168512 1192 raft_consensus.cc:383] T dd566a49d3ff4d3889b80d8d2ccbee0e P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:38.168566 1192 raft_consensus.cc:738] T dd566a49d3ff4d3889b80d8d2ccbee0e P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:38.171418 1198 ts_tablet_manager.cc:1918] T 4c615b0b83d74fcfae0c51047dbfbbf4 P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250113 02:51:38.171564 1198 log.cc:1198] T 4c615b0b83d74fcfae0c51047dbfbbf4 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/4c615b0b83d74fcfae0c51047dbfbbf4
I20250113 02:51:38.172163 1198 ts_tablet_manager.cc:1939] T 4c615b0b83d74fcfae0c51047dbfbbf4 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:38.173538 1191 tablet_bootstrap.cc:492] T b33025a462e74f2aa26cc3779f36615b P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:38.173717 1191 ts_tablet_manager.cc:1397] T b33025a462e74f2aa26cc3779f36615b P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.098s user 0.006s sys 0.005s
I20250113 02:51:38.174258 1891 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" instance_seqno: 1736736674635228) as {username='slave'} at 127.0.104.2:39445; Asking this server to re-register.
I20250113 02:51:38.174966 1191 raft_consensus.cc:357] T b33025a462e74f2aa26cc3779f36615b P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:38.175204 1191 raft_consensus.cc:383] T b33025a462e74f2aa26cc3779f36615b P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:38.175303 1191 raft_consensus.cc:738] T b33025a462e74f2aa26cc3779f36615b P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:38.175578 964 heartbeater.cc:463] Registering TS with master...
I20250113 02:51:38.175560 1191 consensus_queue.cc:260] T b33025a462e74f2aa26cc3779f36615b P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:38.177253 1195 ts_tablet_manager.cc:1918] T c7842d6f2c074e3b9d1a3378db9856be P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:38.178139 1891 ts_manager.cc:194] Registered new tserver with Master: df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:38.180200 1194 ts_tablet_manager.cc:1428] T 0fe5421dc3ea4be684cab084f13d8df0 P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.019s user 0.002s sys 0.000s
I20250113 02:51:38.180794 464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "45e9d43ecac0401fb2c18fe5eaf089b5" candidate_term: 5 candidate_status { last_received { term: 4 index: 994 } } ignore_live_leader: false dest_uuid: "e13f5bcdcee34225a46c6d2651e5ede9" is_pre_election: true
I20250113 02:51:38.181555 1192 consensus_queue.cc:260] T dd566a49d3ff4d3889b80d8d2ccbee0e P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:38.182099 1192 ts_tablet_manager.cc:1428] T dd566a49d3ff4d3889b80d8d2ccbee0e P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.014s user 0.001s sys 0.000s
I20250113 02:51:38.182343 1192 tablet_bootstrap.cc:492] T bc533518faa24dffa3306ff092e8d507 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:38.181872 1344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "45e9d43ecac0401fb2c18fe5eaf089b5" candidate_term: 5 candidate_status { last_received { term: 4 index: 994 } } ignore_live_leader: false dest_uuid: "6bbc3541604a455584bb32dfc2cc4efd" is_pre_election: true
I20250113 02:51:38.183737 1194 tablet_bootstrap.cc:492] T 0539cbbe47e64c039ad39c4b52edcc4b P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:38.184015 1192 tablet_bootstrap.cc:654] T bc533518faa24dffa3306ff092e8d507 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:38.187158 1841 leader_election.cc:304] T 00000000000000000000000000000000 P 45e9d43ecac0401fb2c18fe5eaf089b5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 45e9d43ecac0401fb2c18fe5eaf089b5; no voters: 6bbc3541604a455584bb32dfc2cc4efd, e13f5bcdcee34225a46c6d2651e5ede9
I20250113 02:51:38.187940 2325 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 45e9d43ecac0401fb2c18fe5eaf089b5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250113 02:51:38.185573 1194 tablet_bootstrap.cc:654] T 0539cbbe47e64c039ad39c4b52edcc4b P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:38.190330 1196 log.cc:1198] T 3b1910496e0849b193e390a5ef1a6b2b P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/3b1910496e0849b193e390a5ef1a6b2b
I20250113 02:51:38.191570 1196 ts_tablet_manager.cc:1939] T 3b1910496e0849b193e390a5ef1a6b2b P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:38.192521 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_af07bf28ba6943afa90fcc5287e7d8b1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:38.193115 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_af07bf28ba6943afa90fcc5287e7d8b1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:38.195075 448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_16ae427ef10542e28949285e970acb17" } new_table_name: "default.Table_921e31e7ce184317873cbfa17f6f32ac" modify_external_catalogs: true
I20250113 02:51:38.205832 1195 log.cc:1198] T c7842d6f2c074e3b9d1a3378db9856be P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/c7842d6f2c074e3b9d1a3378db9856be
I20250113 02:51:38.205397 736 tablet_service.cc:1514] Processing DeleteTablet for tablet b33025a462e74f2aa26cc3779f36615b with delete_type TABLET_DATA_DELETED (replaced by tablet f9269967dea94fc3a337546ad0842e11) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:38.200013 1191 ts_tablet_manager.cc:1428] T b33025a462e74f2aa26cc3779f36615b P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.026s user 0.001s sys 0.001s
I20250113 02:51:38.207073 1191 tablet_bootstrap.cc:492] T ba8980ad3a364c888a825affbcbc9b1f P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:38.206100 2228 raft_consensus.cc:491] T 3d85539cf0074a03b167d3279e940079 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:38.207438 2228 raft_consensus.cc:513] T 3d85539cf0074a03b167d3279e940079 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:38.208240 2228 leader_election.cc:290] T 3d85539cf0074a03b167d3279e940079 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:38.208904 1191 tablet_bootstrap.cc:654] T ba8980ad3a364c888a825affbcbc9b1f P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:38.209486 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet ed9e6ba0729b483fa6599dd6dde016e7
I20250113 02:51:38.210424 2286 raft_consensus.cc:491] T aa5c172a69bf4ca79999f4b58df08de8 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:38.210547 2286 raft_consensus.cc:513] T aa5c172a69bf4ca79999f4b58df08de8 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:38.211108 2286 leader_election.cc:290] T aa5c172a69bf4ca79999f4b58df08de8 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:38.211724 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_3525402d375e4d5fa1b2c0f61e27a40e" } new_table_name: "default.Table_d93ff82f449f4a96b688393e8b298b42" modify_external_catalogs: true
I20250113 02:51:38.212136 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d85539cf0074a03b167d3279e940079" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:38.214479 2227 raft_consensus.cc:491] T 64aa8390b1b1475c8743b6491d40674c P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:38.209306 1195 ts_tablet_manager.cc:1939] T c7842d6f2c074e3b9d1a3378db9856be P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:38.214682 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d85539cf0074a03b167d3279e940079" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:38.214643 2227 raft_consensus.cc:513] T 64aa8390b1b1475c8743b6491d40674c P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:38.215287 2227 leader_election.cc:290] T 64aa8390b1b1475c8743b6491d40674c P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:38.215443 736 tablet_service.cc:1514] Processing DeleteTablet for tablet ed9e6ba0729b483fa6599dd6dde016e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:38 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:38.211671 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5c172a69bf4ca79999f4b58df08de8" candidate_uuid: "df69f48656b44bb890f76587ee69cc0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
I20250113 02:51:38.215624 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1053) took 1870 ms. Trace:
I20250113 02:51:38.215710 1195 rpcz_store.cc:276] 0113 02:51:36.344784 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:36.344860 (+ 76us) service_pool.cc:225] Handling call
0113 02:51:38.111304 (+1766444us) ts_tablet_manager.cc:1145] Deleting tablet c7842d6f2c074e3b9d1a3378db9856be
0113 02:51:38.111311 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:38.174277 (+ 62966us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:38.177176 (+ 2899us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:38.215562 (+ 38386us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1766178}
I20250113 02:51:38.212795 2228 raft_consensus.cc:491] T b33025a462e74f2aa26cc3779f36615b P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:38.216178 1041 tablet_service.cc:1514] Processing DeleteTablet for tablet ed9e6ba0729b483fa6599dd6dde016e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:38 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:38.216094 2228 raft_consensus.cc:513] T b33025a462e74f2aa26cc3779f36615b P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:38.216715 2228 leader_election.cc:290] T b33025a462e74f2aa26cc3779f36615b P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:38.216681 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5c172a69bf4ca79999f4b58df08de8" candidate_uuid: "df69f48656b44bb890f76587ee69cc0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:38.217129 2286 raft_consensus.cc:491] T 6d4d8b5cc8ad4c9e935d8afaf5766e66 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:38.217260 2286 raft_consensus.cc:513] T 6d4d8b5cc8ad4c9e935d8afaf5766e66 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:38.217590 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet c7842d6f2c074e3b9d1a3378db9856be (table default.Table_59401ddb940e4e789a5bd54f96548ba5 [id=0ac370ed485e4d64a3e1aa1e8d67ea23]) successfully deleted
I20250113 02:51:38.217808 2286 leader_election.cc:290] T 6d4d8b5cc8ad4c9e935d8afaf5766e66 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587), 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
W20250113 02:51:38.218029 709 leader_election.cc:343] T 3d85539cf0074a03b167d3279e940079 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
W20250113 02:51:38.218586 851 leader_election.cc:343] T 6d4d8b5cc8ad4c9e935d8afaf5766e66 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Not found: Tablet not found: 6d4d8b5cc8ad4c9e935d8afaf5766e66
I20250113 02:51:38.218575 1062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b33025a462e74f2aa26cc3779f36615b" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:38.210191 447 catalog_manager.cc:5618] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587) does not have the latest schema for tablet 6e1284c8aaba44b5810b0d06d36757eb (table default.Table_bfc5031f68f64006aceefe2ee986223a [id=aa4679fc0d7148e3bef21342b3a0ea22]). Expected version 2 got 0
I20250113 02:51:38.219542 852 leader_election.cc:304] T aa5c172a69bf4ca79999f4b58df08de8 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df69f48656b44bb890f76587ee69cc0f; no voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca
I20250113 02:51:38.220288 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b33025a462e74f2aa26cc3779f36615b" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:38.220625 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64aa8390b1b1475c8743b6491d40674c" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:38.220669 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_b3f0fce26a8d48f8a7660c9edba52f3d" } new_table_name: "default.Table_5a88ca4488b9487ba1b91df5fbc57ae1" modify_external_catalogs: true
W20250113 02:51:38.220948 711 leader_election.cc:343] T 3d85539cf0074a03b167d3279e940079 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:38.221081 711 leader_election.cc:304] T 3d85539cf0074a03b167d3279e940079 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:38.222146 1041 tablet_service.cc:1514] Processing DeleteTablet for tablet dd566a49d3ff4d3889b80d8d2ccbee0e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:35 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:38.224941 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64aa8390b1b1475c8743b6491d40674c" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
W20250113 02:51:38.225682 709 leader_election.cc:343] T b33025a462e74f2aa26cc3779f36615b P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
W20250113 02:51:38.225880 709 leader_election.cc:343] T 64aa8390b1b1475c8743b6491d40674c P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:38.227073 2286 raft_consensus.cc:2743] T aa5c172a69bf4ca79999f4b58df08de8 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250113 02:51:38.227643 852 leader_election.cc:343] T 6d4d8b5cc8ad4c9e935d8afaf5766e66 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Not found: Tablet not found: 6d4d8b5cc8ad4c9e935d8afaf5766e66
I20250113 02:51:38.227800 852 leader_election.cc:304] T 6d4d8b5cc8ad4c9e935d8afaf5766e66 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df69f48656b44bb890f76587ee69cc0f; no voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca
I20250113 02:51:38.228144 1196 tablet_replica.cc:331] T 6666922d677a4c678f80d7c6c47fd2a5 P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
W20250113 02:51:38.228255 711 leader_election.cc:343] T b33025a462e74f2aa26cc3779f36615b P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:38.228344 1196 raft_consensus.cc:2238] T 6666922d677a4c678f80d7c6c47fd2a5 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:38.228387 711 leader_election.cc:304] T b33025a462e74f2aa26cc3779f36615b P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:38.228462 1196 raft_consensus.cc:2267] T 6666922d677a4c678f80d7c6c47fd2a5 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:38.228654 2227 raft_consensus.cc:2743] T 3d85539cf0074a03b167d3279e940079 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250113 02:51:38.228837 711 leader_election.cc:343] T 64aa8390b1b1475c8743b6491d40674c P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:38.228857 2227 raft_consensus.cc:2743] T b33025a462e74f2aa26cc3779f36615b P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:38.229004 711 leader_election.cc:304] T 64aa8390b1b1475c8743b6491d40674c P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:38.229295 2227 raft_consensus.cc:2743] T 64aa8390b1b1475c8743b6491d40674c P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:38.229700 1196 ts_tablet_manager.cc:1905] T 6666922d677a4c678f80d7c6c47fd2a5 P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:38.233783 2286 raft_consensus.cc:2743] T 6d4d8b5cc8ad4c9e935d8afaf5766e66 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:38.233477 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_f1d37dd0db604e66abe5918b55426fa2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:38.237686 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f1d37dd0db604e66abe5918b55426fa2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:38.242676 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_5473033c0c4a464785b6f224feec7506" } new_table_name: "default.Table_4df52fe227454c53ac300352248d016d" modify_external_catalogs: true
W20250113 02:51:38.243913 448 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1c3c3b7cf89e4f22b8a5dd7ba4f8f5ec has no consensus state
I20250113 02:51:38.244174 448 master_service.cc:946] ReplaceTablet: received request to replace tablet 0c408921a09a463eb464e08edcf042b3 from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:38.244755 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_b4e63f7e8868448ebd839ba0352acd2c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:38.245265 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b4e63f7e8868448ebd839ba0352acd2c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250113 02:51:38.250602 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 913c427669d24405896826e016138f83 has no consensus state
I20250113 02:51:38.251149 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_7a15797b8bd04c2e911c0803d28cecf6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:38.251662 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7a15797b8bd04c2e911c0803d28cecf6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250113 02:51:38.254248 451 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet aa85362980bb4c4db18d5d55a545046a consensus state has no leader
I20250113 02:51:38.254688 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_2cbb88c393cc492ea908e11fea450a58" } new_table_name: "default.Table_dd419a5795284345929ba3de5c1a0087" modify_external_catalogs: true
W20250113 02:51:38.258183 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e0cdc5e4f4ca4f30b634e4dd2bf862d9 consensus state has no leader
I20250113 02:51:38.258651 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_4d8efcc9d773454bb7158a4d396ec5bc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:38.259315 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4d8efcc9d773454bb7158a4d396ec5bc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:38.216102 1195 tablet_replica.cc:331] T 80d31d5ead804ddaa032969e401a8f7e P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:38.263629 1195 raft_consensus.cc:2238] T 80d31d5ead804ddaa032969e401a8f7e P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:38.263893 1195 raft_consensus.cc:2267] T 80d31d5ead804ddaa032969e401a8f7e P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:38.265395 453 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_892ad2448c3846d08d3e8fffd46084b4" } modify_external_catalogs: true
I20250113 02:51:38.265628 453 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_892ad2448c3846d08d3e8fffd46084b4" } modify_external_catalogs: true
I20250113 02:51:38.266312 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_babc6b8af3474518b339d149d6a2ccb9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:38.266871 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_babc6b8af3474518b339d149d6a2ccb9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:38.268635 1890 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "1186fa869eb14812832fa6562531f196" instance_seqno: 1736736674118640) as {username='slave'} at 127.0.104.1:51103; Asking this server to re-register.
W20250113 02:51:38.270210 451 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ccc24d92091a4643a66959d1f8ddb865 consensus state has no leader
I20250113 02:51:38.270854 820 heartbeater.cc:463] Registering TS with master...
I20250113 02:51:38.271479 888 tablet_service.cc:1514] Processing DeleteTablet for tablet 0fe5421dc3ea4be684cab084f13d8df0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:24 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:38.271564 899 tablet_service.cc:1514] Processing DeleteTablet for tablet ed9e6ba0729b483fa6599dd6dde016e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:38 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:38.271956 1195 ts_tablet_manager.cc:1905] T 80d31d5ead804ddaa032969e401a8f7e P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:38.275789 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet 3b1910496e0849b193e390a5ef1a6b2b (table default.Table_2f3720d8d897466388c3f9dde157d701 [id=8af6fc6e393d4cd3a5eb0671346df592]) successfully deleted
I20250113 02:51:38.276046 451 master_service.cc:946] ReplaceTablet: received request to replace tablet 23476dc111ae430abdce3855d0ef54fe from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:38.278614 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1035) took 2374 ms. Trace:
I20250113 02:51:38.280028 1194 tablet_bootstrap.cc:492] T 0539cbbe47e64c039ad39c4b52edcc4b P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:38.280226 1194 ts_tablet_manager.cc:1397] T 0539cbbe47e64c039ad39c4b52edcc4b P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.097s user 0.003s sys 0.000s
I20250113 02:51:38.279692 1198 rpcz_store.cc:276] 0113 02:51:35.904634 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:35.904686 (+ 52us) service_pool.cc:225] Handling call
0113 02:51:38.162274 (+2257588us) ts_tablet_manager.cc:1145] Deleting tablet 4c615b0b83d74fcfae0c51047dbfbbf4
0113 02:51:38.162443 (+ 169us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:38.167840 (+ 5397us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:38.171331 (+ 3491us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:38.278591 (+107260us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2257409}
I20250113 02:51:38.280936 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet 4c615b0b83d74fcfae0c51047dbfbbf4 (table default.Table_bf689f8eb8884e88a72e9b50598fc389 [id=a956f95f6dd94b578f6bd3cb3b97f9da]) successfully deleted
I20250113 02:51:38.281070 1194 raft_consensus.cc:357] T 0539cbbe47e64c039ad39c4b52edcc4b P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:38.281345 1198 tablet_replica.cc:331] T a92f9b1c9c1a4cbdbf0cc7b2a3f2275b P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:38.281347 1194 raft_consensus.cc:383] T 0539cbbe47e64c039ad39c4b52edcc4b P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:38.281404 1194 raft_consensus.cc:738] T 0539cbbe47e64c039ad39c4b52edcc4b P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:38.281610 1198 raft_consensus.cc:2238] T a92f9b1c9c1a4cbdbf0cc7b2a3f2275b P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:38.281688 1194 consensus_queue.cc:260] T 0539cbbe47e64c039ad39c4b52edcc4b P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:38.281832 1198 raft_consensus.cc:2267] T a92f9b1c9c1a4cbdbf0cc7b2a3f2275b P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:38.282541 1198 ts_tablet_manager.cc:1905] T a92f9b1c9c1a4cbdbf0cc7b2a3f2275b P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:38.285080 2307 consensus_queue.cc:1035] T 7527e6e0b65e4a43b7f48899a34cdf7f P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 02:51:38.290763 2307 raft_consensus.cc:491] T dd566a49d3ff4d3889b80d8d2ccbee0e P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:38.290908 2307 raft_consensus.cc:513] T dd566a49d3ff4d3889b80d8d2ccbee0e P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:38.291450 2307 leader_election.cc:290] T dd566a49d3ff4d3889b80d8d2ccbee0e P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
I20250113 02:51:38.292114 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd566a49d3ff4d3889b80d8d2ccbee0e" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:38.292343 998 consensus_peers.cc:487] T fb18c16fb3ba44ec8a190b88d3bbd1b1 P 4216a7f853d642b7bc3e5cff458f3aca -> Peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Couldn't send request to peer df69f48656b44bb890f76587ee69cc0f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250113 02:51:38.292537 995 leader_election.cc:343] T dd566a49d3ff4d3889b80d8d2ccbee0e P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Not found: Tablet not found: dd566a49d3ff4d3889b80d8d2ccbee0e
W20250113 02:51:38.292790 998 leader_election.cc:343] T dd566a49d3ff4d3889b80d8d2ccbee0e P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:38.292910 998 leader_election.cc:304] T dd566a49d3ff4d3889b80d8d2ccbee0e P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4216a7f853d642b7bc3e5cff458f3aca; no voters: 1186fa869eb14812832fa6562531f196, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:38.293388 2307 raft_consensus.cc:2743] T dd566a49d3ff4d3889b80d8d2ccbee0e P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:38.296958 1194 ts_tablet_manager.cc:1428] T 0539cbbe47e64c039ad39c4b52edcc4b P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.017s user 0.001s sys 0.000s
I20250113 02:51:38.298542 899 tablet_service.cc:1514] Processing DeleteTablet for tablet 0539cbbe47e64c039ad39c4b52edcc4b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:27 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:38.300505 1194 tablet_bootstrap.cc:492] T 99c8046f5d124ca49899df365114c69f P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
W20250113 02:51:38.307277 440 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet ccc24d92091a4643a66959d1f8ddb865 (table default.Table_dd419a5795284345929ba3de5c1a0087 [id=fdaa2293fd5f44539ce4f447af10396a], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet ccc24d92091a4643a66959d1f8ddb865 consensus state has no leader
I20250113 02:51:38.312806 1192 tablet_bootstrap.cc:492] T bc533518faa24dffa3306ff092e8d507 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:38.312983 1192 ts_tablet_manager.cc:1397] T bc533518faa24dffa3306ff092e8d507 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.131s user 0.011s sys 0.009s
I20250113 02:51:38.313779 1192 raft_consensus.cc:357] T bc533518faa24dffa3306ff092e8d507 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:38.314008 1192 raft_consensus.cc:383] T bc533518faa24dffa3306ff092e8d507 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:38.314054 1192 raft_consensus.cc:738] T bc533518faa24dffa3306ff092e8d507 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:38.314287 1192 consensus_queue.cc:260] T bc533518faa24dffa3306ff092e8d507 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:38.314783 1192 ts_tablet_manager.cc:1428] T bc533518faa24dffa3306ff092e8d507 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
W20250113 02:51:38.317951 998 consensus_peers.cc:487] T 7527e6e0b65e4a43b7f48899a34cdf7f P 4216a7f853d642b7bc3e5cff458f3aca -> Peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Couldn't send request to peer df69f48656b44bb890f76587ee69cc0f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250113 02:51:38.321542 1198 ts_tablet_manager.cc:1918] T a92f9b1c9c1a4cbdbf0cc7b2a3f2275b P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:38.321702 1198 log.cc:1198] T a92f9b1c9c1a4cbdbf0cc7b2a3f2275b P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/a92f9b1c9c1a4cbdbf0cc7b2a3f2275b
I20250113 02:51:38.322088 1198 ts_tablet_manager.cc:1939] T a92f9b1c9c1a4cbdbf0cc7b2a3f2275b P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:38.333595 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1037) took 2373 ms. Trace:
I20250113 02:51:38.333838 1198 rpcz_store.cc:276] 0113 02:51:35.960138 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:35.960437 (+ 299us) service_pool.cc:225] Handling call
0113 02:51:38.281290 (+2320853us) ts_tablet_manager.cc:1145] Deleting tablet a92f9b1c9c1a4cbdbf0cc7b2a3f2275b
0113 02:51:38.281302 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:38.318425 (+ 37123us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:38.321446 (+ 3021us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:38.333572 (+ 12126us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2320687}
I20250113 02:51:38.334339 1198 tablet_replica.cc:331] T eebad045e7bb43fa9ca9c9af8b3b7ea5 P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:38.334573 1198 raft_consensus.cc:2238] T eebad045e7bb43fa9ca9c9af8b3b7ea5 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:38.334743 1198 raft_consensus.cc:2267] T eebad045e7bb43fa9ca9c9af8b3b7ea5 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:38.335278 1198 ts_tablet_manager.cc:1905] T eebad045e7bb43fa9ca9c9af8b3b7ea5 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:38.335572 1194 tablet_bootstrap.cc:654] T 99c8046f5d124ca49899df365114c69f P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:38.339876 2300 consensus_queue.cc:1035] T 9438472464304da9a0c7658d4f0736b6 P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 02:51:38.341173 1192 tablet_bootstrap.cc:492] T 655b63ff61ff4f4aa08e0449df39352a P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:38.342857 1192 tablet_bootstrap.cc:654] T 655b63ff61ff4f4aa08e0449df39352a P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:38.345257 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_781f7956cbc4450ab623dc03830f909f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:38.345335 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_5678b31026a4415e9c30223de0518da3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:38.345839 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5678b31026a4415e9c30223de0518da3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:38.347620 2228 raft_consensus.cc:491] T 438604e8859f422f9caccb61c5b5b125 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:38.347797 2228 raft_consensus.cc:513] T 438604e8859f422f9caccb61c5b5b125 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:38.350013 2228 leader_election.cc:290] T 438604e8859f422f9caccb61c5b5b125 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:38.350658 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet a92f9b1c9c1a4cbdbf0cc7b2a3f2275b (table default.Table_6185ed1a90a44576bc65ed60294c2537 [id=5634c185da26460f93cde9c65fbdb24a]) successfully deleted
I20250113 02:51:38.350617 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "438604e8859f422f9caccb61c5b5b125" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:38.350679 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "438604e8859f422f9caccb61c5b5b125" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
W20250113 02:51:38.345846 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_781f7956cbc4450ab623dc03830f909f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250113 02:51:38.352613 711 leader_election.cc:343] T 438604e8859f422f9caccb61c5b5b125 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:38.353708 1191 tablet_bootstrap.cc:492] T ba8980ad3a364c888a825affbcbc9b1f P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:38.353905 1191 ts_tablet_manager.cc:1397] T ba8980ad3a364c888a825affbcbc9b1f P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.147s user 0.004s sys 0.013s
I20250113 02:51:38.354737 1191 raft_consensus.cc:357] T ba8980ad3a364c888a825affbcbc9b1f P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:38.355020 1191 raft_consensus.cc:383] T ba8980ad3a364c888a825affbcbc9b1f P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:38.355150 1191 raft_consensus.cc:738] T ba8980ad3a364c888a825affbcbc9b1f P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:38.355392 1191 consensus_queue.cc:260] T ba8980ad3a364c888a825affbcbc9b1f P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
W20250113 02:51:38.355850 709 leader_election.cc:343] T 438604e8859f422f9caccb61c5b5b125 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:38.355994 709 leader_election.cc:304] T 438604e8859f422f9caccb61c5b5b125 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:38.356088 1191 ts_tablet_manager.cc:1428] T ba8980ad3a364c888a825affbcbc9b1f P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250113 02:51:38.356384 1191 tablet_bootstrap.cc:492] T b38cd34d67834a3f9bf5b5636bb79217 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:38.356999 2228 raft_consensus.cc:2743] T 438604e8859f422f9caccb61c5b5b125 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:38.358904 1041 tablet_service.cc:1514] Processing DeleteTablet for tablet bc533518faa24dffa3306ff092e8d507 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:24 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:38.361382 1191 tablet_bootstrap.cc:654] T b38cd34d67834a3f9bf5b5636bb79217 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:38.366667 2218 raft_consensus.cc:491] T f8a6eebb6b6d4b63903e68750e181707 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:38.366837 2218 raft_consensus.cc:513] T f8a6eebb6b6d4b63903e68750e181707 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:38.367465 2218 leader_election.cc:290] T f8a6eebb6b6d4b63903e68750e181707 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
I20250113 02:51:38.367903 452 catalog_manager.cc:5618] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587) does not have the latest schema for tablet 7527e6e0b65e4a43b7f48899a34cdf7f (table default.Table_c8b93363989c4e109540af6bef426f73 [id=b7846a8ebee84005b1b5d259560a26bd]). Expected version 1 got 0
I20250113 02:51:38.368178 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8a6eebb6b6d4b63903e68750e181707" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
I20250113 02:51:38.368409 772 raft_consensus.cc:2463] T f8a6eebb6b6d4b63903e68750e181707 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 0.
I20250113 02:51:38.368868 995 leader_election.cc:304] T f8a6eebb6b6d4b63903e68750e181707 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters:
I20250113 02:51:38.369258 2218 raft_consensus.cc:2798] T f8a6eebb6b6d4b63903e68750e181707 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 02:51:38.369344 2218 raft_consensus.cc:491] T f8a6eebb6b6d4b63903e68750e181707 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 02:51:38.369418 2218 raft_consensus.cc:3054] T f8a6eebb6b6d4b63903e68750e181707 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:38.372402 2218 raft_consensus.cc:513] T f8a6eebb6b6d4b63903e68750e181707 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:38.373526 1198 ts_tablet_manager.cc:1918] T eebad045e7bb43fa9ca9c9af8b3b7ea5 P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:38.373785 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8a6eebb6b6d4b63903e68750e181707" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196"
I20250113 02:51:38.374028 772 raft_consensus.cc:3054] T f8a6eebb6b6d4b63903e68750e181707 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:38.374191 2228 raft_consensus.cc:491] T 42dbd37e20fa47318bc6497423dbe565 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:38.374346 2228 raft_consensus.cc:513] T 42dbd37e20fa47318bc6497423dbe565 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:38.374936 2228 leader_election.cc:290] T 42dbd37e20fa47318bc6497423dbe565 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:38.375522 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42dbd37e20fa47318bc6497423dbe565" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:38.376504 1198 log.cc:1198] T eebad045e7bb43fa9ca9c9af8b3b7ea5 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/eebad045e7bb43fa9ca9c9af8b3b7ea5
I20250113 02:51:38.376410 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42dbd37e20fa47318bc6497423dbe565" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:38.376657 772 raft_consensus.cc:2463] T f8a6eebb6b6d4b63903e68750e181707 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 1.
I20250113 02:51:38.376801 1198 ts_tablet_manager.cc:1939] T eebad045e7bb43fa9ca9c9af8b3b7ea5 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:38.377013 2300 raft_consensus.cc:491] T d9df314ffaf042ec80180868d7e7bdf7 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250113 02:51:38.377065 711 leader_election.cc:343] T 42dbd37e20fa47318bc6497423dbe565 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:38.377125 2300 raft_consensus.cc:513] T d9df314ffaf042ec80180868d7e7bdf7 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:38.377547 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1038) took 2345 ms. Trace:
I20250113 02:51:38.377671 2300 leader_election.cc:290] T d9df314ffaf042ec80180868d7e7bdf7 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
I20250113 02:51:38.378376 995 leader_election.cc:304] T f8a6eebb6b6d4b63903e68750e181707 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters:
I20250113 02:51:38.378914 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9df314ffaf042ec80180868d7e7bdf7" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
I20250113 02:51:38.379145 772 raft_consensus.cc:2463] T d9df314ffaf042ec80180868d7e7bdf7 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 0.
W20250113 02:51:38.379246 709 leader_election.cc:343] T 42dbd37e20fa47318bc6497423dbe565 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:38.379392 709 leader_election.cc:304] T 42dbd37e20fa47318bc6497423dbe565 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:38.380031 2300 raft_consensus.cc:2798] T f8a6eebb6b6d4b63903e68750e181707 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Leader election won for term 1
I20250113 02:51:38.380151 2300 raft_consensus.cc:695] T f8a6eebb6b6d4b63903e68750e181707 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: Becoming Leader. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Running, Role: LEADER
I20250113 02:51:38.380391 2300 consensus_queue.cc:237] T f8a6eebb6b6d4b63903e68750e181707 P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:38.381914 995 leader_election.cc:304] T d9df314ffaf042ec80180868d7e7bdf7 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters:
I20250113 02:51:38.382391 2306 raft_consensus.cc:2798] T d9df314ffaf042ec80180868d7e7bdf7 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 02:51:38.382458 2306 raft_consensus.cc:491] T d9df314ffaf042ec80180868d7e7bdf7 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 02:51:38.382522 2306 raft_consensus.cc:3054] T d9df314ffaf042ec80180868d7e7bdf7 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:38.383703 1192 tablet_bootstrap.cc:492] T 655b63ff61ff4f4aa08e0449df39352a P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:38.383810 1192 ts_tablet_manager.cc:1397] T 655b63ff61ff4f4aa08e0449df39352a P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.043s user 0.003s sys 0.002s
I20250113 02:51:38.384596 1192 raft_consensus.cc:357] T 655b63ff61ff4f4aa08e0449df39352a P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:38.384750 1192 raft_consensus.cc:383] T 655b63ff61ff4f4aa08e0449df39352a P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:38.384791 1192 raft_consensus.cc:738] T 655b63ff61ff4f4aa08e0449df39352a P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:38.384953 1192 consensus_queue.cc:260] T 655b63ff61ff4f4aa08e0449df39352a P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:38.385075 2306 raft_consensus.cc:513] T d9df314ffaf042ec80180868d7e7bdf7 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:38.385428 1192 ts_tablet_manager.cc:1428] T 655b63ff61ff4f4aa08e0449df39352a P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250113 02:51:38.385560 2306 leader_election.cc:290] T d9df314ffaf042ec80180868d7e7bdf7 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Requested vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
I20250113 02:51:38.385588 1196 ts_tablet_manager.cc:1918] T 6666922d677a4c678f80d7c6c47fd2a5 P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:38.385684 1196 log.cc:1198] T 6666922d677a4c678f80d7c6c47fd2a5 P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/6666922d677a4c678f80d7c6c47fd2a5
I20250113 02:51:38.385694 1192 tablet_bootstrap.cc:492] T b5c1882125b5491ea79a29a6dac6771e P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:38.385938 1196 ts_tablet_manager.cc:1939] T 6666922d677a4c678f80d7c6c47fd2a5 P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:38.387027 1196 tablet_replica.cc:331] T 0a7e7713bc8648378b668d8b769ac927 P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:38.387210 1196 raft_consensus.cc:2238] T 0a7e7713bc8648378b668d8b769ac927 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:38.387260 1192 tablet_bootstrap.cc:654] T b5c1882125b5491ea79a29a6dac6771e P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:38.387193 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9df314ffaf042ec80180868d7e7bdf7" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196"
I20250113 02:51:38.387360 1196 raft_consensus.cc:2267] T 0a7e7713bc8648378b668d8b769ac927 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:38.387394 771 raft_consensus.cc:3054] T d9df314ffaf042ec80180868d7e7bdf7 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:38.387799 2228 raft_consensus.cc:2743] T 42dbd37e20fa47318bc6497423dbe565 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:38.387840 1196 ts_tablet_manager.cc:1905] T 0a7e7713bc8648378b668d8b769ac927 P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:38.377625 1198 rpcz_store.cc:276] 0113 02:51:36.032541 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:36.032701 (+ 160us) service_pool.cc:225] Handling call
0113 02:51:38.334294 (+2301593us) ts_tablet_manager.cc:1145] Deleting tablet eebad045e7bb43fa9ca9c9af8b3b7ea5
0113 02:51:38.334304 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:38.358021 (+ 23717us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:38.373421 (+ 15400us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:38.377531 (+ 4110us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2301424}
I20250113 02:51:38.388861 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet eebad045e7bb43fa9ca9c9af8b3b7ea5 (table default.Table_abd20980e42242719c1529668d68128b [id=043d63161d5e4454af0782a50d64d256]) successfully deleted
I20250113 02:51:38.390024 771 raft_consensus.cc:2463] T d9df314ffaf042ec80180868d7e7bdf7 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 1.
I20250113 02:51:38.390388 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1040) took 2137 ms. Trace:
I20250113 02:51:38.390420 995 leader_election.cc:304] T d9df314ffaf042ec80180868d7e7bdf7 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters:
I20250113 02:51:38.391177 1198 rpcz_store.cc:276] 0113 02:51:36.252906 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:36.252974 (+ 68us) service_pool.cc:225] Handling call
0113 02:51:38.390256 (+2137282us) ts_tablet_manager.cc:1145] Deleting tablet 39a9c90677964441acb1e59d16bd1505
0113 02:51:38.390265 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:38.390359 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2137089}
I20250113 02:51:38.391462 2307 raft_consensus.cc:2798] T d9df314ffaf042ec80180868d7e7bdf7 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Leader election won for term 1
W20250113 02:51:38.391631 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 39a9c90677964441acb1e59d16bd1505 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 39a9c90677964441acb1e59d16bd1505 already in progress: creating tablet
I20250113 02:51:38.393462 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1041) took 2139 ms. Trace:
I20250113 02:51:38.393563 1198 rpcz_store.cc:276] 0113 02:51:36.253793 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:36.253842 (+ 49us) service_pool.cc:225] Handling call
0113 02:51:38.391902 (+2138060us) ts_tablet_manager.cc:1145] Deleting tablet 9938f5866a6149e3a1456b03389064b2
0113 02:51:38.391910 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:38.393439 (+ 1529us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2137726}
W20250113 02:51:38.393998 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 9938f5866a6149e3a1456b03389064b2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9938f5866a6149e3a1456b03389064b2 already in progress: creating tablet
I20250113 02:51:38.394399 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1042) took 2136 ms. Trace:
I20250113 02:51:38.394482 1198 rpcz_store.cc:276] 0113 02:51:36.258120 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:36.258167 (+ 47us) service_pool.cc:225] Handling call
0113 02:51:38.394276 (+2136109us) ts_tablet_manager.cc:1145] Deleting tablet 3178924445d64fea89a9721be7d241e9
0113 02:51:38.394288 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:38.394380 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2135950}
W20250113 02:51:38.394874 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 3178924445d64fea89a9721be7d241e9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3178924445d64fea89a9721be7d241e9 already in progress: creating tablet
I20250113 02:51:38.396770 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1044) took 2124 ms. Trace:
I20250113 02:51:38.396865 1198 rpcz_store.cc:276] 0113 02:51:36.272545 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:36.272717 (+ 172us) service_pool.cc:225] Handling call
0113 02:51:38.395924 (+2123207us) ts_tablet_manager.cc:1145] Deleting tablet ddf0295c286340bcaea28a1468f3c557
0113 02:51:38.395937 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:38.396750 (+ 813us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2123001}
I20250113 02:51:38.397156 452 catalog_manager.cc:5618] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095) does not have the latest schema for tablet 7527e6e0b65e4a43b7f48899a34cdf7f (table default.Table_c8b93363989c4e109540af6bef426f73 [id=b7846a8ebee84005b1b5d259560a26bd]). Expected version 1 got 0
W20250113 02:51:38.397308 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet ddf0295c286340bcaea28a1468f3c557 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ddf0295c286340bcaea28a1468f3c557 already in progress: creating tablet
I20250113 02:51:38.397715 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1045) took 2125 ms. Trace:
I20250113 02:51:38.397795 1198 rpcz_store.cc:276] 0113 02:51:36.272660 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:36.272973 (+ 313us) service_pool.cc:225] Handling call
0113 02:51:38.397598 (+2124625us) ts_tablet_manager.cc:1145] Deleting tablet 6770a0cb45fb420d9fbf24d9a5fe4256
0113 02:51:38.397610 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:38.397697 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2124471}
W20250113 02:51:38.398177 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 6770a0cb45fb420d9fbf24d9a5fe4256 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6770a0cb45fb420d9fbf24d9a5fe4256 already in progress: creating tablet
I20250113 02:51:38.399322 1191 tablet_bootstrap.cc:492] T b38cd34d67834a3f9bf5b5636bb79217 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:38.399525 1191 ts_tablet_manager.cc:1397] T b38cd34d67834a3f9bf5b5636bb79217 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.043s user 0.003s sys 0.003s
I20250113 02:51:38.373626 2218 leader_election.cc:290] T f8a6eebb6b6d4b63903e68750e181707 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Requested vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
I20250113 02:51:38.400652 1191 raft_consensus.cc:357] T b38cd34d67834a3f9bf5b5636bb79217 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:38.400928 1191 raft_consensus.cc:383] T b38cd34d67834a3f9bf5b5636bb79217 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:38.401026 1191 raft_consensus.cc:738] T b38cd34d67834a3f9bf5b5636bb79217 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:38.401811 1191 consensus_queue.cc:260] T b38cd34d67834a3f9bf5b5636bb79217 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:38.402387 1191 ts_tablet_manager.cc:1428] T b38cd34d67834a3f9bf5b5636bb79217 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250113 02:51:38.402731 1191 tablet_bootstrap.cc:492] T c13e9a51c1e74b44b352bade78390ba5 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:38.404830 1191 tablet_bootstrap.cc:654] T c13e9a51c1e74b44b352bade78390ba5 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:38.406035 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1043) took 2133 ms. Trace:
I20250113 02:51:38.406137 1198 rpcz_store.cc:276] 0113 02:51:36.272409 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:36.273154 (+ 745us) service_pool.cc:225] Handling call
0113 02:51:38.405865 (+2132711us) ts_tablet_manager.cc:1145] Deleting tablet f3771f0fe7ff4d54b433640c76bc07ae
0113 02:51:38.405879 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:38.406011 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2132584}
W20250113 02:51:38.406651 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet f3771f0fe7ff4d54b433640c76bc07ae because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f3771f0fe7ff4d54b433640c76bc07ae already in progress: creating tablet
I20250113 02:51:38.406996 1198 tablet_replica.cc:331] T e70658c90555463498d3fc2e2f13ea7f P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:38.407225 1198 raft_consensus.cc:2238] T e70658c90555463498d3fc2e2f13ea7f P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:38.407428 1198 raft_consensus.cc:2267] T e70658c90555463498d3fc2e2f13ea7f P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:38.408167 1198 ts_tablet_manager.cc:1905] T e70658c90555463498d3fc2e2f13ea7f P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:38.408530 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_8c80aefb21ab41c6a1e42a722943ce86"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:38.409268 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8c80aefb21ab41c6a1e42a722943ce86 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:38.419255 1198 ts_tablet_manager.cc:1918] T e70658c90555463498d3fc2e2f13ea7f P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:38.420209 1195 ts_tablet_manager.cc:1918] T 80d31d5ead804ddaa032969e401a8f7e P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:38.420459 1195 log.cc:1198] T 80d31d5ead804ddaa032969e401a8f7e P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/80d31d5ead804ddaa032969e401a8f7e
I20250113 02:51:38.420884 1195 ts_tablet_manager.cc:1939] T 80d31d5ead804ddaa032969e401a8f7e P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:38.422019 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1054) took 2072 ms. Trace:
I20250113 02:51:38.425048 1198 log.cc:1198] T e70658c90555463498d3fc2e2f13ea7f P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/e70658c90555463498d3fc2e2f13ea7f
I20250113 02:51:38.425827 1198 ts_tablet_manager.cc:1939] T e70658c90555463498d3fc2e2f13ea7f P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:38.427469 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1047) took 2132 ms. Trace:
I20250113 02:51:38.429206 1195 rpcz_store.cc:276] 0113 02:51:36.349379 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:36.349912 (+ 533us) service_pool.cc:225] Handling call
0113 02:51:38.216007 (+1866095us) ts_tablet_manager.cc:1145] Deleting tablet 80d31d5ead804ddaa032969e401a8f7e
0113 02:51:38.216018 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:38.343540 (+127522us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:38.420064 (+ 76524us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:38.421981 (+ 1917us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1865812}
I20250113 02:51:38.429313 1198 rpcz_store.cc:276] 0113 02:51:36.295070 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:36.295122 (+ 52us) service_pool.cc:225] Handling call
0113 02:51:38.406941 (+2111819us) ts_tablet_manager.cc:1145] Deleting tablet e70658c90555463498d3fc2e2f13ea7f
0113 02:51:38.406950 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:38.412443 (+ 5493us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:38.419154 (+ 6711us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:38.427445 (+ 8291us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2111636}
I20250113 02:51:38.430198 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet 80d31d5ead804ddaa032969e401a8f7e (table default.Table_2d201bcde7f04c76b66dd5c6a09ddbff [id=e0d56fb485af49c8bcf9152f95f89b2a]) successfully deleted
I20250113 02:51:38.430641 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet e70658c90555463498d3fc2e2f13ea7f (table default.Table_2565c0551f9943a084ace8f7e463e59d [id=fe6f8d302c3e489d8bb832503fd05642]) successfully deleted
I20250113 02:51:38.430816 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1055) took 2068 ms. Trace:
I20250113 02:51:38.431308 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1050) took 2064 ms. Trace:
I20250113 02:51:38.432184 1198 rpcz_store.cc:276] 0113 02:51:36.367346 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:36.373538 (+ 6192us) service_pool.cc:225] Handling call
0113 02:51:38.431129 (+2057591us) ts_tablet_manager.cc:1145] Deleting tablet 9fa6a1c4f6ef4f98b4de114210af08c5
0113 02:51:38.431143 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:38.431285 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2057284}
I20250113 02:51:38.433013 1195 rpcz_store.cc:276] 0113 02:51:36.362778 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:36.363872 (+ 1094us) service_pool.cc:225] Handling call
0113 02:51:38.430652 (+2066780us) ts_tablet_manager.cc:1145] Deleting tablet 9fa6a1c4f6ef4f98b4de114210af08c5
0113 02:51:38.430664 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:38.430779 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2066255}
W20250113 02:51:38.433187 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 9fa6a1c4f6ef4f98b4de114210af08c5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9fa6a1c4f6ef4f98b4de114210af08c5 already in progress: creating tablet
I20250113 02:51:38.433632 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1051) took 2020 ms. Trace:
I20250113 02:51:38.434289 1198 rpcz_store.cc:276] 0113 02:51:36.412943 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:36.413008 (+ 65us) service_pool.cc:225] Handling call
0113 02:51:38.433512 (+2020504us) ts_tablet_manager.cc:1145] Deleting tablet d7193e2799a54950902c4ceac65b663b
0113 02:51:38.433525 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:38.433617 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2020326}
W20250113 02:51:38.435101 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet d7193e2799a54950902c4ceac65b663b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d7193e2799a54950902c4ceac65b663b already in progress: creating tablet
I20250113 02:51:38.435499 1198 tablet_replica.cc:331] T 3025935f9743442eb3e6f2db22eb52ad P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:38.436071 1198 raft_consensus.cc:2238] T 3025935f9743442eb3e6f2db22eb52ad P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus shutting down.
W20250113 02:51:38.436470 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 9fa6a1c4f6ef4f98b4de114210af08c5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9fa6a1c4f6ef4f98b4de114210af08c5 already in progress: creating tablet
I20250113 02:51:38.436694 1198 raft_consensus.cc:2267] T 3025935f9743442eb3e6f2db22eb52ad P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:38.436901 1195 tablet_replica.cc:331] T c518f791b4d74c54ab0c0364d29b4a64 P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:38.437382 1195 raft_consensus.cc:2238] T c518f791b4d74c54ab0c0364d29b4a64 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:38.437506 1198 ts_tablet_manager.cc:1905] T 3025935f9743442eb3e6f2db22eb52ad P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:38.438001 1195 raft_consensus.cc:2267] T c518f791b4d74c54ab0c0364d29b4a64 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:38.438645 1195 ts_tablet_manager.cc:1905] T c518f791b4d74c54ab0c0364d29b4a64 P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:38.442513 2307 raft_consensus.cc:695] T d9df314ffaf042ec80180868d7e7bdf7 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: Becoming Leader. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Running, Role: LEADER
I20250113 02:51:38.442678 1041 tablet_service.cc:1514] Processing DeleteTablet for tablet 655b63ff61ff4f4aa08e0449df39352a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:24 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:38.442852 2307 consensus_queue.cc:237] T d9df314ffaf042ec80180868d7e7bdf7 P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:38.445134 1194 tablet_bootstrap.cc:492] T 99c8046f5d124ca49899df365114c69f P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:38.445322 1194 ts_tablet_manager.cc:1397] T 99c8046f5d124ca49899df365114c69f P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.145s user 0.007s sys 0.000s
I20250113 02:51:38.446269 1194 raft_consensus.cc:357] T 99c8046f5d124ca49899df365114c69f P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:38.446527 1194 raft_consensus.cc:383] T 99c8046f5d124ca49899df365114c69f P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:38.446616 1194 raft_consensus.cc:738] T 99c8046f5d124ca49899df365114c69f P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:38.446936 1194 consensus_queue.cc:260] T 99c8046f5d124ca49899df365114c69f P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:38.452353 1198 ts_tablet_manager.cc:1918] T 3025935f9743442eb3e6f2db22eb52ad P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:38.453794 1194 ts_tablet_manager.cc:1428] T 99c8046f5d124ca49899df365114c69f P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20250113 02:51:38.455756 1194 tablet_bootstrap.cc:492] T 42cf1e708644427f9d4dcd41ae2bcb37 P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:38.457474 1194 tablet_bootstrap.cc:654] T 42cf1e708644427f9d4dcd41ae2bcb37 P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:38.458137 1198 log.cc:1198] T 3025935f9743442eb3e6f2db22eb52ad P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/3025935f9743442eb3e6f2db22eb52ad
I20250113 02:51:38.458750 1198 ts_tablet_manager.cc:1939] T 3025935f9743442eb3e6f2db22eb52ad P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:38.464970 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1054) took 1984 ms. Trace:
I20250113 02:51:38.465626 1198 rpcz_store.cc:276] 0113 02:51:36.480078 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:36.496103 (+ 16025us) service_pool.cc:225] Handling call
0113 02:51:38.435454 (+1939351us) ts_tablet_manager.cc:1145] Deleting tablet 3025935f9743442eb3e6f2db22eb52ad
0113 02:51:38.435463 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:38.441888 (+ 6425us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:38.452244 (+ 10356us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:38.464946 (+ 12702us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1939131}
I20250113 02:51:38.466214 452 catalog_manager.cc:5618] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095) does not have the latest schema for tablet 7527e6e0b65e4a43b7f48899a34cdf7f (table default.Table_c8b93363989c4e109540af6bef426f73 [id=b7846a8ebee84005b1b5d259560a26bd]). Expected version 1 got 0
I20250113 02:51:38.466545 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet 3025935f9743442eb3e6f2db22eb52ad (table default.Table_854e525352dd47c889aaabfc312e33c7 [id=c0b628d230fd4eae83a20530cd998993]) successfully deleted
I20250113 02:51:38.466655 452 catalog_manager.cc:5618] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095) does not have the latest schema for tablet 9438472464304da9a0c7658d4f0736b6 (table default.Table_acdc5be1660742d7a356474b86e8b35f [id=aaf6558a15124f2b8eb9f01d692b845d]). Expected version 2 got 0
I20250113 02:51:38.467113 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1057) took 1910 ms. Trace:
I20250113 02:51:38.467823 1198 rpcz_store.cc:276] 0113 02:51:36.556637 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:36.556695 (+ 58us) service_pool.cc:225] Handling call
0113 02:51:38.466968 (+1910273us) ts_tablet_manager.cc:1145] Deleting tablet c4b7196ef5d54459921977e81a2432db
0113 02:51:38.466991 (+ 23us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:38.467094 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1773727}
W20250113 02:51:38.468739 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet c4b7196ef5d54459921977e81a2432db because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c4b7196ef5d54459921977e81a2432db already in progress: creating tablet
I20250113 02:51:38.469647 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1058) took 1912 ms. Trace:
I20250113 02:51:38.470404 1198 rpcz_store.cc:276] 0113 02:51:36.557241 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:36.558191 (+ 950us) service_pool.cc:225] Handling call
0113 02:51:38.469532 (+1911341us) ts_tablet_manager.cc:1145] Deleting tablet ff5e40047f9e43fbb980b5fedfb36ade
0113 02:51:38.469544 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:38.469632 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1776188}
I20250113 02:51:38.472965 448 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_4e05dc66d38a47b982d5b2389cfc6582" } modify_external_catalogs: true
I20250113 02:51:38.473179 448 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_4e05dc66d38a47b982d5b2389cfc6582" } modify_external_catalogs: true
I20250113 02:51:38.473558 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_ac9027839dca4defb2176e380dac44d9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:38.473695 2306 consensus_queue.cc:1035] T fb18c16fb3ba44ec8a190b88d3bbd1b1 P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 02:51:38.478180 1196 ts_tablet_manager.cc:1918] T 0a7e7713bc8648378b668d8b769ac927 P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:38.478289 1196 log.cc:1198] T 0a7e7713bc8648378b668d8b769ac927 P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/0a7e7713bc8648378b668d8b769ac927
I20250113 02:51:38.478555 1196 ts_tablet_manager.cc:1939] T 0a7e7713bc8648378b668d8b769ac927 P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
W20250113 02:51:38.478562 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet ff5e40047f9e43fbb980b5fedfb36ade because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ff5e40047f9e43fbb980b5fedfb36ade already in progress: creating tablet
I20250113 02:51:38.479249 1196 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37366 (request call id 1041) took 1017 ms. Trace:
I20250113 02:51:38.475983 2218 consensus_queue.cc:1035] T 0b119704bc774ca9b4155c9619d4765b P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250113 02:51:38.482374 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1059) took 1924 ms. Trace:
I20250113 02:51:38.481451 1891 ts_manager.cc:194] Registered new tserver with Master: 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
I20250113 02:51:38.482985 1198 rpcz_store.cc:276] 0113 02:51:36.557797 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:36.558258 (+ 461us) service_pool.cc:225] Handling call
0113 02:51:38.479387 (+1921129us) ts_tablet_manager.cc:1145] Deleting tablet f426887763ed48e3bc87f59d80986672
0113 02:51:38.479405 (+ 18us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:38.482249 (+ 2844us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1785730}
W20250113 02:51:38.484714 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet f426887763ed48e3bc87f59d80986672 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f426887763ed48e3bc87f59d80986672 already in progress: creating tablet
I20250113 02:51:38.485450 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1062) took 1883 ms. Trace:
I20250113 02:51:38.485683 1198 rpcz_store.cc:276] 0113 02:51:36.602216 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:36.602270 (+ 54us) service_pool.cc:225] Handling call
0113 02:51:38.485179 (+1882909us) ts_tablet_manager.cc:1145] Deleting tablet fe8f075d4c9a4f9abe372bfdb81a829c
0113 02:51:38.485196 (+ 17us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:38.485401 (+ 205us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1788718}
W20250113 02:51:38.486367 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet fe8f075d4c9a4f9abe372bfdb81a829c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fe8f075d4c9a4f9abe372bfdb81a829c already in progress: creating tablet
I20250113 02:51:38.486893 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1063) took 1883 ms. Trace:
I20250113 02:51:38.487154 1198 rpcz_store.cc:276] 0113 02:51:36.603607 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:36.603656 (+ 49us) service_pool.cc:225] Handling call
0113 02:51:38.486692 (+1883036us) ts_tablet_manager.cc:1145] Deleting tablet ba8980ad3a364c888a825affbcbc9b1f
0113 02:51:38.486726 (+ 34us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:38.486870 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1790169}
W20250113 02:51:38.487856 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet ba8980ad3a364c888a825affbcbc9b1f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ba8980ad3a364c888a825affbcbc9b1f already in progress: creating tablet
I20250113 02:51:38.476156 2300 raft_consensus.cc:491] T eceed0f52c9f46bbaa78ca773a7afde5 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:38.488368 1198 tablet_replica.cc:331] T 33c665e149dd48e8868acd935de581bf P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:38.497363 1191 tablet_bootstrap.cc:492] T c13e9a51c1e74b44b352bade78390ba5 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:38.488585 2300 raft_consensus.cc:513] T eceed0f52c9f46bbaa78ca773a7afde5 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:38.500028 2300 leader_election.cc:290] T eceed0f52c9f46bbaa78ca773a7afde5 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:38.494637 1192 tablet_bootstrap.cc:492] T b5c1882125b5491ea79a29a6dac6771e P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:38.500453 1192 ts_tablet_manager.cc:1397] T b5c1882125b5491ea79a29a6dac6771e P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.115s user 0.005s sys 0.018s
I20250113 02:51:38.504189 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eceed0f52c9f46bbaa78ca773a7afde5" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
I20250113 02:51:38.504431 771 raft_consensus.cc:2463] T eceed0f52c9f46bbaa78ca773a7afde5 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 0.
I20250113 02:51:38.505031 995 leader_election.cc:304] T eceed0f52c9f46bbaa78ca773a7afde5 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters:
I20250113 02:51:38.505402 2300 raft_consensus.cc:2798] T eceed0f52c9f46bbaa78ca773a7afde5 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 02:51:38.505577 2300 raft_consensus.cc:491] T eceed0f52c9f46bbaa78ca773a7afde5 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 02:51:38.505749 2300 raft_consensus.cc:3054] T eceed0f52c9f46bbaa78ca773a7afde5 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:38.510001 2300 raft_consensus.cc:513] T eceed0f52c9f46bbaa78ca773a7afde5 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:38.511626 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eceed0f52c9f46bbaa78ca773a7afde5" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196"
I20250113 02:51:38.511839 771 raft_consensus.cc:3054] T eceed0f52c9f46bbaa78ca773a7afde5 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:38.512044 2300 leader_election.cc:290] T eceed0f52c9f46bbaa78ca773a7afde5 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Requested vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:38.513437 1191 ts_tablet_manager.cc:1397] T c13e9a51c1e74b44b352bade78390ba5 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.110s user 0.014s sys 0.009s
I20250113 02:51:38.514472 1198 raft_consensus.cc:2238] T 33c665e149dd48e8868acd935de581bf P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:38.516229 1198 raft_consensus.cc:2267] T 33c665e149dd48e8868acd935de581bf P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:38.516605 2228 raft_consensus.cc:491] T 1d2fdc75d00f4e749d322ac6cd5fc410 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:38.516613 771 raft_consensus.cc:2463] T eceed0f52c9f46bbaa78ca773a7afde5 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 1.
I20250113 02:51:38.515602 1192 raft_consensus.cc:357] T b5c1882125b5491ea79a29a6dac6771e P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:38.517382 1192 raft_consensus.cc:383] T b5c1882125b5491ea79a29a6dac6771e P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:38.517752 1192 raft_consensus.cc:738] T b5c1882125b5491ea79a29a6dac6771e P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:38.519048 1198 ts_tablet_manager.cc:1905] T 33c665e149dd48e8868acd935de581bf P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:38.518719 1192 consensus_queue.cc:260] T b5c1882125b5491ea79a29a6dac6771e P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:38.519930 2228 raft_consensus.cc:513] T 1d2fdc75d00f4e749d322ac6cd5fc410 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:38.520651 2228 leader_election.cc:290] T 1d2fdc75d00f4e749d322ac6cd5fc410 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:38.521088 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d2fdc75d00f4e749d322ac6cd5fc410" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:38.522603 995 leader_election.cc:304] T eceed0f52c9f46bbaa78ca773a7afde5 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters:
W20250113 02:51:38.522578 709 leader_election.cc:343] T 1d2fdc75d00f4e749d322ac6cd5fc410 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:38.523249 2218 raft_consensus.cc:2798] T eceed0f52c9f46bbaa78ca773a7afde5 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Leader election won for term 1
I20250113 02:51:38.523437 2218 raft_consensus.cc:695] T eceed0f52c9f46bbaa78ca773a7afde5 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: Becoming Leader. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Running, Role: LEADER
I20250113 02:51:38.522537 2228 raft_consensus.cc:491] T ba8980ad3a364c888a825affbcbc9b1f P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:38.525009 2228 raft_consensus.cc:513] T ba8980ad3a364c888a825affbcbc9b1f P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:38.525638 2218 consensus_queue.cc:237] T eceed0f52c9f46bbaa78ca773a7afde5 P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:38.526907 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba8980ad3a364c888a825affbcbc9b1f" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:38.522562 1191 raft_consensus.cc:357] T c13e9a51c1e74b44b352bade78390ba5 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:38.529215 1192 ts_tablet_manager.cc:1428] T b5c1882125b5491ea79a29a6dac6771e P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.028s user 0.000s sys 0.003s
I20250113 02:51:38.530784 1192 tablet_bootstrap.cc:492] T ee240d838d4a4bbba97f8ed61f69de78 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:38.533818 2228 leader_election.cc:290] T ba8980ad3a364c888a825affbcbc9b1f P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
W20250113 02:51:38.533833 709 leader_election.cc:343] T ba8980ad3a364c888a825affbcbc9b1f P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:38.534530 1191 raft_consensus.cc:383] T c13e9a51c1e74b44b352bade78390ba5 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:38.534790 1191 raft_consensus.cc:738] T c13e9a51c1e74b44b352bade78390ba5 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:38.535238 1191 consensus_queue.cc:260] T c13e9a51c1e74b44b352bade78390ba5 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:38.536430 1191 ts_tablet_manager.cc:1428] T c13e9a51c1e74b44b352bade78390ba5 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.022s user 0.003s sys 0.000s
I20250113 02:51:38.540089 1191 tablet_bootstrap.cc:492] T 6a4c6dc08df744e6856bdf99ab65c575 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:38.551290 1192 tablet_bootstrap.cc:654] T ee240d838d4a4bbba97f8ed61f69de78 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
W20250113 02:51:38.553798 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ac9027839dca4defb2176e380dac44d9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:38.561157 1198 ts_tablet_manager.cc:1918] T 33c665e149dd48e8868acd935de581bf P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:38.562454 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet 6666922d677a4c678f80d7c6c47fd2a5 (table default.Table_03b7491a47674f7187f4952feb9b44e2 [id=6e014400813c4402891b32bcbe3555e7]) successfully deleted
I20250113 02:51:38.564904 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_a306bb67a3c44ddb86b707c1660cf369"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:38.565241 998 consensus_peers.cc:487] T 9438472464304da9a0c7658d4f0736b6 P 4216a7f853d642b7bc3e5cff458f3aca -> Peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Couldn't send request to peer df69f48656b44bb890f76587ee69cc0f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250113 02:51:38.565883 998 consensus_peers.cc:487] T 0b119704bc774ca9b4155c9619d4765b P 4216a7f853d642b7bc3e5cff458f3aca -> Peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Couldn't send request to peer df69f48656b44bb890f76587ee69cc0f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250113 02:51:38.566032 1198 log.cc:1198] T 33c665e149dd48e8868acd935de581bf P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/33c665e149dd48e8868acd935de581bf
I20250113 02:51:38.566352 1198 ts_tablet_manager.cc:1939] T 33c665e149dd48e8868acd935de581bf P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
W20250113 02:51:38.568826 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a306bb67a3c44ddb86b707c1660cf369 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250113 02:51:38.580749 998 consensus_peers.cc:487] T 4eb5f1944cb44fd5a40ca92a6a3d7214 P 4216a7f853d642b7bc3e5cff458f3aca -> Peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Couldn't send request to peer df69f48656b44bb890f76587ee69cc0f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250113 02:51:38.583545 2375 tablet.cc:1721] T 7527e6e0b65e4a43b7f48899a34cdf7f P 4216a7f853d642b7bc3e5cff458f3aca: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250113 02:51:38.584748 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_7e24f64b8bff42a0a429a531f7b588b6" } new_table_name: "default.Table_fda8e0268b174c6f81a15f80e24825b0" modify_external_catalogs: true
I20250113 02:51:38.585850 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1069) took 1869 ms. Trace:
I20250113 02:51:38.484988 2286 raft_consensus.cc:491] T 51d9483ecacb413796d0fd9e7560eac1 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:38.587026 2286 raft_consensus.cc:513] T 51d9483ecacb413796d0fd9e7560eac1 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:38.587674 2286 leader_election.cc:290] T 51d9483ecacb413796d0fd9e7560eac1 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:38.587792 2286 raft_consensus.cc:491] T 377ebf66bbef40988043f6200ae0893c P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:38.587888 2286 raft_consensus.cc:513] T 377ebf66bbef40988043f6200ae0893c P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:38.585949 1198 rpcz_store.cc:276] 0113 02:51:36.716734 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:36.720021 (+ 3287us) service_pool.cc:225] Handling call
0113 02:51:38.488281 (+1768260us) ts_tablet_manager.cc:1145] Deleting tablet 33c665e149dd48e8868acd935de581bf
0113 02:51:38.488295 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:38.558169 (+ 69874us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:38.561061 (+ 2892us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:38.585830 (+ 24769us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1768039}
I20250113 02:51:38.588450 2286 leader_election.cc:290] T 377ebf66bbef40988043f6200ae0893c P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:38.521234 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d2fdc75d00f4e749d322ac6cd5fc410" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:38.526120 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba8980ad3a364c888a825affbcbc9b1f" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:38.563141 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8a6eebb6b6d4b63903e68750e181707" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:38.588692 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1081) took 1469 ms. Trace:
I20250113 02:51:38.589429 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet 33c665e149dd48e8868acd935de581bf (table default.Table_ec91552e358547e99d908cef4bcb7c36 [id=0ebda43313f249d6b35f5beda3d5fc0e]) successfully deleted
I20250113 02:51:38.563573 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9df314ffaf042ec80180868d7e7bdf7" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:38.563872 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9df314ffaf042ec80180868d7e7bdf7" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f"
I20250113 02:51:38.563915 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8a6eebb6b6d4b63903e68750e181707" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f"
I20250113 02:51:38.564327 899 tablet_service.cc:1514] Processing DeleteTablet for tablet 99c8046f5d124ca49899df365114c69f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:23 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:38.564352 914 raft_consensus.cc:3054] T aa5c172a69bf4ca79999f4b58df08de8 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:38.589408 1198 rpcz_store.cc:276] 0113 02:51:37.119359 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:37.120478 (+ 1119us) service_pool.cc:225] Handling call
0113 02:51:38.588571 (+1468093us) ts_tablet_manager.cc:1145] Deleting tablet cb141fb2e9894757b3ae423242ad93d2
0113 02:51:38.588581 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:38.588675 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1467915}
W20250113 02:51:38.590368 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet cb141fb2e9894757b3ae423242ad93d2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cb141fb2e9894757b3ae423242ad93d2 already in progress: creating tablet
W20250113 02:51:38.592185 998 leader_election.cc:343] T f8a6eebb6b6d4b63903e68750e181707 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
W20250113 02:51:38.592684 998 leader_election.cc:343] T d9df314ffaf042ec80180868d7e7bdf7 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:38.592882 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1082) took 1473 ms. Trace:
I20250113 02:51:38.564747 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eceed0f52c9f46bbaa78ca773a7afde5" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f"
I20250113 02:51:38.592958 1198 rpcz_store.cc:276] 0113 02:51:37.119447 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:37.120696 (+ 1249us) service_pool.cc:225] Handling call
0113 02:51:38.592768 (+1472072us) ts_tablet_manager.cc:1145] Deleting tablet 60fac96c00d64ec193b0c706ec64bfaf
0113 02:51:38.592778 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:38.592865 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1471925}
W20250113 02:51:38.593078 998 leader_election.cc:343] T d9df314ffaf042ec80180868d7e7bdf7 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:38.564942 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eceed0f52c9f46bbaa78ca773a7afde5" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:38.593405 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 60fac96c00d64ec193b0c706ec64bfaf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 60fac96c00d64ec193b0c706ec64bfaf already in progress: creating tablet
W20250113 02:51:38.593446 998 leader_election.cc:343] T f8a6eebb6b6d4b63903e68750e181707 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
W20250113 02:51:38.593818 998 leader_election.cc:343] T eceed0f52c9f46bbaa78ca773a7afde5 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
W20250113 02:51:38.594170 998 leader_election.cc:343] T eceed0f52c9f46bbaa78ca773a7afde5 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:38.586879 1194 tablet_bootstrap.cc:492] T 42cf1e708644427f9d4dcd41ae2bcb37 P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:38.594194 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1089) took 1407 ms. Trace:
I20250113 02:51:38.594548 1194 ts_tablet_manager.cc:1397] T 42cf1e708644427f9d4dcd41ae2bcb37 P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.139s user 0.003s sys 0.000s
I20250113 02:51:38.594511 1198 rpcz_store.cc:276] 0113 02:51:37.187199 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:37.187248 (+ 49us) service_pool.cc:225] Handling call
0113 02:51:38.594096 (+1406848us) ts_tablet_manager.cc:1145] Deleting tablet fe3d35d9aaf64188aaa3c1ac38d279d8
0113 02:51:38.594106 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:38.594180 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1385291}
I20250113 02:51:38.595129 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1093) took 1079 ms. Trace:
I20250113 02:51:38.595201 1198 rpcz_store.cc:276] 0113 02:51:37.515169 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:37.515212 (+ 43us) service_pool.cc:225] Handling call
0113 02:51:38.595010 (+1079798us) ts_tablet_manager.cc:1145] Deleting tablet 93c0ce9f457e431193a2ceb0ad4d4af6
0113 02:51:38.595020 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:38.595115 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1064631}
I20250113 02:51:38.595335 1194 raft_consensus.cc:357] T 42cf1e708644427f9d4dcd41ae2bcb37 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:38.595504 1194 raft_consensus.cc:383] T 42cf1e708644427f9d4dcd41ae2bcb37 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:38.595564 1194 raft_consensus.cc:738] T 42cf1e708644427f9d4dcd41ae2bcb37 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:38.595775 1194 consensus_queue.cc:260] T 42cf1e708644427f9d4dcd41ae2bcb37 P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
W20250113 02:51:38.595932 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet fe3d35d9aaf64188aaa3c1ac38d279d8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fe3d35d9aaf64188aaa3c1ac38d279d8 already in progress: creating tablet
W20250113 02:51:38.596206 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 93c0ce9f457e431193a2ceb0ad4d4af6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 93c0ce9f457e431193a2ceb0ad4d4af6 already in progress: creating tablet
I20250113 02:51:38.596995 1194 ts_tablet_manager.cc:1428] T 42cf1e708644427f9d4dcd41ae2bcb37 P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250113 02:51:38.597273 1194 tablet_bootstrap.cc:492] T b8b6ffe690fc459db78b2b3bc12671dd P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:38.598843 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1094) took 1083 ms. Trace:
I20250113 02:51:38.599000 1194 tablet_bootstrap.cc:654] T b8b6ffe690fc459db78b2b3bc12671dd P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:38.598919 1198 rpcz_store.cc:276] 0113 02:51:37.515468 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:37.515533 (+ 65us) service_pool.cc:225] Handling call
0113 02:51:38.596405 (+1080872us) ts_tablet_manager.cc:1145] Deleting tablet e51b750440a24570b734bb51e7a6f923
0113 02:51:38.596415 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:38.598810 (+ 2395us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1065992}
W20250113 02:51:38.599324 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet e51b750440a24570b734bb51e7a6f923 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e51b750440a24570b734bb51e7a6f923 already in progress: creating tablet
I20250113 02:51:38.599931 1198 tablet_replica.cc:331] T f21eb3fa0f414816a29e3d4560cdc0fe P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:38.602180 1198 raft_consensus.cc:2238] T f21eb3fa0f414816a29e3d4560cdc0fe P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:38.602344 1198 raft_consensus.cc:2267] T f21eb3fa0f414816a29e3d4560cdc0fe P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:38.603564 1198 ts_tablet_manager.cc:1905] T f21eb3fa0f414816a29e3d4560cdc0fe P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250113 02:51:38.603791 852 leader_election.cc:343] T 51d9483ecacb413796d0fd9e7560eac1 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Not found: Tablet not found: 51d9483ecacb413796d0fd9e7560eac1
W20250113 02:51:38.604221 852 leader_election.cc:343] T 377ebf66bbef40988043f6200ae0893c P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Not found: Tablet not found: 377ebf66bbef40988043f6200ae0893c
W20250113 02:51:38.605576 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6001369444894b48b51a48fb60ddc52a consensus state has no leader
I20250113 02:51:38.606786 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_cd22304dd0d64c369da3a7566f56bc16" } new_table_name: "default.Table_ca7c8c6ee46346229bf5a7696f4b8165" modify_external_catalogs: true
I20250113 02:51:38.590425 1196 rpcz_store.cc:276] 0113 02:51:37.461376 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:37.463908 (+ 2532us) service_pool.cc:225] Handling call
0113 02:51:38.386982 (+923074us) ts_tablet_manager.cc:1145] Deleting tablet 0a7e7713bc8648378b668d8b769ac927
0113 02:51:38.386991 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:38.440218 (+ 53227us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:38.478072 (+ 37854us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:38.479216 (+ 1144us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":922910}
I20250113 02:51:38.611145 1196 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37366 (request call id 1047) took 1104 ms. Trace:
I20250113 02:51:38.611140 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet 0a7e7713bc8648378b668d8b769ac927 (table default.Table_93a7b6abed0446a4baf21e5e7c8f0c1b [id=9fd4d65e7b9d443a82f06a1f660a7275]) successfully deleted
I20250113 02:51:38.611229 1196 rpcz_store.cc:276] 0113 02:51:37.506266 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:37.536413 (+ 30147us) service_pool.cc:225] Handling call
0113 02:51:38.611016 (+1074603us) ts_tablet_manager.cc:1145] Deleting tablet a1d6cbfb61564e929efab798d4681409
0113 02:51:38.611026 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:38.611126 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1074395}
I20250113 02:51:38.611536 1196 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37366 (request call id 1049) took 1075 ms. Trace:
I20250113 02:51:38.611600 1196 rpcz_store.cc:276] 0113 02:51:37.536312 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:37.536688 (+ 376us) service_pool.cc:225] Handling call
0113 02:51:38.611439 (+1074751us) ts_tablet_manager.cc:1145] Deleting tablet 097cc4bd230046e2ac8551ca23d8f3e8
0113 02:51:38.611449 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:38.611520 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1074588}
W20250113 02:51:38.611670 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet a1d6cbfb61564e929efab798d4681409 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a1d6cbfb61564e929efab798d4681409 already in progress: creating tablet
I20250113 02:51:38.611825 1196 tablet_replica.cc:331] T 6d4d8b5cc8ad4c9e935d8afaf5766e66 P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:38.612027 1196 raft_consensus.cc:2238] T 6d4d8b5cc8ad4c9e935d8afaf5766e66 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
W20250113 02:51:38.612108 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 097cc4bd230046e2ac8551ca23d8f3e8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 097cc4bd230046e2ac8551ca23d8f3e8 already in progress: creating tablet
I20250113 02:51:38.612188 1196 raft_consensus.cc:2267] T 6d4d8b5cc8ad4c9e935d8afaf5766e66 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:38.612671 1196 ts_tablet_manager.cc:1905] T 6d4d8b5cc8ad4c9e935d8afaf5766e66 P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:38.617729 1194 tablet_bootstrap.cc:492] T b8b6ffe690fc459db78b2b3bc12671dd P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:38.617859 1194 ts_tablet_manager.cc:1397] T b8b6ffe690fc459db78b2b3bc12671dd P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.021s user 0.001s sys 0.001s
I20250113 02:51:38.618521 1194 raft_consensus.cc:357] T b8b6ffe690fc459db78b2b3bc12671dd P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:38.618662 1194 raft_consensus.cc:383] T b8b6ffe690fc459db78b2b3bc12671dd P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:38.618718 1194 raft_consensus.cc:738] T b8b6ffe690fc459db78b2b3bc12671dd P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:38.618927 1194 consensus_queue.cc:260] T b8b6ffe690fc459db78b2b3bc12671dd P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:38.619366 1194 ts_tablet_manager.cc:1428] T b8b6ffe690fc459db78b2b3bc12671dd P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250113 02:51:38.619597 1194 tablet_bootstrap.cc:492] T 4b07a985897641c3ac48021c2a4daedb P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:38.632108 1191 tablet_bootstrap.cc:654] T 6a4c6dc08df744e6856bdf99ab65c575 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
W20250113 02:51:38.632098 711 leader_election.cc:343] T 1d2fdc75d00f4e749d322ac6cd5fc410 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
W20250113 02:51:38.632925 851 leader_election.cc:343] T 51d9483ecacb413796d0fd9e7560eac1 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Not found: Tablet not found: 51d9483ecacb413796d0fd9e7560eac1
I20250113 02:51:38.633047 851 leader_election.cc:304] T 51d9483ecacb413796d0fd9e7560eac1 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df69f48656b44bb890f76587ee69cc0f; no voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca
W20250113 02:51:38.633603 851 leader_election.cc:343] T 377ebf66bbef40988043f6200ae0893c P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Not found: Tablet not found: 377ebf66bbef40988043f6200ae0893c
I20250113 02:51:38.633703 851 leader_election.cc:304] T 377ebf66bbef40988043f6200ae0893c P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df69f48656b44bb890f76587ee69cc0f; no voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca
I20250113 02:51:38.634016 2286 raft_consensus.cc:2743] T 51d9483ecacb413796d0fd9e7560eac1 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:38.634140 2286 raft_consensus.cc:2743] T 377ebf66bbef40988043f6200ae0893c P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:38.635170 711 leader_election.cc:304] T 1d2fdc75d00f4e749d322ac6cd5fc410 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
W20250113 02:51:38.635519 711 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:38467, user_credentials={real_user=slave}} blocked reactor thread for 46332.1us
W20250113 02:51:38.635833 711 leader_election.cc:343] T ba8980ad3a364c888a825affbcbc9b1f P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:38.635941 711 leader_election.cc:304] T ba8980ad3a364c888a825affbcbc9b1f P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
W20250113 02:51:38.637485 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ad0b1e35cc824daf8927f79ed36b0616 has no consensus state
I20250113 02:51:38.639004 2378 tablet.cc:1721] T 7527e6e0b65e4a43b7f48899a34cdf7f P 1186fa869eb14812832fa6562531f196: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250113 02:51:38.639081 1196 ts_tablet_manager.cc:1918] T 6d4d8b5cc8ad4c9e935d8afaf5766e66 P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:38.639187 1196 log.cc:1198] T 6d4d8b5cc8ad4c9e935d8afaf5766e66 P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/6d4d8b5cc8ad4c9e935d8afaf5766e66
I20250113 02:51:38.639285 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_86e441512733438dae0893de5acd5c92" } new_table_name: "default.Table_c32890b1c89b40158e226670a74c04c8" modify_external_catalogs: true
I20250113 02:51:38.639475 1196 ts_tablet_manager.cc:1939] T 6d4d8b5cc8ad4c9e935d8afaf5766e66 P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:38.639513 2228 raft_consensus.cc:2743] T 1d2fdc75d00f4e749d322ac6cd5fc410 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:38.639772 1196 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37366 (request call id 1050) took 1103 ms. Trace:
I20250113 02:51:38.639840 1196 rpcz_store.cc:276] 0113 02:51:37.536487 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:37.536886 (+ 399us) service_pool.cc:225] Handling call
0113 02:51:38.611786 (+1074900us) ts_tablet_manager.cc:1145] Deleting tablet 6d4d8b5cc8ad4c9e935d8afaf5766e66
0113 02:51:38.611794 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:38.615412 (+ 3618us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:38.638947 (+ 23535us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:38.639758 (+ 811us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1074780}
I20250113 02:51:38.639948 2227 raft_consensus.cc:2743] T ba8980ad3a364c888a825affbcbc9b1f P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:38.645431 2368 raft_consensus.cc:491] T b0113b16110e4787ac1c4676b2c8bd1e P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:38.645579 2368 raft_consensus.cc:513] T b0113b16110e4787ac1c4676b2c8bd1e P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:38.646206 2368 leader_election.cc:290] T b0113b16110e4787ac1c4676b2c8bd1e P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
W20250113 02:51:38.647699 995 leader_election.cc:343] T b0113b16110e4787ac1c4676b2c8bd1e P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Not found: Tablet not found: b0113b16110e4787ac1c4676b2c8bd1e
I20250113 02:51:38.650144 1195 ts_tablet_manager.cc:1918] T c518f791b4d74c54ab0c0364d29b4a64 P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:38.650353 1195 log.cc:1198] T c518f791b4d74c54ab0c0364d29b4a64 P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/c518f791b4d74c54ab0c0364d29b4a64
I20250113 02:51:38.650732 1195 ts_tablet_manager.cc:1939] T c518f791b4d74c54ab0c0364d29b4a64 P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
W20250113 02:51:38.651712 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 50c4d4783e6d4a55bd3bd355613e1ea0 consensus state has no leader
I20250113 02:51:38.652729 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_7b2714b0471c4a3eb2a9f0590a27562d" } new_table_name: "default.Table_e27f8b6f8232415388a5e53ab15494b7" modify_external_catalogs: true
I20250113 02:51:38.646596 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0113b16110e4787ac1c4676b2c8bd1e" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:38.655915 1196 tablet_replica.cc:331] T 51d9483ecacb413796d0fd9e7560eac1 P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
W20250113 02:51:38.656320 998 leader_election.cc:343] T b0113b16110e4787ac1c4676b2c8bd1e P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:38.656437 998 leader_election.cc:304] T b0113b16110e4787ac1c4676b2c8bd1e P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4216a7f853d642b7bc3e5cff458f3aca; no voters: 1186fa869eb14812832fa6562531f196, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:38.656774 1191 tablet_bootstrap.cc:492] T 6a4c6dc08df744e6856bdf99ab65c575 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:38.656796 2368 raft_consensus.cc:2743] T b0113b16110e4787ac1c4676b2c8bd1e P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:38.656893 1191 ts_tablet_manager.cc:1397] T 6a4c6dc08df744e6856bdf99ab65c575 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.118s user 0.007s sys 0.000s
I20250113 02:51:38.657045 1196 raft_consensus.cc:2238] T 51d9483ecacb413796d0fd9e7560eac1 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:38.657035 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet 6d4d8b5cc8ad4c9e935d8afaf5766e66 (table default.Table_54119752ebc147b8b34429f1c38b71d9 [id=2ab34074f7624b419c1cd36e9a09b4c6]) successfully deleted
I20250113 02:51:38.657598 1191 raft_consensus.cc:357] T 6a4c6dc08df744e6856bdf99ab65c575 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:38.657853 1191 raft_consensus.cc:383] T 6a4c6dc08df744e6856bdf99ab65c575 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:38.657953 1191 raft_consensus.cc:738] T 6a4c6dc08df744e6856bdf99ab65c575 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:38.658238 1191 consensus_queue.cc:260] T 6a4c6dc08df744e6856bdf99ab65c575 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:38.658931 1191 ts_tablet_manager.cc:1428] T 6a4c6dc08df744e6856bdf99ab65c575 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250113 02:51:38.659294 1191 tablet_bootstrap.cc:492] T 3f62f344c5a648328ca2de8ca1069c19 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:38.647658 2377 raft_consensus.cc:491] T 0fe5421dc3ea4be684cab084f13d8df0 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:38.659734 2377 raft_consensus.cc:513] T 0fe5421dc3ea4be684cab084f13d8df0 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:38.660620 2377 leader_election.cc:290] T 0fe5421dc3ea4be684cab084f13d8df0 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
W20250113 02:51:38.662326 851 leader_election.cc:343] T 0fe5421dc3ea4be684cab084f13d8df0 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Not found: Tablet not found: 0fe5421dc3ea4be684cab084f13d8df0
I20250113 02:51:38.662626 1191 tablet_bootstrap.cc:654] T 3f62f344c5a648328ca2de8ca1069c19 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:38.663002 1196 raft_consensus.cc:2267] T 51d9483ecacb413796d0fd9e7560eac1 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
W20250113 02:51:38.663321 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e01e7f8db83d461bbfbb2e901419ce0b consensus state has no leader
I20250113 02:51:38.663606 1196 ts_tablet_manager.cc:1905] T 51d9483ecacb413796d0fd9e7560eac1 P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250113 02:51:38.663805 852 leader_election.cc:343] T 0fe5421dc3ea4be684cab084f13d8df0 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Not found: Tablet not found: 0fe5421dc3ea4be684cab084f13d8df0
I20250113 02:51:38.663898 852 leader_election.cc:304] T 0fe5421dc3ea4be684cab084f13d8df0 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df69f48656b44bb890f76587ee69cc0f; no voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca
I20250113 02:51:38.664410 2377 raft_consensus.cc:2743] T 0fe5421dc3ea4be684cab084f13d8df0 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:38.664366 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_df54b4888fd444f0ad01bc7f7b13c336"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:38.665042 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_df54b4888fd444f0ad01bc7f7b13c336 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:38.674140 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_213c7f5ea7184ab5ab5d3699b4a684bd" } new_table_name: "default.Table_db017462a20b40a99f30d62e7f590a2d" modify_external_catalogs: true
W20250113 02:51:38.680470 711 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:38467, user_credentials={real_user=slave}} blocked reactor thread for 44680.3us
I20250113 02:51:38.683699 1191 tablet_bootstrap.cc:492] T 3f62f344c5a648328ca2de8ca1069c19 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:38.683842 1191 ts_tablet_manager.cc:1397] T 3f62f344c5a648328ca2de8ca1069c19 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.025s user 0.008s sys 0.000s
I20250113 02:51:38.684545 1191 raft_consensus.cc:357] T 3f62f344c5a648328ca2de8ca1069c19 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:38.684705 1191 raft_consensus.cc:383] T 3f62f344c5a648328ca2de8ca1069c19 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:38.684744 1191 raft_consensus.cc:738] T 3f62f344c5a648328ca2de8ca1069c19 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:38.692277 1191 consensus_queue.cc:260] T 3f62f344c5a648328ca2de8ca1069c19 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:38.691532 1192 tablet_bootstrap.cc:492] T ee240d838d4a4bbba97f8ed61f69de78 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:38.692840 1191 ts_tablet_manager.cc:1428] T 3f62f344c5a648328ca2de8ca1069c19 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20250113 02:51:38.692931 1192 ts_tablet_manager.cc:1397] T ee240d838d4a4bbba97f8ed61f69de78 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.163s user 0.005s sys 0.010s
I20250113 02:51:38.693101 1191 tablet_bootstrap.cc:492] T acd83024592e431898e9675c17f50d4f P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:38.693646 1192 raft_consensus.cc:357] T ee240d838d4a4bbba97f8ed61f69de78 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:38.693847 1192 raft_consensus.cc:383] T ee240d838d4a4bbba97f8ed61f69de78 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:38.693897 1192 raft_consensus.cc:738] T ee240d838d4a4bbba97f8ed61f69de78 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:38.694134 1192 consensus_queue.cc:260] T ee240d838d4a4bbba97f8ed61f69de78 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:38.694671 1192 ts_tablet_manager.cc:1428] T ee240d838d4a4bbba97f8ed61f69de78 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250113 02:51:38.690348 1198 ts_tablet_manager.cc:1918] T f21eb3fa0f414816a29e3d4560cdc0fe P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:38.694829 1198 log.cc:1198] T f21eb3fa0f414816a29e3d4560cdc0fe P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/f21eb3fa0f414816a29e3d4560cdc0fe
I20250113 02:51:38.695122 1198 ts_tablet_manager.cc:1939] T f21eb3fa0f414816a29e3d4560cdc0fe P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:38.698729 1192 tablet_bootstrap.cc:492] T 30ca8a716bfc4ab789b216e17567785e P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:38.703028 1192 tablet_bootstrap.cc:654] T 30ca8a716bfc4ab789b216e17567785e P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:38.704505 1191 tablet_bootstrap.cc:654] T acd83024592e431898e9675c17f50d4f P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:38.705818 1196 ts_tablet_manager.cc:1918] T 51d9483ecacb413796d0fd9e7560eac1 P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:38.705933 1196 log.cc:1198] T 51d9483ecacb413796d0fd9e7560eac1 P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/51d9483ecacb413796d0fd9e7560eac1
I20250113 02:51:38.706224 1196 ts_tablet_manager.cc:1939] T 51d9483ecacb413796d0fd9e7560eac1 P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:38.707022 1196 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37366 (request call id 1052) took 1156 ms. Trace:
I20250113 02:51:38.707108 1196 rpcz_store.cc:276] 0113 02:51:37.550095 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:37.550156 (+ 61us) service_pool.cc:225] Handling call
0113 02:51:38.655859 (+1105703us) ts_tablet_manager.cc:1145] Deleting tablet 51d9483ecacb413796d0fd9e7560eac1
0113 02:51:38.655872 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:38.689066 (+ 33194us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:38.705704 (+ 16638us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:38.706983 (+ 1279us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1105501}
I20250113 02:51:38.707471 1196 tablet_replica.cc:331] T c7243d27a1c642269a3d0e67632e458c P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:38.707628 1196 raft_consensus.cc:2238] T c7243d27a1c642269a3d0e67632e458c P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:38.707759 1196 raft_consensus.cc:2267] T c7243d27a1c642269a3d0e67632e458c P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:38.708637 899 tablet_service.cc:1514] Processing DeleteTablet for tablet 42cf1e708644427f9d4dcd41ae2bcb37 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:22 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:38.712750 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1096) took 1196 ms. Trace:
I20250113 02:51:38.712869 1198 rpcz_store.cc:276] 0113 02:51:37.515839 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:37.515892 (+ 53us) service_pool.cc:225] Handling call
0113 02:51:38.599885 (+1083993us) ts_tablet_manager.cc:1145] Deleting tablet f21eb3fa0f414816a29e3d4560cdc0fe
0113 02:51:38.599895 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:38.606680 (+ 6785us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:38.690256 (+ 83576us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:38.712725 (+ 22469us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1067514}
I20250113 02:51:38.714126 1192 tablet_bootstrap.cc:492] T 30ca8a716bfc4ab789b216e17567785e P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:38.714257 1192 ts_tablet_manager.cc:1397] T 30ca8a716bfc4ab789b216e17567785e P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.016s user 0.001s sys 0.005s
I20250113 02:51:38.714898 1192 raft_consensus.cc:357] T 30ca8a716bfc4ab789b216e17567785e P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:38.715085 1192 raft_consensus.cc:383] T 30ca8a716bfc4ab789b216e17567785e P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:38.715134 1192 raft_consensus.cc:738] T 30ca8a716bfc4ab789b216e17567785e P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:38.715348 1192 consensus_queue.cc:260] T 30ca8a716bfc4ab789b216e17567785e P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:38.715759 1192 ts_tablet_manager.cc:1428] T 30ca8a716bfc4ab789b216e17567785e P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20250113 02:51:38.716030 1192 tablet_bootstrap.cc:492] T 1014e51f1c184d729552a5b34292f232 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:38.717475 1194 tablet_bootstrap.cc:654] T 4b07a985897641c3ac48021c2a4daedb P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:38.718871 1191 tablet_bootstrap.cc:492] T acd83024592e431898e9675c17f50d4f P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:38.719000 1191 ts_tablet_manager.cc:1397] T acd83024592e431898e9675c17f50d4f P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.026s user 0.005s sys 0.000s
I20250113 02:51:38.719652 1191 raft_consensus.cc:357] T acd83024592e431898e9675c17f50d4f P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:38.719832 1191 raft_consensus.cc:383] T acd83024592e431898e9675c17f50d4f P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:38.719882 1191 raft_consensus.cc:738] T acd83024592e431898e9675c17f50d4f P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:38.720090 1191 consensus_queue.cc:260] T acd83024592e431898e9675c17f50d4f P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:38.720571 1191 ts_tablet_manager.cc:1428] T acd83024592e431898e9675c17f50d4f P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250113 02:51:38.722756 2368 consensus_queue.cc:1035] T 4eb5f1944cb44fd5a40ca92a6a3d7214 P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 02:51:38.725848 1191 tablet_bootstrap.cc:492] T 1968c7e9357c4c6e901d334ace0467fb P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:38.727427 1191 tablet_bootstrap.cc:654] T 1968c7e9357c4c6e901d334ace0467fb P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:38.728049 736 tablet_service.cc:1467] Processing CreateTablet for tablet 31dadcb6601549c196762a041184bd2d (DEFAULT_TABLE table=default.Table_e031078584204383b805a4e04f36395a [id=bb100490f3d94d7ba7d2494914bf89ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:38.728329 1192 tablet_bootstrap.cc:654] T 1014e51f1c184d729552a5b34292f232 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:38.728422 736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31dadcb6601549c196762a041184bd2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:38.733561 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1099) took 1217 ms. Trace:
I20250113 02:51:38.733686 1198 rpcz_store.cc:276] 0113 02:51:37.516197 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:37.516261 (+ 64us) service_pool.cc:225] Handling call
0113 02:51:38.733409 (+1217148us) ts_tablet_manager.cc:1145] Deleting tablet 5d2d14548b734b1fbe57d68c0ea3e2f8
0113 02:51:38.733419 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:38.733535 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1200943}
I20250113 02:51:38.736368 1198 tablet_replica.cc:331] T 0752fde3a30a4947a2a688c56ceadab5 P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:38.736586 1198 raft_consensus.cc:2238] T 0752fde3a30a4947a2a688c56ceadab5 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: Raft consensus shutting down.
I20250113 02:51:38.709064 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet 51d9483ecacb413796d0fd9e7560eac1 (table default.Table_9417f16fb6384fde8e191ca02808aa78 [id=b49a91d3b86d44e1824a987db3de13b6]) successfully deleted
I20250113 02:51:38.713378 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet f21eb3fa0f414816a29e3d4560cdc0fe (table default.Table_6a3d6ef326e74c4ab0a5ed1e1bec2ecc [id=02e7cb8431884f67989a39c2ffab28ad]) successfully deleted
W20250113 02:51:38.737463 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 5d2d14548b734b1fbe57d68c0ea3e2f8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5d2d14548b734b1fbe57d68c0ea3e2f8 already in progress: creating tablet
I20250113 02:51:38.737811 899 tablet_service.cc:1467] Processing CreateTablet for tablet 31dadcb6601549c196762a041184bd2d (DEFAULT_TABLE table=default.Table_e031078584204383b805a4e04f36395a [id=bb100490f3d94d7ba7d2494914bf89ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:38.738214 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31dadcb6601549c196762a041184bd2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:38.738315 1041 tablet_service.cc:1467] Processing CreateTablet for tablet 31dadcb6601549c196762a041184bd2d (DEFAULT_TABLE table=default.Table_e031078584204383b805a4e04f36395a [id=bb100490f3d94d7ba7d2494914bf89ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:38.738726 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31dadcb6601549c196762a041184bd2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:38.738895 888 tablet_service.cc:1467] Processing CreateTablet for tablet 495064a93dfb41ff999fc73b579baca0 (DEFAULT_TABLE table=default.Table_8c95a106e0684de587166acbc0de43e1 [id=6987c696415646779e6c9816503bb318]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:38.739207 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 495064a93dfb41ff999fc73b579baca0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:38.741130 1024 tablet_service.cc:1467] Processing CreateTablet for tablet 495064a93dfb41ff999fc73b579baca0 (DEFAULT_TABLE table=default.Table_8c95a106e0684de587166acbc0de43e1 [id=6987c696415646779e6c9816503bb318]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:38.741474 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 495064a93dfb41ff999fc73b579baca0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:38.713521 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_370f637005ce4fef8bac91353769e3ec"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:38.725788 451 catalog_manager.cc:5526] T eceed0f52c9f46bbaa78ca773a7afde5 P 4216a7f853d642b7bc3e5cff458f3aca reported cstate change: term changed from 0 to 1, leader changed from <none> to 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3). New cstate: current_term: 1 leader_uuid: "4216a7f853d642b7bc3e5cff458f3aca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } health_report { overall_health: HEALTHY } } }
I20250113 02:51:38.744367 451 catalog_manager.cc:5618] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587) does not have the latest schema for tablet eceed0f52c9f46bbaa78ca773a7afde5 (table default.Table_6a0195b054a043269e133f6ed8dd5979 [id=4b2bd999368c4b628c70cb3ee8c2b1f3]). Expected version 1 got 0
I20250113 02:51:38.744683 451 catalog_manager.cc:5618] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587) does not have the latest schema for tablet 0b119704bc774ca9b4155c9619d4765b (table default.Table_b95a9b1d12284c9aac701a8ef9c93482 [id=843aabcff7074acd98e5801b21282faa]). Expected version 1 got 0
I20250113 02:51:38.744834 451 catalog_manager.cc:5618] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587) does not have the latest schema for tablet b5c1882125b5491ea79a29a6dac6771e (table default.Table_99031e0341fd4ad8a3bb04a6e222a1d0 [id=3237c1be7e1c48918ff095162d191dc1]). Expected version 3 got 0
I20250113 02:51:38.745011 451 catalog_manager.cc:5618] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587) does not have the latest schema for tablet 9438472464304da9a0c7658d4f0736b6 (table default.Table_acdc5be1660742d7a356474b86e8b35f [id=aaf6558a15124f2b8eb9f01d692b845d]). Expected version 2 got 0
W20250113 02:51:38.745985 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_370f637005ce4fef8bac91353769e3ec in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:38.747630 736 tablet_service.cc:1467] Processing CreateTablet for tablet 495064a93dfb41ff999fc73b579baca0 (DEFAULT_TABLE table=default.Table_8c95a106e0684de587166acbc0de43e1 [id=6987c696415646779e6c9816503bb318]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:38.748102 736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 495064a93dfb41ff999fc73b579baca0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:38.748072 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 13f9711b16ae4e36be00db07d9bde7fc
I20250113 02:51:38.749285 1191 tablet_bootstrap.cc:492] T 1968c7e9357c4c6e901d334ace0467fb P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:38.749418 1191 ts_tablet_manager.cc:1397] T 1968c7e9357c4c6e901d334ace0467fb P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.024s user 0.002s sys 0.005s
I20250113 02:51:38.750334 1191 raft_consensus.cc:357] T 1968c7e9357c4c6e901d334ace0467fb P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:38.750515 1191 raft_consensus.cc:383] T 1968c7e9357c4c6e901d334ace0467fb P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:38.750571 1191 raft_consensus.cc:738] T 1968c7e9357c4c6e901d334ace0467fb P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:38.750815 1191 consensus_queue.cc:260] T 1968c7e9357c4c6e901d334ace0467fb P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:38.750969 754 tablet_service.cc:1514] Processing DeleteTablet for tablet 13f9711b16ae4e36be00db07d9bde7fc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:38 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:38.751125 2227 raft_consensus.cc:491] T b815003a27f64809baaa2d43d6293cef P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:38.751214 2227 raft_consensus.cc:513] T b815003a27f64809baaa2d43d6293cef P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:38.751328 1191 ts_tablet_manager.cc:1428] T 1968c7e9357c4c6e901d334ace0467fb P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250113 02:51:38.751549 1191 tablet_bootstrap.cc:492] T aa1ebdb92a5f4c92a5195b614689c100 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:38.751789 2227 leader_election.cc:290] T b815003a27f64809baaa2d43d6293cef P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:38.752245 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b815003a27f64809baaa2d43d6293cef" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:38.752933 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b815003a27f64809baaa2d43d6293cef" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:38.753041 709 leader_election.cc:343] T b815003a27f64809baaa2d43d6293cef P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
W20250113 02:51:38.753371 711 leader_election.cc:343] T b815003a27f64809baaa2d43d6293cef P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:38.753468 711 leader_election.cc:304] T b815003a27f64809baaa2d43d6293cef P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:38.753484 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet 13f9711b16ae4e36be00db07d9bde7fc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:38 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:38.753800 2227 raft_consensus.cc:2743] T b815003a27f64809baaa2d43d6293cef P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:38.754341 1191 tablet_bootstrap.cc:654] T aa1ebdb92a5f4c92a5195b614689c100 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:38.755690 1192 tablet_bootstrap.cc:492] T 1014e51f1c184d729552a5b34292f232 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:38.755820 1192 ts_tablet_manager.cc:1397] T 1014e51f1c184d729552a5b34292f232 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.040s user 0.006s sys 0.002s
I20250113 02:51:38.756716 1192 raft_consensus.cc:357] T 1014e51f1c184d729552a5b34292f232 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:38.756974 1192 raft_consensus.cc:383] T 1014e51f1c184d729552a5b34292f232 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:38.757031 1192 raft_consensus.cc:738] T 1014e51f1c184d729552a5b34292f232 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:38.757257 1192 consensus_queue.cc:260] T 1014e51f1c184d729552a5b34292f232 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:38.759711 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 49c2e648a49f4034a521f38e7d98edc4
I20250113 02:51:38.760902 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 368e878e89b54694823863a49074c11e
I20250113 02:51:38.761458 450 master_service.cc:946] ReplaceTablet: received request to replace tablet 50c4d4783e6d4a55bd3bd355613e1ea0 from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:38.762017 754 tablet_service.cc:1514] Processing DeleteTablet for tablet 368e878e89b54694823863a49074c11e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:38 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:38.762358 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 71befed8b5c74b41b199919b82325891
I20250113 02:51:38.762782 449 catalog_manager.cc:6596] ReplaceTablet: tablet 71befed8b5c74b41b199919b82325891 deleted and replaced by tablet ffb27f71b1464e18977f46d954194bef
I20250113 02:51:38.763489 754 tablet_service.cc:1514] Processing DeleteTablet for tablet 71befed8b5c74b41b199919b82325891 with delete_type TABLET_DATA_DELETED (replaced by tablet ffb27f71b1464e18977f46d954194bef) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:38.763669 1192 ts_tablet_manager.cc:1428] T 1014e51f1c184d729552a5b34292f232 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20250113 02:51:38.764017 1192 tablet_bootstrap.cc:492] T 80d31d5ead804ddaa032969e401a8f7e P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:38.764819 888 tablet_service.cc:1514] Processing DeleteTablet for tablet 49c2e648a49f4034a521f38e7d98edc4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:38 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:38.765024 890 tablet_service.cc:1514] Processing DeleteTablet for tablet 368e878e89b54694823863a49074c11e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:38 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:38.765182 891 tablet_service.cc:1514] Processing DeleteTablet for tablet 71befed8b5c74b41b199919b82325891 with delete_type TABLET_DATA_DELETED (replaced by tablet ffb27f71b1464e18977f46d954194bef) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:38.766880 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet 49c2e648a49f4034a521f38e7d98edc4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:38 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:38.767797 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet 368e878e89b54694823863a49074c11e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:38 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:38.767834 1040 tablet_service.cc:1514] Processing DeleteTablet for tablet 71befed8b5c74b41b199919b82325891 with delete_type TABLET_DATA_DELETED (replaced by tablet ffb27f71b1464e18977f46d954194bef) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:38.768153 891 tablet_service.cc:1467] Processing CreateTablet for tablet bb2fb6d313d7492fbf1e79cea45ab454 (DEFAULT_TABLE table=default.Table_b761dd2912d143b886fa84c37f0047a8 [id=cda1fccf3b1749a7ad6b85f8c1a3f470]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:38.768599 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bb2fb6d313d7492fbf1e79cea45ab454. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:38.769063 890 tablet_service.cc:1467] Processing CreateTablet for tablet cc32610016d14e5f85a37a61c803f85b (DEFAULT_TABLE table=default.Table_2eb9374b5361463297e0aba5d4432877 [id=f8e883a340e946aca13b379acc65b012]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:38.769380 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc32610016d14e5f85a37a61c803f85b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:38.769608 1192 tablet_bootstrap.cc:654] T 80d31d5ead804ddaa032969e401a8f7e P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:38.772557 754 tablet_service.cc:1467] Processing CreateTablet for tablet bb2fb6d313d7492fbf1e79cea45ab454 (DEFAULT_TABLE table=default.Table_b761dd2912d143b886fa84c37f0047a8 [id=cda1fccf3b1749a7ad6b85f8c1a3f470]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:38.772900 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bb2fb6d313d7492fbf1e79cea45ab454. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:38.775801 1196 ts_tablet_manager.cc:1905] T c7243d27a1c642269a3d0e67632e458c P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:38.776520 2227 raft_consensus.cc:491] T 3f62f344c5a648328ca2de8ca1069c19 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:38.776669 2227 raft_consensus.cc:513] T 3f62f344c5a648328ca2de8ca1069c19 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:38.777400 2227 leader_election.cc:290] T 3f62f344c5a648328ca2de8ca1069c19 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:38.778043 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f62f344c5a648328ca2de8ca1069c19" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
W20250113 02:51:38.778642 709 leader_election.cc:343] T 3f62f344c5a648328ca2de8ca1069c19 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:38.783435 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f62f344c5a648328ca2de8ca1069c19" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:38.784649 1040 tablet_service.cc:1467] Processing CreateTablet for tablet bb2fb6d313d7492fbf1e79cea45ab454 (DEFAULT_TABLE table=default.Table_b761dd2912d143b886fa84c37f0047a8 [id=cda1fccf3b1749a7ad6b85f8c1a3f470]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:38.785020 1040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bb2fb6d313d7492fbf1e79cea45ab454. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:38.788506 711 leader_election.cc:343] T 3f62f344c5a648328ca2de8ca1069c19 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:38.788661 711 leader_election.cc:304] T 3f62f344c5a648328ca2de8ca1069c19 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:38.791713 2382 raft_consensus.cc:2743] T 3f62f344c5a648328ca2de8ca1069c19 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:38.792191 1040 tablet_service.cc:1467] Processing CreateTablet for tablet cc32610016d14e5f85a37a61c803f85b (DEFAULT_TABLE table=default.Table_2eb9374b5361463297e0aba5d4432877 [id=f8e883a340e946aca13b379acc65b012]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:38.792326 741 tablet_service.cc:1467] Processing CreateTablet for tablet cc32610016d14e5f85a37a61c803f85b (DEFAULT_TABLE table=default.Table_2eb9374b5361463297e0aba5d4432877 [id=f8e883a340e946aca13b379acc65b012]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:38.792331 890 tablet_service.cc:1467] Processing CreateTablet for tablet acb0ce94537f477a97fef095f7d16a09 (DEFAULT_TABLE table=default.Table_026ff82c780e4fb5803728166c8952ce [id=e388aa4f3e8f4b0ba570afdcc7bfc5d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:38.792659 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc32610016d14e5f85a37a61c803f85b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:38.792659 1040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc32610016d14e5f85a37a61c803f85b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:38.806699 899 tablet_service.cc:1514] Processing DeleteTablet for tablet 13f9711b16ae4e36be00db07d9bde7fc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:38 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:38.808043 1192 tablet_bootstrap.cc:492] T 80d31d5ead804ddaa032969e401a8f7e P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:38.808192 1192 ts_tablet_manager.cc:1397] T 80d31d5ead804ddaa032969e401a8f7e P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.044s user 0.008s sys 0.001s
I20250113 02:51:38.812069 1192 raft_consensus.cc:357] T 80d31d5ead804ddaa032969e401a8f7e P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:38.812296 1192 raft_consensus.cc:383] T 80d31d5ead804ddaa032969e401a8f7e P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:38.812366 1192 raft_consensus.cc:738] T 80d31d5ead804ddaa032969e401a8f7e P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:38.812979 1192 consensus_queue.cc:260] T 80d31d5ead804ddaa032969e401a8f7e P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:38.813999 754 tablet_service.cc:1467] Processing CreateTablet for tablet acb0ce94537f477a97fef095f7d16a09 (DEFAULT_TABLE table=default.Table_026ff82c780e4fb5803728166c8952ce [id=e388aa4f3e8f4b0ba570afdcc7bfc5d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:38.814270 1041 tablet_service.cc:1467] Processing CreateTablet for tablet acb0ce94537f477a97fef095f7d16a09 (DEFAULT_TABLE table=default.Table_026ff82c780e4fb5803728166c8952ce [id=e388aa4f3e8f4b0ba570afdcc7bfc5d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:38.814476 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet acb0ce94537f477a97fef095f7d16a09. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:38.814965 1192 ts_tablet_manager.cc:1428] T 80d31d5ead804ddaa032969e401a8f7e P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20250113 02:51:38.815821 1035 tablet_service.cc:1467] Processing CreateTablet for tablet 501129a689d543609032289c3c5e9ad3 (DEFAULT_TABLE table=default.Table_1b352c8cf2224b2e9b9f4e1322ee8b37 [id=7ceb8d73d00a4bf6bf063e67d0a8a8e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:38.815956 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet acb0ce94537f477a97fef095f7d16a09. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:38.816356 891 tablet_service.cc:1467] Processing CreateTablet for tablet 501129a689d543609032289c3c5e9ad3 (DEFAULT_TABLE table=default.Table_1b352c8cf2224b2e9b9f4e1322ee8b37 [id=7ceb8d73d00a4bf6bf063e67d0a8a8e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:38.820444 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 501129a689d543609032289c3c5e9ad3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:38.824720 736 tablet_service.cc:1514] Processing DeleteTablet for tablet 49c2e648a49f4034a521f38e7d98edc4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:38 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:38.825779 1192 tablet_bootstrap.cc:492] T 44d2c7d5334c476fac60fdd57e2c02b6 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:38.826164 1198 raft_consensus.cc:2267] T 0752fde3a30a4947a2a688c56ceadab5 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:38.828267 1192 tablet_bootstrap.cc:654] T 44d2c7d5334c476fac60fdd57e2c02b6 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:38.828884 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 9fad58cccf1744b0a476523e0af0036e
I20250113 02:51:38.831965 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet 9fad58cccf1744b0a476523e0af0036e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:38 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:38.832703 888 tablet_service.cc:1467] Processing CreateTablet for tablet 77036be46b574f73a02ca656bf5cc0a3 (DEFAULT_TABLE table=default.Table_cdfc4b389e7c4bd984d3dd2c3b33b1c8 [id=2c56f5e5633a4d668357c0a9851fc115]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:38.833030 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet a6d0f9a0a7104b8fb3d22aac976b7d99
I20250113 02:51:38.833519 454 catalog_manager.cc:6596] ReplaceTablet: tablet a6d0f9a0a7104b8fb3d22aac976b7d99 deleted and replaced by tablet 636f2cf30a3f4e509d39f99c6828639c
I20250113 02:51:38.833904 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet f4acbe59400247678d2441b1b844d976
I20250113 02:51:38.833998 886 tablet_service.cc:1514] Processing DeleteTablet for tablet a6d0f9a0a7104b8fb3d22aac976b7d99 with delete_type TABLET_DATA_DELETED (replaced by tablet 636f2cf30a3f4e509d39f99c6828639c) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:38.834254 751 tablet_service.cc:1514] Processing DeleteTablet for tablet a6d0f9a0a7104b8fb3d22aac976b7d99 with delete_type TABLET_DATA_DELETED (replaced by tablet 636f2cf30a3f4e509d39f99c6828639c) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:38.834357 1040 tablet_service.cc:1514] Processing DeleteTablet for tablet a6d0f9a0a7104b8fb3d22aac976b7d99 with delete_type TABLET_DATA_DELETED (replaced by tablet 636f2cf30a3f4e509d39f99c6828639c) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:38.834637 886 tablet_service.cc:1514] Processing DeleteTablet for tablet f4acbe59400247678d2441b1b844d976 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:38 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:38.835093 1035 tablet_service.cc:1467] Processing CreateTablet for tablet 77036be46b574f73a02ca656bf5cc0a3 (DEFAULT_TABLE table=default.Table_cdfc4b389e7c4bd984d3dd2c3b33b1c8 [id=2c56f5e5633a4d668357c0a9851fc115]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:38.835244 1040 tablet_service.cc:1514] Processing DeleteTablet for tablet f4acbe59400247678d2441b1b844d976 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:38 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:38.835461 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 77036be46b574f73a02ca656bf5cc0a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:38.835940 752 tablet_service.cc:1467] Processing CreateTablet for tablet 77036be46b574f73a02ca656bf5cc0a3 (DEFAULT_TABLE table=default.Table_cdfc4b389e7c4bd984d3dd2c3b33b1c8 [id=2c56f5e5633a4d668357c0a9851fc115]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:38.836335 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 77036be46b574f73a02ca656bf5cc0a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:38.837458 1040 tablet_service.cc:1467] Processing CreateTablet for tablet 90f02076f38d4d4780df6b1d90f5a2cc (DEFAULT_TABLE table=default.Table_695dfeac867647bab36791ab4dac6bdc [id=d607bd5d5a434bbb8c164cb296931f57]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:38.837599 751 tablet_service.cc:1514] Processing DeleteTablet for tablet f4acbe59400247678d2441b1b844d976 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:38 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:38.837915 1040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90f02076f38d4d4780df6b1d90f5a2cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:38.838027 886 tablet_service.cc:1467] Processing CreateTablet for tablet 90f02076f38d4d4780df6b1d90f5a2cc (DEFAULT_TABLE table=default.Table_695dfeac867647bab36791ab4dac6bdc [id=d607bd5d5a434bbb8c164cb296931f57]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:38.838569 751 tablet_service.cc:1467] Processing CreateTablet for tablet 90f02076f38d4d4780df6b1d90f5a2cc (DEFAULT_TABLE table=default.Table_695dfeac867647bab36791ab4dac6bdc [id=d607bd5d5a434bbb8c164cb296931f57]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:38.838889 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90f02076f38d4d4780df6b1d90f5a2cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:38.839457 1024 tablet_service.cc:1467] Processing CreateTablet for tablet c61449185ce34ca9bcf4a592255f4f9e (DEFAULT_TABLE table=default.Table_adea383eb53e46f7a4dc11af1e7f2fc7 [id=e873271e43954c93b1acc38fb044e3f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:38.839742 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c61449185ce34ca9bcf4a592255f4f9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:38.840490 881 tablet_service.cc:1467] Processing CreateTablet for tablet c61449185ce34ca9bcf4a592255f4f9e (DEFAULT_TABLE table=default.Table_adea383eb53e46f7a4dc11af1e7f2fc7 [id=e873271e43954c93b1acc38fb044e3f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:38.845575 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet acb0ce94537f477a97fef095f7d16a09. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:38.845791 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c61449185ce34ca9bcf4a592255f4f9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:38.845988 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 77036be46b574f73a02ca656bf5cc0a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:38.848073 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90f02076f38d4d4780df6b1d90f5a2cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:38.849066 2377 raft_consensus.cc:491] T b8b6ffe690fc459db78b2b3bc12671dd P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:38.849192 2377 raft_consensus.cc:513] T b8b6ffe690fc459db78b2b3bc12671dd P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:38.849785 2377 leader_election.cc:290] T b8b6ffe690fc459db78b2b3bc12671dd P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:38.835623 741 tablet_service.cc:1467] Processing CreateTablet for tablet 501129a689d543609032289c3c5e9ad3 (DEFAULT_TABLE table=default.Table_1b352c8cf2224b2e9b9f4e1322ee8b37 [id=7ceb8d73d00a4bf6bf063e67d0a8a8e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:38.852051 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 501129a689d543609032289c3c5e9ad3. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:38.852030 851 leader_election.cc:343] T b8b6ffe690fc459db78b2b3bc12671dd P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Not found: Tablet not found: b8b6ffe690fc459db78b2b3bc12671dd
W20250113 02:51:38.852550 852 leader_election.cc:343] T b8b6ffe690fc459db78b2b3bc12671dd P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Not found: Tablet not found: b8b6ffe690fc459db78b2b3bc12671dd
I20250113 02:51:38.852636 852 leader_election.cc:304] T b8b6ffe690fc459db78b2b3bc12671dd P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df69f48656b44bb890f76587ee69cc0f; no voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca
I20250113 02:51:38.852865 2377 raft_consensus.cc:2743] T b8b6ffe690fc459db78b2b3bc12671dd P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:38.855867 1196 ts_tablet_manager.cc:1918] T c7243d27a1c642269a3d0e67632e458c P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:38.856328 736 tablet_service.cc:1514] Processing DeleteTablet for tablet 9fad58cccf1744b0a476523e0af0036e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:38 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:38.858767 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_8eeeda2b6ca04de78b01bb716651d49c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:38.859488 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8eeeda2b6ca04de78b01bb716651d49c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250113 02:51:38.860086 421 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.104.2:38467, user_credentials={real_user=slave}} blocked reactor thread for 60746us
I20250113 02:51:38.860414 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_302b76de6d984d2586c6a6e182f85156"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:38.861027 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_302b76de6d984d2586c6a6e182f85156 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:38.862648 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_4f8f4a48900d4aa7bd05738d54c4f309"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:38.863205 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4f8f4a48900d4aa7bd05738d54c4f309 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:38.864143 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_830a393895fb4abc877fe4c60f6ed04a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:38.864652 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_830a393895fb4abc877fe4c60f6ed04a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:38.868180 1191 tablet_bootstrap.cc:492] T aa1ebdb92a5f4c92a5195b614689c100 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:38.868388 1191 ts_tablet_manager.cc:1397] T aa1ebdb92a5f4c92a5195b614689c100 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.117s user 0.005s sys 0.016s
I20250113 02:51:38.869139 1191 raft_consensus.cc:357] T aa1ebdb92a5f4c92a5195b614689c100 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:38.869192 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_20be4eba4c334ea69d38f3336e2873e0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:38.869493 1191 raft_consensus.cc:383] T aa1ebdb92a5f4c92a5195b614689c100 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:38.869592 1191 raft_consensus.cc:738] T aa1ebdb92a5f4c92a5195b614689c100 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
W20250113 02:51:38.869671 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_20be4eba4c334ea69d38f3336e2873e0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:38.869967 1191 consensus_queue.cc:260] T aa1ebdb92a5f4c92a5195b614689c100 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:38.874516 751 tablet_service.cc:1467] Processing CreateTablet for tablet c61449185ce34ca9bcf4a592255f4f9e (DEFAULT_TABLE table=default.Table_adea383eb53e46f7a4dc11af1e7f2fc7 [id=e873271e43954c93b1acc38fb044e3f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:38.874991 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c61449185ce34ca9bcf4a592255f4f9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:38.875473 1191 ts_tablet_manager.cc:1428] T aa1ebdb92a5f4c92a5195b614689c100 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20250113 02:51:38.876276 1191 tablet_bootstrap.cc:492] T 938cc1ccd8dd449996836188ba9e64ff P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:38.877642 1191 tablet_bootstrap.cc:654] T 938cc1ccd8dd449996836188ba9e64ff P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:38.879089 452 master_service.cc:946] ReplaceTablet: received request to replace tablet 17fc320eb01f4ac2b9361aeab5b452fc from {username='slave'} at 127.0.0.1:49258
W20250113 02:51:38.879747 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 04dd182ad3564f34989932e463e4d9e4 consensus state has no leader
I20250113 02:51:38.880604 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 50c4d4783e6d4a55bd3bd355613e1ea0
I20250113 02:51:38.881127 450 catalog_manager.cc:6596] ReplaceTablet: tablet 50c4d4783e6d4a55bd3bd355613e1ea0 deleted and replaced by tablet e11387e33dbf4d759f4943eed1d77917
I20250113 02:51:38.882870 453 catalog_manager.cc:5618] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095) does not have the latest schema for tablet 9438472464304da9a0c7658d4f0736b6 (table default.Table_acdc5be1660742d7a356474b86e8b35f [id=aaf6558a15124f2b8eb9f01d692b845d]). Expected version 2 got 0
I20250113 02:51:38.883100 453 catalog_manager.cc:5618] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095) does not have the latest schema for tablet 0b119704bc774ca9b4155c9619d4765b (table default.Table_b95a9b1d12284c9aac701a8ef9c93482 [id=843aabcff7074acd98e5801b21282faa]). Expected version 1 got 0
I20250113 02:51:38.883440 453 catalog_manager.cc:5618] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095) does not have the latest schema for tablet c13e9a51c1e74b44b352bade78390ba5 (table default.Table_0468611ec68e47109f5357935e50874d [id=caf098e2b57849a997bd7c8eaf1f9326]). Expected version 1 got 0
I20250113 02:51:38.883638 453 catalog_manager.cc:5618] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095) does not have the latest schema for tablet fb18c16fb3ba44ec8a190b88d3bbd1b1 (table default.Table_7a4628807cc44ebdadf927ff709de3a2 [id=75ca7cbd7b9548d784f9a77d0f0cb64f]). Expected version 1 got 0
I20250113 02:51:38.885082 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_fe974dd3772d4afd880bc574c573a649"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:38.885363 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet 50c4d4783e6d4a55bd3bd355613e1ea0 with delete_type TABLET_DATA_DELETED (replaced by tablet e11387e33dbf4d759f4943eed1d77917) from {username='slave'} at 127.0.0.1:60230
W20250113 02:51:38.885690 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fe974dd3772d4afd880bc574c573a649 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:38.887272 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_f6e2816287cf46e1b05c86ba38199184"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:38.887823 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f6e2816287cf46e1b05c86ba38199184 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:38.889438 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_9887bd371c8046b49c4ae84f97f16802"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:38.893975 1191 tablet_bootstrap.cc:492] T 938cc1ccd8dd449996836188ba9e64ff P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:38.894163 1191 ts_tablet_manager.cc:1397] T 938cc1ccd8dd449996836188ba9e64ff P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.004s
I20250113 02:51:38.894932 1191 raft_consensus.cc:357] T 938cc1ccd8dd449996836188ba9e64ff P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:38.895181 1191 raft_consensus.cc:383] T 938cc1ccd8dd449996836188ba9e64ff P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:38.895251 1191 raft_consensus.cc:738] T 938cc1ccd8dd449996836188ba9e64ff P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:38.895591 1191 consensus_queue.cc:260] T 938cc1ccd8dd449996836188ba9e64ff P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:38.896639 2382 raft_consensus.cc:491] T 82505f1389184d0ca12ec0e2d6084d22 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:38.896706 1196 log.cc:1198] T c7243d27a1c642269a3d0e67632e458c P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/c7243d27a1c642269a3d0e67632e458c
I20250113 02:51:38.896755 2382 raft_consensus.cc:513] T 82505f1389184d0ca12ec0e2d6084d22 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:38.897047 1196 ts_tablet_manager.cc:1939] T c7243d27a1c642269a3d0e67632e458c P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:38.897377 2382 leader_election.cc:290] T 82505f1389184d0ca12ec0e2d6084d22 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:38.856113 899 tablet_service.cc:1514] Processing DeleteTablet for tablet 9fad58cccf1744b0a476523e0af0036e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:38 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:38.897877 1196 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37366 (request call id 1051) took 1347 ms. Trace:
I20250113 02:51:38.898037 1196 rpcz_store.cc:276] 0113 02:51:37.549987 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:37.551065 (+ 1078us) service_pool.cc:225] Handling call
0113 02:51:38.707410 (+1156345us) ts_tablet_manager.cc:1145] Deleting tablet c7243d27a1c642269a3d0e67632e458c
0113 02:51:38.707424 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:38.778954 (+ 71530us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:38.855793 (+ 76839us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:38.897850 (+ 42057us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1154342}
I20250113 02:51:38.898316 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82505f1389184d0ca12ec0e2d6084d22" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:38.898458 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82505f1389184d0ca12ec0e2d6084d22" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:38.898595 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet c7243d27a1c642269a3d0e67632e458c (table default.Table_7157800e639b45d8bb947e53d4319cfc [id=39997cddb4684b6c984eeeb48d715002]) successfully deleted
W20250113 02:51:38.898842 709 leader_election.cc:343] T 82505f1389184d0ca12ec0e2d6084d22 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
W20250113 02:51:38.899206 711 leader_election.cc:343] T 82505f1389184d0ca12ec0e2d6084d22 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:38.899338 1196 tablet_replica.cc:331] T e468b652f71f44e3945f8d919ac0e1a8 P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:38.899322 711 leader_election.cc:304] T 82505f1389184d0ca12ec0e2d6084d22 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:38.899495 1196 raft_consensus.cc:2238] T e468b652f71f44e3945f8d919ac0e1a8 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:38.899629 1196 raft_consensus.cc:2267] T e468b652f71f44e3945f8d919ac0e1a8 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:38.899799 2382 raft_consensus.cc:2743] T 82505f1389184d0ca12ec0e2d6084d22 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:38.900125 1196 ts_tablet_manager.cc:1905] T e468b652f71f44e3945f8d919ac0e1a8 P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:38.856691 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 501129a689d543609032289c3c5e9ad3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:38.881302 881 tablet_service.cc:1514] Processing DeleteTablet for tablet 50c4d4783e6d4a55bd3bd355613e1ea0 with delete_type TABLET_DATA_DELETED (replaced by tablet e11387e33dbf4d759f4943eed1d77917) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:38.905301 1191 ts_tablet_manager.cc:1428] T 938cc1ccd8dd449996836188ba9e64ff P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.011s user 0.001s sys 0.001s
I20250113 02:51:38.905418 736 tablet_service.cc:1514] Processing DeleteTablet for tablet 50c4d4783e6d4a55bd3bd355613e1ea0 with delete_type TABLET_DATA_DELETED (replaced by tablet e11387e33dbf4d759f4943eed1d77917) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:38.910804 1191 tablet_bootstrap.cc:492] T 283fc7f1c74e480c8a63c74d3ae9eb7d P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:38.911470 890 tablet_service.cc:1514] Processing DeleteTablet for tablet b8b6ffe690fc459db78b2b3bc12671dd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:23 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:38.912433 1191 tablet_bootstrap.cc:654] T 283fc7f1c74e480c8a63c74d3ae9eb7d P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:38.912065 881 tablet_service.cc:1467] Processing CreateTablet for tablet 8b2d63589216421b9d6fa56fb8e0c883 (DEFAULT_TABLE table=default.Table_737a54941ce54064a9191c33fe730efb [id=755a279f0b7f445c9c2644cabcaaee23]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:38.912734 1035 tablet_service.cc:1467] Processing CreateTablet for tablet 8b2d63589216421b9d6fa56fb8e0c883 (DEFAULT_TABLE table=default.Table_737a54941ce54064a9191c33fe730efb [id=755a279f0b7f445c9c2644cabcaaee23]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:38.913039 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b2d63589216421b9d6fa56fb8e0c883. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:38.913107 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b2d63589216421b9d6fa56fb8e0c883. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:38.913887 899 tablet_service.cc:1467] Processing CreateTablet for tablet 54a35608cd8846188ef8f8f0278be68a (DEFAULT_TABLE table=default.Table_c4e4c90fea79464ca6cdb5decd2c4a5c [id=3e9ebb7b8dba4873bd0aa34694c135c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:38.914211 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 54a35608cd8846188ef8f8f0278be68a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:38.916760 1040 tablet_service.cc:1467] Processing CreateTablet for tablet 54a35608cd8846188ef8f8f0278be68a (DEFAULT_TABLE table=default.Table_c4e4c90fea79464ca6cdb5decd2c4a5c [id=3e9ebb7b8dba4873bd0aa34694c135c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:38.917112 1040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 54a35608cd8846188ef8f8f0278be68a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:38.917760 886 tablet_service.cc:1467] Processing CreateTablet for tablet 9241fa233f1b446db4de33943c45e269 (DEFAULT_TABLE table=default.Table_d1195c2276654128b31755efa960e72c [id=91123648dac947e7a8d775990d0fc035]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:38.918098 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9241fa233f1b446db4de33943c45e269. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:38.918337 1024 tablet_service.cc:1467] Processing CreateTablet for tablet 9241fa233f1b446db4de33943c45e269 (DEFAULT_TABLE table=default.Table_d1195c2276654128b31755efa960e72c [id=91123648dac947e7a8d775990d0fc035]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:38.918634 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9241fa233f1b446db4de33943c45e269. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:38.919802 752 tablet_service.cc:1467] Processing CreateTablet for tablet 54a35608cd8846188ef8f8f0278be68a (DEFAULT_TABLE table=default.Table_c4e4c90fea79464ca6cdb5decd2c4a5c [id=3e9ebb7b8dba4873bd0aa34694c135c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:38.920183 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 54a35608cd8846188ef8f8f0278be68a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:38.924741 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 17fc320eb01f4ac2b9361aeab5b452fc
I20250113 02:51:38.925249 452 catalog_manager.cc:6596] ReplaceTablet: tablet 17fc320eb01f4ac2b9361aeab5b452fc deleted and replaced by tablet 284ca2fe0c6a4b79a3bc6d096a90ccf1
I20250113 02:51:38.925597 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet 17fc320eb01f4ac2b9361aeab5b452fc with delete_type TABLET_DATA_DELETED (replaced by tablet 284ca2fe0c6a4b79a3bc6d096a90ccf1) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:38.925835 736 tablet_service.cc:1467] Processing CreateTablet for tablet 8b2d63589216421b9d6fa56fb8e0c883 (DEFAULT_TABLE table=default.Table_737a54941ce54064a9191c33fe730efb [id=755a279f0b7f445c9c2644cabcaaee23]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:38.926179 736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b2d63589216421b9d6fa56fb8e0c883. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:38.927243 886 tablet_service.cc:1514] Processing DeleteTablet for tablet 17fc320eb01f4ac2b9361aeab5b452fc with delete_type TABLET_DATA_DELETED (replaced by tablet 284ca2fe0c6a4b79a3bc6d096a90ccf1) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:38.927603 744 tablet_service.cc:1514] Processing DeleteTablet for tablet 17fc320eb01f4ac2b9361aeab5b452fc with delete_type TABLET_DATA_DELETED (replaced by tablet 284ca2fe0c6a4b79a3bc6d096a90ccf1) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:38.930639 453 catalog_manager.cc:6071] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: default.Table_c8b93363989c4e109540af6bef426f73 [id=b7846a8ebee84005b1b5d259560a26bd] alter complete (version 1)
I20250113 02:51:38.933226 744 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a4c6dc08df744e6856bdf99ab65c575 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:26 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:38.937433 2377 raft_consensus.cc:491] T 0539cbbe47e64c039ad39c4b52edcc4b P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:38.937588 2377 raft_consensus.cc:513] T 0539cbbe47e64c039ad39c4b52edcc4b P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:38.938251 2377 leader_election.cc:290] T 0539cbbe47e64c039ad39c4b52edcc4b P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
W20250113 02:51:38.939869 852 leader_election.cc:343] T 0539cbbe47e64c039ad39c4b52edcc4b P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Not found: Tablet not found: 0539cbbe47e64c039ad39c4b52edcc4b
W20250113 02:51:38.940295 851 leader_election.cc:343] T 0539cbbe47e64c039ad39c4b52edcc4b P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Not found: Tablet not found: 0539cbbe47e64c039ad39c4b52edcc4b
I20250113 02:51:38.940388 851 leader_election.cc:304] T 0539cbbe47e64c039ad39c4b52edcc4b P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df69f48656b44bb890f76587ee69cc0f; no voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca
I20250113 02:51:38.940793 2377 raft_consensus.cc:2743] T 0539cbbe47e64c039ad39c4b52edcc4b P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:38.875946 2397 raft_consensus.cc:491] T 00000000000000000000000000000000 P 45e9d43ecac0401fb2c18fe5eaf089b5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader e13f5bcdcee34225a46c6d2651e5ede9)
I20250113 02:51:38.941789 2397 raft_consensus.cc:513] T 00000000000000000000000000000000 P 45e9d43ecac0401fb2c18fe5eaf089b5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e13f5bcdcee34225a46c6d2651e5ede9" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44217 } } peers { permanent_uuid: "6bbc3541604a455584bb32dfc2cc4efd" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 35271 } } peers { permanent_uuid: "45e9d43ecac0401fb2c18fe5eaf089b5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42067 } }
I20250113 02:51:38.942458 2397 leader_election.cc:290] T 00000000000000000000000000000000 P 45e9d43ecac0401fb2c18fe5eaf089b5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers e13f5bcdcee34225a46c6d2651e5ede9 (127.0.104.62:44217), 6bbc3541604a455584bb32dfc2cc4efd (127.0.104.61:35271)
I20250113 02:51:38.944108 744 tablet_service.cc:1467] Processing CreateTablet for tablet 9241fa233f1b446db4de33943c45e269 (DEFAULT_TABLE table=default.Table_d1195c2276654128b31755efa960e72c [id=91123648dac947e7a8d775990d0fc035]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:38.944501 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9241fa233f1b446db4de33943c45e269. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:38.944628 1194 tablet_bootstrap.cc:492] T 4b07a985897641c3ac48021c2a4daedb P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:38.945374 1198 ts_tablet_manager.cc:1905] T 0752fde3a30a4947a2a688c56ceadab5 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:38.948266 886 tablet_service.cc:1467] Processing CreateTablet for tablet 3080f09d5849406d994152ef9544cbdc (DEFAULT_TABLE table=default.Table_cb9f6aa12bff4666bd309bc132674898 [id=8a6a38af0ce04a56aefceee520f29755]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:38.948606 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3080f09d5849406d994152ef9544cbdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:38.950480 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1059) took 2529 ms. Trace:
I20250113 02:51:38.950563 1195 rpcz_store.cc:276] 0113 02:51:36.420638 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:36.420779 (+ 141us) service_pool.cc:225] Handling call
0113 02:51:38.436825 (+2016046us) ts_tablet_manager.cc:1145] Deleting tablet c518f791b4d74c54ab0c0364d29b4a64
0113 02:51:38.436836 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:38.551677 (+114841us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:38.634887 (+ 83210us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:38.950459 (+315572us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2015842}
I20250113 02:51:38.950965 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet c518f791b4d74c54ab0c0364d29b4a64 (table default.Table_157d1350cce74502a990462ac9bd0e5f [id=f96a39bfcdf442f7965a7b16d9aa432f]) successfully deleted
I20250113 02:51:38.952922 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_4edd5e339c0a437c88823ed8388a400d" } new_table_name: "default.Table_926f5386ec3c43dfa2b4230c28784c26" modify_external_catalogs: true
I20250113 02:51:38.955291 464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "45e9d43ecac0401fb2c18fe5eaf089b5" candidate_term: 5 candidate_status { last_received { term: 4 index: 994 } } ignore_live_leader: false dest_uuid: "e13f5bcdcee34225a46c6d2651e5ede9" is_pre_election: true
I20250113 02:51:38.956984 452 catalog_manager.cc:5618] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587) does not have the latest schema for tablet fb18c16fb3ba44ec8a190b88d3bbd1b1 (table default.Table_7a4628807cc44ebdadf927ff709de3a2 [id=75ca7cbd7b9548d784f9a77d0f0cb64f]). Expected version 1 got 0
I20250113 02:51:38.960345 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet 30ca8a716bfc4ab789b216e17567785e with delete_type TABLET_DATA_DELETED (replaced by tablet 5f7c91d8c018455e847e5969f7b5e1cc) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:38.960816 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet ee240d838d4a4bbba97f8ed61f69de78 with delete_type TABLET_DATA_DELETED (replaced by tablet 95e47db8a4d74615800ba6c049ec7131) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:38.961385 1041 tablet_service.cc:1514] Processing DeleteTablet for tablet 1014e51f1c184d729552a5b34292f232 with delete_type TABLET_DATA_DELETED (replaced by tablet ec8196505dab4e05bb6487b566cf3144) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:38.961545 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet 80d31d5ead804ddaa032969e401a8f7e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:22 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:38.961833 1344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "45e9d43ecac0401fb2c18fe5eaf089b5" candidate_term: 5 candidate_status { last_received { term: 4 index: 994 } } ignore_live_leader: false dest_uuid: "6bbc3541604a455584bb32dfc2cc4efd" is_pre_election: true
I20250113 02:51:38.963089 1024 tablet_service.cc:1467] Processing CreateTablet for tablet 3080f09d5849406d994152ef9544cbdc (DEFAULT_TABLE table=default.Table_cb9f6aa12bff4666bd309bc132674898 [id=8a6a38af0ce04a56aefceee520f29755]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:38.963546 452 catalog_manager.cc:5618] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095) does not have the latest schema for tablet 938cc1ccd8dd449996836188ba9e64ff (table default.Table_e5b4d614e4b743d7982b025ca6d297e5 [id=5824acd778ed47d09f087bbf8e7822d6]). Expected version 1 got 0
I20250113 02:51:38.963738 1841 leader_election.cc:304] T 00000000000000000000000000000000 P 45e9d43ecac0401fb2c18fe5eaf089b5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 45e9d43ecac0401fb2c18fe5eaf089b5; no voters: 6bbc3541604a455584bb32dfc2cc4efd, e13f5bcdcee34225a46c6d2651e5ede9
I20250113 02:51:38.964287 2397 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 45e9d43ecac0401fb2c18fe5eaf089b5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250113 02:51:38.966828 774 raft_consensus.cc:1270] T eceed0f52c9f46bbaa78ca773a7afde5 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Refusing update from remote peer 4216a7f853d642b7bc3e5cff458f3aca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250113 02:51:38.967074 744 tablet_service.cc:1467] Processing CreateTablet for tablet 3080f09d5849406d994152ef9544cbdc (DEFAULT_TABLE table=default.Table_cb9f6aa12bff4666bd309bc132674898 [id=8a6a38af0ce04a56aefceee520f29755]), partition=RANGE (key) PARTITION UNBOUNDED
W20250113 02:51:38.967247 998 consensus_peers.cc:487] T eceed0f52c9f46bbaa78ca773a7afde5 P 4216a7f853d642b7bc3e5cff458f3aca -> Peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Couldn't send request to peer df69f48656b44bb890f76587ee69cc0f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250113 02:51:38.967432 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3080f09d5849406d994152ef9544cbdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:38.967690 2368 consensus_queue.cc:1035] T eceed0f52c9f46bbaa78ca773a7afde5 P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 02:51:38.972529 1041 tablet_service.cc:1467] Processing CreateTablet for tablet ca144327ca8d48d48a072caaff74212f (DEFAULT_TABLE table=default.Table_ffa9bfdfd58f4159b17a5185a1d2192e [id=442296f98c864b519c3d4ca70c323199]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:38.972934 2368 raft_consensus.cc:491] T 0644f5600c5b47e29f3fa73060ed96e3 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:38.973042 2368 raft_consensus.cc:513] T 0644f5600c5b47e29f3fa73060ed96e3 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:38.973582 2368 leader_election.cc:290] T 0644f5600c5b47e29f3fa73060ed96e3 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
I20250113 02:51:38.974537 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0644f5600c5b47e29f3fa73060ed96e3" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:38.974965 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0644f5600c5b47e29f3fa73060ed96e3" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
I20250113 02:51:38.975147 771 raft_consensus.cc:2463] T 0644f5600c5b47e29f3fa73060ed96e3 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 0.
I20250113 02:51:38.975523 995 leader_election.cc:304] T 0644f5600c5b47e29f3fa73060ed96e3 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters:
I20250113 02:51:38.975952 2368 raft_consensus.cc:2798] T 0644f5600c5b47e29f3fa73060ed96e3 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 02:51:38.976052 2368 raft_consensus.cc:491] T 0644f5600c5b47e29f3fa73060ed96e3 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 02:51:38.976070 1194 ts_tablet_manager.cc:1397] T 4b07a985897641c3ac48021c2a4daedb P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.357s user 0.005s sys 0.009s
I20250113 02:51:38.976143 2368 raft_consensus.cc:3054] T 0644f5600c5b47e29f3fa73060ed96e3 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:38.977053 1194 raft_consensus.cc:357] T 4b07a985897641c3ac48021c2a4daedb P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:38.977284 1194 raft_consensus.cc:383] T 4b07a985897641c3ac48021c2a4daedb P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:38.977341 1194 raft_consensus.cc:738] T 4b07a985897641c3ac48021c2a4daedb P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:38.977573 1194 consensus_queue.cc:260] T 4b07a985897641c3ac48021c2a4daedb P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
W20250113 02:51:38.980244 998 leader_election.cc:343] T 0644f5600c5b47e29f3fa73060ed96e3 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:38.980613 1194 ts_tablet_manager.cc:1428] T 4b07a985897641c3ac48021c2a4daedb P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250113 02:51:38.980929 1194 tablet_bootstrap.cc:492] T f129bb3fbbfd492d98822be0a6626e44 P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:38.982586 1194 tablet_bootstrap.cc:654] T f129bb3fbbfd492d98822be0a6626e44 P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:38.984686 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca144327ca8d48d48a072caaff74212f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:38.987810 2368 raft_consensus.cc:513] T 0644f5600c5b47e29f3fa73060ed96e3 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:38.989084 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0644f5600c5b47e29f3fa73060ed96e3" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f"
W20250113 02:51:38.989555 998 leader_election.cc:343] T 0644f5600c5b47e29f3fa73060ed96e3 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:38.989980 886 tablet_service.cc:1467] Processing CreateTablet for tablet ca144327ca8d48d48a072caaff74212f (DEFAULT_TABLE table=default.Table_ffa9bfdfd58f4159b17a5185a1d2192e [id=442296f98c864b519c3d4ca70c323199]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:38.990700 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0644f5600c5b47e29f3fa73060ed96e3" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196"
I20250113 02:51:38.990872 771 raft_consensus.cc:3054] T 0644f5600c5b47e29f3fa73060ed96e3 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:38.992519 2218 raft_consensus.cc:491] T 80d31d5ead804ddaa032969e401a8f7e P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:38.992626 2218 raft_consensus.cc:513] T 80d31d5ead804ddaa032969e401a8f7e P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:38.993204 2218 leader_election.cc:290] T 80d31d5ead804ddaa032969e401a8f7e P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
I20250113 02:51:38.993813 1195 tablet_replica.cc:331] T 655b63ff61ff4f4aa08e0449df39352a P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:38.994040 771 raft_consensus.cc:2463] T 0644f5600c5b47e29f3fa73060ed96e3 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 1.
I20250113 02:51:38.994585 738 tablet_service.cc:1514] Processing DeleteTablet for tablet 1968c7e9357c4c6e901d334ace0467fb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:26 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:38.994690 746 tablet_service.cc:1514] Processing DeleteTablet for tablet aa1ebdb92a5f4c92a5195b614689c100 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:29 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:38.995164 741 tablet_service.cc:1514] Processing DeleteTablet for tablet acd83024592e431898e9675c17f50d4f with delete_type TABLET_DATA_DELETED (replaced by tablet 18716752335043e89038f01b4d65d711) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:38.995239 995 leader_election.cc:304] T 0644f5600c5b47e29f3fa73060ed96e3 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters: df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:38.995576 2218 raft_consensus.cc:2798] T 0644f5600c5b47e29f3fa73060ed96e3 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Leader election won for term 1
I20250113 02:51:38.995692 2218 raft_consensus.cc:695] T 0644f5600c5b47e29f3fa73060ed96e3 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: Becoming Leader. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Running, Role: LEADER
I20250113 02:51:38.995913 2218 consensus_queue.cc:237] T 0644f5600c5b47e29f3fa73060ed96e3 P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:38.994055 1195 raft_consensus.cc:2238] T 655b63ff61ff4f4aa08e0449df39352a P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:39.000644 1195 raft_consensus.cc:2267] T 655b63ff61ff4f4aa08e0449df39352a P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:39.001199 1195 ts_tablet_manager.cc:1905] T 655b63ff61ff4f4aa08e0449df39352a P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:39.001500 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3080f09d5849406d994152ef9544cbdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:39.001773 2375 tablet.cc:1721] T eceed0f52c9f46bbaa78ca773a7afde5 P 4216a7f853d642b7bc3e5cff458f3aca: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250113 02:51:39.009135 2368 leader_election.cc:290] T 0644f5600c5b47e29f3fa73060ed96e3 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Requested vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
I20250113 02:51:38.993873 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80d31d5ead804ddaa032969e401a8f7e" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:39.010176 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca144327ca8d48d48a072caaff74212f. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:39.009536 995 leader_election.cc:343] T 80d31d5ead804ddaa032969e401a8f7e P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Not found: Tablet not found: 80d31d5ead804ddaa032969e401a8f7e
W20250113 02:51:39.010426 998 leader_election.cc:343] T 80d31d5ead804ddaa032969e401a8f7e P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:39.010527 998 leader_election.cc:304] T 80d31d5ead804ddaa032969e401a8f7e P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4216a7f853d642b7bc3e5cff458f3aca; no voters: 1186fa869eb14812832fa6562531f196, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:39.013073 1192 tablet_bootstrap.cc:492] T 44d2c7d5334c476fac60fdd57e2c02b6 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:39.013202 1192 ts_tablet_manager.cc:1397] T 44d2c7d5334c476fac60fdd57e2c02b6 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.187s user 0.015s sys 0.041s
I20250113 02:51:39.013989 1192 raft_consensus.cc:357] T 44d2c7d5334c476fac60fdd57e2c02b6 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:39.014182 1192 raft_consensus.cc:383] T 44d2c7d5334c476fac60fdd57e2c02b6 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:39.014238 1192 raft_consensus.cc:738] T 44d2c7d5334c476fac60fdd57e2c02b6 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:39.014443 1192 consensus_queue.cc:260] T 44d2c7d5334c476fac60fdd57e2c02b6 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:39.015336 1192 ts_tablet_manager.cc:1428] T 44d2c7d5334c476fac60fdd57e2c02b6 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250113 02:51:39.015580 1192 tablet_bootstrap.cc:492] T f1e32e09a8244957a99bdfea5812ff31 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
W20250113 02:51:39.016417 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9887bd371c8046b49c4ae84f97f16802 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:39.019074 1192 tablet_bootstrap.cc:654] T f1e32e09a8244957a99bdfea5812ff31 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:39.019650 1198 ts_tablet_manager.cc:1918] T 0752fde3a30a4947a2a688c56ceadab5 P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250113 02:51:39.021019 2378 tablet.cc:1721] T eceed0f52c9f46bbaa78ca773a7afde5 P 1186fa869eb14812832fa6562531f196: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250113 02:51:39.021746 1191 tablet_bootstrap.cc:492] T 283fc7f1c74e480c8a63c74d3ae9eb7d P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:39.021868 1191 ts_tablet_manager.cc:1397] T 283fc7f1c74e480c8a63c74d3ae9eb7d P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.111s user 0.013s sys 0.022s
I20250113 02:51:39.022504 1191 raft_consensus.cc:357] T 283fc7f1c74e480c8a63c74d3ae9eb7d P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:39.022677 1191 raft_consensus.cc:383] T 283fc7f1c74e480c8a63c74d3ae9eb7d P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:39.022722 1191 raft_consensus.cc:738] T 283fc7f1c74e480c8a63c74d3ae9eb7d P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:39.022949 1191 consensus_queue.cc:260] T 283fc7f1c74e480c8a63c74d3ae9eb7d P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:39.023401 1191 ts_tablet_manager.cc:1428] T 283fc7f1c74e480c8a63c74d3ae9eb7d P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20250113 02:51:39.023619 1191 tablet_bootstrap.cc:492] T fd0b8592997d4a61b8ca854ecbb67c55 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:39.024070 1198 log.cc:1198] T 0752fde3a30a4947a2a688c56ceadab5 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/0752fde3a30a4947a2a688c56ceadab5
I20250113 02:51:39.024464 1198 ts_tablet_manager.cc:1939] T 0752fde3a30a4947a2a688c56ceadab5 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:39.027483 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_34138ec336f245abafeb5e3ba18706e8" } new_table_name: "default.Table_634c9d690a0d42a1ac0653a55e6b98d0" modify_external_catalogs: true
I20250113 02:51:39.018841 452 catalog_manager.cc:5618] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095) does not have the latest schema for tablet eceed0f52c9f46bbaa78ca773a7afde5 (table default.Table_6a0195b054a043269e133f6ed8dd5979 [id=4b2bd999368c4b628c70cb3ee8c2b1f3]). Expected version 1 got 0
I20250113 02:51:39.034657 2368 raft_consensus.cc:491] T bc533518faa24dffa3306ff092e8d507 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:39.035133 2368 raft_consensus.cc:513] T bc533518faa24dffa3306ff092e8d507 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:39.014073 891 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b07a985897641c3ac48021c2a4daedb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:30 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:39.036526 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc533518faa24dffa3306ff092e8d507" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
I20250113 02:51:39.036773 771 raft_consensus.cc:2463] T bc533518faa24dffa3306ff092e8d507 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 0.
I20250113 02:51:39.037218 995 leader_election.cc:304] T bc533518faa24dffa3306ff092e8d507 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters:
I20250113 02:51:39.037464 2306 raft_consensus.cc:2798] T bc533518faa24dffa3306ff092e8d507 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 02:51:39.037575 2306 raft_consensus.cc:491] T bc533518faa24dffa3306ff092e8d507 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 02:51:39.037642 2306 raft_consensus.cc:3054] T bc533518faa24dffa3306ff092e8d507 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:39.044332 454 master_service.cc:946] ReplaceTablet: received request to replace tablet 8cc78bf7d6324b7ba10a1b1045923f2b from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:39.045161 2306 raft_consensus.cc:513] T bc533518faa24dffa3306ff092e8d507 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:39.045758 2306 leader_election.cc:290] T bc533518faa24dffa3306ff092e8d507 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Requested vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:39.046674 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc533518faa24dffa3306ff092e8d507" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196"
I20250113 02:51:39.046912 771 raft_consensus.cc:3054] T bc533518faa24dffa3306ff092e8d507 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:39.047827 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_e03d954cc20048e4b0d598282689643d" } new_table_name: "default.Table_c506ab8e150647aaaa2baac66ae763b5" modify_external_catalogs: true
I20250113 02:51:39.049728 771 raft_consensus.cc:2463] T bc533518faa24dffa3306ff092e8d507 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 1.
I20250113 02:51:39.050465 2368 leader_election.cc:290] T bc533518faa24dffa3306ff092e8d507 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:39.052258 448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_d6dd014b2a3f4de5aaca5efe047b5819" } new_table_name: "default.Table_567d446e6ae54abf9ae7586692b5780a" modify_external_catalogs: true
I20250113 02:51:39.054214 454 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_9cc116d6861b4480b32eb7fa206ae7c0" } modify_external_catalogs: true
I20250113 02:51:39.054426 454 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_9cc116d6861b4480b32eb7fa206ae7c0" } modify_external_catalogs: true
I20250113 02:51:39.056058 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_5bbda7be1ad04c32864d6d49a342e41f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:39.056641 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5bbda7be1ad04c32864d6d49a342e41f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250113 02:51:39.066656 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f3a8c7a25aca4496919386396626de00 consensus state has no leader
I20250113 02:51:39.068634 995 leader_election.cc:304] T bc533518faa24dffa3306ff092e8d507 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters:
I20250113 02:51:39.069164 2306 raft_consensus.cc:2798] T bc533518faa24dffa3306ff092e8d507 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Leader election won for term 1
I20250113 02:51:39.069613 2306 raft_consensus.cc:695] T bc533518faa24dffa3306ff092e8d507 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: Becoming Leader. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Running, Role: LEADER
I20250113 02:51:39.069847 2306 consensus_queue.cc:237] T bc533518faa24dffa3306ff092e8d507 P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:39.070544 453 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_5678b31026a4415e9c30223de0518da3" } modify_external_catalogs: true
I20250113 02:51:39.070714 453 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_5678b31026a4415e9c30223de0518da3" } modify_external_catalogs: true
I20250113 02:51:39.072129 455 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_1871943d605842118e5ac9c6f594f30c" } modify_external_catalogs: true
I20250113 02:51:39.072331 455 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_1871943d605842118e5ac9c6f594f30c" } modify_external_catalogs: true
I20250113 02:51:39.073016 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_07e3e48b34b040ebbcf9a5dffdbb6bd9" } modify_external_catalogs: true
I20250113 02:51:39.073740 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_07e3e48b34b040ebbcf9a5dffdbb6bd9" } modify_external_catalogs: true
W20250113 02:51:39.074465 451 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0269663b35f148c4a2a93cfa6132d0f8 consensus state has no leader
I20250113 02:51:39.075596 1035 tablet_service.cc:1467] Processing CreateTablet for tablet 5abfbac8a9ce44f59afbc4b21d1e8e87 (DEFAULT_TABLE table=default.Table_33d55ba79fc14c53be88aac64de3b16a [id=b7ae3215fb854b0e84cac3705f63de30]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:39.076013 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5abfbac8a9ce44f59afbc4b21d1e8e87. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:39.076138 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_56ac841ab1804eafbd61ea8fa855824f" } modify_external_catalogs: true
I20250113 02:51:39.076261 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_56ac841ab1804eafbd61ea8fa855824f" } modify_external_catalogs: true
I20250113 02:51:39.076637 1040 tablet_service.cc:1467] Processing CreateTablet for tablet 194b032bd6624ede85d6a3006811ce59 (DEFAULT_TABLE table=default.Table_e975ec9af8ab463f911e916f9aec089b [id=9c5ac4f938c142b9a5fd11dcae8bc0b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:39.076931 1040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 194b032bd6624ede85d6a3006811ce59. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:39.080147 899 tablet_service.cc:1467] Processing CreateTablet for tablet 5abfbac8a9ce44f59afbc4b21d1e8e87 (DEFAULT_TABLE table=default.Table_33d55ba79fc14c53be88aac64de3b16a [id=b7ae3215fb854b0e84cac3705f63de30]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:39.080559 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5abfbac8a9ce44f59afbc4b21d1e8e87. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:39.082770 891 tablet_service.cc:1467] Processing CreateTablet for tablet 194b032bd6624ede85d6a3006811ce59 (DEFAULT_TABLE table=default.Table_e975ec9af8ab463f911e916f9aec089b [id=9c5ac4f938c142b9a5fd11dcae8bc0b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:39.083170 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 194b032bd6624ede85d6a3006811ce59. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:39.084789 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_5a88ca4488b9487ba1b91df5fbc57ae1" } new_table_name: "default.Table_7555ebcfe8f644d69109539f67ccb96a" modify_external_catalogs: true
I20250113 02:51:39.089185 1191 tablet_bootstrap.cc:654] T fd0b8592997d4a61b8ca854ecbb67c55 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:39.093288 751 tablet_service.cc:1467] Processing CreateTablet for tablet ca144327ca8d48d48a072caaff74212f (DEFAULT_TABLE table=default.Table_ffa9bfdfd58f4159b17a5185a1d2192e [id=442296f98c864b519c3d4ca70c323199]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:39.095420 890 tablet_service.cc:1467] Processing CreateTablet for tablet 4b0e7558f9d546b7a87c5728a5f14442 (DEFAULT_TABLE table=default.Table_f85400f8b51646018d553e67a5fdcffb [id=3fcd1bf7f6d146ea8fa172af337f61a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:39.095952 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b0e7558f9d546b7a87c5728a5f14442. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:39.096066 752 tablet_service.cc:1467] Processing CreateTablet for tablet 5abfbac8a9ce44f59afbc4b21d1e8e87 (DEFAULT_TABLE table=default.Table_33d55ba79fc14c53be88aac64de3b16a [id=b7ae3215fb854b0e84cac3705f63de30]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:39.099126 741 tablet_service.cc:1467] Processing CreateTablet for tablet 194b032bd6624ede85d6a3006811ce59 (DEFAULT_TABLE table=default.Table_e975ec9af8ab463f911e916f9aec089b [id=9c5ac4f938c142b9a5fd11dcae8bc0b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:39.101020 1195 ts_tablet_manager.cc:1918] T 655b63ff61ff4f4aa08e0449df39352a P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:39.104553 1195 log.cc:1198] T 655b63ff61ff4f4aa08e0449df39352a P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/655b63ff61ff4f4aa08e0449df39352a
I20250113 02:51:39.104486 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_1e8aa807c9e642c8950cff0dccb6e83a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:39.105051 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1e8aa807c9e642c8950cff0dccb6e83a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:39.105386 1195 ts_tablet_manager.cc:1939] T 655b63ff61ff4f4aa08e0449df39352a P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:39.107321 881 tablet_service.cc:1467] Processing CreateTablet for tablet 7adc82d54b4740ce9a2b143f961a7818 (DEFAULT_TABLE table=default.Table_ac317bba2577464095cac3d5fe1374eb [id=96b19ffacfa24f118bf4ec4c52a22a82]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:39.107867 888 tablet_service.cc:1467] Processing CreateTablet for tablet c75f5773e0ca4554807b302cec1db90a (DEFAULT_TABLE table=default.Table_de89db4ee495473993f7cb9709fcd706 [id=15538635367049c7adf6a6eb9f96e176]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:39.107882 754 tablet_service.cc:1467] Processing CreateTablet for tablet 4b0e7558f9d546b7a87c5728a5f14442 (DEFAULT_TABLE table=default.Table_f85400f8b51646018d553e67a5fdcffb [id=3fcd1bf7f6d146ea8fa172af337f61a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:39.108716 746 tablet_service.cc:1467] Processing CreateTablet for tablet c75f5773e0ca4554807b302cec1db90a (DEFAULT_TABLE table=default.Table_de89db4ee495473993f7cb9709fcd706 [id=15538635367049c7adf6a6eb9f96e176]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:39.108814 882 tablet_service.cc:1467] Processing CreateTablet for tablet bd3935f168e74990892bfea011714dab (DEFAULT_TABLE table=default.Table_bccd061fec0d4a9e91a3b126aec48c92 [id=87a475439c6b4c788ff5ce10485d91c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:39.109617 883 tablet_service.cc:1467] Processing CreateTablet for tablet dfb6ef2ccc2d481c8ce65470d953d5a2 (DEFAULT_TABLE table=default.Table_c91c831eacb14ed9bbb303b6f7fcd6ca [id=1bd5367e7c85495c828f6a29143f144a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:39.109723 1031 tablet_service.cc:1467] Processing CreateTablet for tablet 790cc9449d1a47328854eeceb44f7718 (DEFAULT_TABLE table=default.Table_8d91c57d1c1941d4bf5ea4e798361591 [id=33bf00d2884d4956a7a94ccf1bb8fd10]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:39.109848 1035 tablet_service.cc:1467] Processing CreateTablet for tablet 4b0e7558f9d546b7a87c5728a5f14442 (DEFAULT_TABLE table=default.Table_f85400f8b51646018d553e67a5fdcffb [id=3fcd1bf7f6d146ea8fa172af337f61a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:39.110131 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 790cc9449d1a47328854eeceb44f7718. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:39.110343 736 tablet_service.cc:1467] Processing CreateTablet for tablet 790cc9449d1a47328854eeceb44f7718 (DEFAULT_TABLE table=default.Table_8d91c57d1c1941d4bf5ea4e798361591 [id=33bf00d2884d4956a7a94ccf1bb8fd10]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:39.110466 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b0e7558f9d546b7a87c5728a5f14442. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:39.110716 738 tablet_service.cc:1467] Processing CreateTablet for tablet bd3935f168e74990892bfea011714dab (DEFAULT_TABLE table=default.Table_bccd061fec0d4a9e91a3b126aec48c92 [id=87a475439c6b4c788ff5ce10485d91c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:39.111379 744 tablet_service.cc:1467] Processing CreateTablet for tablet 7adc82d54b4740ce9a2b143f961a7818 (DEFAULT_TABLE table=default.Table_ac317bba2577464095cac3d5fe1374eb [id=96b19ffacfa24f118bf4ec4c52a22a82]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:39.112082 890 tablet_service.cc:1467] Processing CreateTablet for tablet 790cc9449d1a47328854eeceb44f7718 (DEFAULT_TABLE table=default.Table_8d91c57d1c1941d4bf5ea4e798361591 [id=33bf00d2884d4956a7a94ccf1bb8fd10]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:39.114887 1196 ts_tablet_manager.cc:1918] T e468b652f71f44e3945f8d919ac0e1a8 P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:39.115028 1196 log.cc:1198] T e468b652f71f44e3945f8d919ac0e1a8 P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/e468b652f71f44e3945f8d919ac0e1a8
I20250113 02:51:39.115599 1196 ts_tablet_manager.cc:1939] T e468b652f71f44e3945f8d919ac0e1a8 P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:39.116466 1196 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37366 (request call id 1056) took 1533 ms. Trace:
I20250113 02:51:39.119410 2368 raft_consensus.cc:491] T 655b63ff61ff4f4aa08e0449df39352a P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:39.120544 2368 raft_consensus.cc:513] T 655b63ff61ff4f4aa08e0449df39352a P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:39.125463 1040 tablet_service.cc:1467] Processing CreateTablet for tablet 7adc82d54b4740ce9a2b143f961a7818 (DEFAULT_TABLE table=default.Table_ac317bba2577464095cac3d5fe1374eb [id=96b19ffacfa24f118bf4ec4c52a22a82]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:39.125823 1030 tablet_service.cc:1467] Processing CreateTablet for tablet c75f5773e0ca4554807b302cec1db90a (DEFAULT_TABLE table=default.Table_de89db4ee495473993f7cb9709fcd706 [id=15538635367049c7adf6a6eb9f96e176]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:39.126164 1040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7adc82d54b4740ce9a2b143f961a7818. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:39.126461 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c75f5773e0ca4554807b302cec1db90a. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:39.127875 995 leader_election.cc:343] T 655b63ff61ff4f4aa08e0449df39352a P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Not found: Tablet not found: 655b63ff61ff4f4aa08e0449df39352a
I20250113 02:51:39.128324 2368 leader_election.cc:290] T 655b63ff61ff4f4aa08e0449df39352a P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
I20250113 02:51:39.102068 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca144327ca8d48d48a072caaff74212f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:39.130193 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c75f5773e0ca4554807b302cec1db90a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:39.131543 736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 790cc9449d1a47328854eeceb44f7718. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:39.132242 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd3935f168e74990892bfea011714dab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:39.136324 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7adc82d54b4740ce9a2b143f961a7818. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:39.137377 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 194b032bd6624ede85d6a3006811ce59. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:39.138581 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b0e7558f9d546b7a87c5728a5f14442. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:39.124933 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1057) took 2704 ms. Trace:
I20250113 02:51:39.139340 1195 rpcz_store.cc:276] 0113 02:51:36.420481 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:36.430453 (+ 9972us) service_pool.cc:225] Handling call
0113 02:51:38.993529 (+2563076us) ts_tablet_manager.cc:1145] Deleting tablet 655b63ff61ff4f4aa08e0449df39352a
0113 02:51:38.993542 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:39.005596 (+ 12054us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:39.100922 (+ 95326us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:39.124816 (+ 23894us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2562879}
I20250113 02:51:39.140218 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1056) took 2727 ms. Trace:
I20250113 02:51:39.140411 1195 rpcz_store.cc:276] 0113 02:51:36.412713 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:36.430734 (+ 18021us) service_pool.cc:225] Handling call
0113 02:51:39.139956 (+2709222us) ts_tablet_manager.cc:1145] Deleting tablet d7193e2799a54950902c4ceac65b663b
0113 02:51:39.140020 (+ 64us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:39.140177 (+ 157us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2709090}
I20250113 02:51:39.140952 1195 tablet_replica.cc:331] T ee240d838d4a4bbba97f8ed61f69de78 P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:39.141302 1195 raft_consensus.cc:2238] T ee240d838d4a4bbba97f8ed61f69de78 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:39.141585 1195 raft_consensus.cc:2267] T ee240d838d4a4bbba97f8ed61f69de78 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:39.142210 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet 655b63ff61ff4f4aa08e0449df39352a (table default.Table_c6c64517b49e4843ba63567a871d22bd [id=ec4bdc424404427589852603bb6d8112]) successfully deleted
W20250113 02:51:39.142455 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet d7193e2799a54950902c4ceac65b663b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d7193e2799a54950902c4ceac65b663b already in progress: creating tablet
I20250113 02:51:39.143827 1030 tablet_service.cc:1467] Processing CreateTablet for tablet bd3935f168e74990892bfea011714dab (DEFAULT_TABLE table=default.Table_bccd061fec0d4a9e91a3b126aec48c92 [id=87a475439c6b4c788ff5ce10485d91c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:39.144876 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd3935f168e74990892bfea011714dab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:39.148212 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5abfbac8a9ce44f59afbc4b21d1e8e87. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:39.151199 1041 tablet_service.cc:1467] Processing CreateTablet for tablet 9dd01e01d99b4856b10cf014c4c0c075 (DEFAULT_TABLE table=default.Table_9d1fda1b56c74fbb80a73da524dbbe7a [id=8ee30223b82d4befbc98eb980bc6b148]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:39.152021 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9dd01e01d99b4856b10cf014c4c0c075. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:39.153463 1191 tablet_bootstrap.cc:492] T fd0b8592997d4a61b8ca854ecbb67c55 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:39.153750 1191 ts_tablet_manager.cc:1397] T fd0b8592997d4a61b8ca854ecbb67c55 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.130s user 0.009s sys 0.005s
I20250113 02:51:39.157680 1024 tablet_service.cc:1467] Processing CreateTablet for tablet dfb6ef2ccc2d481c8ce65470d953d5a2 (DEFAULT_TABLE table=default.Table_c91c831eacb14ed9bbb303b6f7fcd6ca [id=1bd5367e7c85495c828f6a29143f144a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:39.158392 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dfb6ef2ccc2d481c8ce65470d953d5a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:39.159296 1195 ts_tablet_manager.cc:1905] T ee240d838d4a4bbba97f8ed61f69de78 P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:39.163127 1191 raft_consensus.cc:357] T fd0b8592997d4a61b8ca854ecbb67c55 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:39.164508 1191 raft_consensus.cc:383] T fd0b8592997d4a61b8ca854ecbb67c55 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:39.165011 1191 raft_consensus.cc:738] T fd0b8592997d4a61b8ca854ecbb67c55 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:39.166903 1191 consensus_queue.cc:260] T fd0b8592997d4a61b8ca854ecbb67c55 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:39.171675 1191 ts_tablet_manager.cc:1428] T fd0b8592997d4a61b8ca854ecbb67c55 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.018s user 0.001s sys 0.002s
I20250113 02:51:39.173465 1191 tablet_bootstrap.cc:492] T e570465a3d4046b1ac19a3a034fe91f1 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:39.148978 895 tablet_service.cc:1467] Processing CreateTablet for tablet 9dd01e01d99b4856b10cf014c4c0c075 (DEFAULT_TABLE table=default.Table_9d1fda1b56c74fbb80a73da524dbbe7a [id=8ee30223b82d4befbc98eb980bc6b148]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:39.163592 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7adc82d54b4740ce9a2b143f961a7818. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:39.183584 1196 rpcz_store.cc:276] 0113 02:51:37.582957 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:37.583007 (+ 50us) service_pool.cc:225] Handling call
0113 02:51:38.899281 (+1316274us) ts_tablet_manager.cc:1145] Deleting tablet e468b652f71f44e3945f8d919ac0e1a8
0113 02:51:38.899292 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:39.006351 (+107059us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:39.114694 (+108343us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:39.116405 (+ 1711us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1316091}
I20250113 02:51:39.184149 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9dd01e01d99b4856b10cf014c4c0c075. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:39.184314 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c75f5773e0ca4554807b302cec1db90a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:39.184491 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet e468b652f71f44e3945f8d919ac0e1a8 (table default.Table_c52b68d0d3d74003bf10bc8ff205d25b [id=caf9972c69f14619be22fb3a9e563b05]) successfully deleted
I20250113 02:51:39.184811 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dfb6ef2ccc2d481c8ce65470d953d5a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:39.190924 1191 tablet_bootstrap.cc:654] T e570465a3d4046b1ac19a3a034fe91f1 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:39.193912 2306 raft_consensus.cc:491] T ee240d838d4a4bbba97f8ed61f69de78 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:39.194109 2306 raft_consensus.cc:513] T ee240d838d4a4bbba97f8ed61f69de78 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:39.194852 2306 leader_election.cc:290] T ee240d838d4a4bbba97f8ed61f69de78 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
I20250113 02:51:39.196758 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd3935f168e74990892bfea011714dab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:39.196946 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 790cc9449d1a47328854eeceb44f7718. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:39.206039 2382 raft_consensus.cc:491] T acd83024592e431898e9675c17f50d4f P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:39.206204 2382 raft_consensus.cc:513] T acd83024592e431898e9675c17f50d4f P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:39.206862 2382 leader_election.cc:290] T acd83024592e431898e9675c17f50d4f P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:39.207459 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acd83024592e431898e9675c17f50d4f" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:39.208079 711 leader_election.cc:343] T acd83024592e431898e9675c17f50d4f P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:39.213642 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 960a7078181f4621a5ff4f3eb3050de3
I20250113 02:51:39.214460 1196 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37366 (request call id 1058) took 1619 ms. Trace:
I20250113 02:51:39.215212 1196 rpcz_store.cc:276] 0113 02:51:37.595195 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:37.595238 (+ 43us) service_pool.cc:225] Handling call
0113 02:51:39.214332 (+1619094us) ts_tablet_manager.cc:1145] Deleting tablet abc49f968ea94bdbb0443fba6864a04a
0113 02:51:39.214343 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:39.214440 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1618884}
W20250113 02:51:39.215718 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet abc49f968ea94bdbb0443fba6864a04a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet abc49f968ea94bdbb0443fba6864a04a already in progress: creating tablet
I20250113 02:51:39.215952 1191 tablet_bootstrap.cc:492] T e570465a3d4046b1ac19a3a034fe91f1 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:39.216125 1196 tablet_replica.cc:331] T 378d59d314a84ddb84cf2ab041c51cf1 P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:39.216321 1196 raft_consensus.cc:2238] T 378d59d314a84ddb84cf2ab041c51cf1 P df69f48656b44bb890f76587ee69cc0f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:39.216436 1191 ts_tablet_manager.cc:1397] T e570465a3d4046b1ac19a3a034fe91f1 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.043s user 0.001s sys 0.005s
I20250113 02:51:39.216570 1196 raft_consensus.cc:2267] T 378d59d314a84ddb84cf2ab041c51cf1 P df69f48656b44bb890f76587ee69cc0f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:39.217391 1196 ts_tablet_manager.cc:1905] T 378d59d314a84ddb84cf2ab041c51cf1 P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:39.217654 1191 raft_consensus.cc:357] T e570465a3d4046b1ac19a3a034fe91f1 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:39.218892 1191 raft_consensus.cc:383] T e570465a3d4046b1ac19a3a034fe91f1 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:39.219017 1191 raft_consensus.cc:738] T e570465a3d4046b1ac19a3a034fe91f1 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:39.219273 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet 960a7078181f4621a5ff4f3eb3050de3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:39 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:39.222589 2218 raft_consensus.cc:2743] T 80d31d5ead804ddaa032969e401a8f7e P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:39.222813 2218 raft_consensus.cc:491] T b5c1882125b5491ea79a29a6dac6771e P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:39.222965 2218 raft_consensus.cc:513] T b5c1882125b5491ea79a29a6dac6771e P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:39.223768 2218 leader_election.cc:290] T b5c1882125b5491ea79a29a6dac6771e P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
W20250113 02:51:39.224085 998 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:38467, user_credentials={real_user=slave}} blocked reactor thread for 214015us
I20250113 02:51:39.228032 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc533518faa24dffa3306ff092e8d507" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:39.228130 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc533518faa24dffa3306ff092e8d507" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f"
I20250113 02:51:39.228367 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "655b63ff61ff4f4aa08e0449df39352a" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:39.228655 2377 raft_consensus.cc:491] T 99c8046f5d124ca49899df365114c69f P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:39.228775 2377 raft_consensus.cc:513] T 99c8046f5d124ca49899df365114c69f P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:39.229437 2377 leader_election.cc:290] T 99c8046f5d124ca49899df365114c69f P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:39.229962 1194 tablet_bootstrap.cc:492] T f129bb3fbbfd492d98822be0a6626e44 P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:39.230098 1194 ts_tablet_manager.cc:1397] T f129bb3fbbfd492d98822be0a6626e44 P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.249s user 0.005s sys 0.016s
I20250113 02:51:39.230297 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee240d838d4a4bbba97f8ed61f69de78" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:39.230911 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_2e04cc940d734e3bbe72ae90690357d6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:39.231145 852 leader_election.cc:343] T 99c8046f5d124ca49899df365114c69f P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Not found: Tablet not found: 99c8046f5d124ca49899df365114c69f
I20250113 02:51:39.231375 454 catalog_manager.cc:5618] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587) does not have the latest schema for tablet 44d2c7d5334c476fac60fdd57e2c02b6 (table default.Table_6daec99b05744d0caed86e51e86f24c2 [id=2931f217041c46cc8ae52bd6ce8cfd17]). Expected version 1 got 0
W20250113 02:51:39.231470 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2e04cc940d734e3bbe72ae90690357d6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250113 02:51:39.231601 851 leader_election.cc:343] T 99c8046f5d124ca49899df365114c69f P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Not found: Tablet not found: 99c8046f5d124ca49899df365114c69f
I20250113 02:51:39.231702 851 leader_election.cc:304] T 99c8046f5d124ca49899df365114c69f P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df69f48656b44bb890f76587ee69cc0f; no voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca
I20250113 02:51:39.231627 1191 consensus_queue.cc:260] T e570465a3d4046b1ac19a3a034fe91f1 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:39.231163 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5c1882125b5491ea79a29a6dac6771e" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:39.232340 2377 raft_consensus.cc:2743] T 99c8046f5d124ca49899df365114c69f P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:39.232827 1192 tablet_bootstrap.cc:492] T f1e32e09a8244957a99bdfea5812ff31 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:39.232965 1192 ts_tablet_manager.cc:1397] T f1e32e09a8244957a99bdfea5812ff31 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.217s user 0.023s sys 0.032s
I20250113 02:51:39.235350 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acd83024592e431898e9675c17f50d4f" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:39.236061 1192 raft_consensus.cc:357] T f1e32e09a8244957a99bdfea5812ff31 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:39.236249 1192 raft_consensus.cc:383] T f1e32e09a8244957a99bdfea5812ff31 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:39.236301 1192 raft_consensus.cc:738] T f1e32e09a8244957a99bdfea5812ff31 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:39.236457 1192 consensus_queue.cc:260] T f1e32e09a8244957a99bdfea5812ff31 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:39.237243 895 tablet_service.cc:1467] Processing CreateTablet for tablet 5edebea1ccfa4c5d8bf038604b32f803 (DEFAULT_TABLE table=default.Table_41aed46571c449228b1f98fb1bae0b4d [id=9732d8c3b9e1439ca60be105328d33de]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:39.237685 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5edebea1ccfa4c5d8bf038604b32f803. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:39.239696 2382 raft_consensus.cc:491] T b38cd34d67834a3f9bf5b5636bb79217 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:39.239836 2382 raft_consensus.cc:513] T b38cd34d67834a3f9bf5b5636bb79217 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:39.240530 2382 leader_election.cc:290] T b38cd34d67834a3f9bf5b5636bb79217 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:39.241343 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b38cd34d67834a3f9bf5b5636bb79217" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:39.241807 711 leader_election.cc:343] T b38cd34d67834a3f9bf5b5636bb79217 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:39.243069 2382 raft_consensus.cc:491] T 283fc7f1c74e480c8a63c74d3ae9eb7d P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:39.243182 2382 raft_consensus.cc:513] T 283fc7f1c74e480c8a63c74d3ae9eb7d P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:39.243533 1035 tablet_service.cc:1467] Processing CreateTablet for tablet 3bb458e47a0b4a06ab23e6709bb0edef (DEFAULT_TABLE table=default.Table_2b473336a0b14ae6a2cff5109f945c4f [id=5edb11ef33a9419c9688334c45ac7903]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:39.243755 2382 leader_election.cc:290] T 283fc7f1c74e480c8a63c74d3ae9eb7d P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:39.244002 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3bb458e47a0b4a06ab23e6709bb0edef. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:39.244414 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "283fc7f1c74e480c8a63c74d3ae9eb7d" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:39.244808 1040 tablet_service.cc:1467] Processing CreateTablet for tablet 5edebea1ccfa4c5d8bf038604b32f803 (DEFAULT_TABLE table=default.Table_41aed46571c449228b1f98fb1bae0b4d [id=9732d8c3b9e1439ca60be105328d33de]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:39.245132 1040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5edebea1ccfa4c5d8bf038604b32f803. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:39.247015 883 tablet_service.cc:1467] Processing CreateTablet for tablet 3bb458e47a0b4a06ab23e6709bb0edef (DEFAULT_TABLE table=default.Table_2b473336a0b14ae6a2cff5109f945c4f [id=5edb11ef33a9419c9688334c45ac7903]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:39.247329 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3bb458e47a0b4a06ab23e6709bb0edef. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:39.248291 711 leader_election.cc:343] T 283fc7f1c74e480c8a63c74d3ae9eb7d P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
W20250113 02:51:39.248389 709 leader_election.cc:343] T acd83024592e431898e9675c17f50d4f P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:39.248473 709 leader_election.cc:304] T acd83024592e431898e9675c17f50d4f P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:39.249125 2227 raft_consensus.cc:2743] T acd83024592e431898e9675c17f50d4f P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250113 02:51:39.255556 998 leader_election.cc:343] T bc533518faa24dffa3306ff092e8d507 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
W20250113 02:51:39.255932 998 leader_election.cc:343] T bc533518faa24dffa3306ff092e8d507 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:39.256366 1192 ts_tablet_manager.cc:1428] T f1e32e09a8244957a99bdfea5812ff31 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.023s user 0.002s sys 0.000s
I20250113 02:51:39.257308 1192 tablet_bootstrap.cc:492] T bc6a7322c0c34adb85457750526d8955 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:39.260342 891 tablet_service.cc:1514] Processing DeleteTablet for tablet 960a7078181f4621a5ff4f3eb3050de3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:39 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:39.262084 1192 tablet_bootstrap.cc:654] T bc6a7322c0c34adb85457750526d8955 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:39.263760 2377 raft_consensus.cc:491] T 42cf1e708644427f9d4dcd41ae2bcb37 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:39.263882 2377 raft_consensus.cc:513] T 42cf1e708644427f9d4dcd41ae2bcb37 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:39.264482 2377 leader_election.cc:290] T 42cf1e708644427f9d4dcd41ae2bcb37 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
W20250113 02:51:39.265203 852 leader_election.cc:343] T 42cf1e708644427f9d4dcd41ae2bcb37 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Not found: Tablet not found: 42cf1e708644427f9d4dcd41ae2bcb37
W20250113 02:51:39.265362 851 leader_election.cc:343] T 42cf1e708644427f9d4dcd41ae2bcb37 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Not found: Tablet not found: 42cf1e708644427f9d4dcd41ae2bcb37
I20250113 02:51:39.265458 851 leader_election.cc:304] T 42cf1e708644427f9d4dcd41ae2bcb37 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df69f48656b44bb890f76587ee69cc0f; no voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca
I20250113 02:51:39.265872 2377 raft_consensus.cc:2743] T 42cf1e708644427f9d4dcd41ae2bcb37 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:39.265360 1194 raft_consensus.cc:357] T f129bb3fbbfd492d98822be0a6626e44 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:39.266582 1194 raft_consensus.cc:383] T f129bb3fbbfd492d98822be0a6626e44 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:39.264317 1191 ts_tablet_manager.cc:1428] T e570465a3d4046b1ac19a3a034fe91f1 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.047s user 0.000s sys 0.002s
I20250113 02:51:39.266819 1194 raft_consensus.cc:738] T f129bb3fbbfd492d98822be0a6626e44 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:39.267191 1191 tablet_bootstrap.cc:492] T 9f8eb43a39774d2c8e9cb4e88a405b0f P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
W20250113 02:51:39.259809 998 leader_election.cc:343] T 655b63ff61ff4f4aa08e0449df39352a P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:39.269163 998 leader_election.cc:304] T 655b63ff61ff4f4aa08e0449df39352a P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4216a7f853d642b7bc3e5cff458f3aca; no voters: 1186fa869eb14812832fa6562531f196, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:39.269452 2218 raft_consensus.cc:2743] T 655b63ff61ff4f4aa08e0449df39352a P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:39.270395 1040 tablet_service.cc:1467] Processing CreateTablet for tablet ab0eeacaa387474d8752934d5001adc6 (DEFAULT_TABLE table=default.Table_9e8c1b27c2f74c919d2e9224a1db676a [id=7ae5af0b630c452583e12f2592acf9c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:39.270741 1040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab0eeacaa387474d8752934d5001adc6. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:39.273115 998 leader_election.cc:343] T ee240d838d4a4bbba97f8ed61f69de78 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
W20250113 02:51:39.273347 998 leader_election.cc:343] T b5c1882125b5491ea79a29a6dac6771e P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:39.274081 1062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b38cd34d67834a3f9bf5b5636bb79217" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:39.274430 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "283fc7f1c74e480c8a63c74d3ae9eb7d" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:39.274816 1194 consensus_queue.cc:260] T f129bb3fbbfd492d98822be0a6626e44 P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:39.276849 2218 raft_consensus.cc:491] T 44d2c7d5334c476fac60fdd57e2c02b6 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:39.277112 2218 raft_consensus.cc:513] T 44d2c7d5334c476fac60fdd57e2c02b6 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:39.277931 2306 raft_consensus.cc:491] T 30ca8a716bfc4ab789b216e17567785e P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:39.278039 2306 raft_consensus.cc:513] T 30ca8a716bfc4ab789b216e17567785e P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:39.278745 2306 leader_election.cc:290] T 30ca8a716bfc4ab789b216e17567785e P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
I20250113 02:51:39.276294 891 tablet_service.cc:1467] Processing CreateTablet for tablet ab0eeacaa387474d8752934d5001adc6 (DEFAULT_TABLE table=default.Table_9e8c1b27c2f74c919d2e9224a1db676a [id=7ae5af0b630c452583e12f2592acf9c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:39.279537 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab0eeacaa387474d8752934d5001adc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:39.279577 2218 leader_election.cc:290] T 44d2c7d5334c476fac60fdd57e2c02b6 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
I20250113 02:51:39.281172 1042 tablet_service.cc:1467] Processing CreateTablet for tablet 546ee73435444c90ad286943807af984 (DEFAULT_TABLE table=default.Table_7698d677addf448fa62c659dad28b1e4 [id=870c8ed489464cb6bcec113842e2c49f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:39.281581 1042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 546ee73435444c90ad286943807af984. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:39.284205 890 tablet_service.cc:1467] Processing CreateTablet for tablet 546ee73435444c90ad286943807af984 (DEFAULT_TABLE table=default.Table_7698d677addf448fa62c659dad28b1e4 [id=870c8ed489464cb6bcec113842e2c49f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:39.287143 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30ca8a716bfc4ab789b216e17567785e" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:39.288839 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44d2c7d5334c476fac60fdd57e2c02b6" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:39.290952 998 leader_election.cc:343] T 30ca8a716bfc4ab789b216e17567785e P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
W20250113 02:51:39.292147 998 leader_election.cc:343] T 44d2c7d5334c476fac60fdd57e2c02b6 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
W20250113 02:51:39.294103 998 consensus_peers.cc:487] T 378d59d314a84ddb84cf2ab041c51cf1 P 4216a7f853d642b7bc3e5cff458f3aca -> Peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Couldn't send request to peer df69f48656b44bb890f76587ee69cc0f. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250113 02:51:39.294064 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee240d838d4a4bbba97f8ed61f69de78" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
I20250113 02:51:39.294664 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 546ee73435444c90ad286943807af984. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:39.293561 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1098) took 1777 ms. Trace:
I20250113 02:51:39.294826 1198 rpcz_store.cc:276] 0113 02:51:37.516098 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:37.516391 (+ 293us) service_pool.cc:225] Handling call
0113 02:51:38.736302 (+1219911us) ts_tablet_manager.cc:1145] Deleting tablet 0752fde3a30a4947a2a688c56ceadab5
0113 02:51:38.736317 (+ 15us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:38.997215 (+260898us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:39.019550 (+ 22335us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:39.293541 (+273991us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1199971}
I20250113 02:51:39.295440 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet 0752fde3a30a4947a2a688c56ceadab5 (table default.Table_da3e49fa1e654145a05858466ec2260f [id=94704429b49b46e38a48258a450d2705]) successfully deleted
I20250113 02:51:39.295899 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1104) took 1759 ms. Trace:
I20250113 02:51:39.296039 1198 rpcz_store.cc:276] 0113 02:51:37.536312 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:37.536679 (+ 367us) service_pool.cc:225] Handling call
0113 02:51:39.295793 (+1759114us) ts_tablet_manager.cc:1145] Deleting tablet 097cc4bd230046e2ac8551ca23d8f3e8
0113 02:51:39.295802 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:39.295886 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1758937}
I20250113 02:51:39.296311 1194 ts_tablet_manager.cc:1428] T f129bb3fbbfd492d98822be0a6626e44 P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.066s user 0.001s sys 0.001s
I20250113 02:51:39.296341 449 catalog_manager.cc:6071] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: default.Table_6a0195b054a043269e133f6ed8dd5979 [id=4b2bd999368c4b628c70cb3ee8c2b1f3] alter complete (version 1)
W20250113 02:51:39.296476 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 097cc4bd230046e2ac8551ca23d8f3e8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 097cc4bd230046e2ac8551ca23d8f3e8 already in progress: creating tablet
I20250113 02:51:39.296904 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1103) took 1779 ms. Trace:
I20250113 02:51:39.297034 1198 rpcz_store.cc:276] 0113 02:51:37.517685 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:37.517748 (+ 63us) service_pool.cc:225] Handling call
0113 02:51:39.296776 (+1779028us) ts_tablet_manager.cc:1145] Deleting tablet a1d6cbfb61564e929efab798d4681409
0113 02:51:39.296785 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:39.296890 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1752203}
W20250113 02:51:39.297437 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet a1d6cbfb61564e929efab798d4681409 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a1d6cbfb61564e929efab798d4681409 already in progress: creating tablet
I20250113 02:51:39.297765 1198 tablet_replica.cc:331] T 7213e3284b4d4cb581d9fb946a9f87e9 P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:39.297964 1198 raft_consensus.cc:2238] T 7213e3284b4d4cb581d9fb946a9f87e9 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:39.298131 1198 raft_consensus.cc:2267] T 7213e3284b4d4cb581d9fb946a9f87e9 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:39.298631 1198 ts_tablet_manager.cc:1905] T 7213e3284b4d4cb581d9fb946a9f87e9 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:39.299046 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_d6974c08d3a44b0d93d7eb4404b475b5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:39.299685 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d6974c08d3a44b0d93d7eb4404b475b5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:39.300964 2382 raft_consensus.cc:491] T 938cc1ccd8dd449996836188ba9e64ff P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:39.301095 2382 raft_consensus.cc:513] T 938cc1ccd8dd449996836188ba9e64ff P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:39.301746 2382 leader_election.cc:290] T 938cc1ccd8dd449996836188ba9e64ff P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:39.300915 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_c89db70b70a64f88a0203c8c66867911"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:39.302470 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c89db70b70a64f88a0203c8c66867911 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:39.302987 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "938cc1ccd8dd449996836188ba9e64ff" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:39.303465 711 leader_election.cc:343] T 938cc1ccd8dd449996836188ba9e64ff P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:39.304484 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5c1882125b5491ea79a29a6dac6771e" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
I20250113 02:51:39.304662 774 raft_consensus.cc:2463] T b5c1882125b5491ea79a29a6dac6771e P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 0.
I20250113 02:51:39.309363 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet f1e32e09a8244957a99bdfea5812ff31 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:25 UTC) from {username='slave'} at 127.0.0.1:60230
W20250113 02:51:39.309933 439 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 5d2d14548b734b1fbe57d68c0ea3e2f8 (table default.Table_854e525352dd47c889aaabfc312e33c7 [id=c0b628d230fd4eae83a20530cd998993], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 5d2d14548b734b1fbe57d68c0ea3e2f8 consensus state has no leader
I20250113 02:51:39.310082 1194 tablet_bootstrap.cc:492] T 56b022c94847411785a7e9db3757a2d9 P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:39.311702 1194 tablet_bootstrap.cc:654] T 56b022c94847411785a7e9db3757a2d9 P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:39.312184 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44d2c7d5334c476fac60fdd57e2c02b6" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
I20250113 02:51:39.312386 774 raft_consensus.cc:2463] T 44d2c7d5334c476fac60fdd57e2c02b6 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 0.
I20250113 02:51:39.313222 881 tablet_service.cc:1514] Processing DeleteTablet for tablet f129bb3fbbfd492d98822be0a6626e44 with delete_type TABLET_DATA_DELETED (replaced by tablet 67baaab1e36e415381579d7385d4ab5a) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:39.315323 754 tablet_service.cc:1514] Processing DeleteTablet for tablet 960a7078181f4621a5ff4f3eb3050de3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:39 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:39.315650 741 tablet_service.cc:1467] Processing CreateTablet for tablet 5edebea1ccfa4c5d8bf038604b32f803 (DEFAULT_TABLE table=default.Table_41aed46571c449228b1f98fb1bae0b4d [id=9732d8c3b9e1439ca60be105328d33de]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:39.316017 736 tablet_service.cc:1467] Processing CreateTablet for tablet dfb6ef2ccc2d481c8ce65470d953d5a2 (DEFAULT_TABLE table=default.Table_c91c831eacb14ed9bbb303b6f7fcd6ca [id=1bd5367e7c85495c828f6a29143f144a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:39.316370 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5edebea1ccfa4c5d8bf038604b32f803. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:39.317364 1191 tablet_bootstrap.cc:654] T 9f8eb43a39774d2c8e9cb4e88a405b0f P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:39.320645 881 tablet_service.cc:1467] Processing CreateTablet for tablet f405e1ae381348b093b2bb5e9f05be91 (DEFAULT_TABLE table=default.Table_e7e2f8efbe58458c968a5a649f789847 [id=a3b579edea134effb782efeba888385b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:39.320966 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f405e1ae381348b093b2bb5e9f05be91. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:39.322140 1031 tablet_service.cc:1467] Processing CreateTablet for tablet 0104fb13c40f44a6b26fb525ec26b3fc (DEFAULT_TABLE table=default.Table_9b1d5bafaf7b49599008dc137ff247b9 [id=3d2bbd97625740829561f62b0c294f0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:39.322274 1026 tablet_service.cc:1467] Processing CreateTablet for tablet 3e307ac1e6784abd821f2f0e0360738e (DEFAULT_TABLE table=default.Table_2ab49714961e452d8adfa6f8acb9be86 [id=c50eac8f23024cb8962c117bf3b62a7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:39.322624 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0104fb13c40f44a6b26fb525ec26b3fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:39.323060 1035 tablet_service.cc:1467] Processing CreateTablet for tablet f405e1ae381348b093b2bb5e9f05be91 (DEFAULT_TABLE table=default.Table_e7e2f8efbe58458c968a5a649f789847 [id=a3b579edea134effb782efeba888385b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:39.323426 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f405e1ae381348b093b2bb5e9f05be91. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:39.315716 744 tablet_service.cc:1467] Processing CreateTablet for tablet 9dd01e01d99b4856b10cf014c4c0c075 (DEFAULT_TABLE table=default.Table_9d1fda1b56c74fbb80a73da524dbbe7a [id=8ee30223b82d4befbc98eb980bc6b148]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:39.323696 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e307ac1e6784abd821f2f0e0360738e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:39.315966 751 tablet_service.cc:1467] Processing CreateTablet for tablet 3bb458e47a0b4a06ab23e6709bb0edef (DEFAULT_TABLE table=default.Table_2b473336a0b14ae6a2cff5109f945c4f [id=5edb11ef33a9419c9688334c45ac7903]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:39.324156 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3bb458e47a0b4a06ab23e6709bb0edef. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:39.322669 888 tablet_service.cc:1467] Processing CreateTablet for tablet 3e307ac1e6784abd821f2f0e0360738e (DEFAULT_TABLE table=default.Table_2ab49714961e452d8adfa6f8acb9be86 [id=c50eac8f23024cb8962c117bf3b62a7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:39.328469 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e307ac1e6784abd821f2f0e0360738e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:39.329774 738 tablet_service.cc:1467] Processing CreateTablet for tablet ab0eeacaa387474d8752934d5001adc6 (DEFAULT_TABLE table=default.Table_9e8c1b27c2f74c919d2e9224a1db676a [id=7ae5af0b630c452583e12f2592acf9c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:39.330097 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab0eeacaa387474d8752934d5001adc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:39.333648 1039 tablet_service.cc:1467] Processing CreateTablet for tablet 368e5f00861b40ffb76e79376245b885 (DEFAULT_TABLE table=default.Table_bb52873ae0724253b10583549f8f3596 [id=6b163dc88bb24653a4b738c1a2616ad2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:39.333993 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 368e5f00861b40ffb76e79376245b885. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:39.335901 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9dd01e01d99b4856b10cf014c4c0c075. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:39.336226 1191 tablet_bootstrap.cc:492] T 9f8eb43a39774d2c8e9cb4e88a405b0f P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:39.336282 882 tablet_service.cc:1467] Processing CreateTablet for tablet 368e5f00861b40ffb76e79376245b885 (DEFAULT_TABLE table=default.Table_bb52873ae0724253b10583549f8f3596 [id=6b163dc88bb24653a4b738c1a2616ad2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:39.336669 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 368e5f00861b40ffb76e79376245b885. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:39.336874 1191 ts_tablet_manager.cc:1397] T 9f8eb43a39774d2c8e9cb4e88a405b0f P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.070s user 0.001s sys 0.003s
I20250113 02:51:39.337877 1191 raft_consensus.cc:357] T 9f8eb43a39774d2c8e9cb4e88a405b0f P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:39.339670 1191 raft_consensus.cc:383] T 9f8eb43a39774d2c8e9cb4e88a405b0f P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:39.338590 751 tablet_service.cc:1514] Processing DeleteTablet for tablet 283fc7f1c74e480c8a63c74d3ae9eb7d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:24 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:39.340113 1191 raft_consensus.cc:738] T 9f8eb43a39774d2c8e9cb4e88a405b0f P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:39.340927 741 tablet_service.cc:1467] Processing CreateTablet for tablet f405e1ae381348b093b2bb5e9f05be91 (DEFAULT_TABLE table=default.Table_e7e2f8efbe58458c968a5a649f789847 [id=a3b579edea134effb782efeba888385b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:39.340154 736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dfb6ef2ccc2d481c8ce65470d953d5a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:39.341295 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f405e1ae381348b093b2bb5e9f05be91. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:39.344578 746 tablet_service.cc:1467] Processing CreateTablet for tablet 368e5f00861b40ffb76e79376245b885 (DEFAULT_TABLE table=default.Table_bb52873ae0724253b10583549f8f3596 [id=6b163dc88bb24653a4b738c1a2616ad2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:39.344923 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 368e5f00861b40ffb76e79376245b885. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:39.344877 995 leader_election.cc:343] T ee240d838d4a4bbba97f8ed61f69de78 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Illegal state: Tablet not RUNNING: STOPPED
I20250113 02:51:39.344993 995 leader_election.cc:304] T ee240d838d4a4bbba97f8ed61f69de78 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4216a7f853d642b7bc3e5cff458f3aca; no voters: 1186fa869eb14812832fa6562531f196, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:39.345517 995 leader_election.cc:304] T b5c1882125b5491ea79a29a6dac6771e P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters: df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:39.345968 2368 raft_consensus.cc:2743] T ee240d838d4a4bbba97f8ed61f69de78 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:39.346122 2368 raft_consensus.cc:2798] T b5c1882125b5491ea79a29a6dac6771e P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 02:51:39.346175 2368 raft_consensus.cc:491] T b5c1882125b5491ea79a29a6dac6771e P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 02:51:39.346249 2368 raft_consensus.cc:3054] T b5c1882125b5491ea79a29a6dac6771e P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:39.348421 754 tablet_service.cc:1467] Processing CreateTablet for tablet 3e307ac1e6784abd821f2f0e0360738e (DEFAULT_TABLE table=default.Table_2ab49714961e452d8adfa6f8acb9be86 [id=c50eac8f23024cb8962c117bf3b62a7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:39.348846 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e307ac1e6784abd821f2f0e0360738e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:39.348722 2368 raft_consensus.cc:513] T b5c1882125b5491ea79a29a6dac6771e P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:39.349400 2368 leader_election.cc:290] T b5c1882125b5491ea79a29a6dac6771e P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Requested vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
I20250113 02:51:39.349267 1191 consensus_queue.cc:260] T 9f8eb43a39774d2c8e9cb4e88a405b0f P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:39.350474 1191 ts_tablet_manager.cc:1428] T 9f8eb43a39774d2c8e9cb4e88a405b0f P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.013s user 0.001s sys 0.001s
W20250113 02:51:39.352016 995 leader_election.cc:343] T 30ca8a716bfc4ab789b216e17567785e P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Not found: Tablet not found: 30ca8a716bfc4ab789b216e17567785e
I20250113 02:51:39.352313 995 leader_election.cc:304] T 30ca8a716bfc4ab789b216e17567785e P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4216a7f853d642b7bc3e5cff458f3aca; no voters: 1186fa869eb14812832fa6562531f196, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:39.352774 995 leader_election.cc:304] T 44d2c7d5334c476fac60fdd57e2c02b6 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters: df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:39.353391 2306 raft_consensus.cc:2743] T 30ca8a716bfc4ab789b216e17567785e P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:39.353929 2306 raft_consensus.cc:2798] T 44d2c7d5334c476fac60fdd57e2c02b6 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 02:51:39.354012 2306 raft_consensus.cc:491] T 44d2c7d5334c476fac60fdd57e2c02b6 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 02:51:39.354080 2306 raft_consensus.cc:3054] T 44d2c7d5334c476fac60fdd57e2c02b6 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:39.354372 2375 tablet.cc:1721] T 9438472464304da9a0c7658d4f0736b6 P 4216a7f853d642b7bc3e5cff458f3aca: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250113 02:51:39.356498 2306 raft_consensus.cc:513] T 44d2c7d5334c476fac60fdd57e2c02b6 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:39.357007 2306 leader_election.cc:290] T 44d2c7d5334c476fac60fdd57e2c02b6 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Requested vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
W20250113 02:51:39.357352 709 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.3:42587, user_credentials={real_user=slave}} blocked reactor thread for 108955us
I20250113 02:51:39.357800 1191 tablet_bootstrap.cc:492] T 67baaab1e36e415381579d7385d4ab5a P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:39.358271 2382 raft_consensus.cc:491] T 6a4c6dc08df744e6856bdf99ab65c575 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:39.358371 2382 raft_consensus.cc:513] T 6a4c6dc08df744e6856bdf99ab65c575 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:39.360270 1042 tablet_service.cc:1467] Processing CreateTablet for tablet b48eea73b4d5429f9159ffffe94ae24a (DEFAULT_TABLE table=default.Table_e8c6d21d2c8d4abeaca3e0f79c0fc63d [id=d1203769a7424e9296b9931403f51cbe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:39.361625 2375 tablet.cc:1721] T 4fe54131aafb40ee9d8bc5ea299af193 P 4216a7f853d642b7bc3e5cff458f3aca: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20250113 02:51:39.366628 1042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b48eea73b4d5429f9159ffffe94ae24a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:39.379451 2227 raft_consensus.cc:491] T 1968c7e9357c4c6e901d334ace0467fb P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:39.380949 1196 ts_tablet_manager.cc:1918] T 378d59d314a84ddb84cf2ab041c51cf1 P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250113 02:51:39.381328 1192 tablet_bootstrap.cc:492] T bc6a7322c0c34adb85457750526d8955 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:39.381671 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "938cc1ccd8dd449996836188ba9e64ff" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:39.381759 1192 ts_tablet_manager.cc:1397] T bc6a7322c0c34adb85457750526d8955 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.125s user 0.002s sys 0.004s
I20250113 02:51:39.382542 2342 catalog_manager.cc:1260] Loaded cluster ID: 00860a399ece46318a55d1901f0159c4
I20250113 02:51:39.382681 1192 raft_consensus.cc:357] T bc6a7322c0c34adb85457750526d8955 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:39.383008 1196 log.cc:1198] T 378d59d314a84ddb84cf2ab041c51cf1 P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/378d59d314a84ddb84cf2ab041c51cf1
I20250113 02:51:39.383610 1196 ts_tablet_manager.cc:1939] T 378d59d314a84ddb84cf2ab041c51cf1 P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:39.383816 1192 raft_consensus.cc:383] T bc6a7322c0c34adb85457750526d8955 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:39.383906 2342 catalog_manager.cc:1553] T 00000000000000000000000000000000 P 45e9d43ecac0401fb2c18fe5eaf089b5: loading cluster ID for follower catalog manager: success
I20250113 02:51:39.384035 1192 raft_consensus.cc:738] T bc6a7322c0c34adb85457750526d8955 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:39.384446 1192 consensus_queue.cc:260] T bc6a7322c0c34adb85457750526d8955 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:39.385031 1041 tablet_service.cc:1467] Processing CreateTablet for tablet 332b2f5dad0047dc8fa4bd3e4053e96d (DEFAULT_TABLE table=default.Table_6a6779b59f8541b086c5d17a372fd0c0 [id=2a02d03acbe24e20aad16259f82ee736]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:39.385394 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 332b2f5dad0047dc8fa4bd3e4053e96d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:39.385879 1192 ts_tablet_manager.cc:1428] T bc6a7322c0c34adb85457750526d8955 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250113 02:51:39.386652 1192 tablet_bootstrap.cc:492] T c7842d6f2c074e3b9d1a3378db9856be P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:39.381266 895 tablet_service.cc:1467] Processing CreateTablet for tablet 0104fb13c40f44a6b26fb525ec26b3fc (DEFAULT_TABLE table=default.Table_9b1d5bafaf7b49599008dc137ff247b9 [id=3d2bbd97625740829561f62b0c294f0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:39.388512 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0104fb13c40f44a6b26fb525ec26b3fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:39.389076 2382 leader_election.cc:290] T 6a4c6dc08df744e6856bdf99ab65c575 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:39.390003 752 tablet_service.cc:1467] Processing CreateTablet for tablet 546ee73435444c90ad286943807af984 (DEFAULT_TABLE table=default.Table_7698d677addf448fa62c659dad28b1e4 [id=870c8ed489464cb6bcec113842e2c49f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:39.390758 890 tablet_service.cc:1467] Processing CreateTablet for tablet 332b2f5dad0047dc8fa4bd3e4053e96d (DEFAULT_TABLE table=default.Table_6a6779b59f8541b086c5d17a372fd0c0 [id=2a02d03acbe24e20aad16259f82ee736]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:39.390873 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5c1882125b5491ea79a29a6dac6771e" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f"
I20250113 02:51:39.391096 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 332b2f5dad0047dc8fa4bd3e4053e96d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:39.392851 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 546ee73435444c90ad286943807af984. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:39.395001 1192 tablet_bootstrap.cc:654] T c7842d6f2c074e3b9d1a3378db9856be P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:39.402037 1042 tablet_service.cc:1467] Processing CreateTablet for tablet 189dd9665f754a5d858dc9ffc227440d (DEFAULT_TABLE table=default.Table_39421f9a919a4fe0851a559994412d3a [id=581289610ec443d1916ef1daa0010f65]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:39.402449 1042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 189dd9665f754a5d858dc9ffc227440d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:39.405867 1196 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37366 (request call id 1059) took 1778 ms. Trace:
I20250113 02:51:39.406657 1196 rpcz_store.cc:276] 0113 02:51:37.627618 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:37.628019 (+ 401us) service_pool.cc:225] Handling call
0113 02:51:39.216065 (+1588046us) ts_tablet_manager.cc:1145] Deleting tablet 378d59d314a84ddb84cf2ab041c51cf1
0113 02:51:39.216082 (+ 17us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:39.221388 (+ 5306us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:39.380867 (+159479us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:39.405851 (+ 24984us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1587759}
I20250113 02:51:39.408221 1196 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37366 (request call id 1060) took 1767 ms. Trace:
I20250113 02:51:39.414533 1040 tablet_service.cc:1467] Processing CreateTablet for tablet 8833856830b9492191181984356d14f8 (DEFAULT_TABLE table=default.Table_d795516e6fbc46f5b2fe2b81042b1d62 [id=1fcb4fea0a4c4395ac474a2fb16c9ca9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:39.414892 1040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8833856830b9492191181984356d14f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:39.416870 1030 tablet_service.cc:1467] Processing CreateTablet for tablet 1c82cf9373d24fedad88b5df7d5a8b64 (DEFAULT_TABLE table=default.Table_3cf8774e45444063a6da1fec1b7ed5e2 [id=2b7cd9368495489da417f617bd3be7c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:39.417323 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c82cf9373d24fedad88b5df7d5a8b64. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:39.394131 2382 raft_consensus.cc:491] T 9f8eb43a39774d2c8e9cb4e88a405b0f P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:39.422348 2382 raft_consensus.cc:513] T 9f8eb43a39774d2c8e9cb4e88a405b0f P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:39.422997 2382 leader_election.cc:290] T 9f8eb43a39774d2c8e9cb4e88a405b0f P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:39.394857 751 tablet_service.cc:1467] Processing CreateTablet for tablet 0104fb13c40f44a6b26fb525ec26b3fc (DEFAULT_TABLE table=default.Table_9b1d5bafaf7b49599008dc137ff247b9 [id=3d2bbd97625740829561f62b0c294f0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:39.395565 2413 raft_consensus.cc:491] T c13e9a51c1e74b44b352bade78390ba5 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250113 02:51:39.395669 709 leader_election.cc:343] T b38cd34d67834a3f9bf5b5636bb79217 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:39.393815 2227 raft_consensus.cc:513] T 1968c7e9357c4c6e901d334ace0467fb P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:39.423847 2413 raft_consensus.cc:513] T c13e9a51c1e74b44b352bade78390ba5 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:39.424252 2227 leader_election.cc:290] T 1968c7e9357c4c6e901d334ace0467fb P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:39.424320 2413 leader_election.cc:290] T c13e9a51c1e74b44b352bade78390ba5 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:39.424916 709 leader_election.cc:304] T b38cd34d67834a3f9bf5b5636bb79217 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:39.425238 2413 raft_consensus.cc:2743] T b38cd34d67834a3f9bf5b5636bb79217 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250113 02:51:39.425724 709 leader_election.cc:343] T 283fc7f1c74e480c8a63c74d3ae9eb7d P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:39.425885 709 leader_election.cc:304] T 283fc7f1c74e480c8a63c74d3ae9eb7d P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:39.426319 2413 raft_consensus.cc:2743] T 283fc7f1c74e480c8a63c74d3ae9eb7d P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:39.427573 2423 tablet.cc:1721] T 9438472464304da9a0c7658d4f0736b6 P 1186fa869eb14812832fa6562531f196: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20250113 02:51:39.427800 709 leader_election.cc:343] T 938cc1ccd8dd449996836188ba9e64ff P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:39.427996 709 leader_election.cc:304] T 938cc1ccd8dd449996836188ba9e64ff P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:39.428284 2413 raft_consensus.cc:2743] T 938cc1ccd8dd449996836188ba9e64ff P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:39.429841 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a4c6dc08df744e6856bdf99ab65c575" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:39.390314 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a4c6dc08df744e6856bdf99ab65c575" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:39.430668 1024 tablet_service.cc:1467] Processing CreateTablet for tablet becee3bc8d824a6d94e58730416aa076 (DEFAULT_TABLE table=default.Table_80b52e8cafc549b1b530660a5fd2bb1d [id=413cdddd25cb4a80b5b9a32fba699a35]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:39.414459 1196 rpcz_store.cc:276] 0113 02:51:37.641073 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:37.641744 (+ 671us) service_pool.cc:225] Handling call
0113 02:51:39.408099 (+1766355us) ts_tablet_manager.cc:1145] Deleting tablet dff74d7c093b45eba7a47d8eceb0c1d1
0113 02:51:39.408108 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:39.408205 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1766182}
I20250113 02:51:39.431139 1196 tablet_replica.cc:331] T aa5c172a69bf4ca79999f4b58df08de8 P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:39.431388 1196 raft_consensus.cc:2238] T aa5c172a69bf4ca79999f4b58df08de8 P df69f48656b44bb890f76587ee69cc0f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:39.431603 1196 raft_consensus.cc:2267] T aa5c172a69bf4ca79999f4b58df08de8 P df69f48656b44bb890f76587ee69cc0f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:39.432348 1196 ts_tablet_manager.cc:1905] T aa5c172a69bf4ca79999f4b58df08de8 P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:39.390578 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44d2c7d5334c476fac60fdd57e2c02b6" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f"
W20250113 02:51:39.433023 711 leader_election.cc:343] T 6a4c6dc08df744e6856bdf99ab65c575 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:39.433640 2413 raft_consensus.cc:491] T aa1ebdb92a5f4c92a5195b614689c100 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:39.433756 2413 raft_consensus.cc:513] T aa1ebdb92a5f4c92a5195b614689c100 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:39.434327 2413 leader_election.cc:290] T aa1ebdb92a5f4c92a5195b614689c100 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
W20250113 02:51:39.434614 709 leader_election.cc:343] T 6a4c6dc08df744e6856bdf99ab65c575 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:39.434724 709 leader_election.cc:304] T 6a4c6dc08df744e6856bdf99ab65c575 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:39.435918 738 tablet_service.cc:1467] Processing CreateTablet for tablet 332b2f5dad0047dc8fa4bd3e4053e96d (DEFAULT_TABLE table=default.Table_6a6779b59f8541b086c5d17a372fd0c0 [id=2a02d03acbe24e20aad16259f82ee736]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:39.436560 2413 raft_consensus.cc:2743] T 6a4c6dc08df744e6856bdf99ab65c575 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:39.437397 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f8eb43a39774d2c8e9cb4e88a405b0f" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:39.437525 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1968c7e9357c4c6e901d334ace0467fb" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:39.437700 1062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa1ebdb92a5f4c92a5195b614689c100" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:39.437974 2368 consensus_queue.cc:1035] T f8a6eebb6b6d4b63903e68750e181707 P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 02:51:39.438102 743 tablet_service.cc:1467] Processing CreateTablet for tablet 8833856830b9492191181984356d14f8 (DEFAULT_TABLE table=default.Table_d795516e6fbc46f5b2fe2b81042b1d62 [id=1fcb4fea0a4c4395ac474a2fb16c9ca9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:39.438261 736 tablet_service.cc:1467] Processing CreateTablet for tablet b48eea73b4d5429f9159ffffe94ae24a (DEFAULT_TABLE table=default.Table_e8c6d21d2c8d4abeaca3e0f79c0fc63d [id=d1203769a7424e9296b9931403f51cbe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:39.438556 747 tablet_service.cc:1467] Processing CreateTablet for tablet 189dd9665f754a5d858dc9ffc227440d (DEFAULT_TABLE table=default.Table_39421f9a919a4fe0851a559994412d3a [id=581289610ec443d1916ef1daa0010f65]), partition=RANGE (key) PARTITION UNBOUNDED
W20250113 02:51:39.438611 998 leader_election.cc:343] T b5c1882125b5491ea79a29a6dac6771e P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:39.438584 886 tablet_service.cc:1467] Processing CreateTablet for tablet becee3bc8d824a6d94e58730416aa076 (DEFAULT_TABLE table=default.Table_80b52e8cafc549b1b530660a5fd2bb1d [id=413cdddd25cb4a80b5b9a32fba699a35]), partition=RANGE (key) PARTITION UNBOUNDED
W20250113 02:51:39.438874 998 leader_election.cc:343] T 44d2c7d5334c476fac60fdd57e2c02b6 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:39.438910 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet becee3bc8d824a6d94e58730416aa076. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:39.438889 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet 378d59d314a84ddb84cf2ab041c51cf1 (table default.Table_49dafeaa3df04cb789a3e54f469973e8 [id=0c01441cec7043b687d6744e72a0e45d]) successfully deleted
W20250113 02:51:39.439142 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet dff74d7c093b45eba7a47d8eceb0c1d1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dff74d7c093b45eba7a47d8eceb0c1d1 already in progress: creating tablet
I20250113 02:51:39.439245 891 tablet_service.cc:1467] Processing CreateTablet for tablet 1c82cf9373d24fedad88b5df7d5a8b64 (DEFAULT_TABLE table=default.Table_3cf8774e45444063a6da1fec1b7ed5e2 [id=2b7cd9368495489da417f617bd3be7c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:39.439579 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c82cf9373d24fedad88b5df7d5a8b64. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:39.439710 748 tablet_service.cc:1467] Processing CreateTablet for tablet becee3bc8d824a6d94e58730416aa076 (DEFAULT_TABLE table=default.Table_80b52e8cafc549b1b530660a5fd2bb1d [id=413cdddd25cb4a80b5b9a32fba699a35]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:39.440194 742 tablet_service.cc:1467] Processing CreateTablet for tablet 1c82cf9373d24fedad88b5df7d5a8b64 (DEFAULT_TABLE table=default.Table_3cf8774e45444063a6da1fec1b7ed5e2 [id=2b7cd9368495489da417f617bd3be7c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:39.441861 888 tablet_service.cc:1467] Processing CreateTablet for tablet b48eea73b4d5429f9159ffffe94ae24a (DEFAULT_TABLE table=default.Table_e8c6d21d2c8d4abeaca3e0f79c0fc63d [id=d1203769a7424e9296b9931403f51cbe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:39.442189 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b48eea73b4d5429f9159ffffe94ae24a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:39.443898 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 332b2f5dad0047dc8fa4bd3e4053e96d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:39.445294 890 tablet_service.cc:1467] Processing CreateTablet for tablet 189dd9665f754a5d858dc9ffc227440d (DEFAULT_TABLE table=default.Table_39421f9a919a4fe0851a559994412d3a [id=581289610ec443d1916ef1daa0010f65]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:39.445813 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 189dd9665f754a5d858dc9ffc227440d. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:39.446035 711 leader_election.cc:343] T 9f8eb43a39774d2c8e9cb4e88a405b0f P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
W20250113 02:51:39.446377 711 leader_election.cc:343] T 1968c7e9357c4c6e901d334ace0467fb P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
W20250113 02:51:39.446962 709 leader_election.cc:343] T aa1ebdb92a5f4c92a5195b614689c100 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:39.398816 1195 ts_tablet_manager.cc:1918] T ee240d838d4a4bbba97f8ed61f69de78 P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:39.399546 1191 tablet_bootstrap.cc:654] T 67baaab1e36e415381579d7385d4ab5a P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:39.438812 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c13e9a51c1e74b44b352bade78390ba5" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:39.414340 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44d2c7d5334c476fac60fdd57e2c02b6" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196"
I20250113 02:51:39.448738 775 raft_consensus.cc:3054] T 44d2c7d5334c476fac60fdd57e2c02b6 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:39.448792 899 tablet_service.cc:1467] Processing CreateTablet for tablet 8833856830b9492191181984356d14f8 (DEFAULT_TABLE table=default.Table_d795516e6fbc46f5b2fe2b81042b1d62 [id=1fcb4fea0a4c4395ac474a2fb16c9ca9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:39.449162 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8833856830b9492191181984356d14f8. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:39.450904 998 consensus_peers.cc:487] T f8a6eebb6b6d4b63903e68750e181707 P 4216a7f853d642b7bc3e5cff458f3aca -> Peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Couldn't send request to peer df69f48656b44bb890f76587ee69cc0f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250113 02:51:39.451331 775 raft_consensus.cc:2463] T 44d2c7d5334c476fac60fdd57e2c02b6 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 1.
I20250113 02:51:39.415166 2378 tablet.cc:1721] T 4fe54131aafb40ee9d8bc5ea299af193 P 1186fa869eb14812832fa6562531f196: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20250113 02:51:39.416586 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5c1882125b5491ea79a29a6dac6771e" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196"
I20250113 02:51:39.452061 771 raft_consensus.cc:3054] T b5c1882125b5491ea79a29a6dac6771e P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:39.452011 883 tablet_service.cc:1467] Processing CreateTablet for tablet cd0d96a29efc4f85887afdbed6d48e64 (DEFAULT_TABLE table=default.Table_ccac11f0e0014fe7a1277b59017d92e4 [id=6f7c6844697e4853af14297e5c0e2007]), partition=RANGE (key) PARTITION UNBOUNDED
W20250113 02:51:39.452498 998 consensus_peers.cc:487] T d9df314ffaf042ec80180868d7e7bdf7 P 4216a7f853d642b7bc3e5cff458f3aca -> Peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Couldn't send request to peer df69f48656b44bb890f76587ee69cc0f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250113 02:51:39.454437 771 raft_consensus.cc:2463] T b5c1882125b5491ea79a29a6dac6771e P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 1.
I20250113 02:51:39.454802 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cd0d96a29efc4f85887afdbed6d48e64. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:39.458477 2378 tablet.cc:1721] T d304b65eba8c40c6922333506a60540c P 1186fa869eb14812832fa6562531f196: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250113 02:51:39.464033 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa1ebdb92a5f4c92a5195b614689c100" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:39.464452 711 leader_election.cc:343] T c13e9a51c1e74b44b352bade78390ba5 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
W20250113 02:51:39.464829 711 leader_election.cc:343] T aa1ebdb92a5f4c92a5195b614689c100 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:39.464926 711 leader_election.cc:304] T aa1ebdb92a5f4c92a5195b614689c100 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:39.465286 2421 raft_consensus.cc:2743] T aa1ebdb92a5f4c92a5195b614689c100 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:39.468182 883 tablet_service.cc:1467] Processing CreateTablet for tablet 0e2d8726a2ab489cae80fb1685d6951a (DEFAULT_TABLE table=default.Table_8a3f2660341f4712aa264ffc3e3a9a0d [id=5b5c93f281ee42a59d3e7b3e94b740cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:39.472268 1195 log.cc:1198] T ee240d838d4a4bbba97f8ed61f69de78 P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/ee240d838d4a4bbba97f8ed61f69de78
I20250113 02:51:39.477291 1195 ts_tablet_manager.cc:1939] T ee240d838d4a4bbba97f8ed61f69de78 P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:39.497197 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 16f547e3cec54baf86b96ad40e3afb6a
I20250113 02:51:39.497839 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 128f74fdc98f4135b39dbf23f1d987b0
I20250113 02:51:39.498749 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet e5ed7016af184b86be4c234f42e2a6ef
I20250113 02:51:39.500053 453 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet 958f4791a98948ddbe88b98013894f0a
I20250113 02:51:39.481196 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8833856830b9492191181984356d14f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:39.512277 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c82cf9373d24fedad88b5df7d5a8b64. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:39.512472 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 189dd9665f754a5d858dc9ffc227440d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:39.512751 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet becee3bc8d824a6d94e58730416aa076. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:39.512928 736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b48eea73b4d5429f9159ffffe94ae24a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:39.516084 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0104fb13c40f44a6b26fb525ec26b3fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:39.520763 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1058) took 3100 ms. Trace:
W20250113 02:51:39.521072 998 consensus_peers.cc:487] T aa5c172a69bf4ca79999f4b58df08de8 P 4216a7f853d642b7bc3e5cff458f3aca -> Peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Couldn't send request to peer df69f48656b44bb890f76587ee69cc0f. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250113 02:51:39.524132 881 tablet_service.cc:1514] Processing DeleteTablet for tablet 128f74fdc98f4135b39dbf23f1d987b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:39 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:39.524363 891 tablet_service.cc:1514] Processing DeleteTablet for tablet e5ed7016af184b86be4c234f42e2a6ef with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:39 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:39.531687 995 leader_election.cc:304] T 44d2c7d5334c476fac60fdd57e2c02b6 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters: df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:39.532258 2363 raft_consensus.cc:2798] T 44d2c7d5334c476fac60fdd57e2c02b6 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Leader election won for term 1
I20250113 02:51:39.532393 2363 raft_consensus.cc:695] T 44d2c7d5334c476fac60fdd57e2c02b6 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: Becoming Leader. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Running, Role: LEADER
I20250113 02:51:39.532747 890 tablet_service.cc:1467] Processing CreateTablet for tablet 6c4868fe133043169238e2d67ea7276b (DEFAULT_TABLE table=default.Table_854e525352dd47c889aaabfc312e33c7 [id=c0b628d230fd4eae83a20530cd998993]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:39.533260 995 leader_election.cc:304] T b5c1882125b5491ea79a29a6dac6771e P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters: df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:39.533675 2368 raft_consensus.cc:2798] T b5c1882125b5491ea79a29a6dac6771e P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Leader election won for term 1
I20250113 02:51:39.533783 2368 raft_consensus.cc:695] T b5c1882125b5491ea79a29a6dac6771e P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: Becoming Leader. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Running, Role: LEADER
I20250113 02:51:39.534054 2368 consensus_queue.cc:237] T b5c1882125b5491ea79a29a6dac6771e P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:39.538516 736 tablet_service.cc:1467] Processing CreateTablet for tablet cd0d96a29efc4f85887afdbed6d48e64 (DEFAULT_TABLE table=default.Table_ccac11f0e0014fe7a1277b59017d92e4 [id=6f7c6844697e4853af14297e5c0e2007]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:39.538579 751 tablet_service.cc:1467] Processing CreateTablet for tablet 0e2d8726a2ab489cae80fb1685d6951a (DEFAULT_TABLE table=default.Table_8a3f2660341f4712aa264ffc3e3a9a0d [id=5b5c93f281ee42a59d3e7b3e94b740cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:39.538642 747 tablet_service.cc:1467] Processing CreateTablet for tablet e5ed7016af184b86be4c234f42e2a6ef (DEFAULT_TABLE table=default.Table_56ac841ab1804eafbd61ea8fa855824f [id=0fd27b71c4b64352a5d0ea764e489107]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:39.539108 742 tablet_service.cc:1467] Processing CreateTablet for tablet 104d491605f64657a936174f5a889ffd (DEFAULT_TABLE table=default.Table_0820d78702cd41a3bfa9b7544842d8d4 [id=63464d10972f4bab8454fb09eb7a029b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:39.545156 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5ed7016af184b86be4c234f42e2a6ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:39.539353 743 tablet_service.cc:1514] Processing DeleteTablet for tablet 16f547e3cec54baf86b96ad40e3afb6a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:39 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:39.545329 736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cd0d96a29efc4f85887afdbed6d48e64. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:39.539816 741 tablet_service.cc:1514] Processing DeleteTablet for tablet e5ed7016af184b86be4c234f42e2a6ef with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:39 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:39.544661 1195 rpcz_store.cc:276] 0113 02:51:36.420572 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:36.433067 (+ 12495us) service_pool.cc:225] Handling call
0113 02:51:39.140849 (+2707782us) ts_tablet_manager.cc:1145] Deleting tablet ee240d838d4a4bbba97f8ed61f69de78
0113 02:51:39.140883 (+ 34us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:39.267016 (+126133us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:39.398708 (+131692us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:39.520742 (+122034us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2707628}
I20250113 02:51:39.545599 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 104d491605f64657a936174f5a889ffd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:39.545717 1195 tablet_replica.cc:331] T d9df314ffaf042ec80180868d7e7bdf7 P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:39.545917 1195 raft_consensus.cc:2238] T d9df314ffaf042ec80180868d7e7bdf7 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:39.546085 1195 raft_consensus.cc:2267] T d9df314ffaf042ec80180868d7e7bdf7 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:39.548065 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e2d8726a2ab489cae80fb1685d6951a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:39.429895 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f8eb43a39774d2c8e9cb4e88a405b0f" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:39.554319 1060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1968c7e9357c4c6e901d334ace0467fb" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:39.555480 1026 tablet_service.cc:1467] Processing CreateTablet for tablet e5ed7016af184b86be4c234f42e2a6ef (DEFAULT_TABLE table=default.Table_56ac841ab1804eafbd61ea8fa855824f [id=0fd27b71c4b64352a5d0ea764e489107]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:39.555842 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5ed7016af184b86be4c234f42e2a6ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:39.556752 1198 ts_tablet_manager.cc:1918] T 7213e3284b4d4cb581d9fb946a9f87e9 P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:39.556849 1198 log.cc:1198] T 7213e3284b4d4cb581d9fb946a9f87e9 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/7213e3284b4d4cb581d9fb946a9f87e9
I20250113 02:51:39.557139 1198 ts_tablet_manager.cc:1939] T 7213e3284b4d4cb581d9fb946a9f87e9 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:39.557906 1041 tablet_service.cc:1467] Processing CreateTablet for tablet cd0d96a29efc4f85887afdbed6d48e64 (DEFAULT_TABLE table=default.Table_ccac11f0e0014fe7a1277b59017d92e4 [id=6f7c6844697e4853af14297e5c0e2007]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:39.558384 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cd0d96a29efc4f85887afdbed6d48e64. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:39.560530 709 leader_election.cc:343] T 9f8eb43a39774d2c8e9cb4e88a405b0f P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:39.560664 709 leader_election.cc:304] T 9f8eb43a39774d2c8e9cb4e88a405b0f P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:39.532688 2363 consensus_queue.cc:237] T 44d2c7d5334c476fac60fdd57e2c02b6 P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:39.560927 2421 raft_consensus.cc:2743] T 9f8eb43a39774d2c8e9cb4e88a405b0f P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:39.430668 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c13e9a51c1e74b44b352bade78390ba5" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
W20250113 02:51:39.561376 709 leader_election.cc:343] T 1968c7e9357c4c6e901d334ace0467fb P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:39.561503 709 leader_election.cc:304] T 1968c7e9357c4c6e901d334ace0467fb P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:39.561755 2421 raft_consensus.cc:2743] T 1968c7e9357c4c6e901d334ace0467fb P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250113 02:51:39.562198 709 leader_election.cc:343] T c13e9a51c1e74b44b352bade78390ba5 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:39.562337 709 leader_election.cc:304] T c13e9a51c1e74b44b352bade78390ba5 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:39.562600 2421 raft_consensus.cc:2743] T c13e9a51c1e74b44b352bade78390ba5 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:39.540141 754 tablet_service.cc:1467] Processing CreateTablet for tablet 6c4868fe133043169238e2d67ea7276b (DEFAULT_TABLE table=default.Table_854e525352dd47c889aaabfc312e33c7 [id=c0b628d230fd4eae83a20530cd998993]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:39.563268 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet becee3bc8d824a6d94e58730416aa076. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:39.563431 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c4868fe133043169238e2d67ea7276b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:39.566799 1191 tablet_bootstrap.cc:492] T 67baaab1e36e415381579d7385d4ab5a P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:39.566931 1191 ts_tablet_manager.cc:1397] T 67baaab1e36e415381579d7385d4ab5a P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.209s user 0.001s sys 0.002s
I20250113 02:51:39.566802 748 tablet_service.cc:1514] Processing DeleteTablet for tablet 128f74fdc98f4135b39dbf23f1d987b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:39 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:39.567632 1191 raft_consensus.cc:357] T 67baaab1e36e415381579d7385d4ab5a P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:39.567795 1191 raft_consensus.cc:383] T 67baaab1e36e415381579d7385d4ab5a P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:39.567836 1191 raft_consensus.cc:738] T 67baaab1e36e415381579d7385d4ab5a P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:39.568035 1191 consensus_queue.cc:260] T 67baaab1e36e415381579d7385d4ab5a P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:39.568542 1191 ts_tablet_manager.cc:1428] T 67baaab1e36e415381579d7385d4ab5a P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250113 02:51:39.568815 1191 tablet_bootstrap.cc:492] T 21bcdd741104474bad7023426856d18f P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:39.568994 1037 tablet_service.cc:1514] Processing DeleteTablet for tablet e5ed7016af184b86be4c234f42e2a6ef with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:39 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:39.569366 888 tablet_service.cc:1467] Processing CreateTablet for tablet e5ed7016af184b86be4c234f42e2a6ef (DEFAULT_TABLE table=default.Table_56ac841ab1804eafbd61ea8fa855824f [id=0fd27b71c4b64352a5d0ea764e489107]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:39.569922 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c4868fe133043169238e2d67ea7276b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:39.570155 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1095) took 2054 ms. Trace:
I20250113 02:51:39.570389 1191 tablet_bootstrap.cc:654] T 21bcdd741104474bad7023426856d18f P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:39.570238 1198 rpcz_store.cc:276] 0113 02:51:37.515582 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:37.515627 (+ 45us) service_pool.cc:225] Handling call
0113 02:51:39.297725 (+1782098us) ts_tablet_manager.cc:1145] Deleting tablet 7213e3284b4d4cb581d9fb946a9f87e9
0113 02:51:39.297734 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:39.311970 (+ 14236us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:39.556668 (+244698us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:39.570136 (+ 13468us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1746566}
I20250113 02:51:39.570560 1198 tablet_replica.cc:331] T aa5c172a69bf4ca79999f4b58df08de8 P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:39.570750 1198 raft_consensus.cc:2238] T aa5c172a69bf4ca79999f4b58df08de8 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: Raft consensus shutting down.
I20250113 02:51:39.571080 1198 raft_consensus.cc:2267] T aa5c172a69bf4ca79999f4b58df08de8 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:39.571712 1198 ts_tablet_manager.cc:1905] T aa5c172a69bf4ca79999f4b58df08de8 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:39.572235 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet 16f547e3cec54baf86b96ad40e3afb6a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:39 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:39.573405 1195 ts_tablet_manager.cc:1905] T d9df314ffaf042ec80180868d7e7bdf7 P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:39.576047 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5ed7016af184b86be4c234f42e2a6ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:39.577555 886 tablet_service.cc:1514] Processing DeleteTablet for tablet 16f547e3cec54baf86b96ad40e3afb6a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:39 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:39.577719 895 tablet_service.cc:1467] Processing CreateTablet for tablet 104d491605f64657a936174f5a889ffd (DEFAULT_TABLE table=default.Table_0820d78702cd41a3bfa9b7544842d8d4 [id=63464d10972f4bab8454fb09eb7a029b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:39.578178 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 104d491605f64657a936174f5a889ffd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:39.586068 1039 tablet_service.cc:1467] Processing CreateTablet for tablet 0e2d8726a2ab489cae80fb1685d6951a (DEFAULT_TABLE table=default.Table_8a3f2660341f4712aa264ffc3e3a9a0d [id=5b5c93f281ee42a59d3e7b3e94b740cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:39.586359 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet 128f74fdc98f4135b39dbf23f1d987b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:39 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:39.590031 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e2d8726a2ab489cae80fb1685d6951a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:39.592223 1042 tablet_service.cc:1467] Processing CreateTablet for tablet 104d491605f64657a936174f5a889ffd (DEFAULT_TABLE table=default.Table_0820d78702cd41a3bfa9b7544842d8d4 [id=63464d10972f4bab8454fb09eb7a029b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:39.593627 1042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 104d491605f64657a936174f5a889ffd. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:39.593907 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e0cdc5e4f4ca4f30b634e4dd2bf862d9 consensus state has no leader
I20250113 02:51:39.595481 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet 7213e3284b4d4cb581d9fb946a9f87e9 (table default.Table_ff6df4afcf4d4f9a964d079ee24f02b5 [id=8a5901fc685345579a254a0f3af32051]) successfully deleted
I20250113 02:51:39.596474 1031 tablet_service.cc:1467] Processing CreateTablet for tablet 6c4868fe133043169238e2d67ea7276b (DEFAULT_TABLE table=default.Table_854e525352dd47c889aaabfc312e33c7 [id=c0b628d230fd4eae83a20530cd998993]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:39.596666 2306 raft_consensus.cc:491] T 1014e51f1c184d729552a5b34292f232 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:39.596808 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c4868fe133043169238e2d67ea7276b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:39.596802 2306 raft_consensus.cc:513] T 1014e51f1c184d729552a5b34292f232 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:39.597391 2375 tablet.cc:1721] T d304b65eba8c40c6922333506a60540c P 4216a7f853d642b7bc3e5cff458f3aca: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250113 02:51:39.597623 2306 leader_election.cc:290] T 1014e51f1c184d729552a5b34292f232 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
I20250113 02:51:39.599663 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1014e51f1c184d729552a5b34292f232" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:39.600375 998 leader_election.cc:343] T 1014e51f1c184d729552a5b34292f232 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:39.612816 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet ee240d838d4a4bbba97f8ed61f69de78 (table default.Table_1b71e9ea2fca48daa60e8d692eab8e49 [id=383148ed836144c1bc5403dae46aad50]) successfully deleted
I20250113 02:51:39.614140 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_3b79620cd21744a28c4292aaebc54307"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:39.614804 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3b79620cd21744a28c4292aaebc54307 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:39.614737 2427 tablet.cc:1721] T d304b65eba8c40c6922333506a60540c P df69f48656b44bb890f76587ee69cc0f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250113 02:51:39.615000 1198 ts_tablet_manager.cc:1918] T aa5c172a69bf4ca79999f4b58df08de8 P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250113 02:51:39.615097 1198 log.cc:1198] T aa5c172a69bf4ca79999f4b58df08de8 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/aa5c172a69bf4ca79999f4b58df08de8
I20250113 02:51:39.615432 1198 ts_tablet_manager.cc:1939] T aa5c172a69bf4ca79999f4b58df08de8 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:39.616173 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1108) took 2023 ms. Trace:
I20250113 02:51:39.616267 1198 rpcz_store.cc:276] 0113 02:51:37.593159 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:37.593218 (+ 59us) service_pool.cc:225] Handling call
0113 02:51:39.570513 (+1977295us) ts_tablet_manager.cc:1145] Deleting tablet aa5c172a69bf4ca79999f4b58df08de8
0113 02:51:39.570526 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:39.588952 (+ 18426us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:39.614925 (+ 25973us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:39.616153 (+ 1228us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1905942}
I20250113 02:51:39.616657 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1109) took 2022 ms. Trace:
I20250113 02:51:39.616694 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet aa5c172a69bf4ca79999f4b58df08de8 (table default.Table_3f1e7146fed34d53896f138e43a4d26e [id=5dceb76e1dbe422ab98d4da5bf6c45a0]) successfully deleted
I20250113 02:51:39.616608 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_6c3417d70cd4422993f46368566b2d51"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:39.616729 1198 rpcz_store.cc:276] 0113 02:51:37.594184 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:37.594239 (+ 55us) service_pool.cc:225] Handling call
0113 02:51:39.616538 (+2022299us) ts_tablet_manager.cc:1145] Deleting tablet abc49f968ea94bdbb0443fba6864a04a
0113 02:51:39.616547 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:39.616643 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1951864}
I20250113 02:51:39.616981 1198 tablet_replica.cc:331] T 378d59d314a84ddb84cf2ab041c51cf1 P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
W20250113 02:51:39.617094 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6c3417d70cd4422993f46368566b2d51 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250113 02:51:39.617134 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet abc49f968ea94bdbb0443fba6864a04a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet abc49f968ea94bdbb0443fba6864a04a already in progress: creating tablet
I20250113 02:51:39.617147 1198 raft_consensus.cc:2238] T 378d59d314a84ddb84cf2ab041c51cf1 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: Raft consensus shutting down.
I20250113 02:51:39.617442 1198 raft_consensus.cc:2267] T 378d59d314a84ddb84cf2ab041c51cf1 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:39.617738 895 tablet_service.cc:1467] Processing CreateTablet for tablet a66c6636f9f2492684b975cbfcb7530e (DEFAULT_TABLE table=default.Table_923d4e514de74882b7cb333bee6c14b5 [id=15a8a374fa64473289ac0b51149149ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:39.618093 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a66c6636f9f2492684b975cbfcb7530e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:39.618371 1198 ts_tablet_manager.cc:1905] T 378d59d314a84ddb84cf2ab041c51cf1 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:39.618891 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_2d9099633e344b5886574d99e90fe2b7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:39.619436 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2d9099633e344b5886574d99e90fe2b7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:39.620848 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_96b4ae94f08f485695f19b9d92f92bec" } new_table_name: "default.Table_0894b27199a241e594a6f314094d0010" modify_external_catalogs: true
I20250113 02:51:39.622696 448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_345d3448247f4d818397c10f9e3c96de" } new_table_name: "default.Table_5dea789d6bb64579bde4b03309ab08a7" modify_external_catalogs: true
I20250113 02:51:39.626989 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_d70d62ca846e4071af60ad816ded897f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:39.627743 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d70d62ca846e4071af60ad816ded897f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:39.636876 1192 tablet_bootstrap.cc:492] T c7842d6f2c074e3b9d1a3378db9856be P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:39.637081 1192 ts_tablet_manager.cc:1397] T c7842d6f2c074e3b9d1a3378db9856be P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.250s user 0.008s sys 0.021s
I20250113 02:51:39.639400 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_7843edf8e2e14e51b6ab66031605949e" } new_table_name: "default.Table_dae13183fa724859bf379288d5a35ded" modify_external_catalogs: true
I20250113 02:51:39.642372 454 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_685277f86f8f48fc9577c4adef84e006" } new_table_name: "default.Table_25cb4fc43dc6485ca0e4ec0cf048105d" modify_external_catalogs: true
I20250113 02:51:39.646778 1192 raft_consensus.cc:357] T c7842d6f2c074e3b9d1a3378db9856be P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:39.647040 1192 raft_consensus.cc:383] T c7842d6f2c074e3b9d1a3378db9856be P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:39.647095 1192 raft_consensus.cc:738] T c7842d6f2c074e3b9d1a3378db9856be P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:39.647305 1192 consensus_queue.cc:260] T c7842d6f2c074e3b9d1a3378db9856be P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:39.648313 1192 ts_tablet_manager.cc:1428] T c7842d6f2c074e3b9d1a3378db9856be P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.011s user 0.001s sys 0.002s
I20250113 02:51:39.650408 1192 tablet_bootstrap.cc:492] T c518f791b4d74c54ab0c0364d29b4a64 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:39.651033 2421 raft_consensus.cc:491] T fd0b8592997d4a61b8ca854ecbb67c55 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:39.651181 2421 raft_consensus.cc:513] T fd0b8592997d4a61b8ca854ecbb67c55 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:39.651834 2421 leader_election.cc:290] T fd0b8592997d4a61b8ca854ecbb67c55 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:39.652578 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd0b8592997d4a61b8ca854ecbb67c55" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:39.653631 711 leader_election.cc:343] T fd0b8592997d4a61b8ca854ecbb67c55 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:39.658022 1192 tablet_bootstrap.cc:654] T c518f791b4d74c54ab0c0364d29b4a64 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:39.660941 1198 ts_tablet_manager.cc:1918] T 378d59d314a84ddb84cf2ab041c51cf1 P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250113 02:51:39.666354 899 tablet_service.cc:1467] Processing CreateTablet for tablet e19d831879584f1aaebaff0e33d63aaf (DEFAULT_TABLE table=default.Table_458c7f623bc4476c918b4a71f74e3170 [id=b85984ed55904ca2826eacd9abcc16d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:39.667236 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e19d831879584f1aaebaff0e33d63aaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:39.668241 886 tablet_service.cc:1467] Processing CreateTablet for tablet 593c54696ea7473d8450744a7c4a394d (DEFAULT_TABLE table=default.Table_b1bd611e962141f19470cb3296e4bd00 [id=a701c59d52634e78a17f4210655d9971]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:39.668555 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 593c54696ea7473d8450744a7c4a394d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:39.675217 891 tablet_service.cc:1467] Processing CreateTablet for tablet 6f73a254674b4cdcb137659069ba0257 (DEFAULT_TABLE table=default.Table_aab0b6c811674e558515778d0dd658a1 [id=5fd6aa5a7e4a4aa98432ea4c879a0f6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:39.675293 2427 tablet.cc:1721] T 4fe54131aafb40ee9d8bc5ea299af193 P df69f48656b44bb890f76587ee69cc0f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20250113 02:51:39.675536 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f73a254674b4cdcb137659069ba0257. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:39.682024 881 tablet_service.cc:1467] Processing CreateTablet for tablet 9fcd4b72b66c4373aa3e8168ab438b56 (DEFAULT_TABLE table=default.Table_96b4ae94f08f485695f19b9d92f92bec [id=56c62880977a433aaeaa1fa6fe227fe6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:39.682592 2377 raft_consensus.cc:491] T 4b07a985897641c3ac48021c2a4daedb P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:39.682726 2377 raft_consensus.cc:513] T 4b07a985897641c3ac48021c2a4daedb P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:39.683326 2377 leader_election.cc:290] T 4b07a985897641c3ac48021c2a4daedb P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
W20250113 02:51:39.685724 851 leader_election.cc:343] T 4b07a985897641c3ac48021c2a4daedb P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Not found: Tablet not found: 4b07a985897641c3ac48021c2a4daedb
I20250113 02:51:39.688194 895 tablet_service.cc:1467] Processing CreateTablet for tablet 78e7e343081c4905acc1abc4dd78ed6c (DEFAULT_TABLE table=default.Table_d6b0044411fc480ebcb1a635d4438a07 [id=6759bdc1f879476bbcb63b4969cd99f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:39.692770 1191 tablet_bootstrap.cc:492] T 21bcdd741104474bad7023426856d18f P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:39.693465 1194 tablet_bootstrap.cc:492] T 56b022c94847411785a7e9db3757a2d9 P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:39.693553 2342 catalog_manager.cc:1575] T 00000000000000000000000000000000 P 45e9d43ecac0401fb2c18fe5eaf089b5: acquiring CA information for follower catalog manager: success
I20250113 02:51:39.694787 1191 ts_tablet_manager.cc:1397] T 21bcdd741104474bad7023426856d18f P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.126s user 0.001s sys 0.008s
I20250113 02:51:39.695837 1191 raft_consensus.cc:357] T 21bcdd741104474bad7023426856d18f P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:39.696256 1194 ts_tablet_manager.cc:1397] T 56b022c94847411785a7e9db3757a2d9 P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.386s user 0.005s sys 0.010s
W20250113 02:51:39.697397 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9fcd4b72b66c4373aa3e8168ab438b56 consensus state has no leader
I20250113 02:51:39.697858 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_f1f79d9ce8f24b5cb56d1ce5082196c6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:39.698429 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f1f79d9ce8f24b5cb56d1ce5082196c6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:39.700510 455 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_e9f6c43232944cb090b686edb4607e6a" } modify_external_catalogs: true
I20250113 02:51:39.700664 455 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_e9f6c43232944cb090b686edb4607e6a" } modify_external_catalogs: true
I20250113 02:51:39.702230 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_344e7583f72042fb926d0f2e1aa2ce87"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:39.702787 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_344e7583f72042fb926d0f2e1aa2ce87 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:39.704476 447 master_service.cc:946] ReplaceTablet: received request to replace tablet 61c79029dfb74f01aaac0f7f828880d0 from {username='slave'} at 127.0.0.1:49258
W20250113 02:51:39.709398 448 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7aa4a8fbf5204ee89aa8bdc3047bc7e5 consensus state has no leader
I20250113 02:51:39.709724 448 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_d8e5fcedafb14988b94b1114b38e35d3" } modify_external_catalogs: true
I20250113 02:51:39.709856 448 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_d8e5fcedafb14988b94b1114b38e35d3" } modify_external_catalogs: true
W20250113 02:51:39.711560 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a78081b7efe34af8a053f5272cfaf346 consensus state has no leader
I20250113 02:51:39.711853 446 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_3935379456cf4f08b2e36f4ba95af912" } modify_external_catalogs: true
I20250113 02:51:39.712003 446 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_3935379456cf4f08b2e36f4ba95af912" } modify_external_catalogs: true
I20250113 02:51:39.665470 1041 tablet_service.cc:1467] Processing CreateTablet for tablet a66c6636f9f2492684b975cbfcb7530e (DEFAULT_TABLE table=default.Table_923d4e514de74882b7cb333bee6c14b5 [id=15a8a374fa64473289ac0b51149149ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:39.713485 1198 log.cc:1198] T 378d59d314a84ddb84cf2ab041c51cf1 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/378d59d314a84ddb84cf2ab041c51cf1
I20250113 02:51:39.713680 452 catalog_manager.cc:6071] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: default.Table_acdc5be1660742d7a356474b86e8b35f [id=aaf6558a15124f2b8eb9f01d692b845d] alter complete (version 2)
I20250113 02:51:39.713742 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a66c6636f9f2492684b975cbfcb7530e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:39.714151 1198 ts_tablet_manager.cc:1939] T 378d59d314a84ddb84cf2ab041c51cf1 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:39.714102 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_daf955e3f47f4bdf8c76884ea4054463" } modify_external_catalogs: true
I20250113 02:51:39.714249 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_daf955e3f47f4bdf8c76884ea4054463" } modify_external_catalogs: true
I20250113 02:51:39.715268 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1110) took 2088 ms. Trace:
I20250113 02:51:39.715402 1198 rpcz_store.cc:276] 0113 02:51:37.626562 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:37.626621 (+ 59us) service_pool.cc:225] Handling call
0113 02:51:39.616941 (+1990320us) ts_tablet_manager.cc:1145] Deleting tablet 378d59d314a84ddb84cf2ab041c51cf1
0113 02:51:39.616950 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:39.628200 (+ 11250us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:39.660817 (+ 32617us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:39.715248 (+ 54431us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1952210}
I20250113 02:51:39.715871 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet 378d59d314a84ddb84cf2ab041c51cf1 (table default.Table_49dafeaa3df04cb789a3e54f469973e8 [id=0c01441cec7043b687d6744e72a0e45d]) successfully deleted
I20250113 02:51:39.716264 1198 tablet_replica.cc:331] T d9df314ffaf042ec80180868d7e7bdf7 P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:39.716409 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_90d1fd99656d4ad89beb35946d8ad9d4" } modify_external_catalogs: true
I20250113 02:51:39.716504 1198 raft_consensus.cc:2238] T d9df314ffaf042ec80180868d7e7bdf7 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: Raft consensus shutting down.
I20250113 02:51:39.716604 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_90d1fd99656d4ad89beb35946d8ad9d4" } modify_external_catalogs: true
I20250113 02:51:39.716794 1198 pending_rounds.cc:62] T d9df314ffaf042ec80180868d7e7bdf7 P 4216a7f853d642b7bc3e5cff458f3aca: Trying to abort 1 pending ops.
I20250113 02:51:39.717288 1198 pending_rounds.cc:69] T d9df314ffaf042ec80180868d7e7bdf7 P 4216a7f853d642b7bc3e5cff458f3aca: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113673516850061312 op_type: NO_OP noop_request { }
I20250113 02:51:39.717464 1198 raft_consensus.cc:2883] T d9df314ffaf042ec80180868d7e7bdf7 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250113 02:51:39.717617 1198 raft_consensus.cc:2267] T d9df314ffaf042ec80180868d7e7bdf7 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:39.718238 1198 ts_tablet_manager.cc:1905] T d9df314ffaf042ec80180868d7e7bdf7 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:39.720309 1040 tablet_service.cc:1467] Processing CreateTablet for tablet e19d831879584f1aaebaff0e33d63aaf (DEFAULT_TABLE table=default.Table_458c7f623bc4476c918b4a71f74e3170 [id=b85984ed55904ca2826eacd9abcc16d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:39.720741 1040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e19d831879584f1aaebaff0e33d63aaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:39.720573 1194 raft_consensus.cc:357] T 56b022c94847411785a7e9db3757a2d9 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:39.721757 1191 raft_consensus.cc:383] T 21bcdd741104474bad7023426856d18f P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:39.721966 1191 raft_consensus.cc:738] T 21bcdd741104474bad7023426856d18f P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:39.697007 754 tablet_service.cc:1467] Processing CreateTablet for tablet e19d831879584f1aaebaff0e33d63aaf (DEFAULT_TABLE table=default.Table_458c7f623bc4476c918b4a71f74e3170 [id=b85984ed55904ca2826eacd9abcc16d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:39.697353 742 tablet_service.cc:1467] Processing CreateTablet for tablet 78e7e343081c4905acc1abc4dd78ed6c (DEFAULT_TABLE table=default.Table_d6b0044411fc480ebcb1a635d4438a07 [id=6759bdc1f879476bbcb63b4969cd99f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:39.697636 751 tablet_service.cc:1467] Processing CreateTablet for tablet 593c54696ea7473d8450744a7c4a394d (DEFAULT_TABLE table=default.Table_b1bd611e962141f19470cb3296e4bd00 [id=a701c59d52634e78a17f4210655d9971]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:39.717388 748 tablet_service.cc:1467] Processing CreateTablet for tablet 6f73a254674b4cdcb137659069ba0257 (DEFAULT_TABLE table=default.Table_aab0b6c811674e558515778d0dd658a1 [id=5fd6aa5a7e4a4aa98432ea4c879a0f6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:39.667165 1031 tablet_service.cc:1467] Processing CreateTablet for tablet 78e7e343081c4905acc1abc4dd78ed6c (DEFAULT_TABLE table=default.Table_d6b0044411fc480ebcb1a635d4438a07 [id=6759bdc1f879476bbcb63b4969cd99f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:39.668287 1039 tablet_service.cc:1467] Processing CreateTablet for tablet 593c54696ea7473d8450744a7c4a394d (DEFAULT_TABLE table=default.Table_b1bd611e962141f19470cb3296e4bd00 [id=a701c59d52634e78a17f4210655d9971]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:39.724633 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 593c54696ea7473d8450744a7c4a394d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:39.725834 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e19d831879584f1aaebaff0e33d63aaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:39.669308 1026 tablet_service.cc:1467] Processing CreateTablet for tablet 6f73a254674b4cdcb137659069ba0257 (DEFAULT_TABLE table=default.Table_aab0b6c811674e558515778d0dd658a1 [id=5fd6aa5a7e4a4aa98432ea4c879a0f6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:39.728191 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 78e7e343081c4905acc1abc4dd78ed6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:39.728439 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f73a254674b4cdcb137659069ba0257. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:39.670343 1042 tablet_service.cc:1467] Processing CreateTablet for tablet 9fcd4b72b66c4373aa3e8168ab438b56 (DEFAULT_TABLE table=default.Table_96b4ae94f08f485695f19b9d92f92bec [id=56c62880977a433aaeaa1fa6fe227fe6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:39.732255 1042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9fcd4b72b66c4373aa3e8168ab438b56. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:39.674700 1058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd0b8592997d4a61b8ca854ecbb67c55" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
W20250113 02:51:39.681581 995 leader_election.cc:343] T 1014e51f1c184d729552a5b34292f232 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Not found: Tablet not found: 1014e51f1c184d729552a5b34292f232
I20250113 02:51:39.736094 995 leader_election.cc:304] T 1014e51f1c184d729552a5b34292f232 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4216a7f853d642b7bc3e5cff458f3aca; no voters: 1186fa869eb14812832fa6562531f196, df69f48656b44bb890f76587ee69cc0f
W20250113 02:51:39.736462 995 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:40095, user_credentials={real_user=slave}} blocked reactor thread for 54894.3us
I20250113 02:51:39.736682 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 593c54696ea7473d8450744a7c4a394d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:39.737461 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f73a254674b4cdcb137659069ba0257. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:39.737542 2368 raft_consensus.cc:2743] T 1014e51f1c184d729552a5b34292f232 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:39.740239 742 tablet_service.cc:1467] Processing CreateTablet for tablet 9fcd4b72b66c4373aa3e8168ab438b56 (DEFAULT_TABLE table=default.Table_0894b27199a241e594a6f314094d0010 [id=56c62880977a433aaeaa1fa6fe227fe6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:39.740989 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9fcd4b72b66c4373aa3e8168ab438b56. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:39.747778 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 78e7e343081c4905acc1abc4dd78ed6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:38.099210 1911 rpcz_store.cc:276] 0113 02:51:31.354252 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:31.354312 (+ 60us) service_pool.cc:225] Handling call
0113 02:51:37.122139 (+5767827us) raft_consensus.cc:1466] Updating replica for 427 ops
0113 02:51:37.149827 (+ 27688us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 02:51:37.149837 (+ 10us) raft_consensus.cc:1534] Triggering prepare for 427 ops
0113 02:51:37.165136 (+ 15299us) write_op.cc:270] Start()
0113 02:51:37.180515 (+ 15379us) write_op.cc:276] Timestamp: P: 1736736684161517 usec, L: 0
0113 02:51:37.202431 (+ 21916us) write_op.cc:270] Start()
0113 02:51:37.202468 (+ 37us) write_op.cc:276] Timestamp: P: 1736736684162401 usec, L: 0
0113 02:51:37.202689 (+ 221us) write_op.cc:270] Start()
0113 02:51:37.202747 (+ 58us) write_op.cc:276] Timestamp: P: 1736736684163725 usec, L: 0
0113 02:51:37.202916 (+ 169us) write_op.cc:270] Start()
0113 02:51:37.202935 (+ 19us) write_op.cc:276] Timestamp: P: 1736736684167085 usec, L: 0
0113 02:51:37.203065 (+ 130us) write_op.cc:270] Start()
0113 02:51:37.203082 (+ 17us) write_op.cc:276] Timestamp: P: 1736736684167981 usec, L: 0
0113 02:51:37.203338 (+ 256us) write_op.cc:270] Start()
0113 02:51:37.203358 (+ 20us) write_op.cc:276] Timestamp: P: 1736736684173465 usec, L: 0
0113 02:51:37.203470 (+ 112us) write_op.cc:270] Start()
0113 02:51:37.203519 (+ 49us) write_op.cc:276] Timestamp: P: 1736736684174338 usec, L: 0
0113 02:51:37.203671 (+ 152us) write_op.cc:270] Start()
0113 02:51:37.203691 (+ 20us) write_op.cc:276] Timestamp: P: 1736736684175129 usec, L: 0
0113 02:51:37.203830 (+ 139us) write_op.cc:270] Start()
0113 02:51:37.203861 (+ 31us) write_op.cc:276] Timestamp: P: 1736736684176926 usec, L: 0
0113 02:51:37.204019 (+ 158us) write_op.cc:270] Start()
0113 02:51:37.204036 (+ 17us) write_op.cc:276] Timestamp: P: 1736736684189661 usec, L: 0
0113 02:51:37.204198 (+ 162us) write_op.cc:270] Start()
0113 02:51:37.204216 (+ 18us) write_op.cc:276] Timestamp: P: 1736736684214126 usec, L: 0
0113 02:51:37.204465 (+ 249us) write_op.cc:270] Start()
0113 02:51:37.204484 (+ 19us) write_op.cc:276] Timestamp: P: 1736736684215999 usec, L: 0
0113 02:51:37.204591 (+ 107us) write_op.cc:270] Start()
0113 02:51:37.204607 (+ 16us) write_op.cc:276] Timestamp: P: 1736736684234340 usec, L: 0
0113 02:51:37.204742 (+ 135us) write_op.cc:270] Start()
0113 02:51:37.204758 (+ 16us) write_op.cc:276] Timestamp: P: 1736736684236196 usec, L: 0
0113 02:51:37.204859 (+ 101us) write_op.cc:270] Start()
0113 02:51:37.204878 (+ 19us) write_op.cc:276] Timestamp: P: 1736736684236737 usec, L: 0
0113 02:51:37.205018 (+ 140us) write_op.cc:270] Start()
0113 02:51:37.205035 (+ 17us) write_op.cc:276] Timestamp: P: 1736736684308807 usec, L: 0
0113 02:51:37.205159 (+ 124us) write_op.cc:270] Start()
0113 02:51:37.205182 (+ 23us) write_op.cc:276] Timestamp: P: 1736736684309528 usec, L: 0
0113 02:51:37.205301 (+ 119us) write_op.cc:270] Start()
0113 02:51:37.205439 (+ 138us) write_op.cc:276] Timestamp: P: 1736736684310043 usec, L: 0
0113 02:51:37.205604 (+ 165us) write_op.cc:270] Start()
0113 02:51:37.205629 (+ 25us) write_op.cc:276] Timestamp: P: 1736736684310546 usec, L: 0
0113 02:51:37.205896 (+ 267us) write_op.cc:270] Start()
0113 02:51:37.205917 (+ 21us) write_op.cc:276] Timestamp: P: 1736736684311032 usec, L: 0
0113 02:51:37.206048 (+ 131us) write_op.cc:270] Start()
0113 02:51:37.206064 (+ 16us) write_op.cc:276] Timestamp: P: 1736736684345501 usec, L: 0
0113 02:51:37.206168 (+ 104us) write_op.cc:270] Start()
0113 02:51:37.206184 (+ 16us) write_op.cc:276] Timestamp: P: 1736736684399503 usec, L: 0
0113 02:51:37.206319 (+ 135us) write_op.cc:270] Start()
0113 02:51:37.206335 (+ 16us) write_op.cc:276] Timestamp: P: 1736736684400219 usec, L: 0
0113 02:51:37.206472 (+ 137us) write_op.cc:270] Start()
0113 02:51:37.206492 (+ 20us) write_op.cc:276] Timestamp: P: 1736736684400824 usec, L: 0
0113 02:51:37.206645 (+ 153us) write_op.cc:270] Start()
0113 02:51:37.206666 (+ 21us) write_op.cc:276] Timestamp: P: 1736736684416527 usec, L: 0
0113 02:51:37.219195 (+ 12529us) write_op.cc:270] Start()
0113 02:51:37.219244 (+ 49us) write_op.cc:276] Timestamp: P: 1736736684422314 usec, L: 0
0113 02:51:37.219591 (+ 347us) write_op.cc:270] Start()
0113 02:51:37.219632 (+ 41us) write_op.cc:276] Timestamp: P: 1736736684422979 usec, L: 0
0113 02:51:37.219782 (+ 150us) write_op.cc:270] Start()
0113 02:51:37.219801 (+ 19us) write_op.cc:276] Timestamp: P: 1736736684487324 usec, L: 0
0113 02:51:37.219938 (+ 137us) write_op.cc:270] Start()
0113 02:51:37.219955 (+ 17us) write_op.cc:276] Timestamp: P: 1736736684488064 usec, L: 0
0113 02:51:37.220092 (+ 137us) write_op.cc:270] Start()
0113 02:51:37.220111 (+ 19us) write_op.cc:276] Timestamp: P: 1736736684497810 usec, L: 0
0113 02:51:37.220223 (+ 112us) write_op.cc:270] Start()
0113 02:51:37.220240 (+ 17us) write_op.cc:276] Timestamp: P: 1736736684522209 usec, L: 0
0113 02:51:37.220335 (+ 95us) write_op.cc:270] Start()
0113 02:51:37.220350 (+ 15us) write_op.cc:276] Timestamp: P: 1736736684523216 usec, L: 0
0113 02:51:37.220490 (+ 140us) write_op.cc:270] Start()
0113 02:51:37.220506 (+ 16us) write_op.cc:276] Timestamp: P: 1736736684524084 usec, L: 0
0113 02:51:37.220611 (+ 105us) write_op.cc:270] Start()
0113 02:51:37.220627 (+ 16us) write_op.cc:276] Timestamp: P: 1736736684524916 usec, L: 0
0113 02:51:37.220740 (+ 113us) write_op.cc:270] Start()
0113 02:51:37.220755 (+ 15us) write_op.cc:276] Timestamp: P: 1736736684525792 usec, L: 0
0113 02:51:37.220856 (+ 101us) write_op.cc:270] Start()
0113 02:51:37.220870 (+ 14us) write_op.cc:276] Timestamp: P: 1736736684526718 usec, L: 0
0113 02:51:37.220996 (+ 126us) write_op.cc:270] Start()
0113 02:51:37.221022 (+ 26us) write_op.cc:276] Timestamp: P: 1736736684527614 usec, L: 0
0113 02:51:37.221138 (+ 116us) write_op.cc:270] Start()
0113 02:51:37.221154 (+ 16us) write_op.cc:276] Timestamp: P: 1736736684528777 usec, L: 0
0113 02:51:37.221282 (+ 128us) write_op.cc:270] Start()
0113 02:51:37.221298 (+ 16us) write_op.cc:276] Timestamp: P: 1736736684572600 usec, L: 0
0113 02:51:37.221409 (+ 111us) write_op.cc:270] Start()
0113 02:51:37.221429 (+ 20us) write_op.cc:276] Timestamp: P: 1736736684576709 usec, L: 0
0113 02:51:37.221618 (+ 189us) write_op.cc:270] Start()
0113 02:51:37.221645 (+ 27us) write_op.cc:276] Timestamp: P: 1736736684638333 usec, L: 0
0113 02:51:37.221787 (+ 142us) write_op.cc:270] Start()
0113 02:51:37.221806 (+ 19us) write_op.cc:276] Timestamp: P: 1736736684691742 usec, L: 0
0113 02:51:37.221950 (+ 144us) write_op.cc:270] Start()
0113 02:51:37.221969 (+ 19us) write_op.cc:276] Timestamp: P: 1736736684713091 usec, L: 0
0113 02:51:37.222094 (+ 125us) write_op.cc:270] Start()
0113 02:51:37.222111 (+ 17us) write_op.cc:276] Timestamp: P: 1736736684714229 usec, L: 0
0113 02:51:37.222268 (+ 157us) write_op.cc:270] Start()
0113 02:51:37.222287 (+ 19us) write_op.cc:276] Timestamp: P: 1736736684715538 usec, L: 0
0113 02:51:37.222414 (+ 127us) write_op.cc:270] Start()
0113 02:51:37.222432 (+ 18us) write_op.cc:276] Timestamp: P: 1736736684717187 usec, L: 0
0113 02:51:37.222567 (+ 135us) write_op.cc:270] Start()
0113 02:51:37.222584 (+ 17us) write_op.cc:276] Timestamp: P: 1736736684719129 usec, L: 0
0113 02:51:37.222713 (+ 129us) write_op.cc:270] Start()
0113 02:51:37.222731 (+ 18us) write_op.cc:276] Timestamp: P: 1736736684796834 usec, L: 0
0113 02:51:37.222883 (+ 152us) write_op.cc:270] Start()
0113 02:51:37.222911 (+ 28us) write_op.cc:276] Timestamp: P: 1736736684797654 usec, L: 0
0113 02:51:37.223051 (+ 140us) write_op.cc:270] Start()
0113 02:51:37.223070 (+ 19us) write_op.cc:276] Timestamp: P: 1736736684798322 usec, L: 0
0113 02:51:37.223289 (+ 219us) write_op.cc:270] Start()
0113 02:51:37.223311 (+ 22us) write_op.cc:276] Timestamp: P: 1736736684798992 usec, L: 0
0113 02:51:37.223435 (+ 124us) write_op.cc:270] Start()
0113 02:51:37.223452 (+ 17us) write_op.cc:276] Timestamp: P: 1736736684811050 usec, L: 0
0113 02:51:37.223556 (+ 104us) write_op.cc:270] Start()
0113 02:51:37.223573 (+ 17us) write_op.cc:276] Timestamp: P: 1736736684811650 usec, L: 0
0113 02:51:37.223686 (+ 113us) write_op.cc:270] Start()
0113 02:51:37.223705 (+ 19us) write_op.cc:276] Timestamp: P: 1736736684838142 usec, L: 0
0113 02:51:37.223828 (+ 123us) write_op.cc:270] Start()
0113 02:51:37.223847 (+ 19us) write_op.cc:276] Timestamp: P: 1736736684839685 usec, L: 0
0113 02:51:37.224003 (+ 156us) write_op.cc:270] Start()
0113 02:51:37.224024 (+ 21us) write_op.cc:276] Timestamp: P: 1736736684841132 usec, L: 0
0113 02:51:37.224165 (+ 141us) write_op.cc:270] Start()
0113 02:51:37.224189 (+ 24us) write_op.cc:276] Timestamp: P: 1736736684862795 usec, L: 0
0113 02:51:37.224313 (+ 124us) write_op.cc:270] Start()
0113 02:51:37.224331 (+ 18us) write_op.cc:276] Timestamp: P: 1736736684999571 usec, L: 0
0113 02:51:37.224451 (+ 120us) write_op.cc:270] Start()
0113 02:51:37.224468 (+ 17us) write_op.cc:276] Timestamp: P: 1736736685118230 usec, L: 0
0113 02:51:37.224579 (+ 111us) write_op.cc:270] Start()
0113 02:51:37.224622 (+ 43us) write_op.cc:276] Timestamp: P: 1736736685120246 usec, L: 0
0113 02:51:37.224739 (+ 117us) write_op.cc:270] Start()
0113 02:51:37.229724 (+ 4985us) write_op.cc:276] Timestamp: P: 1736736685121400 usec, L: 0
0113 02:51:37.229989 (+ 265us) write_op.cc:270] Start()
0113 02:51:37.230017 (+ 28us) write_op.cc:276] Timestamp: P: 1736736685122866 usec, L: 0
0113 02:51:37.238921 (+ 8904us) write_op.cc:270] Start()
0113 02:51:37.238963 (+ 42us) write_op.cc:276] Timestamp: P: 1736736685124448 usec, L: 0
0113 02:51:37.239209 (+ 246us) write_op.cc:270] Start()
0113 02:51:37.239237 (+ 28us) write_op.cc:276] Timestamp: P: 1736736685129258 usec, L: 0
0113 02:51:37.258217 (+ 18980us) write_op.cc:270] Start()
0113 02:51:37.258255 (+ 38us) write_op.cc:276] Timestamp: P: 1736736685132713 usec, L: 0
0113 02:51:37.258424 (+ 169us) write_op.cc:270] Start()
0113 02:51:37.260381 (+ 1957us) write_op.cc:276] Timestamp: P: 1736736685134073 usec, L: 0
0113 02:51:37.260595 (+ 214us) write_op.cc:270] Start()
0113 02:51:37.260616 (+ 21us) write_op.cc:276] Timestamp: P: 1736736685135442 usec, L: 0
0113 02:51:37.260765 (+ 149us) write_op.cc:270] Start()
0113 02:51:37.260786 (+ 21us) write_op.cc:276] Timestamp: P: 1736736685140380 usec, L: 0
0113 02:51:37.260963 (+ 177us) write_op.cc:270] Start()
0113 02:51:37.260989 (+ 26us) write_op.cc:276] Timestamp: P: 1736736685402134 usec, L: 0
0113 02:51:37.261135 (+ 146us) write_op.cc:270] Start()
0113 02:51:37.261156 (+ 21us) write_op.cc:276] Timestamp: P: 1736736685402688 usec, L: 0
0113 02:51:37.261306 (+ 150us) write_op.cc:270] Start()
0113 02:51:37.261327 (+ 21us) write_op.cc:276] Timestamp: P: 1736736685403320 usec, L: 0
0113 02:51:37.288266 (+ 26939us) write_op.cc:270] Start()
0113 02:51:37.291411 (+ 3145us) write_op.cc:276] Timestamp: P: 1736736685425290 usec, L: 0
0113 02:51:37.291648 (+ 237us) write_op.cc:270] Start()
0113 02:51:37.291674 (+ 26us) write_op.cc:276] Timestamp: P: 1736736685425990 usec, L: 0
0113 02:51:37.291847 (+ 173us) write_op.cc:270] Start()
0113 02:51:37.291868 (+ 21us) write_op.cc:276] Timestamp: P: 1736736685426478 usec, L: 0
0113 02:51:37.294158 (+ 2290us) write_op.cc:270] Start()
0113 02:51:37.294184 (+ 26us) write_op.cc:276] Timestamp: P: 1736736685439534 usec, L: 0
0113 02:51:37.294352 (+ 168us) write_op.cc:270] Start()
0113 02:51:37.295464 (+ 1112us) write_op.cc:276] Timestamp: P: 1736736685441238 usec, L: 0
0113 02:51:37.295664 (+ 200us) write_op.cc:270] Start()
0113 02:51:37.295684 (+ 20us) write_op.cc:276] Timestamp: P: 1736736685444078 usec, L: 0
0113 02:51:37.295821 (+ 137us) write_op.cc:270] Start()
0113 02:51:37.331463 (+ 35642us) write_op.cc:276] Timestamp: P: 1736736685446014 usec, L: 0
0113 02:51:37.331885 (+ 422us) write_op.cc:270] Start()
0113 02:51:37.331944 (+ 59us) write_op.cc:276] Timestamp: P: 1736736685451054 usec, L: 0
0113 02:51:37.332184 (+ 240us) write_op.cc:270] Start()
0113 02:51:37.332206 (+ 22us) write_op.cc:276] Timestamp: P: 1736736685576676 usec, L: 0
0113 02:51:37.332345 (+ 139us) write_op.cc:270] Start()
0113 02:51:37.332365 (+ 20us) write_op.cc:276] Timestamp: P: 1736736685577583 usec, L: 0
0113 02:51:37.332504 (+ 139us) write_op.cc:270] Start()
0113 02:51:37.332530 (+ 26us) write_op.cc:276] Timestamp: P: 1736736685578200 usec, L: 0
0113 02:51:37.332656 (+ 126us) write_op.cc:270] Start()
0113 02:51:37.332673 (+ 17us) write_op.cc:276] Timestamp: P: 1736736685592408 usec, L: 0
0113 02:51:37.332778 (+ 105us) write_op.cc:270] Start()
0113 02:51:37.332794 (+ 16us) write_op.cc:276] Timestamp: P: 1736736685599389 usec, L: 0
0113 02:51:37.332901 (+ 107us) write_op.cc:270] Start()
0113 02:51:37.332926 (+ 25us) write_op.cc:276] Timestamp: P: 1736736685600766 usec, L: 0
0113 02:51:37.342807 (+ 9881us) write_op.cc:270] Start()
0113 02:51:37.342850 (+ 43us) write_op.cc:276] Timestamp: P: 1736736685601504 usec, L: 0
0113 02:51:37.344520 (+ 1670us) write_op.cc:270] Start()
0113 02:51:37.344551 (+ 31us) write_op.cc:276] Timestamp: P: 1736736685605390 usec, L: 0
0113 02:51:37.344759 (+ 208us) write_op.cc:270] Start()
0113 02:51:37.344779 (+ 20us) write_op.cc:276] Timestamp: P: 1736736685606472 usec, L: 0
0113 02:51:37.344928 (+ 149us) write_op.cc:270] Start()
0113 02:51:37.344949 (+ 21us) write_op.cc:276] Timestamp: P: 1736736685607555 usec, L: 0
0113 02:51:37.345135 (+ 186us) write_op.cc:270] Start()
0113 02:51:37.345156 (+ 21us) write_op.cc:276] Timestamp: P: 1736736685616219 usec, L: 0
0113 02:51:37.345279 (+ 123us) write_op.cc:270] Start()
0113 02:51:37.345296 (+ 17us) write_op.cc:276] Timestamp: P: 1736736685618313 usec, L: 0
0113 02:51:37.345423 (+ 127us) write_op.cc:270] Start()
0113 02:51:37.345441 (+ 18us) write_op.cc:276] Timestamp: P: 1736736685620635 usec, L: 0
0113 02:51:37.345597 (+ 156us) write_op.cc:270] Start()
0113 02:51:37.345617 (+ 20us) write_op.cc:276] Timestamp: P: 1736736685770533 usec, L: 0
0113 02:51:37.345758 (+ 141us) write_op.cc:270] Start()
0113 02:51:37.345786 (+ 28us) write_op.cc:276] Timestamp: P: 1736736685774208 usec, L: 0
0113 02:51:37.345945 (+ 159us) write_op.cc:270] Start()
0113 02:51:37.345967 (+ 22us) write_op.cc:276] Timestamp: P: 1736736685774904 usec, L: 0
0113 02:51:37.346141 (+ 174us) write_op.cc:270] Start()
0113 02:51:37.346163 (+ 22us) write_op.cc:276] Timestamp: P: 1736736685835271 usec, L: 0
0113 02:51:37.346284 (+ 121us) write_op.cc:270] Start()
0113 02:51:37.346301 (+ 17us) write_op.cc:276] Timestamp: P: 1736736685844555 usec, L: 0
0113 02:51:37.346427 (+ 126us) write_op.cc:270] Start()
0113 02:51:37.346446 (+ 19us) write_op.cc:276] Timestamp: P: 1736736685845205 usec, L: 0
0113 02:51:37.346587 (+ 141us) write_op.cc:270] Start()
0113 02:51:37.346607 (+ 20us) write_op.cc:276] Timestamp: P: 1736736685850139 usec, L: 0
0113 02:51:37.346778 (+ 171us) write_op.cc:270] Start()
0113 02:51:37.346799 (+ 21us) write_op.cc:276] Timestamp: P: 1736736685850852 usec, L: 0
0113 02:51:37.346970 (+ 171us) write_op.cc:270] Start()
0113 02:51:37.346994 (+ 24us) write_op.cc:276] Timestamp: P: 1736736685851588 usec, L: 0
0113 02:51:37.347120 (+ 126us) write_op.cc:270] Start()
0113 02:51:37.347138 (+ 18us) write_op.cc:276] Timestamp: P: 1736736685852530 usec, L: 0
0113 02:51:37.347289 (+ 151us) write_op.cc:270] Start()
0113 02:51:37.347314 (+ 25us) write_op.cc:276] Timestamp: P: 1736736685913422 usec, L: 0
0113 02:51:37.347468 (+ 154us) write_op.cc:270] Start()
0113 02:51:37.347490 (+ 22us) write_op.cc:276] Timestamp: P: 1736736685914860 usec, L: 0
0113 02:51:37.347601 (+ 111us) write_op.cc:270] Start()
0113 02:51:37.347617 (+ 16us) write_op.cc:276] Timestamp: P: 1736736685921741 usec, L: 0
0113 02:51:37.347742 (+ 125us) write_op.cc:270] Start()
0113 02:51:37.347776 (+ 34us) write_op.cc:276] Timestamp: P: 1736736685924083 usec, L: 0
0113 02:51:37.347941 (+ 165us) write_op.cc:270] Start()
0113 02:51:37.347964 (+ 23us) write_op.cc:276] Timestamp: P: 1736736685933254 usec, L: 0
0113 02:51:37.403264 (+ 55300us) write_op.cc:270] Start()
0113 02:51:37.403301 (+ 37us) write_op.cc:276] Timestamp: P: 1736736685976381 usec, L: 0
0113 02:51:37.403495 (+ 194us) write_op.cc:270] Start()
0113 02:51:37.403518 (+ 23us) write_op.cc:276] Timestamp: P: 1736736685977058 usec, L: 0
0113 02:51:37.403653 (+ 135us) write_op.cc:270] Start()
0113 02:51:37.403674 (+ 21us) write_op.cc:276] Timestamp: P: 1736736685977721 usec, L: 0
0113 02:51:37.403826 (+ 152us) write_op.cc:270] Start()
0113 02:51:37.403846 (+ 20us) write_op.cc:276] Timestamp: P: 1736736685981116 usec, L: 0
0113 02:51:37.404201 (+ 355us) write_op.cc:270] Start()
0113 02:51:37.404230 (+ 29us) write_op.cc:276] Timestamp: P: 1736736686190133 usec, L: 0
0113 02:51:37.422371 (+ 18141us) write_op.cc:270] Start()
0113 02:51:37.422409 (+ 38us) write_op.cc:276] Timestamp: P: 1736736686195686 usec, L: 0
0113 02:51:37.422652 (+ 243us) write_op.cc:270] Start()
0113 02:51:37.426178 (+ 3526us) write_op.cc:276] Timestamp: P: 1736736686198746 usec, L: 0
0113 02:51:37.426395 (+ 217us) write_op.cc:270] Start()
0113 02:51:37.426417 (+ 22us) write_op.cc:276] Timestamp: P: 1736736686200969 usec, L: 0
0113 02:51:37.426586 (+ 169us) write_op.cc:270] Start()
0113 02:51:37.426604 (+ 18us) write_op.cc:276] Timestamp: P: 1736736686206005 usec, L: 0
0113 02:51:37.426714 (+ 110us) write_op.cc:270] Start()
0113 02:51:37.426729 (+ 15us) write_op.cc:276] Timestamp: P: 1736736686213032 usec, L: 0
0113 02:51:37.432232 (+ 5503us) write_op.cc:270] Start()
0113 02:51:37.432264 (+ 32us) write_op.cc:276] Timestamp: P: 1736736686214366 usec, L: 0
0113 02:51:37.432433 (+ 169us) write_op.cc:270] Start()
0113 02:51:37.432453 (+ 20us) write_op.cc:276] Timestamp: P: 1736736686218821 usec, L: 0
0113 02:51:37.432582 (+ 129us) write_op.cc:270] Start()
0113 02:51:37.432600 (+ 18us) write_op.cc:276] Timestamp: P: 1736736686224644 usec, L: 0
0113 02:51:37.432733 (+ 133us) write_op.cc:270] Start()
0113 02:51:37.432754 (+ 21us) write_op.cc:276] Timestamp: P: 1736736686234041 usec, L: 0
0113 02:51:37.432884 (+ 130us) write_op.cc:270] Start()
0113 02:51:37.432904 (+ 20us) write_op.cc:276] Timestamp: P: 1736736686240073 usec, L: 0
0113 02:51:37.433027 (+ 123us) write_op.cc:270] Start()
0113 02:51:37.433045 (+ 18us) write_op.cc:276] Timestamp: P: 1736736686245605 usec, L: 0
0113 02:51:37.433919 (+ 874us) write_op.cc:270] Start()
0113 02:51:37.433945 (+ 26us) write_op.cc:276] Timestamp: P: 1736736686246467 usec, L: 0
0113 02:51:37.434143 (+ 198us) write_op.cc:270] Start()
0113 02:51:37.434165 (+ 22us) write_op.cc:276] Timestamp: P: 1736736686247070 usec, L: 0
0113 02:51:37.434324 (+ 159us) write_op.cc:270] Start()
0113 02:51:37.434346 (+ 22us) write_op.cc:276] Timestamp: P: 1736736686248134 usec, L: 0
0113 02:51:37.434482 (+ 136us) write_op.cc:270] Start()
0113 02:51:37.434501 (+ 19us) write_op.cc:276] Timestamp: P: 1736736686250973 usec, L: 0
0113 02:51:37.434658 (+ 157us) write_op.cc:270] Start()
0113 02:51:37.434678 (+ 20us) write_op.cc:276] Timestamp: P: 1736736686349332 usec, L: 0
0113 02:51:37.434868 (+ 190us) write_op.cc:270] Start()
0113 02:51:37.434889 (+ 21us) write_op.cc:276] Timestamp: P: 1736736686377743 usec, L: 0
0113 02:51:37.435042 (+ 153us) write_op.cc:270] Start()
0113 02:51:37.435061 (+ 19us) write_op.cc:276] Timestamp: P: 1736736686422936 usec, L: 0
0113 02:51:37.435190 (+ 129us) write_op.cc:270] Start()
0113 02:51:37.435228 (+ 38us) write_op.cc:276] Timestamp: P: 1736736686426442 usec, L: 0
0113 02:51:37.435365 (+ 137us) write_op.cc:270] Start()
0113 02:51:37.435384 (+ 19us) write_op.cc:276] Timestamp: P: 1736736686430725 usec, L: 0
0113 02:51:37.435523 (+ 139us) write_op.cc:270] Start()
0113 02:51:37.435547 (+ 24us) write_op.cc:276] Timestamp: P: 1736736686446634 usec, L: 0
0113 02:51:37.435679 (+ 132us) write_op.cc:270] Start()
0113 02:51:37.435698 (+ 19us) write_op.cc:276] Timestamp: P: 1736736686447443 usec, L: 0
0113 02:51:37.435823 (+ 125us) write_op.cc:270] Start()
0113 02:51:37.435841 (+ 18us) write_op.cc:276] Timestamp: P: 1736736686448177 usec, L: 0
0113 02:51:37.435966 (+ 125us) write_op.cc:270] Start()
0113 02:51:37.436007 (+ 41us) write_op.cc:276] Timestamp: P: 1736736686508864 usec, L: 0
0113 02:51:37.436118 (+ 111us) write_op.cc:270] Start()
0113 02:51:37.436139 (+ 21us) write_op.cc:276] Timestamp: P: 1736736686510372 usec, L: 0
0113 02:51:37.436280 (+ 141us) write_op.cc:270] Start()
0113 02:51:37.436301 (+ 21us) write_op.cc:276] Timestamp: P: 1736736686511317 usec, L: 0
0113 02:51:37.436440 (+ 139us) write_op.cc:270] Start()
0113 02:51:37.436458 (+ 18us) write_op.cc:276] Timestamp: P: 1736736686526944 usec, L: 0
0113 02:51:37.436579 (+ 121us) write_op.cc:270] Start()
0113 02:51:37.436597 (+ 18us) write_op.cc:276] Timestamp: P: 1736736686531917 usec, L: 0
0113 02:51:37.436725 (+ 128us) write_op.cc:270] Start()
0113 02:51:37.436741 (+ 16us) write_op.cc:276] Timestamp: P: 1736736686539602 usec, L: 0
0113 02:51:37.436870 (+ 129us) write_op.cc:270] Start()
0113 02:51:37.436887 (+ 17us) write_op.cc:276] Timestamp: P: 1736736686551510 usec, L: 0
0113 02:51:37.437038 (+ 151us) write_op.cc:270] Start()
0113 02:51:37.437057 (+ 19us) write_op.cc:276] Timestamp: P: 1736736686573522 usec, L: 0
0113 02:51:37.437196 (+ 139us) write_op.cc:270] Start()
0113 02:51:37.437216 (+ 20us) write_op.cc:276] Timestamp: P: 1736736686580934 usec, L: 0
0113 02:51:37.437348 (+ 132us) write_op.cc:270] Start()
0113 02:51:37.437365 (+ 17us) write_op.cc:276] Timestamp: P: 1736736686581782 usec, L: 0
0113 02:51:37.437490 (+ 125us) write_op.cc:270] Start()
0113 02:51:37.437509 (+ 19us) write_op.cc:276] Timestamp: P: 1736736686582413 usec, L: 0
0113 02:51:37.437617 (+ 108us) write_op.cc:270] Start()
0113 02:51:37.437633 (+ 16us) write_op.cc:276] Timestamp: P: 1736736686583068 usec, L: 0
0113 02:51:37.437740 (+ 107us) write_op.cc:270] Start()
0113 02:51:37.437755 (+ 15us) write_op.cc:276] Timestamp: P: 1736736686585816 usec, L: 0
0113 02:51:37.437861 (+ 106us) write_op.cc:270] Start()
0113 02:51:37.437876 (+ 15us) write_op.cc:276] Timestamp: P: 1736736686600112 usec, L: 0
0113 02:51:37.438038 (+ 162us) write_op.cc:270] Start()
0113 02:51:37.438061 (+ 23us) write_op.cc:276] Timestamp: P: 1736736686602946 usec, L: 0
0113 02:51:37.438255 (+ 194us) write_op.cc:270] Start()
0113 02:51:37.439503 (+ 1248us) write_op.cc:276] Timestamp: P: 1736736686635446 usec, L: 0
0113 02:51:37.439685 (+ 182us) write_op.cc:270] Start()
0113 02:51:37.439706 (+ 21us) write_op.cc:276] Timestamp: P: 1736736686669064 usec, L: 0
0113 02:51:37.439839 (+ 133us) write_op.cc:270] Start()
0113 02:51:37.439859 (+ 20us) write_op.cc:276] Timestamp: P: 1736736686688564 usec, L: 0
0113 02:51:37.440044 (+ 185us) write_op.cc:270] Start()
0113 02:51:37.440065 (+ 21us) write_op.cc:276] Timestamp: P: 1736736686711093 usec, L: 0
0113 02:51:37.484390 (+ 44325us) write_op.cc:270] Start()
0113 02:51:37.484433 (+ 43us) write_op.cc:276] Timestamp: P: 1736736686711760 usec, L: 0
0113 02:51:37.484637 (+ 204us) write_op.cc:270] Start()
0113 02:51:37.484666 (+ 29us) write_op.cc:276] Timestamp: P: 1736736686712321 usec, L: 0
0113 02:51:37.484853 (+ 187us) write_op.cc:270] Start()
0113 02:51:37.484879 (+ 26us) write_op.cc:276] Timestamp: P: 1736736686712846 usec, L: 0
0113 02:51:37.485025 (+ 146us) write_op.cc:270] Start()
0113 02:51:37.485045 (+ 20us) write_op.cc:276] Timestamp: P: 1736736686714972 usec, L: 0
0113 02:51:37.485181 (+ 136us) write_op.cc:270] Start()
0113 02:51:37.485202 (+ 21us) write_op.cc:276] Timestamp: P: 1736736686719510 usec, L: 0
0113 02:51:37.485337 (+ 135us) write_op.cc:270] Start()
0113 02:51:37.485358 (+ 21us) write_op.cc:276] Timestamp: P: 1736736686722640 usec, L: 0
0113 02:51:37.485490 (+ 132us) write_op.cc:270] Start()
0113 02:51:37.485509 (+ 19us) write_op.cc:276] Timestamp: P: 1736736686727496 usec, L: 0
0113 02:51:37.485653 (+ 144us) write_op.cc:270] Start()
0113 02:51:37.485673 (+ 20us) write_op.cc:276] Timestamp: P: 1736736686763655 usec, L: 0
0113 02:51:37.489606 (+ 3933us) write_op.cc:270] Start()
0113 02:51:37.489648 (+ 42us) write_op.cc:276] Timestamp: P: 1736736686771174 usec, L: 0
0113 02:51:37.489846 (+ 198us) write_op.cc:270] Start()
0113 02:51:37.489869 (+ 23us) write_op.cc:276] Timestamp: P: 1736736686773960 usec, L: 0
0113 02:51:37.490039 (+ 170us) write_op.cc:270] Start()
0113 02:51:37.490059 (+ 20us) write_op.cc:276] Timestamp: P: 1736736686814288 usec, L: 0
0113 02:51:37.490182 (+ 123us) write_op.cc:270] Start()
0113 02:51:37.490202 (+ 20us) write_op.cc:276] Timestamp: P: 1736736686815330 usec, L: 0
0113 02:51:37.490364 (+ 162us) write_op.cc:270] Start()
0113 02:51:37.490437 (+ 73us) write_op.cc:276] Timestamp: P: 1736736686816494 usec, L: 0
0113 02:51:37.490663 (+ 226us) write_op.cc:270] Start()
0113 02:51:37.490686 (+ 23us) write_op.cc:276] Timestamp: P: 1736736686874429 usec, L: 0
0113 02:51:37.490817 (+ 131us) write_op.cc:270] Start()
0113 02:51:37.490837 (+ 20us) write_op.cc:276] Timestamp: P: 1736736686888722 usec, L: 0
0113 02:51:37.490998 (+ 161us) write_op.cc:270] Start()
0113 02:51:37.491019 (+ 21us) write_op.cc:276] Timestamp: P: 1736736686932133 usec, L: 0
0113 02:51:37.491157 (+ 138us) write_op.cc:270] Start()
0113 02:51:37.491177 (+ 20us) write_op.cc:276] Timestamp: P: 1736736686934426 usec, L: 0
0113 02:51:37.491324 (+ 147us) write_op.cc:270] Start()
0113 02:51:37.491346 (+ 22us) write_op.cc:276] Timestamp: P: 1736736686956675 usec, L: 0
0113 02:51:37.491498 (+ 152us) write_op.cc:270] Start()
0113 02:51:37.491524 (+ 26us) write_op.cc:276] Timestamp: P: 1736736686962166 usec, L: 0
0113 02:51:37.491670 (+ 146us) write_op.cc:270] Start()
0113 02:51:37.491697 (+ 27us) write_op.cc:276] Timestamp: P: 1736736686962774 usec, L: 0
0113 02:51:37.491834 (+ 137us) write_op.cc:270] Start()
0113 02:51:37.491859 (+ 25us) write_op.cc:276] Timestamp: P: 1736736686980334 usec, L: 0
0113 02:51:37.496691 (+ 4832us) write_op.cc:270] Start()
0113 02:51:37.496724 (+ 33us) write_op.cc:276] Timestamp: P: 1736736686981126 usec, L: 0
0113 02:51:37.496933 (+ 209us) write_op.cc:270] Start()
0113 02:51:37.496961 (+ 28us) write_op.cc:276] Timestamp: P: 1736736686981743 usec, L: 0
0113 02:51:37.497136 (+ 175us) write_op.cc:270] Start()
0113 02:51:37.497158 (+ 22us) write_op.cc:276] Timestamp: P: 1736736687017530 usec, L: 0
0113 02:51:37.497288 (+ 130us) write_op.cc:270] Start()
0113 02:51:37.497307 (+ 19us) write_op.cc:276] Timestamp: P: 1736736687018177 usec, L: 0
0113 02:51:37.497462 (+ 155us) write_op.cc:270] Start()
0113 02:51:37.497485 (+ 23us) write_op.cc:276] Timestamp: P: 1736736687018725 usec, L: 0
0113 02:51:37.497622 (+ 137us) write_op.cc:270] Start()
0113 02:51:37.497641 (+ 19us) write_op.cc:276] Timestamp: P: 1736736687027676 usec, L: 0
0113 02:51:37.497766 (+ 125us) write_op.cc:270] Start()
0113 02:51:37.497785 (+ 19us) write_op.cc:276] Timestamp: P: 1736736687030179 usec, L: 0
0113 02:51:37.497938 (+ 153us) write_op.cc:270] Start()
0113 02:51:37.497957 (+ 19us) write_op.cc:276] Timestamp: P: 1736736687031089 usec, L: 0
0113 02:51:37.498098 (+ 141us) write_op.cc:270] Start()
0113 02:51:37.498120 (+ 22us) write_op.cc:276] Timestamp: P: 1736736687032194 usec, L: 0
0113 02:51:37.498260 (+ 140us) write_op.cc:270] Start()
0113 02:51:37.498288 (+ 28us) write_op.cc:276] Timestamp: P: 1736736687191084 usec, L: 0
0113 02:51:37.498427 (+ 139us) write_op.cc:270] Start()
0113 02:51:37.498449 (+ 22us) write_op.cc:276] Timestamp: P: 1736736687191766 usec, L: 0
0113 02:51:37.498579 (+ 130us) write_op.cc:270] Start()
0113 02:51:37.498598 (+ 19us) write_op.cc:276] Timestamp: P: 1736736687193735 usec, L: 0
0113 02:51:37.498757 (+ 159us) write_op.cc:270] Start()
0113 02:51:37.498777 (+ 20us) write_op.cc:276] Timestamp: P: 1736736687194648 usec, L: 0
0113 02:51:37.498924 (+ 147us) write_op.cc:270] Start()
0113 02:51:37.498945 (+ 21us) write_op.cc:276] Timestamp: P: 1736736687199320 usec, L: 0
0113 02:51:37.499078 (+ 133us) write_op.cc:270] Start()
0113 02:51:37.499097 (+ 19us) write_op.cc:276] Timestamp: P: 1736736687211875 usec, L: 0
0113 02:51:37.499248 (+ 151us) write_op.cc:270] Start()
0113 02:51:37.499268 (+ 20us) write_op.cc:276] Timestamp: P: 1736736687212545 usec, L: 0
0113 02:51:37.499409 (+ 141us) write_op.cc:270] Start()
0113 02:51:37.499429 (+ 20us) write_op.cc:276] Timestamp: P: 1736736687214086 usec, L: 0
0113 02:51:37.503541 (+ 4112us) write_op.cc:270] Start()
0113 02:51:37.503566 (+ 25us) write_op.cc:276] Timestamp: P: 1736736687315021 usec, L: 0
0113 02:51:37.503727 (+ 161us) write_op.cc:270] Start()
0113 02:51:37.503750 (+ 23us) write_op.cc:276] Timestamp: P: 1736736687337106 usec, L: 0
0113 02:51:37.503882 (+ 132us) write_op.cc:270] Start()
0113 02:51:37.503903 (+ 21us) write_op.cc:276] Timestamp: P: 1736736687337886 usec, L: 0
0113 02:51:37.504117 (+ 214us) write_op.cc:270] Start()
0113 02:51:37.504141 (+ 24us) write_op.cc:276] Timestamp: P: 1736736687340301 usec, L: 0
0113 02:51:37.504289 (+ 148us) write_op.cc:270] Start()
0113 02:51:37.504644 (+ 355us) write_op.cc:276] Timestamp: P: 1736736
I20250113 02:51:39.753822 2349 sys_catalog.cc:455] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 4 leader_uuid: "e13f5bcdcee34225a46c6d2651e5ede9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e13f5bcdcee34225a46c6d2651e5ede9" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44217 } } peers { permanent_uuid: "6bbc3541604a455584bb32dfc2cc4efd" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 35271 } } peers { permanent_uuid: "45e9d43ecac0401fb2c18fe5eaf089b5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42067 } } }
I20250113 02:51:39.754040 2349 sys_catalog.cc:458] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9 [sys.catalog]: This master's current role is: LEADER
W20250113 02:51:39.759313 709 leader_election.cc:343] T fd0b8592997d4a61b8ca854ecbb67c55 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:39.759454 709 leader_election.cc:304] T fd0b8592997d4a61b8ca854ecbb67c55 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:39.759763 2421 raft_consensus.cc:2743] T fd0b8592997d4a61b8ca854ecbb67c55 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:39.764456 1042 tablet_service.cc:1467] Processing CreateTablet for tablet 59a447c6d0264ec38390fe688bc9555a (DEFAULT_TABLE table=default.Table_f5b4d64c86db49829574aea1b88e3968 [id=a59f6a19ab9541d7bf386d352f1962ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:39.764847 1042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59a447c6d0264ec38390fe688bc9555a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:39.771225 1026 tablet_service.cc:1467] Processing CreateTablet for tablet 1f4f365250df4532a0a5fa04f8f3f098 (DEFAULT_TABLE table=default.Table_1796b3c54f624e3aa374222d74b931b5 [id=9f5fcd45b39447d3941eca91cffd07d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:39.771606 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1f4f365250df4532a0a5fa04f8f3f098. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:39.779613 1041 tablet_service.cc:1467] Processing CreateTablet for tablet 7da6f8a4b89647968609f73f9ad5ba14 (DEFAULT_TABLE table=default.Table_9c2b9596afed4532858a877b5e954b4d [id=84cbc9ab3d23459988b3681c8c6004da]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:39.780087 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7da6f8a4b89647968609f73f9ad5ba14. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:39.786377 1039 tablet_service.cc:1467] Processing CreateTablet for tablet 4ae2bf8a24dc4143bca81715edadc8ca (DEFAULT_TABLE table=default.Table_67d9e82495c24463ba855129a5e7b1c2 [id=51ed2c3984374e3da4e91be78d412dda]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:39.790650 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ae2bf8a24dc4143bca81715edadc8ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:39.790700 1198 ts_tablet_manager.cc:1918] T d9df314ffaf042ec80180868d7e7bdf7 P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250113 02:51:39.791061 1040 tablet_service.cc:1467] Processing CreateTablet for tablet c29e7cf657794fbdaa385f3b32231b1e (DEFAULT_TABLE table=default.Table_a8de05b02a0342898b051d8c7078f835 [id=f9b4e442919c449ab1d93adb2eb820e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:39.791730 1040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c29e7cf657794fbdaa385f3b32231b1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:39.791728 1198 log.cc:1198] T d9df314ffaf042ec80180868d7e7bdf7 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/d9df314ffaf042ec80180868d7e7bdf7
I20250113 02:51:39.721567 738 tablet_service.cc:1467] Processing CreateTablet for tablet a66c6636f9f2492684b975cbfcb7530e (DEFAULT_TABLE table=default.Table_923d4e514de74882b7cb333bee6c14b5 [id=15a8a374fa64473289ac0b51149149ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:39.792659 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a66c6636f9f2492684b975cbfcb7530e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:39.792711 1192 tablet_bootstrap.cc:492] T c518f791b4d74c54ab0c0364d29b4a64 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:39.792788 1194 raft_consensus.cc:383] T 56b022c94847411785a7e9db3757a2d9 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:39.792881 1194 raft_consensus.cc:738] T 56b022c94847411785a7e9db3757a2d9 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:39.793028 1192 ts_tablet_manager.cc:1397] T c518f791b4d74c54ab0c0364d29b4a64 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.143s user 0.007s sys 0.017s
I20250113 02:51:39.793131 1194 consensus_queue.cc:260] T 56b022c94847411785a7e9db3757a2d9 P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:39.731389 1196 ts_tablet_manager.cc:1918] T aa5c172a69bf4ca79999f4b58df08de8 P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250113 02:51:39.795490 1196 log.cc:1198] T aa5c172a69bf4ca79999f4b58df08de8 P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/aa5c172a69bf4ca79999f4b58df08de8
W20250113 02:51:39.740880 852 leader_election.cc:343] T 4b07a985897641c3ac48021c2a4daedb P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Not found: Tablet not found: 4b07a985897641c3ac48021c2a4daedb
I20250113 02:51:39.795648 852 leader_election.cc:304] T 4b07a985897641c3ac48021c2a4daedb P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df69f48656b44bb890f76587ee69cc0f; no voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca
I20250113 02:51:39.795850 1196 ts_tablet_manager.cc:1939] T aa5c172a69bf4ca79999f4b58df08de8 P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
W20250113 02:51:39.795953 852 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.3:42587, user_credentials={real_user=slave}} blocked reactor thread for 55087.2us
I20250113 02:51:39.796375 2377 raft_consensus.cc:2743] T 4b07a985897641c3ac48021c2a4daedb P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:39.796290 1198 ts_tablet_manager.cc:1939] T d9df314ffaf042ec80180868d7e7bdf7 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:39.762447 899 tablet_service.cc:1467] Processing CreateTablet for tablet 7da6f8a4b89647968609f73f9ad5ba14 (DEFAULT_TABLE table=default.Table_9c2b9596afed4532858a877b5e954b4d [id=84cbc9ab3d23459988b3681c8c6004da]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:39.763412 891 tablet_service.cc:1467] Processing CreateTablet for tablet 59a447c6d0264ec38390fe688bc9555a (DEFAULT_TABLE table=default.Table_f5b4d64c86db49829574aea1b88e3968 [id=a59f6a19ab9541d7bf386d352f1962ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:39.765096 886 tablet_service.cc:1467] Processing CreateTablet for tablet 1f4f365250df4532a0a5fa04f8f3f098 (DEFAULT_TABLE table=default.Table_1796b3c54f624e3aa374222d74b931b5 [id=9f5fcd45b39447d3941eca91cffd07d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:39.766829 890 tablet_service.cc:1467] Processing CreateTablet for tablet c29e7cf657794fbdaa385f3b32231b1e (DEFAULT_TABLE table=default.Table_a8de05b02a0342898b051d8c7078f835 [id=f9b4e442919c449ab1d93adb2eb820e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:39.767094 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9fcd4b72b66c4373aa3e8168ab438b56. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:39.785375 888 tablet_service.cc:1467] Processing CreateTablet for tablet 4ae2bf8a24dc4143bca81715edadc8ca (DEFAULT_TABLE table=default.Table_67d9e82495c24463ba855129a5e7b1c2 [id=51ed2c3984374e3da4e91be78d412dda]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:39.797617 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 78e7e343081c4905acc1abc4dd78ed6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:39.799170 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59a447c6d0264ec38390fe688bc9555a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:39.800216 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1f4f365250df4532a0a5fa04f8f3f098. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:39.808187 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ae2bf8a24dc4143bca81715edadc8ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:39.808898 881 tablet_service.cc:1467] Processing CreateTablet for tablet 80320e2fe442400daa31f166754ff361 (DEFAULT_TABLE table=default.Table_84a7c1edc9204358b6a5c3e818d46321 [id=1f9fa491e6c84c0e9ac8f5b5b81e2531]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:39.809244 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 80320e2fe442400daa31f166754ff361. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:39.810861 882 tablet_service.cc:1467] Processing CreateTablet for tablet 4791a82b5e13429184d0c7192dd7aef6 (DEFAULT_TABLE table=default.Table_f9e0115a922e4a7c825265fc1f7222f9 [id=c73f4c8fe96a4283bbb075eab16beda8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:39.811177 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4791a82b5e13429184d0c7192dd7aef6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:39.812260 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7da6f8a4b89647968609f73f9ad5ba14. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:39.816304 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c29e7cf657794fbdaa385f3b32231b1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:39.809854 1026 tablet_service.cc:1467] Processing CreateTablet for tablet 80320e2fe442400daa31f166754ff361 (DEFAULT_TABLE table=default.Table_84a7c1edc9204358b6a5c3e818d46321 [id=1f9fa491e6c84c0e9ac8f5b5b81e2531]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:39.816646 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 80320e2fe442400daa31f166754ff361. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:39.816864 895 tablet_service.cc:1467] Processing CreateTablet for tablet d9f0fdf99ed94a19b336ef4fb5d38ca0 (DEFAULT_TABLE table=default.Table_24bc600591b2484591266b7da6c66ce0 [id=b4b989a40b8d4f42b3b0a7add3456287]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:39.817167 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d9f0fdf99ed94a19b336ef4fb5d38ca0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:39.723923 1191 consensus_queue.cc:260] T 21bcdd741104474bad7023426856d18f P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:39.817934 882 tablet_service.cc:1467] Processing CreateTablet for tablet 5d56fa4b79154521b7bd9d35b133d391 (DEFAULT_TABLE table=default.Table_6d41dae8ad174b7fa4364a4edb1fc307 [id=6e54a3c1280144ef8a082a5090493a4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:39.818102 1191 ts_tablet_manager.cc:1428] T 21bcdd741104474bad7023426856d18f P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.123s user 0.002s sys 0.000s
I20250113 02:51:39.818215 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d56fa4b79154521b7bd9d35b133d391. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:39.822646 748 tablet_service.cc:1467] Processing CreateTablet for tablet 59a447c6d0264ec38390fe688bc9555a (DEFAULT_TABLE table=default.Table_f5b4d64c86db49829574aea1b88e3968 [id=a59f6a19ab9541d7bf386d352f1962ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:39.822705 742 tablet_service.cc:1467] Processing CreateTablet for tablet 7da6f8a4b89647968609f73f9ad5ba14 (DEFAULT_TABLE table=default.Table_9c2b9596afed4532858a877b5e954b4d [id=84cbc9ab3d23459988b3681c8c6004da]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:39.793681 1194 ts_tablet_manager.cc:1428] T 56b022c94847411785a7e9db3757a2d9 P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.094s user 0.001s sys 0.001s
I20250113 02:51:39.825102 1031 tablet_service.cc:1467] Processing CreateTablet for tablet d9f0fdf99ed94a19b336ef4fb5d38ca0 (DEFAULT_TABLE table=default.Table_24bc600591b2484591266b7da6c66ce0 [id=b4b989a40b8d4f42b3b0a7add3456287]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:39.825495 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d9f0fdf99ed94a19b336ef4fb5d38ca0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:39.825690 751 tablet_service.cc:1467] Processing CreateTablet for tablet 1f4f365250df4532a0a5fa04f8f3f098 (DEFAULT_TABLE table=default.Table_1796b3c54f624e3aa374222d74b931b5 [id=9f5fcd45b39447d3941eca91cffd07d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:39.826920 1194 tablet_bootstrap.cc:492] T fb18c16fb3ba44ec8a190b88d3bbd1b1 P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:39.827351 754 tablet_service.cc:1467] Processing CreateTablet for tablet c29e7cf657794fbdaa385f3b32231b1e (DEFAULT_TABLE table=default.Table_a8de05b02a0342898b051d8c7078f835 [id=f9b4e442919c449ab1d93adb2eb820e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:39.828539 747 tablet_service.cc:1467] Processing CreateTablet for tablet 4ae2bf8a24dc4143bca81715edadc8ca (DEFAULT_TABLE table=default.Table_67d9e82495c24463ba855129a5e7b1c2 [id=51ed2c3984374e3da4e91be78d412dda]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:39.830165 1194 tablet_bootstrap.cc:654] T fb18c16fb3ba44ec8a190b88d3bbd1b1 P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:39.830523 736 tablet_service.cc:1467] Processing CreateTablet for tablet 80320e2fe442400daa31f166754ff361 (DEFAULT_TABLE table=default.Table_84a7c1edc9204358b6a5c3e818d46321 [id=1f9fa491e6c84c0e9ac8f5b5b81e2531]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:39.830950 736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 80320e2fe442400daa31f166754ff361. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:39.836781 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c29e7cf657794fbdaa385f3b32231b1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:39.840180 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59a447c6d0264ec38390fe688bc9555a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:39.844480 2342 catalog_manager.cc:1603] T 00000000000000000000000000000000 P 45e9d43ecac0401fb2c18fe5eaf089b5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250113 02:51:39.846309 2306 raft_consensus.cc:491] T c7842d6f2c074e3b9d1a3378db9856be P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:39.846568 2306 raft_consensus.cc:513] T c7842d6f2c074e3b9d1a3378db9856be P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:39.850770 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1112) took 2216 ms. Trace:
I20250113 02:51:39.850862 1198 rpcz_store.cc:276] 0113 02:51:37.634182 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:37.634239 (+ 57us) service_pool.cc:225] Handling call
0113 02:51:39.716208 (+2081969us) ts_tablet_manager.cc:1145] Deleting tablet d9df314ffaf042ec80180868d7e7bdf7
0113 02:51:39.716220 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:39.753300 (+ 37080us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:39.789704 (+ 36404us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:39.850747 (+ 61043us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2051421}
I20250113 02:51:39.830535 1191 tablet_bootstrap.cc:492] T fb2458e2044d49029ed2d6851db9de25 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:39.852929 1191 tablet_bootstrap.cc:654] T fb2458e2044d49029ed2d6851db9de25 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:39.853915 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7da6f8a4b89647968609f73f9ad5ba14. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:39.855033 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1f4f365250df4532a0a5fa04f8f3f098. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:39.860795 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ae2bf8a24dc4143bca81715edadc8ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:39.868839 1191 tablet_bootstrap.cc:492] T fb2458e2044d49029ed2d6851db9de25 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:39.869683 1042 tablet_service.cc:1467] Processing CreateTablet for tablet 4791a82b5e13429184d0c7192dd7aef6 (DEFAULT_TABLE table=default.Table_f9e0115a922e4a7c825265fc1f7222f9 [id=c73f4c8fe96a4283bbb075eab16beda8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:39.872332 1034 tablet_service.cc:1467] Processing CreateTablet for tablet 21fcae83282d43c28000c5fe581e5ae6 (DEFAULT_TABLE table=default.Table_0f386e0d294749fba0d3c68b7bcf0f46 [id=27f04ea09d024411b36dadf8dcb48e2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:39.872886 1030 tablet_service.cc:1467] Processing CreateTablet for tablet 5d56fa4b79154521b7bd9d35b133d391 (DEFAULT_TABLE table=default.Table_6d41dae8ad174b7fa4364a4edb1fc307 [id=6e54a3c1280144ef8a082a5090493a4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:39.873589 2306 leader_election.cc:290] T c7842d6f2c074e3b9d1a3378db9856be P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
I20250113 02:51:39.873878 1035 tablet_service.cc:1467] Processing CreateTablet for tablet 913c427669d24405896826e016138f83 (DEFAULT_TABLE table=default.Table_d93ff82f449f4a96b688393e8b298b42 [id=36b206c6af624ec78b3dff77eaea319e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:39.874676 1191 ts_tablet_manager.cc:1397] T fb2458e2044d49029ed2d6851db9de25 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.044s user 0.003s sys 0.000s
I20250113 02:51:39.875401 1191 raft_consensus.cc:357] T fb2458e2044d49029ed2d6851db9de25 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:39.875689 1191 raft_consensus.cc:383] T fb2458e2044d49029ed2d6851db9de25 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:39.875787 1191 raft_consensus.cc:738] T fb2458e2044d49029ed2d6851db9de25 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:39.877243 1195 ts_tablet_manager.cc:1918] T d9df314ffaf042ec80180868d7e7bdf7 P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:39.877405 1195 log.cc:1198] T d9df314ffaf042ec80180868d7e7bdf7 P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/d9df314ffaf042ec80180868d7e7bdf7
I20250113 02:51:39.877780 1195 ts_tablet_manager.cc:1939] T d9df314ffaf042ec80180868d7e7bdf7 P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:39.878484 1024 tablet_service.cc:1467] Processing CreateTablet for tablet 228ccf742ffd4504a2bfb1fc38da377f (DEFAULT_TABLE table=default.Table_ecba1621e21641e98ab0246b22af7417 [id=e31997add9334353b2b56eb6869b8856]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:39.878814 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1064) took 3398 ms. Trace:
I20250113 02:51:39.878891 1195 rpcz_store.cc:276] 0113 02:51:36.480621 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:36.480863 (+ 242us) service_pool.cc:225] Handling call
0113 02:51:39.545674 (+3064811us) ts_tablet_manager.cc:1145] Deleting tablet d9df314ffaf042ec80180868d7e7bdf7
0113 02:51:39.545682 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:39.717901 (+172219us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:39.877133 (+159232us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:39.878794 (+ 1661us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3064528}
I20250113 02:51:39.879288 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1067) took 3320 ms. Trace:
I20250113 02:51:39.879379 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1114) took 2245 ms. Trace:
I20250113 02:51:39.879451 1198 rpcz_store.cc:276] 0113 02:51:37.634332 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:37.634451 (+ 119us) service_pool.cc:225] Handling call
0113 02:51:39.851148 (+2216697us) ts_tablet_manager.cc:1145] Deleting tablet dff74d7c093b45eba7a47d8eceb0c1d1
0113 02:51:39.879251 (+ 28103us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:39.879359 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2186309}
I20250113 02:51:39.879359 1195 rpcz_store.cc:276] 0113 02:51:36.558807 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:36.559053 (+ 246us) service_pool.cc:225] Handling call
0113 02:51:39.879150 (+3320097us) ts_tablet_manager.cc:1145] Deleting tablet c4b7196ef5d54459921977e81a2432db
0113 02:51:39.879160 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:39.879273 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3319917}
I20250113 02:51:39.880079 1037 tablet_service.cc:1467] Processing CreateTablet for tablet 99e00224ed7049f88aff91013e92d4e0 (DEFAULT_TABLE table=default.Table_b638c7b0c29a41748e55d806b0924138 [id=ba5822870dd34e6eada4583f0478df63]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:39.880314 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1069) took 3321 ms. Trace:
I20250113 02:51:39.880379 1195 rpcz_store.cc:276] 0113 02:51:36.559005 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:36.559260 (+ 255us) service_pool.cc:225] Handling call
0113 02:51:39.880198 (+3320938us) ts_tablet_manager.cc:1145] Deleting tablet f426887763ed48e3bc87f59d80986672
0113 02:51:39.880208 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:39.880300 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3320834}
I20250113 02:51:39.880605 1195 tablet_replica.cc:331] T bc533518faa24dffa3306ff092e8d507 P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:39.880586 1036 tablet_service.cc:1467] Processing CreateTablet for tablet 89ac5c337f394705a1fc638c9f6835fb (DEFAULT_TABLE table=default.Table_5d54936883fa4a068a24a069ad10dafb [id=6962d4bbc6f04f409a1fd3a760239000]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:39.880787 1195 raft_consensus.cc:2238] T bc533518faa24dffa3306ff092e8d507 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:39.880918 1195 raft_consensus.cc:2267] T bc533518faa24dffa3306ff092e8d507 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:39.880937 1038 tablet_service.cc:1467] Processing CreateTablet for tablet 5771569d4bcd4625954dc1ba2a085b21 (DEFAULT_TABLE table=default.Table_e253836ed32b43b58cf1f7938dadf3dc [id=df9291acc6564f22b739bf3bcbb891e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:39.881261 1023 tablet_service.cc:1467] Processing CreateTablet for tablet 3490334f751f4ecfbcfb8dd7329e169d (DEFAULT_TABLE table=default.Table_86c5f0e60c9347858bc135037c194bb7 [id=9a92afc2804347aebdc68a0f3c25f75e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:39.881407 1195 ts_tablet_manager.cc:1905] T bc533518faa24dffa3306ff092e8d507 P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:39.881661 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet d9df314ffaf042ec80180868d7e7bdf7 (table default.Table_2155bf71d1a04a93a28b13c97f636fab [id=00107bce6ce24feba58176198a222970]) successfully deleted
W20250113 02:51:39.881896 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet dff74d7c093b45eba7a47d8eceb0c1d1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dff74d7c093b45eba7a47d8eceb0c1d1 already in progress: creating tablet
I20250113 02:51:39.888309 751 tablet_service.cc:1467] Processing CreateTablet for tablet d9f0fdf99ed94a19b336ef4fb5d38ca0 (DEFAULT_TABLE table=default.Table_24bc600591b2484591266b7da6c66ce0 [id=b4b989a40b8d4f42b3b0a7add3456287]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:39.888649 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d9f0fdf99ed94a19b336ef4fb5d38ca0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:39.889164 1198 tablet_replica.cc:331] T d8bb60fdd06946739b75b255303d9d24 P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:39.889353 1198 raft_consensus.cc:2238] T d8bb60fdd06946739b75b255303d9d24 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:39.889654 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 21fcae83282d43c28000c5fe581e5ae6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:39.892210 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d56fa4b79154521b7bd9d35b133d391. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:39.893925 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 913c427669d24405896826e016138f83. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:39.894877 1042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4791a82b5e13429184d0c7192dd7aef6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:39.895612 736 tablet_service.cc:1467] Processing CreateTablet for tablet 228ccf742ffd4504a2bfb1fc38da377f (DEFAULT_TABLE table=default.Table_ecba1621e21641e98ab0246b22af7417 [id=e31997add9334353b2b56eb6869b8856]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:39.895928 736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 228ccf742ffd4504a2bfb1fc38da377f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:39.896469 1036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 89ac5c337f394705a1fc638c9f6835fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:39.896659 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5771569d4bcd4625954dc1ba2a085b21. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:39.896865 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3490334f751f4ecfbcfb8dd7329e169d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:39.900168 1037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 99e00224ed7049f88aff91013e92d4e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:39.901285 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 228ccf742ffd4504a2bfb1fc38da377f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:39.902503 748 tablet_service.cc:1467] Processing CreateTablet for tablet 5d56fa4b79154521b7bd9d35b133d391 (DEFAULT_TABLE table=default.Table_6d41dae8ad174b7fa4364a4edb1fc307 [id=6e54a3c1280144ef8a082a5090493a4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:39.902812 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d56fa4b79154521b7bd9d35b133d391. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:39.906577 754 tablet_service.cc:1467] Processing CreateTablet for tablet 21fcae83282d43c28000c5fe581e5ae6 (DEFAULT_TABLE table=default.Table_0f386e0d294749fba0d3c68b7bcf0f46 [id=27f04ea09d024411b36dadf8dcb48e2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:39.907095 751 tablet_service.cc:1467] Processing CreateTablet for tablet 4791a82b5e13429184d0c7192dd7aef6 (DEFAULT_TABLE table=default.Table_f9e0115a922e4a7c825265fc1f7222f9 [id=c73f4c8fe96a4283bbb075eab16beda8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:39.911451 741 tablet_service.cc:1467] Processing CreateTablet for tablet 99e00224ed7049f88aff91013e92d4e0 (DEFAULT_TABLE table=default.Table_b638c7b0c29a41748e55d806b0924138 [id=ba5822870dd34e6eada4583f0478df63]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:39.912101 743 tablet_service.cc:1467] Processing CreateTablet for tablet 89ac5c337f394705a1fc638c9f6835fb (DEFAULT_TABLE table=default.Table_5d54936883fa4a068a24a069ad10dafb [id=6962d4bbc6f04f409a1fd3a760239000]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:39.912804 744 tablet_service.cc:1467] Processing CreateTablet for tablet 5771569d4bcd4625954dc1ba2a085b21 (DEFAULT_TABLE table=default.Table_e253836ed32b43b58cf1f7938dadf3dc [id=df9291acc6564f22b739bf3bcbb891e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:39.912909 746 tablet_service.cc:1467] Processing CreateTablet for tablet 3490334f751f4ecfbcfb8dd7329e169d (DEFAULT_TABLE table=default.Table_86c5f0e60c9347858bc135037c194bb7 [id=9a92afc2804347aebdc68a0f3c25f75e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:39.915164 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet d9df314ffaf042ec80180868d7e7bdf7 (table default.Table_2155bf71d1a04a93a28b13c97f636fab [id=00107bce6ce24feba58176198a222970]) successfully deleted
I20250113 02:51:39.915688 736 tablet_service.cc:1467] Processing CreateTablet for tablet 913c427669d24405896826e016138f83 (DEFAULT_TABLE table=default.Table_d93ff82f449f4a96b688393e8b298b42 [id=36b206c6af624ec78b3dff77eaea319e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:39.845237 1196 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37366 (request call id 1057) took 2251 ms. Trace:
I20250113 02:51:39.920778 1196 rpcz_store.cc:276] 0113 02:51:37.594173 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:37.679487 (+ 85314us) service_pool.cc:225] Handling call
0113 02:51:39.431090 (+1751603us) ts_tablet_manager.cc:1145] Deleting tablet aa5c172a69bf4ca79999f4b58df08de8
0113 02:51:39.431101 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:39.728315 (+297214us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:39.731308 (+ 2993us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:39.845217 (+113909us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1750941}
I20250113 02:51:39.921309 1196 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37366 (request call id 1064) took 2163 ms. Trace:
I20250113 02:51:39.921402 1196 rpcz_store.cc:276] 0113 02:51:37.757773 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:37.761668 (+ 3895us) service_pool.cc:225] Handling call
0113 02:51:39.921170 (+2159502us) ts_tablet_manager.cc:1145] Deleting tablet 97159fccfa154465a33ba90a27576f0a
0113 02:51:39.921182 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:39.921288 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2159308}
I20250113 02:51:39.921708 1196 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37366 (request call id 1066) took 2076 ms. Trace:
I20250113 02:51:39.921778 1196 rpcz_store.cc:276] 0113 02:51:37.844899 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:37.844976 (+ 77us) service_pool.cc:225] Handling call
0113 02:51:39.921621 (+2076645us) ts_tablet_manager.cc:1145] Deleting tablet 524b4bd640a04123b4abbd716e7d40f2
0113 02:51:39.921629 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:39.921695 (+ 66us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2076424}
I20250113 02:51:39.845542 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e2d8726a2ab489cae80fb1685d6951a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:39.922415 1191 consensus_queue.cc:260] T fb2458e2044d49029ed2d6851db9de25 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:39.922538 2373 raft_consensus.cc:491] T f1e32e09a8244957a99bdfea5812ff31 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:39.922654 2373 raft_consensus.cc:513] T f1e32e09a8244957a99bdfea5812ff31 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:39.922910 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 99e00224ed7049f88aff91013e92d4e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:39.923403 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 89ac5c337f394705a1fc638c9f6835fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:39.923751 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3490334f751f4ecfbcfb8dd7329e169d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:39.924161 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 21fcae83282d43c28000c5fe581e5ae6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:39.925014 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4791a82b5e13429184d0c7192dd7aef6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:39.925359 736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 913c427669d24405896826e016138f83. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:39.925798 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet c4b7196ef5d54459921977e81a2432db because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c4b7196ef5d54459921977e81a2432db already in progress: creating tablet
W20250113 02:51:39.926136 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet f426887763ed48e3bc87f59d80986672 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f426887763ed48e3bc87f59d80986672 already in progress: creating tablet
I20250113 02:51:39.926913 2421 raft_consensus.cc:491] T e570465a3d4046b1ac19a3a034fe91f1 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:39.927039 2421 raft_consensus.cc:513] T e570465a3d4046b1ac19a3a034fe91f1 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:39.930272 1192 raft_consensus.cc:357] T c518f791b4d74c54ab0c0364d29b4a64 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:39.930442 1192 raft_consensus.cc:383] T c518f791b4d74c54ab0c0364d29b4a64 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:39.930483 1192 raft_consensus.cc:738] T c518f791b4d74c54ab0c0364d29b4a64 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:39.930756 1198 raft_consensus.cc:2267] T d8bb60fdd06946739b75b255303d9d24 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:39.931275 1198 ts_tablet_manager.cc:1905] T d8bb60fdd06946739b75b255303d9d24 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:39.856076 899 tablet_service.cc:1467] Processing CreateTablet for tablet 228ccf742ffd4504a2bfb1fc38da377f (DEFAULT_TABLE table=default.Table_ecba1621e21641e98ab0246b22af7417 [id=e31997add9334353b2b56eb6869b8856]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:39.856525 890 tablet_service.cc:1467] Processing CreateTablet for tablet 21fcae83282d43c28000c5fe581e5ae6 (DEFAULT_TABLE table=default.Table_0f386e0d294749fba0d3c68b7bcf0f46 [id=27f04ea09d024411b36dadf8dcb48e2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:39.857108 888 tablet_service.cc:1467] Processing CreateTablet for tablet 99e00224ed7049f88aff91013e92d4e0 (DEFAULT_TABLE table=default.Table_b638c7b0c29a41748e55d806b0924138 [id=ba5822870dd34e6eada4583f0478df63]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:39.857373 895 tablet_service.cc:1467] Processing CreateTablet for tablet 913c427669d24405896826e016138f83 (DEFAULT_TABLE table=default.Table_d93ff82f449f4a96b688393e8b298b42 [id=36b206c6af624ec78b3dff77eaea319e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:39.858322 891 tablet_service.cc:1467] Processing CreateTablet for tablet 89ac5c337f394705a1fc638c9f6835fb (DEFAULT_TABLE table=default.Table_5d54936883fa4a068a24a069ad10dafb [id=6962d4bbc6f04f409a1fd3a760239000]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:39.858992 882 tablet_service.cc:1467] Processing CreateTablet for tablet 5771569d4bcd4625954dc1ba2a085b21 (DEFAULT_TABLE table=default.Table_e253836ed32b43b58cf1f7938dadf3dc [id=df9291acc6564f22b739bf3bcbb891e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:39.859463 886 tablet_service.cc:1467] Processing CreateTablet for tablet 3490334f751f4ecfbcfb8dd7329e169d (DEFAULT_TABLE table=default.Table_86c5f0e60c9347858bc135037c194bb7 [id=9a92afc2804347aebdc68a0f3c25f75e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:39.933032 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5771569d4bcd4625954dc1ba2a085b21. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:39.873558 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7842d6f2c074e3b9d1a3378db9856be" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:39.896241 2377 raft_consensus.cc:491] T 56b022c94847411785a7e9db3757a2d9 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:39.933367 2377 raft_consensus.cc:513] T 56b022c94847411785a7e9db3757a2d9 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:39.933643 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet aa5c172a69bf4ca79999f4b58df08de8 (table default.Table_3f1e7146fed34d53896f138e43a4d26e [id=5dceb76e1dbe422ab98d4da5bf6c45a0]) successfully deleted
W20250113 02:51:39.933849 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 97159fccfa154465a33ba90a27576f0a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 97159fccfa154465a33ba90a27576f0a already in progress: creating tablet
W20250113 02:51:39.934041 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 524b4bd640a04123b4abbd716e7d40f2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 524b4bd640a04123b4abbd716e7d40f2 already in progress: creating tablet
I20250113 02:51:39.934159 2377 leader_election.cc:290] T 56b022c94847411785a7e9db3757a2d9 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:39.934751 1196 tablet_replica.cc:331] T 377ebf66bbef40988043f6200ae0893c P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:39.934929 1196 raft_consensus.cc:2238] T 377ebf66bbef40988043f6200ae0893c P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:39.935061 1196 raft_consensus.cc:2267] T 377ebf66bbef40988043f6200ae0893c P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:39.934984 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1e32e09a8244957a99bdfea5812ff31" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:39.935518 998 leader_election.cc:343] T c7842d6f2c074e3b9d1a3378db9856be P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:39.935575 1196 ts_tablet_manager.cc:1905] T 377ebf66bbef40988043f6200ae0893c P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250113 02:51:39.935721 998 leader_election.cc:343] T f1e32e09a8244957a99bdfea5812ff31 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:39.936098 1058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56b022c94847411785a7e9db3757a2d9" candidate_uuid: "df69f48656b44bb890f76587ee69cc0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:39.936717 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56b022c94847411785a7e9db3757a2d9" candidate_uuid: "df69f48656b44bb890f76587ee69cc0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
I20250113 02:51:39.937232 851 leader_election.cc:304] T 56b022c94847411785a7e9db3757a2d9 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df69f48656b44bb890f76587ee69cc0f; no voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca
I20250113 02:51:39.937536 914 raft_consensus.cc:3054] T 56b022c94847411785a7e9db3757a2d9 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:39.940193 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5771569d4bcd4625954dc1ba2a085b21. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:39.940358 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 99e00224ed7049f88aff91013e92d4e0. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:39.943032 454 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 18716752335043e89038f01b4d65d711 consensus state has no leader
I20250113 02:51:39.944245 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 913c427669d24405896826e016138f83. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:39.945739 454 catalog_manager.cc:5526] T 44d2c7d5334c476fac60fdd57e2c02b6 P 4216a7f853d642b7bc3e5cff458f3aca reported cstate change: term changed from 0 to 1, leader changed from <none> to 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3). New cstate: current_term: 1 leader_uuid: "4216a7f853d642b7bc3e5cff458f3aca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } health_report { overall_health: HEALTHY } } }
I20250113 02:51:39.946157 454 catalog_manager.cc:5618] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587) does not have the latest schema for tablet 44d2c7d5334c476fac60fdd57e2c02b6 (table default.Table_6daec99b05744d0caed86e51e86f24c2 [id=2931f217041c46cc8ae52bd6ce8cfd17]). Expected version 1 got 0
I20250113 02:51:39.946434 454 catalog_manager.cc:5526] T b5c1882125b5491ea79a29a6dac6771e P 4216a7f853d642b7bc3e5cff458f3aca reported cstate change: term changed from 0 to 1, leader changed from <none> to 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3). New cstate: current_term: 1 leader_uuid: "4216a7f853d642b7bc3e5cff458f3aca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } health_report { overall_health: HEALTHY } } }
I20250113 02:51:39.946724 454 catalog_manager.cc:5618] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587) does not have the latest schema for tablet b5c1882125b5491ea79a29a6dac6771e (table default.Table_99031e0341fd4ad8a3bb04a6e222a1d0 [id=3237c1be7e1c48918ff095162d191dc1]). Expected version 3 got 0
I20250113 02:51:39.947427 449 catalog_manager.cc:6071] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: default.Table_567d446e6ae54abf9ae7586692b5780a [id=70089f757a1c45f6b983f216fb167478] alter complete (version 5)
I20250113 02:51:39.948165 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 89ac5c337f394705a1fc638c9f6835fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:39.923262 2373 leader_election.cc:290] T f1e32e09a8244957a99bdfea5812ff31 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
I20250113 02:51:39.951730 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 228ccf742ffd4504a2bfb1fc38da377f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:39.955338 1196 ts_tablet_manager.cc:1918] T 377ebf66bbef40988043f6200ae0893c P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:39.955440 1196 log.cc:1198] T 377ebf66bbef40988043f6200ae0893c P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/377ebf66bbef40988043f6200ae0893c
I20250113 02:51:39.955729 1196 ts_tablet_manager.cc:1939] T 377ebf66bbef40988043f6200ae0893c P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:39.956207 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 21fcae83282d43c28000c5fe581e5ae6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:39.957166 1192 consensus_queue.cc:260] T c518f791b4d74c54ab0c0364d29b4a64 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:39.957691 1192 ts_tablet_manager.cc:1428] T c518f791b4d74c54ab0c0364d29b4a64 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.088s user 0.000s sys 0.002s
I20250113 02:51:39.957964 1192 tablet_bootstrap.cc:492] T 788ab82626424b6e833e31761a15b9b2 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:39.959337 1192 tablet_bootstrap.cc:654] T 788ab82626424b6e833e31761a15b9b2 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:39.961894 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3490334f751f4ecfbcfb8dd7329e169d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:39.964103 455 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet ab8bf412a3f741d6a8206e026fd407dd
I20250113 02:51:39.965219 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 61c79029dfb74f01aaac0f7f828880d0
I20250113 02:51:39.965658 447 catalog_manager.cc:6596] ReplaceTablet: tablet 61c79029dfb74f01aaac0f7f828880d0 deleted and replaced by tablet 366e5c5ee708458ba443dd69d65c3988
I20250113 02:51:39.965807 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_5d4d2d0f30f44ba89da01f4af3b1a4c3" } new_table_name: "default.Table_259ab962438241e7a29cb5f526ef8ebd" modify_external_catalogs: true
I20250113 02:51:39.966130 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_dc9db684670a46229bcca7df696bb04c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:39.966750 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dc9db684670a46229bcca7df696bb04c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:39.966912 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 045167a6122a4560b298aae2358a485c
I20250113 02:51:39.967571 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_5a9dd13ac839443f862131a444ab842c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:39.968266 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_e076d93131bb4c78bbc1548986736de4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:39.968649 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e076d93131bb4c78bbc1548986736de4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250113 02:51:39.969331 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5a9dd13ac839443f862131a444ab842c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:39.970311 883 tablet_service.cc:1514] Processing DeleteTablet for tablet 045167a6122a4560b298aae2358a485c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:39 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:39.971107 1196 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37366 (request call id 1067) took 2124 ms. Trace:
I20250113 02:51:39.971340 1196 rpcz_store.cc:276] 0113 02:51:37.846329 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:37.846500 (+ 171us) service_pool.cc:225] Handling call
0113 02:51:39.921958 (+2075458us) ts_tablet_manager.cc:1145] Deleting tablet 377ebf66bbef40988043f6200ae0893c
0113 02:51:39.921966 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:39.947714 (+ 25748us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:39.955239 (+ 7525us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:39.964438 (+ 9199us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2075213}
I20250113 02:51:39.961750 2368 raft_consensus.cc:491] T c518f791b4d74c54ab0c0364d29b4a64 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:39.973038 2368 raft_consensus.cc:513] T c518f791b4d74c54ab0c0364d29b4a64 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:39.973558 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet 377ebf66bbef40988043f6200ae0893c (table default.Table_5ddfdff2cc8c4d12a1c4552e9a302027 [id=8b35e389de9f4917904e87b43618737e]) successfully deleted
I20250113 02:51:39.973986 1196 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37366 (request call id 1070) took 2096 ms. Trace:
I20250113 02:51:39.973970 1041 tablet_service.cc:1514] Processing DeleteTablet for tablet 045167a6122a4560b298aae2358a485c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:39 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:39.974380 1196 rpcz_store.cc:276] 0113 02:51:37.877731 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:37.877785 (+ 54us) service_pool.cc:225] Handling call
0113 02:51:39.973864 (+2096079us) ts_tablet_manager.cc:1145] Deleting tablet de806919192f4691bf9ee088ec77963c
0113 02:51:39.973874 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:39.973972 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2095902}
I20250113 02:51:39.974977 1196 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37366 (request call id 1068) took 2098 ms. Trace:
W20250113 02:51:39.975675 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet de806919192f4691bf9ee088ec77963c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet de806919192f4691bf9ee088ec77963c already in progress: creating tablet
I20250113 02:51:39.976449 2413 raft_consensus.cc:491] T 67baaab1e36e415381579d7385d4ab5a P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:39.976575 2413 raft_consensus.cc:513] T 67baaab1e36e415381579d7385d4ab5a P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:39.977146 2413 leader_election.cc:290] T 67baaab1e36e415381579d7385d4ab5a P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:39.977698 1058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67baaab1e36e415381579d7385d4ab5a" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
W20250113 02:51:39.978186 709 leader_election.cc:343] T 67baaab1e36e415381579d7385d4ab5a P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:39.984478 2421 leader_election.cc:290] T e570465a3d4046b1ac19a3a034fe91f1 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:39.985127 1058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e570465a3d4046b1ac19a3a034fe91f1" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
W20250113 02:51:39.985859 709 leader_election.cc:343] T e570465a3d4046b1ac19a3a034fe91f1 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:39.987644 1191 ts_tablet_manager.cc:1428] T fb2458e2044d49029ed2d6851db9de25 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.113s user 0.002s sys 0.000s
I20250113 02:51:39.994065 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67baaab1e36e415381579d7385d4ab5a" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:39.994269 2368 leader_election.cc:290] T c518f791b4d74c54ab0c0364d29b4a64 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
I20250113 02:51:39.994431 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e570465a3d4046b1ac19a3a034fe91f1" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:39.994761 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c518f791b4d74c54ab0c0364d29b4a64" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:39.994907 711 leader_election.cc:343] T 67baaab1e36e415381579d7385d4ab5a P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:39.995019 711 leader_election.cc:304] T 67baaab1e36e415381579d7385d4ab5a P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
W20250113 02:51:39.995165 998 leader_election.cc:343] T c518f791b4d74c54ab0c0364d29b4a64 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
W20250113 02:51:39.995378 711 leader_election.cc:343] T e570465a3d4046b1ac19a3a034fe91f1 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:39.995463 711 leader_election.cc:304] T e570465a3d4046b1ac19a3a034fe91f1 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:39.995777 2421 raft_consensus.cc:2743] T 67baaab1e36e415381579d7385d4ab5a P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:39.995913 2421 raft_consensus.cc:2743] T e570465a3d4046b1ac19a3a034fe91f1 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250113 02:51:39.998912 998 consensus_peers.cc:487] T bc533518faa24dffa3306ff092e8d507 P 4216a7f853d642b7bc3e5cff458f3aca -> Peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Couldn't send request to peer df69f48656b44bb890f76587ee69cc0f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250113 02:51:39.975453 1196 rpcz_store.cc:276] 0113 02:51:37.876208 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:37.878740 (+ 2532us) service_pool.cc:225] Handling call
0113 02:51:39.974885 (+2096145us) ts_tablet_manager.cc:1145] Deleting tablet a8aedbc8d7bb4183bb1ab9902f07bea7
0113 02:51:39.974893 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:39.974963 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2095663}
I20250113 02:51:40.000064 1196 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37366 (request call id 1069) took 2123 ms. Trace:
I20250113 02:51:40.000213 1196 rpcz_store.cc:276] 0113 02:51:37.876333 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:37.896494 (+ 20161us) service_pool.cc:225] Handling call
0113 02:51:39.999924 (+2103430us) ts_tablet_manager.cc:1145] Deleting tablet b1cd18ee2e2145ba9273f500d468343d
0113 02:51:39.999933 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:40.000048 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2103255}
I20250113 02:51:40.000733 1196 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37366 (request call id 1071) took 2015 ms. Trace:
I20250113 02:51:40.000862 1196 rpcz_store.cc:276] 0113 02:51:37.985306 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:37.985984 (+ 678us) service_pool.cc:225] Handling call
0113 02:51:40.000639 (+2014655us) ts_tablet_manager.cc:1145] Deleting tablet cfa963a6df094ce0b9c54408f75ca6b2
0113 02:51:40.000649 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:40.000719 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2014459}
I20250113 02:51:40.001288 1196 tablet_replica.cc:331] T 0fe5421dc3ea4be684cab084f13d8df0 P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:40.001507 1196 raft_consensus.cc:2238] T 0fe5421dc3ea4be684cab084f13d8df0 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:40.001698 1196 raft_consensus.cc:2267] T 0fe5421dc3ea4be684cab084f13d8df0 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:40.002815 1196 ts_tablet_manager.cc:1905] T 0fe5421dc3ea4be684cab084f13d8df0 P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:40.003362 1191 tablet_bootstrap.cc:492] T fe8f075d4c9a4f9abe372bfdb81a829c P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:40.004776 1191 tablet_bootstrap.cc:654] T fe8f075d4c9a4f9abe372bfdb81a829c P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
W20250113 02:51:40.009127 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet a8aedbc8d7bb4183bb1ab9902f07bea7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a8aedbc8d7bb4183bb1ab9902f07bea7 already in progress: creating tablet
W20250113 02:51:40.009368 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet b1cd18ee2e2145ba9273f500d468343d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b1cd18ee2e2145ba9273f500d468343d already in progress: creating tablet
W20250113 02:51:40.009567 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet cfa963a6df094ce0b9c54408f75ca6b2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cfa963a6df094ce0b9c54408f75ca6b2 already in progress: creating tablet
I20250113 02:51:40.015738 1198 ts_tablet_manager.cc:1918] T d8bb60fdd06946739b75b255303d9d24 P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:40.015852 1198 log.cc:1198] T d8bb60fdd06946739b75b255303d9d24 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/d8bb60fdd06946739b75b255303d9d24
I20250113 02:51:40.016501 886 tablet_service.cc:1514] Processing DeleteTablet for tablet 61c79029dfb74f01aaac0f7f828880d0 with delete_type TABLET_DATA_DELETED (replaced by tablet 366e5c5ee708458ba443dd69d65c3988) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:40.017267 1198 ts_tablet_manager.cc:1939] T d8bb60fdd06946739b75b255303d9d24 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:40.018337 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1111) took 2384 ms. Trace:
I20250113 02:51:40.019637 1198 rpcz_store.cc:276] 0113 02:51:37.634047 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:37.634380 (+ 333us) service_pool.cc:225] Handling call
0113 02:51:39.879656 (+2245276us) ts_tablet_manager.cc:1145] Deleting tablet d8bb60fdd06946739b75b255303d9d24
0113 02:51:39.889117 (+ 9461us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:40.006655 (+117538us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:40.015662 (+ 9007us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:40.018320 (+ 2658us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":41257,"tablet-delete.queue_time_us":2214771}
I20250113 02:51:40.016093 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet 61c79029dfb74f01aaac0f7f828880d0 with delete_type TABLET_DATA_DELETED (replaced by tablet 366e5c5ee708458ba443dd69d65c3988) from {username='slave'} at 127.0.0.1:60230
W20250113 02:51:40.020335 995 leader_election.cc:343] T c7842d6f2c074e3b9d1a3378db9856be P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Not found: Tablet not found: c7842d6f2c074e3b9d1a3378db9856be
I20250113 02:51:40.020476 995 leader_election.cc:304] T c7842d6f2c074e3b9d1a3378db9856be P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4216a7f853d642b7bc3e5cff458f3aca; no voters: 1186fa869eb14812832fa6562531f196, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:40.021018 2373 raft_consensus.cc:2743] T c7842d6f2c074e3b9d1a3378db9856be P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:40.021085 744 tablet_service.cc:1514] Processing DeleteTablet for tablet 045167a6122a4560b298aae2358a485c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:39 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:40.021821 2419 tablet.cc:1721] T fb18c16fb3ba44ec8a190b88d3bbd1b1 P 4216a7f853d642b7bc3e5cff458f3aca: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250113 02:51:40.024209 748 tablet_service.cc:1514] Processing DeleteTablet for tablet 61c79029dfb74f01aaac0f7f828880d0 with delete_type TABLET_DATA_DELETED (replaced by tablet 366e5c5ee708458ba443dd69d65c3988) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:40.020354 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1113) took 2386 ms. Trace:
I20250113 02:51:40.028661 1198 rpcz_store.cc:276] 0113 02:51:37.634263 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:37.634402 (+ 139us) service_pool.cc:225] Handling call
0113 02:51:40.020255 (+2385853us) ts_tablet_manager.cc:1145] Deleting tablet d5bd09a1ede644c59e400e49c8125793
0113 02:51:40.020265 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:40.020337 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2355308}
I20250113 02:51:40.028963 1198 tablet_replica.cc:331] T f8a6eebb6b6d4b63903e68750e181707 P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:40.029135 1198 raft_consensus.cc:2238] T f8a6eebb6b6d4b63903e68750e181707 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: Raft consensus shutting down.
I20250113 02:51:40.029183 1194 tablet_bootstrap.cc:492] T fb18c16fb3ba44ec8a190b88d3bbd1b1 P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:40.029335 1194 ts_tablet_manager.cc:1397] T fb18c16fb3ba44ec8a190b88d3bbd1b1 P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.202s user 0.005s sys 0.000s
I20250113 02:51:40.030164 1194 raft_consensus.cc:357] T fb18c16fb3ba44ec8a190b88d3bbd1b1 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:40.030452 1194 raft_consensus.cc:383] T fb18c16fb3ba44ec8a190b88d3bbd1b1 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:40.030537 1194 raft_consensus.cc:738] T fb18c16fb3ba44ec8a190b88d3bbd1b1 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:40.030757 1194 consensus_queue.cc:260] T fb18c16fb3ba44ec8a190b88d3bbd1b1 P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:40.031296 1194 ts_tablet_manager.cc:1428] T fb18c16fb3ba44ec8a190b88d3bbd1b1 P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250113 02:51:40.031565 1194 tablet_bootstrap.cc:492] T 1d15c7a01f854f0aaabcd30ec3166e31 P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:40.033246 1194 tablet_bootstrap.cc:654] T 1d15c7a01f854f0aaabcd30ec3166e31 P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:40.036525 1198 raft_consensus.cc:2267] T f8a6eebb6b6d4b63903e68750e181707 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:40.037552 1198 ts_tablet_manager.cc:1905] T f8a6eebb6b6d4b63903e68750e181707 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:40.040264 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet d8bb60fdd06946739b75b255303d9d24 (table default.Table_ba98ed25e93a45a288784d4e5bdee029 [id=7c84304f6d1e4b4ea99f0bc38215d74c]) successfully deleted
W20250113 02:51:40.040679 440 catalog_manager.cc:4839] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet d5bd09a1ede644c59e400e49c8125793 because the tablet was not found. No further retry: Not found: Tablet not found: d5bd09a1ede644c59e400e49c8125793
W20250113 02:51:40.063007 995 leader_election.cc:343] T c518f791b4d74c54ab0c0364d29b4a64 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Not found: Tablet not found: c518f791b4d74c54ab0c0364d29b4a64
I20250113 02:51:40.063206 995 leader_election.cc:304] T c518f791b4d74c54ab0c0364d29b4a64 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4216a7f853d642b7bc3e5cff458f3aca; no voters: 1186fa869eb14812832fa6562531f196, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:40.063686 2306 raft_consensus.cc:2743] T c518f791b4d74c54ab0c0364d29b4a64 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:40.063711 1198 ts_tablet_manager.cc:1918] T f8a6eebb6b6d4b63903e68750e181707 P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250113 02:51:40.063937 2442 raft_consensus.cc:491] T f129bb3fbbfd492d98822be0a6626e44 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:40.064077 1198 log.cc:1198] T f8a6eebb6b6d4b63903e68750e181707 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/f8a6eebb6b6d4b63903e68750e181707
W20250113 02:51:40.064344 995 leader_election.cc:343] T f1e32e09a8244957a99bdfea5812ff31 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Not found: Tablet not found: f1e32e09a8244957a99bdfea5812ff31
I20250113 02:51:40.064462 995 leader_election.cc:304] T f1e32e09a8244957a99bdfea5812ff31 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4216a7f853d642b7bc3e5cff458f3aca; no voters: 1186fa869eb14812832fa6562531f196, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:40.064673 1198 ts_tablet_manager.cc:1939] T f8a6eebb6b6d4b63903e68750e181707 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:40.064885 2306 raft_consensus.cc:2743] T f1e32e09a8244957a99bdfea5812ff31 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:40.066644 2422 tablet.cc:1721] T fb18c16fb3ba44ec8a190b88d3bbd1b1 P 1186fa869eb14812832fa6562531f196: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20250113 02:51:40.067876 752 consensus_meta.cc:287] T 546ee73435444c90ad286943807af984 P 1186fa869eb14812832fa6562531f196: Time spent flushing consensus metadata: real 0.506s user 0.014s sys 0.000s
I20250113 02:51:40.071617 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1116) took 2392 ms. Trace:
I20250113 02:51:40.071807 1198 rpcz_store.cc:276] 0113 02:51:37.679016 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:37.680839 (+ 1823us) service_pool.cc:225] Handling call
0113 02:51:40.028909 (+2348070us) ts_tablet_manager.cc:1145] Deleting tablet f8a6eebb6b6d4b63903e68750e181707
0113 02:51:40.028918 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:40.042742 (+ 13824us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:40.063608 (+ 20866us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:40.071594 (+ 7986us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2347837}
I20250113 02:51:40.073912 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 432bcbc7ba6a44ba98cfe588f8292e32
I20250113 02:51:40.074774 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_b96d277cb8434af4a5d2284a69ff62a9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:40.075074 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 797d91d4aae240bd82b00021717772ae
W20250113 02:51:40.075335 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b96d277cb8434af4a5d2284a69ff62a9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:40.075562 446 master_service.cc:946] ReplaceTablet: received request to replace tablet 1014e51f1c184d729552a5b34292f232 from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:40.076184 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_20be4eba4c334ea69d38f3336e2873e0" } new_table_name: "default.Table_4648c0331016457faa041613f30e8a95" modify_external_catalogs: true
W20250113 02:51:40.076455 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3a438fd452ba403ca6283138913b3e03 consensus state has no leader
I20250113 02:51:40.076687 1191 tablet_bootstrap.cc:492] T fe8f075d4c9a4f9abe372bfdb81a829c P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:40.076817 1198 tablet_replica.cc:331] T 9e736211eb3c4536b9fd1d90d2d83838 P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:40.076829 1191 ts_tablet_manager.cc:1397] T fe8f075d4c9a4f9abe372bfdb81a829c P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.074s user 0.002s sys 0.000s
I20250113 02:51:40.076824 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_f7a044df83094686b983b6a11fd518cd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:40.077024 1198 raft_consensus.cc:2238] T 9e736211eb3c4536b9fd1d90d2d83838 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:40.077261 1198 raft_consensus.cc:2267] T 9e736211eb3c4536b9fd1d90d2d83838 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus is shut down!
W20250113 02:51:40.077265 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f7a044df83094686b983b6a11fd518cd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:40.077680 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 67fe2f52d98048dab9f37dd820b7f712
I20250113 02:51:40.077625 1191 raft_consensus.cc:357] T fe8f075d4c9a4f9abe372bfdb81a829c P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:40.077811 1191 raft_consensus.cc:383] T fe8f075d4c9a4f9abe372bfdb81a829c P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:40.077833 1198 ts_tablet_manager.cc:1905] T 9e736211eb3c4536b9fd1d90d2d83838 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:40.077870 1191 raft_consensus.cc:738] T fe8f075d4c9a4f9abe372bfdb81a829c P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:40.078068 1191 consensus_queue.cc:260] T fe8f075d4c9a4f9abe372bfdb81a829c P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:40.078260 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_0e07e7c736e74a51bd66a147c524027f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:40.078604 1191 ts_tablet_manager.cc:1428] T fe8f075d4c9a4f9abe372bfdb81a829c P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
W20250113 02:51:40.078668 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0e07e7c736e74a51bd66a147c524027f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:40.078867 1191 tablet_bootstrap.cc:492] T 4c759831773849f2bc52e484ffa0cf49 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:40.079620 449 catalog_manager.cc:6071] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: default.Table_c506ab8e150647aaaa2baac66ae763b5 [id=e62f8e4c51b04729903b1943275c5f7f] alter complete (version 2)
I20250113 02:51:40.080507 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_5576eb7eae61401bbb7065ffbd571846"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:40.080890 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5576eb7eae61401bbb7065ffbd571846 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:40.082146 453 master_service.cc:946] ReplaceTablet: received request to replace tablet efe5f5ffd2c8423686b4007ded447466 from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:40.085134 454 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_76a8b4c723fb4930bcda1dafea1b055c" } new_table_name: "default.Table_6110cff62ea642d49ffcb38cb488a447" modify_external_catalogs: true
I20250113 02:51:40.085213 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_84475daaf7544a4b83c6846dffec23e0" } new_table_name: "default.Table_0a0f9ff8173c404bac451d9168bf9426" modify_external_catalogs: true
I20250113 02:51:40.085676 1191 tablet_bootstrap.cc:654] T 4c759831773849f2bc52e484ffa0cf49 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:40.085667 1198 ts_tablet_manager.cc:1918] T 9e736211eb3c4536b9fd1d90d2d83838 P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:40.085767 1198 log.cc:1198] T 9e736211eb3c4536b9fd1d90d2d83838 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/9e736211eb3c4536b9fd1d90d2d83838
I20250113 02:51:40.090514 1195 ts_tablet_manager.cc:1918] T bc533518faa24dffa3306ff092e8d507 P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:40.090579 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet f8a6eebb6b6d4b63903e68750e181707 (table default.Table_4484fe40b4454c5ea4943a4d38f49c5e [id=a0e91da6e0ee449b92b7afbc6f68087d]) successfully deleted
I20250113 02:51:40.097967 1195 log.cc:1198] T bc533518faa24dffa3306ff092e8d507 P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/bc533518faa24dffa3306ff092e8d507
I20250113 02:51:40.098623 1195 ts_tablet_manager.cc:1939] T bc533518faa24dffa3306ff092e8d507 P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:40.099471 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1073) took 3507 ms. Trace:
I20250113 02:51:40.102118 1195 rpcz_store.cc:276] 0113 02:51:36.592371 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:36.592435 (+ 64us) service_pool.cc:225] Handling call
0113 02:51:39.880562 (+3288127us) ts_tablet_manager.cc:1145] Deleting tablet bc533518faa24dffa3306ff092e8d507
0113 02:51:39.880573 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:40.060543 (+179970us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:40.090409 (+ 29866us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:40.099453 (+ 9044us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3287910}
I20250113 02:51:40.101401 754 tablet_service.cc:1514] Processing DeleteTablet for tablet 797d91d4aae240bd82b00021717772ae with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:39 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:40.101893 738 tablet_service.cc:1514] Processing DeleteTablet for tablet 67fe2f52d98048dab9f37dd820b7f712 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:39 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:40.102375 743 tablet_service.cc:1514] Processing DeleteTablet for tablet fd0b8592997d4a61b8ca854ecbb67c55 with delete_type TABLET_DATA_DELETED (replaced by tablet e312e70334e04ecd843159ac72f4a2a6) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:40.098227 752 tablet_service.cc:1514] Processing DeleteTablet for tablet 432bcbc7ba6a44ba98cfe588f8292e32 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:39 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:40.102944 743 tablet_service.cc:1514] Processing DeleteTablet for tablet e570465a3d4046b1ac19a3a034fe91f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:27 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:40.103191 752 tablet_service.cc:1514] Processing DeleteTablet for tablet 67baaab1e36e415381579d7385d4ab5a with delete_type TABLET_DATA_DELETED (replaced by tablet c368b5b0c3a940c4b4efb64c946375c3) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:40.103513 738 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f8eb43a39774d2c8e9cb4e88a405b0f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:23 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:40.102936 1195 tablet_replica.cc:331] T fe8f075d4c9a4f9abe372bfdb81a829c P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:40.103847 1195 raft_consensus.cc:2238] T fe8f075d4c9a4f9abe372bfdb81a829c P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:40.106364 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet bc533518faa24dffa3306ff092e8d507 (table default.Table_03babf194cc448f296512a09b2e3fe3e [id=64b42698782f41a589577af20b33501e]) successfully deleted
I20250113 02:51:40.106626 1195 raft_consensus.cc:2267] T fe8f075d4c9a4f9abe372bfdb81a829c P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:40.107203 2413 raft_consensus.cc:491] T fb2458e2044d49029ed2d6851db9de25 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:40.107342 2413 raft_consensus.cc:513] T fb2458e2044d49029ed2d6851db9de25 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:40.107897 2413 leader_election.cc:290] T fb2458e2044d49029ed2d6851db9de25 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:40.110330 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb2458e2044d49029ed2d6851db9de25" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:40.110445 886 tablet_service.cc:1514] Processing DeleteTablet for tablet 432bcbc7ba6a44ba98cfe588f8292e32 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:39 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:40.110622 890 tablet_service.cc:1514] Processing DeleteTablet for tablet 797d91d4aae240bd82b00021717772ae with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:39 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:40.110657 895 tablet_service.cc:1514] Processing DeleteTablet for tablet 67fe2f52d98048dab9f37dd820b7f712 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:39 UTC) from {username='slave'} at 127.0.0.1:37366
W20250113 02:51:40.110975 711 leader_election.cc:343] T fb2458e2044d49029ed2d6851db9de25 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:40.064199 2442 raft_consensus.cc:513] T f129bb3fbbfd492d98822be0a6626e44 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:40.113145 1191 tablet_bootstrap.cc:492] T 4c759831773849f2bc52e484ffa0cf49 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:40.113296 1191 ts_tablet_manager.cc:1397] T 4c759831773849f2bc52e484ffa0cf49 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.034s user 0.006s sys 0.000s
W20250113 02:51:40.113755 851 leader_election.cc:343] T f129bb3fbbfd492d98822be0a6626e44 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Not found: Tablet not found: f129bb3fbbfd492d98822be0a6626e44
I20250113 02:51:40.113945 1191 raft_consensus.cc:357] T 4c759831773849f2bc52e484ffa0cf49 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:40.114135 1191 raft_consensus.cc:383] T 4c759831773849f2bc52e484ffa0cf49 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:40.114187 1191 raft_consensus.cc:738] T 4c759831773849f2bc52e484ffa0cf49 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:40.114382 1191 consensus_queue.cc:260] T 4c759831773849f2bc52e484ffa0cf49 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:40.114854 1191 ts_tablet_manager.cc:1428] T 4c759831773849f2bc52e484ffa0cf49 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250113 02:51:40.115084 1191 tablet_bootstrap.cc:492] T 79aaab73797c474bae06831fa308b066 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
W20250113 02:51:40.115321 852 leader_election.cc:343] T f129bb3fbbfd492d98822be0a6626e44 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Not found: Tablet not found: f129bb3fbbfd492d98822be0a6626e44
I20250113 02:51:40.115424 852 leader_election.cc:304] T f129bb3fbbfd492d98822be0a6626e44 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df69f48656b44bb890f76587ee69cc0f; no voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca
I20250113 02:51:40.116530 1191 tablet_bootstrap.cc:654] T 79aaab73797c474bae06831fa308b066 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:40.117571 1195 ts_tablet_manager.cc:1905] T fe8f075d4c9a4f9abe372bfdb81a829c P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:40.094496 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet 432bcbc7ba6a44ba98cfe588f8292e32 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:39 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:40.094651 1038 tablet_service.cc:1514] Processing DeleteTablet for tablet 797d91d4aae240bd82b00021717772ae with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:39 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:40.094754 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet 67fe2f52d98048dab9f37dd820b7f712 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:39 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:40.094854 1040 tablet_service.cc:1514] Processing DeleteTablet for tablet c7842d6f2c074e3b9d1a3378db9856be with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:24 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:40.124153 1198 ts_tablet_manager.cc:1939] T 9e736211eb3c4536b9fd1d90d2d83838 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:40.109225 2306 consensus_queue.cc:1035] T 0644f5600c5b47e29f3fa73060ed96e3 P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 02:51:40.109362 1058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb2458e2044d49029ed2d6851db9de25" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:40.124938 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1118) took 2418 ms. Trace:
I20250113 02:51:40.125025 1198 rpcz_store.cc:276] 0113 02:51:37.706947 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:37.707002 (+ 55us) service_pool.cc:225] Handling call
0113 02:51:40.076751 (+2369749us) ts_tablet_manager.cc:1145] Deleting tablet 9e736211eb3c4536b9fd1d90d2d83838
0113 02:51:40.076765 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:40.081903 (+ 5138us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:40.085571 (+ 3668us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:40.124921 (+ 39350us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2369546}
W20250113 02:51:40.125221 709 leader_election.cc:343] T fb2458e2044d49029ed2d6851db9de25 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:40.125327 709 leader_election.cc:304] T fb2458e2044d49029ed2d6851db9de25 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:40.125419 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1120) took 2367 ms. Trace:
I20250113 02:51:40.125480 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet 9e736211eb3c4536b9fd1d90d2d83838 (table default.Table_f8af5d3e0dc64d3fae054f82a7b33ccf [id=4b739228564344b8b394a4718750dbb0]) successfully deleted
I20250113 02:51:40.125485 1198 rpcz_store.cc:276] 0113 02:51:37.757700 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:37.757751 (+ 51us) service_pool.cc:225] Handling call
0113 02:51:40.125307 (+2367556us) ts_tablet_manager.cc:1145] Deleting tablet 97159fccfa154465a33ba90a27576f0a
0113 02:51:40.125317 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:40.125404 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2367311}
I20250113 02:51:40.125784 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1123) took 2271 ms. Trace:
W20250113 02:51:40.125887 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 97159fccfa154465a33ba90a27576f0a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 97159fccfa154465a33ba90a27576f0a already in progress: creating tablet
I20250113 02:51:40.125852 1198 rpcz_store.cc:276] 0113 02:51:37.854381 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:37.856127 (+ 1746us) service_pool.cc:225] Handling call
0113 02:51:40.125691 (+2269564us) ts_tablet_manager.cc:1145] Deleting tablet 524b4bd640a04123b4abbd716e7d40f2
0113 02:51:40.125699 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:40.125771 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2250738}
I20250113 02:51:40.126104 2413 raft_consensus.cc:2743] T fb2458e2044d49029ed2d6851db9de25 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:40.126149 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1126) took 2247 ms. Trace:
I20250113 02:51:40.126206 1198 rpcz_store.cc:276] 0113 02:51:37.878735 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:37.878789 (+ 54us) service_pool.cc:225] Handling call
0113 02:51:40.126058 (+2247269us) ts_tablet_manager.cc:1145] Deleting tablet de806919192f4691bf9ee088ec77963c
0113 02:51:40.126066 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:40.126136 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2247084}
I20250113 02:51:40.126514 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1125) took 2248 ms. Trace:
W20250113 02:51:40.126561 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 524b4bd640a04123b4abbd716e7d40f2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 524b4bd640a04123b4abbd716e7d40f2 already in progress: creating tablet
I20250113 02:51:40.126590 1198 rpcz_store.cc:276] 0113 02:51:37.878475 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:37.878538 (+ 63us) service_pool.cc:225] Handling call
0113 02:51:40.126423 (+2247885us) ts_tablet_manager.cc:1145] Deleting tablet b1cd18ee2e2145ba9273f500d468343d
0113 02:51:40.126431 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:40.126501 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2224795}
W20250113 02:51:40.126756 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet de806919192f4691bf9ee088ec77963c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet de806919192f4691bf9ee088ec77963c already in progress: creating tablet
I20250113 02:51:40.126874 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1124) took 2248 ms. Trace:
I20250113 02:51:40.126945 1198 rpcz_store.cc:276] 0113 02:51:37.878037 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:37.878087 (+ 50us) service_pool.cc:225] Handling call
0113 02:51:40.126790 (+2248703us) ts_tablet_manager.cc:1145] Deleting tablet a8aedbc8d7bb4183bb1ab9902f07bea7
0113 02:51:40.126798 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:40.126862 (+ 64us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2216773}
I20250113 02:51:40.127204 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1127) took 2139 ms. Trace:
I20250113 02:51:40.127282 1198 rpcz_store.cc:276] 0113 02:51:37.987658 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:37.992040 (+ 4382us) service_pool.cc:225] Handling call
0113 02:51:40.127124 (+2135084us) ts_tablet_manager.cc:1145] Deleting tablet cfa963a6df094ce0b9c54408f75ca6b2
0113 02:51:40.127131 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:40.127192 (+ 61us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2134712}
I20250113 02:51:40.127516 1198 tablet_replica.cc:331] T b0113b16110e4787ac1c4676b2c8bd1e P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:40.127708 1198 raft_consensus.cc:2238] T b0113b16110e4787ac1c4676b2c8bd1e P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:40.127835 1198 raft_consensus.cc:2267] T b0113b16110e4787ac1c4676b2c8bd1e P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus is shut down!
W20250113 02:51:40.128289 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet b1cd18ee2e2145ba9273f500d468343d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b1cd18ee2e2145ba9273f500d468343d already in progress: creating tablet
W20250113 02:51:40.128486 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet a8aedbc8d7bb4183bb1ab9902f07bea7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a8aedbc8d7bb4183bb1ab9902f07bea7 already in progress: creating tablet
W20250113 02:51:40.128664 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet cfa963a6df094ce0b9c54408f75ca6b2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cfa963a6df094ce0b9c54408f75ca6b2 already in progress: creating tablet
I20250113 02:51:40.117729 1192 tablet_bootstrap.cc:492] T 788ab82626424b6e833e31761a15b9b2 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:40.130499 1192 ts_tablet_manager.cc:1397] T 788ab82626424b6e833e31761a15b9b2 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.173s user 0.003s sys 0.008s
I20250113 02:51:40.131127 1192 raft_consensus.cc:357] T 788ab82626424b6e833e31761a15b9b2 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:40.131286 1192 raft_consensus.cc:383] T 788ab82626424b6e833e31761a15b9b2 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:40.131336 1192 raft_consensus.cc:738] T 788ab82626424b6e833e31761a15b9b2 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:40.131528 1192 consensus_queue.cc:260] T 788ab82626424b6e833e31761a15b9b2 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:40.131934 1192 ts_tablet_manager.cc:1428] T 788ab82626424b6e833e31761a15b9b2 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250113 02:51:40.133975 1192 tablet_bootstrap.cc:492] T bbc7832c50114d6f890fec45a3d994e9 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:40.135373 1192 tablet_bootstrap.cc:654] T bbc7832c50114d6f890fec45a3d994e9 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:40.138789 1196 ts_tablet_manager.cc:1918] T 0fe5421dc3ea4be684cab084f13d8df0 P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:40.138886 1196 log.cc:1198] T 0fe5421dc3ea4be684cab084f13d8df0 P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/0fe5421dc3ea4be684cab084f13d8df0
I20250113 02:51:40.139139 1196 ts_tablet_manager.cc:1939] T 0fe5421dc3ea4be684cab084f13d8df0 P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:40.139137 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_0e6eae4ff7724f6aa85277439a83f3b1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:40.139669 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0e6eae4ff7724f6aa85277439a83f3b1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:40.139818 1196 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37366 (request call id 1072) took 1868 ms. Trace:
I20250113 02:51:40.139890 1196 rpcz_store.cc:276] 0113 02:51:38.271345 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:38.271413 (+ 68us) service_pool.cc:225] Handling call
0113 02:51:40.001244 (+1729831us) ts_tablet_manager.cc:1145] Deleting tablet 0fe5421dc3ea4be684cab084f13d8df0
0113 02:51:40.001253 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:40.007586 (+ 6333us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:40.138687 (+131101us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:40.139798 (+ 1111us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1729623}
I20250113 02:51:40.140283 1196 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37366 (request call id 1073) took 1868 ms. Trace:
I20250113 02:51:40.140360 1196 rpcz_store.cc:276] 0113 02:51:38.271475 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:38.271523 (+ 48us) service_pool.cc:225] Handling call
0113 02:51:40.140158 (+1868635us) ts_tablet_manager.cc:1145] Deleting tablet ed9e6ba0729b483fa6599dd6dde016e7
0113 02:51:40.140170 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:40.140266 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1868482}
I20250113 02:51:40.140599 1196 tablet_replica.cc:331] T 0539cbbe47e64c039ad39c4b52edcc4b P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:40.140753 1196 raft_consensus.cc:2238] T 0539cbbe47e64c039ad39c4b52edcc4b P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:40.140785 1191 tablet_bootstrap.cc:492] T 79aaab73797c474bae06831fa308b066 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:40.140882 1196 raft_consensus.cc:2267] T 0539cbbe47e64c039ad39c4b52edcc4b P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:40.140900 1191 ts_tablet_manager.cc:1397] T 79aaab73797c474bae06831fa308b066 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.026s user 0.002s sys 0.000s
I20250113 02:51:40.141371 1196 ts_tablet_manager.cc:1905] T 0539cbbe47e64c039ad39c4b52edcc4b P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:40.141662 1191 raft_consensus.cc:357] T 79aaab73797c474bae06831fa308b066 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:40.141840 1191 raft_consensus.cc:383] T 79aaab73797c474bae06831fa308b066 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:40.141887 1191 raft_consensus.cc:738] T 79aaab73797c474bae06831fa308b066 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:40.142096 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet efe5f5ffd2c8423686b4007ded447466
I20250113 02:51:40.142500 453 catalog_manager.cc:6596] ReplaceTablet: tablet efe5f5ffd2c8423686b4007ded447466 deleted and replaced by tablet 0136684e4cbe428db182017844e3909d
I20250113 02:51:40.143626 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet 0fe5421dc3ea4be684cab084f13d8df0 (table default.Table_ca04ad8d0d8b40d1ba6b147731579526 [id=beb83fe520f94241949a46d436e6374f]) successfully deleted
W20250113 02:51:40.143828 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet ed9e6ba0729b483fa6599dd6dde016e7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ed9e6ba0729b483fa6599dd6dde016e7 already in progress: creating tablet
I20250113 02:51:40.144750 895 tablet_service.cc:1514] Processing DeleteTablet for tablet efe5f5ffd2c8423686b4007ded447466 with delete_type TABLET_DATA_DELETED (replaced by tablet 0136684e4cbe428db182017844e3909d) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:40.147814 1196 ts_tablet_manager.cc:1918] T 0539cbbe47e64c039ad39c4b52edcc4b P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:40.148010 1196 log.cc:1198] T 0539cbbe47e64c039ad39c4b52edcc4b P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/0539cbbe47e64c039ad39c4b52edcc4b
I20250113 02:51:40.147941 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_edd55c813c94406ea65e9215b40632b6" } new_table_name: "default.Table_e38f2c9d348948238d245b93b82b0dce" modify_external_catalogs: true
I20250113 02:51:40.148415 1196 ts_tablet_manager.cc:1939] T 0539cbbe47e64c039ad39c4b52edcc4b P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:40.149221 1196 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37366 (request call id 1074) took 1850 ms. Trace:
I20250113 02:51:40.149380 1196 rpcz_store.cc:276] 0113 02:51:38.298436 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:38.298492 (+ 56us) service_pool.cc:225] Handling call
0113 02:51:40.140557 (+1842065us) ts_tablet_manager.cc:1145] Deleting tablet 0539cbbe47e64c039ad39c4b52edcc4b
0113 02:51:40.140565 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:40.144859 (+ 4294us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:40.147732 (+ 2873us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:40.149204 (+ 1472us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1841891}
W20250113 02:51:40.149915 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b7247150e20348619fda21ebbdd305c8 has no consensus state
I20250113 02:51:40.150357 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet 0539cbbe47e64c039ad39c4b52edcc4b (table default.Table_d1b93410e6504e71a32397040f50b464 [id=817479ce0ac04442a0f89e6115d24ba4]) successfully deleted
I20250113 02:51:40.150962 1196 tablet_replica.cc:331] T 99c8046f5d124ca49899df365114c69f P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:40.151188 1196 raft_consensus.cc:2238] T 99c8046f5d124ca49899df365114c69f P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:40.151384 1196 raft_consensus.cc:2267] T 99c8046f5d124ca49899df365114c69f P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:40.151928 1196 ts_tablet_manager.cc:1905] T 99c8046f5d124ca49899df365114c69f P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:40.156395 1198 ts_tablet_manager.cc:1905] T b0113b16110e4787ac1c4676b2c8bd1e P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:40.158437 2442 leader_election.cc:290] T f129bb3fbbfd492d98822be0a6626e44 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:40.158680 2442 raft_consensus.cc:2743] T f129bb3fbbfd492d98822be0a6626e44 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:40.158988 2427 tablet.cc:1721] T 56b022c94847411785a7e9db3757a2d9 P df69f48656b44bb890f76587ee69cc0f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250113 02:51:40.160519 2377 raft_consensus.cc:2743] T 56b022c94847411785a7e9db3757a2d9 P df69f48656b44bb890f76587ee69cc0f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:40.164429 1198 ts_tablet_manager.cc:1918] T b0113b16110e4787ac1c4676b2c8bd1e P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:40.164532 1198 log.cc:1198] T b0113b16110e4787ac1c4676b2c8bd1e P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/b0113b16110e4787ac1c4676b2c8bd1e
I20250113 02:51:40.165552 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_9175390f220844ad984ccaa8df33cecf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:40.142071 1191 consensus_queue.cc:260] T 79aaab73797c474bae06831fa308b066 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:40.147965 1195 ts_tablet_manager.cc:1918] T fe8f075d4c9a4f9abe372bfdb81a829c P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:40.167606 1195 log.cc:1198] T fe8f075d4c9a4f9abe372bfdb81a829c P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/fe8f075d4c9a4f9abe372bfdb81a829c
I20250113 02:51:40.167953 1195 ts_tablet_manager.cc:1939] T fe8f075d4c9a4f9abe372bfdb81a829c P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:40.169824 738 tablet_service.cc:1514] Processing DeleteTablet for tablet efe5f5ffd2c8423686b4007ded447466 with delete_type TABLET_DATA_DELETED (replaced by tablet 0136684e4cbe428db182017844e3909d) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:40.170542 446 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_17489ca6f295469eb97e9a3a5c7cb1aa" } modify_external_catalogs: true
I20250113 02:51:40.170745 446 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_17489ca6f295469eb97e9a3a5c7cb1aa" } modify_external_catalogs: true
I20250113 02:51:40.172950 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_03ffbd822433478e8576aa5297fc07ae"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:40.175956 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_bb52873ae0724253b10583549f8f3596" } modify_external_catalogs: true
I20250113 02:51:40.176124 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_bb52873ae0724253b10583549f8f3596" } modify_external_catalogs: true
I20250113 02:51:40.176129 1031 tablet_service.cc:1514] Processing DeleteTablet for tablet efe5f5ffd2c8423686b4007ded447466 with delete_type TABLET_DATA_DELETED (replaced by tablet 0136684e4cbe428db182017844e3909d) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:40.158497 1196 ts_tablet_manager.cc:1918] T 99c8046f5d124ca49899df365114c69f P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:40.176429 1196 log.cc:1198] T 99c8046f5d124ca49899df365114c69f P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/99c8046f5d124ca49899df365114c69f
I20250113 02:51:40.176759 1196 ts_tablet_manager.cc:1939] T 99c8046f5d124ca49899df365114c69f P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:40.177544 1196 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37366 (request call id 1075) took 1614 ms. Trace:
I20250113 02:51:40.177644 1196 rpcz_store.cc:276] 0113 02:51:38.562921 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:38.564284 (+ 1363us) service_pool.cc:225] Handling call
0113 02:51:40.150918 (+1586634us) ts_tablet_manager.cc:1145] Deleting tablet 99c8046f5d124ca49899df365114c69f
0113 02:51:40.150928 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:40.155085 (+ 4157us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:40.158427 (+ 3342us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:40.177524 (+ 19097us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1561048}
I20250113 02:51:40.177979 1196 tablet_replica.cc:331] T 42cf1e708644427f9d4dcd41ae2bcb37 P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:40.178150 1196 raft_consensus.cc:2238] T 42cf1e708644427f9d4dcd41ae2bcb37 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:40.178306 1196 raft_consensus.cc:2267] T 42cf1e708644427f9d4dcd41ae2bcb37 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:40.178802 1196 ts_tablet_manager.cc:1905] T 42cf1e708644427f9d4dcd41ae2bcb37 P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:40.180516 2413 raft_consensus.cc:491] T 21bcdd741104474bad7023426856d18f P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:40.180635 2413 raft_consensus.cc:513] T 21bcdd741104474bad7023426856d18f P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:40.181420 2413 leader_election.cc:290] T 21bcdd741104474bad7023426856d18f P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:40.182161 1058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21bcdd741104474bad7023426856d18f" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
W20250113 02:51:40.183081 709 leader_election.cc:343] T 21bcdd741104474bad7023426856d18f P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:40.183580 1191 ts_tablet_manager.cc:1428] T 79aaab73797c474bae06831fa308b066 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.043s user 0.002s sys 0.000s
I20250113 02:51:40.183571 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet 99c8046f5d124ca49899df365114c69f (table default.Table_7805d72c367b4ef8aa585de05f3d9b90 [id=d4ce999a9a7b47ada946f259228771bb]) successfully deleted
I20250113 02:51:40.184250 1191 tablet_bootstrap.cc:492] T f1237f7b04af4ebbb39b03aa3c78a499 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:40.180646 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1074) took 3579 ms. Trace:
I20250113 02:51:40.184824 1195 rpcz_store.cc:276] 0113 02:51:36.601615 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:36.601669 (+ 54us) service_pool.cc:225] Handling call
0113 02:51:40.102890 (+3501221us) ts_tablet_manager.cc:1145] Deleting tablet fe8f075d4c9a4f9abe372bfdb81a829c
0113 02:51:40.102900 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:40.127148 (+ 24248us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:40.147873 (+ 20725us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:40.180629 (+ 32756us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":8371,"tablet-delete.queue_time_us":3501025}
I20250113 02:51:40.185205 1195 tablet_replica.cc:331] T 0644f5600c5b47e29f3fa73060ed96e3 P 1186fa869eb14812832fa6562531f196: stopping tablet replica
W20250113 02:51:40.188951 852 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.3:42587, user_credentials={real_user=slave}} blocked reactor thread for 73630us
I20250113 02:51:40.190222 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21bcdd741104474bad7023426856d18f" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:40.190651 711 leader_election.cc:343] T 21bcdd741104474bad7023426856d18f P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:40.190769 711 leader_election.cc:304] T 21bcdd741104474bad7023426856d18f P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:40.191248 2413 raft_consensus.cc:2743] T 21bcdd741104474bad7023426856d18f P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:40.191695 1198 ts_tablet_manager.cc:1939] T b0113b16110e4787ac1c4676b2c8bd1e P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:40.194168 1191 tablet_bootstrap.cc:654] T f1237f7b04af4ebbb39b03aa3c78a499 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
W20250113 02:51:40.194355 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 85d13f8abbec484d82c4d0b86480531b consensus state has no leader
I20250113 02:51:40.196538 449 catalog_manager.cc:5618] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467) does not have the latest schema for tablet fb18c16fb3ba44ec8a190b88d3bbd1b1 (table default.Table_7a4628807cc44ebdadf927ff709de3a2 [id=75ca7cbd7b9548d784f9a77d0f0cb64f]). Expected version 1 got 0
I20250113 02:51:40.196718 449 catalog_manager.cc:5618] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467) does not have the latest schema for tablet 56b022c94847411785a7e9db3757a2d9 (table default.Table_6461b062100240d59a5d91a87a74510c [id=40ca872c045b4349b63ba9046e31ed13]). Expected version 2 got 0
W20250113 02:51:40.197525 998 consensus_peers.cc:487] T 0644f5600c5b47e29f3fa73060ed96e3 P 4216a7f853d642b7bc3e5cff458f3aca -> Peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Couldn't send request to peer df69f48656b44bb890f76587ee69cc0f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250113 02:51:40.198736 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_03ffbd822433478e8576aa5297fc07ae in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:40.186808 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet fe8f075d4c9a4f9abe372bfdb81a829c (table default.Table_ce5d67ac15bc4d89a173d58c766a259c [id=d37981b613bd494685de3e48d382f485]) successfully deleted
W20250113 02:51:40.198779 454 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ab73866613c841ee85f3a4a2c4a0733b consensus state has no leader
I20250113 02:51:40.199849 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1128) took 2188 ms. Trace:
W20250113 02:51:40.199914 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9175390f220844ad984ccaa8df33cecf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:40.200532 1195 raft_consensus.cc:2238] T 0644f5600c5b47e29f3fa73060ed96e3 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:40.200843 1195 pending_rounds.cc:62] T 0644f5600c5b47e29f3fa73060ed96e3 P 1186fa869eb14812832fa6562531f196: Trying to abort 1 pending ops.
I20250113 02:51:40.200925 1195 pending_rounds.cc:69] T 0644f5600c5b47e29f3fa73060ed96e3 P 1186fa869eb14812832fa6562531f196: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113673519090003968 op_type: NO_OP noop_request { }
I20250113 02:51:40.201084 1195 raft_consensus.cc:2883] T 0644f5600c5b47e29f3fa73060ed96e3 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20250113 02:51:40.201211 1195 raft_consensus.cc:2267] T 0644f5600c5b47e29f3fa73060ed96e3 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:40.202277 1191 tablet_bootstrap.cc:492] T f1237f7b04af4ebbb39b03aa3c78a499 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:40.207551 1198 rpcz_store.cc:276] 0113 02:51:38.011718 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:38.019828 (+ 8110us) service_pool.cc:225] Handling call
0113 02:51:40.127477 (+2107649us) ts_tablet_manager.cc:1145] Deleting tablet b0113b16110e4787ac1c4676b2c8bd1e
0113 02:51:40.127485 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:40.161653 (+ 34168us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:40.164349 (+ 2696us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:40.199827 (+ 35478us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2107458}
I20250113 02:51:40.208110 1191 ts_tablet_manager.cc:1397] T f1237f7b04af4ebbb39b03aa3c78a499 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.024s user 0.003s sys 0.000s
I20250113 02:51:40.209371 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet b0113b16110e4787ac1c4676b2c8bd1e (table default.Table_d5c2b805dfbf4227bb1ac190525b7dee [id=bf855203fc20403294cc733cdbc593e5]) successfully deleted
I20250113 02:51:40.208983 1192 tablet_bootstrap.cc:492] T bbc7832c50114d6f890fec45a3d994e9 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:40.209802 1194 tablet_bootstrap.cc:492] T 1d15c7a01f854f0aaabcd30ec3166e31 P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:40.209795 1192 ts_tablet_manager.cc:1397] T bbc7832c50114d6f890fec45a3d994e9 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.076s user 0.003s sys 0.000s
I20250113 02:51:40.210038 1194 ts_tablet_manager.cc:1397] T 1d15c7a01f854f0aaabcd30ec3166e31 P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.179s user 0.007s sys 0.007s
I20250113 02:51:40.210799 1192 raft_consensus.cc:357] T bbc7832c50114d6f890fec45a3d994e9 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:40.210965 1194 raft_consensus.cc:357] T 1d15c7a01f854f0aaabcd30ec3166e31 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:40.211113 1192 raft_consensus.cc:383] T bbc7832c50114d6f890fec45a3d994e9 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:40.211206 1192 raft_consensus.cc:738] T bbc7832c50114d6f890fec45a3d994e9 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:40.211226 1194 raft_consensus.cc:383] T 1d15c7a01f854f0aaabcd30ec3166e31 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:40.211333 1194 raft_consensus.cc:738] T 1d15c7a01f854f0aaabcd30ec3166e31 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:40.211571 1194 consensus_queue.cc:260] T 1d15c7a01f854f0aaabcd30ec3166e31 P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:40.211573 1192 consensus_queue.cc:260] T bbc7832c50114d6f890fec45a3d994e9 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:40.212334 1192 ts_tablet_manager.cc:1428] T bbc7832c50114d6f890fec45a3d994e9 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
W20250113 02:51:40.207396 2453 consensus_queue.cc:1159] T 0644f5600c5b47e29f3fa73060ed96e3 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "1186fa869eb14812832fa6562531f196" status { last_received { term: 1 index: 1 } last_committed_idx: 0 last_received_current_leader { term: 1 index: 1 } }
I20250113 02:51:40.212881 1192 tablet_bootstrap.cc:492] T b33025a462e74f2aa26cc3779f36615b P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:40.215154 451 catalog_manager.cc:6071] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: default.Table_7a4628807cc44ebdadf927ff709de3a2 [id=75ca7cbd7b9548d784f9a77d0f0cb64f] alter complete (version 1)
I20250113 02:51:40.215600 1195 ts_tablet_manager.cc:1905] T 0644f5600c5b47e29f3fa73060ed96e3 P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250113 02:51:40.216104 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet eb4a985261f4452db4de74891b7bf65a consensus state has no leader
I20250113 02:51:40.219883 738 tablet_service.cc:1514] Processing DeleteTablet for tablet 21bcdd741104474bad7023426856d18f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:33 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:40.220352 738 tablet_service.cc:1514] Processing DeleteTablet for tablet fb2458e2044d49029ed2d6851db9de25 with delete_type TABLET_DATA_DELETED (replaced by tablet 32e3dc0b7c044415a6343368ffa08552) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:40.220682 1194 ts_tablet_manager.cc:1428] T 1d15c7a01f854f0aaabcd30ec3166e31 P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.011s user 0.001s sys 0.001s
I20250113 02:51:40.220702 738 tablet_service.cc:1514] Processing DeleteTablet for tablet fe8f075d4c9a4f9abe372bfdb81a829c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:36 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:40.220990 1194 tablet_bootstrap.cc:492] T 3025935f9743442eb3e6f2db22eb52ad P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:40.214720 1198 tablet_replica.cc:331] T 0644f5600c5b47e29f3fa73060ed96e3 P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
W20250113 02:51:40.215030 995 consensus_peers.cc:487] T 0644f5600c5b47e29f3fa73060ed96e3 P 4216a7f853d642b7bc3e5cff458f3aca -> Peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Couldn't send request to peer 1186fa869eb14812832fa6562531f196. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: STOPPING. This is attempt 1: this message will repeat every 5th retry.
I20250113 02:51:40.222786 1194 tablet_bootstrap.cc:654] T 3025935f9743442eb3e6f2db22eb52ad P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:40.228133 1191 raft_consensus.cc:357] T f1237f7b04af4ebbb39b03aa3c78a499 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:40.228444 1191 raft_consensus.cc:383] T f1237f7b04af4ebbb39b03aa3c78a499 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:40.228538 1191 raft_consensus.cc:738] T f1237f7b04af4ebbb39b03aa3c78a499 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:40.228804 1191 consensus_queue.cc:260] T f1237f7b04af4ebbb39b03aa3c78a499 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:40.231318 447 master_service.cc:946] ReplaceTablet: received request to replace tablet c368b5b0c3a940c4b4efb64c946375c3 from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:40.231462 738 tablet_service.cc:1514] Processing DeleteTablet for tablet 79aaab73797c474bae06831fa308b066 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:30 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:40.232028 1191 ts_tablet_manager.cc:1428] T f1237f7b04af4ebbb39b03aa3c78a499 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.020s user 0.002s sys 0.000s
I20250113 02:51:40.235067 1191 tablet_bootstrap.cc:492] T cfa963a6df094ce0b9c54408f75ca6b2 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:40.235638 895 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d15c7a01f854f0aaabcd30ec3166e31 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:31 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:40.235896 1031 tablet_service.cc:1514] Processing DeleteTablet for tablet c518f791b4d74c54ab0c0364d29b4a64 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:23 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:40.237112 1191 tablet_bootstrap.cc:654] T cfa963a6df094ce0b9c54408f75ca6b2 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:40.243516 1194 tablet_bootstrap.cc:492] T 3025935f9743442eb3e6f2db22eb52ad P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:40.243752 1194 ts_tablet_manager.cc:1397] T 3025935f9743442eb3e6f2db22eb52ad P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.023s user 0.003s sys 0.003s
I20250113 02:51:40.244673 1194 raft_consensus.cc:357] T 3025935f9743442eb3e6f2db22eb52ad P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:40.244947 1194 raft_consensus.cc:383] T 3025935f9743442eb3e6f2db22eb52ad P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:40.245070 1194 raft_consensus.cc:738] T 3025935f9743442eb3e6f2db22eb52ad P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:40.245404 1194 consensus_queue.cc:260] T 3025935f9743442eb3e6f2db22eb52ad P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:40.246394 1040 tablet_service.cc:1514] Processing DeleteTablet for tablet bbc7832c50114d6f890fec45a3d994e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:22 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:40.249024 1031 tablet_service.cc:1514] Processing DeleteTablet for tablet 788ab82626424b6e833e31761a15b9b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:26 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:40.249270 1191 tablet_bootstrap.cc:492] T cfa963a6df094ce0b9c54408f75ca6b2 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:40.249411 1191 ts_tablet_manager.cc:1397] T cfa963a6df094ce0b9c54408f75ca6b2 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.000s
I20250113 02:51:40.251546 738 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c759831773849f2bc52e484ffa0cf49 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:30 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:40.251586 1191 raft_consensus.cc:357] T cfa963a6df094ce0b9c54408f75ca6b2 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:40.251762 1191 raft_consensus.cc:383] T cfa963a6df094ce0b9c54408f75ca6b2 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:40.251825 1191 raft_consensus.cc:738] T cfa963a6df094ce0b9c54408f75ca6b2 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:40.252101 1191 consensus_queue.cc:260] T cfa963a6df094ce0b9c54408f75ca6b2 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:40.252768 1191 ts_tablet_manager.cc:1428] T cfa963a6df094ce0b9c54408f75ca6b2 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250113 02:51:40.253115 1191 tablet_bootstrap.cc:492] T cf7b8bd6801743f49e9b9473747447c5 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:40.253156 1194 ts_tablet_manager.cc:1428] T 3025935f9743442eb3e6f2db22eb52ad P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20250113 02:51:40.256953 895 tablet_service.cc:1514] Processing DeleteTablet for tablet 3025935f9743442eb3e6f2db22eb52ad with delete_type TABLET_DATA_DELETED (replaced by tablet 5d2d14548b734b1fbe57d68c0ea3e2f8) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:40.260598 1195 ts_tablet_manager.cc:1918] T 0644f5600c5b47e29f3fa73060ed96e3 P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250113 02:51:40.261603 1198 raft_consensus.cc:2238] T 0644f5600c5b47e29f3fa73060ed96e3 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: Raft consensus shutting down.
I20250113 02:51:40.262329 1198 raft_consensus.cc:2267] T 0644f5600c5b47e29f3fa73060ed96e3 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:40.265555 1192 tablet_bootstrap.cc:654] T b33025a462e74f2aa26cc3779f36615b P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:40.271435 1194 tablet_bootstrap.cc:492] T 999fecb3fc65466cad21cc1ff5ef2d44 P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:40.273809 1195 log.cc:1198] T 0644f5600c5b47e29f3fa73060ed96e3 P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/0644f5600c5b47e29f3fa73060ed96e3
I20250113 02:51:40.275697 450 catalog_manager.cc:5618] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587) does not have the latest schema for tablet 0b119704bc774ca9b4155c9619d4765b (table default.Table_b95a9b1d12284c9aac701a8ef9c93482 [id=843aabcff7074acd98e5801b21282faa]). Expected version 1 got 0
I20250113 02:51:40.276466 450 catalog_manager.cc:5618] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587) does not have the latest schema for tablet b5c1882125b5491ea79a29a6dac6771e (table default.Table_99031e0341fd4ad8a3bb04a6e222a1d0 [id=3237c1be7e1c48918ff095162d191dc1]). Expected version 3 got 0
I20250113 02:51:40.277524 450 catalog_manager.cc:5618] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587) does not have the latest schema for tablet 44d2c7d5334c476fac60fdd57e2c02b6 (table default.Table_6daec99b05744d0caed86e51e86f24c2 [id=2931f217041c46cc8ae52bd6ce8cfd17]). Expected version 1 got 0
I20250113 02:51:40.279050 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 368e5f00861b40ffb76e79376245b885
I20250113 02:51:40.272691 1198 ts_tablet_manager.cc:1905] T 0644f5600c5b47e29f3fa73060ed96e3 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:40.280150 1031 tablet_service.cc:1514] Processing DeleteTablet for tablet 368e5f00861b40ffb76e79376245b885 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:40 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:40.285848 455 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_d70d62ca846e4071af60ad816ded897f" } modify_external_catalogs: true
I20250113 02:51:40.286026 455 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_d70d62ca846e4071af60ad816ded897f" } modify_external_catalogs: true
I20250113 02:51:40.287338 448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_165edade161e458c81a066ac578d832c" } new_table_name: "default.Table_7cdc614381e047e6a44f1c316214d076" modify_external_catalogs: true
I20250113 02:51:40.280382 738 tablet_service.cc:1514] Processing DeleteTablet for tablet 368e5f00861b40ffb76e79376245b885 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:40 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:40.261075 1191 tablet_bootstrap.cc:654] T cf7b8bd6801743f49e9b9473747447c5 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:40.288792 1195 ts_tablet_manager.cc:1939] T 0644f5600c5b47e29f3fa73060ed96e3 P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:40.288913 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_b96d277cb8434af4a5d2284a69ff62a9" } modify_external_catalogs: true
I20250113 02:51:40.289072 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_b96d277cb8434af4a5d2284a69ff62a9" } modify_external_catalogs: true
I20250113 02:51:40.290239 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1072) took 3698 ms. Trace:
I20250113 02:51:40.291426 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_4d27f00185134500b7a8461f7efe9961"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:40.291913 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4d27f00185134500b7a8461f7efe9961 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:40.292997 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 59fadc693f554854bc2874f8db7c64c8
I20250113 02:51:40.293036 1195 rpcz_store.cc:276] 0113 02:51:36.592193 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:36.602386 (+ 10193us) service_pool.cc:225] Handling call
0113 02:51:40.185154 (+3582768us) ts_tablet_manager.cc:1145] Deleting tablet 0644f5600c5b47e29f3fa73060ed96e3
0113 02:51:40.185176 (+ 22us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:40.200776 (+ 15600us) spinlock_profiling.cc:243] Waited 15.1 ms on lock 0x6160000e5054. stack: 00007f2b88d31fd6 00007f2b88d31d90 00007f2b87d1148d 000000000096d544 000000000096d50d 000000000096ceda 00007f2b945d4ed0 00007f2b958c9463 00007f2b9836deb7 00007f2b983a0584 00007f2b98381690 00007f2b9838141d 0000000000a71ecc 00007f2b88d78b64 00007f2b88d7c18b 00007f2b88d7c008
0113 02:51:40.218680 (+ 17904us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:40.260492 (+ 41812us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:40.290216 (+ 29724us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":34782976,"tablet-delete.queue_time_us":3582521}
I20250113 02:51:40.293677 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet 0644f5600c5b47e29f3fa73060ed96e3 (table default.Table_cd499f4a8487400690a9ab957c606b7f [id=be5f659ed070438bb6d0e61a3820eb58]) successfully deleted
I20250113 02:51:40.293846 738 tablet_service.cc:1514] Processing DeleteTablet for tablet 59fadc693f554854bc2874f8db7c64c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:40 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:40.294235 1195 tablet_replica.cc:331] T ba8980ad3a364c888a825affbcbc9b1f P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:40.298194 1195 raft_consensus.cc:2238] T ba8980ad3a364c888a825affbcbc9b1f P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:40.284668 895 tablet_service.cc:1514] Processing DeleteTablet for tablet 368e5f00861b40ffb76e79376245b885 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:40 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:40.298403 1195 raft_consensus.cc:2267] T ba8980ad3a364c888a825affbcbc9b1f P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:40.294477 890 tablet_service.cc:1514] Processing DeleteTablet for tablet 59fadc693f554854bc2874f8db7c64c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:40 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:40.298738 453 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_b17de55bafab4c41b91c30d96a90b75c" } modify_external_catalogs: true
I20250113 02:51:40.298900 1195 ts_tablet_manager.cc:1905] T ba8980ad3a364c888a825affbcbc9b1f P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:40.298923 453 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_b17de55bafab4c41b91c30d96a90b75c" } modify_external_catalogs: true
I20250113 02:51:40.300089 1194 tablet_bootstrap.cc:654] T 999fecb3fc65466cad21cc1ff5ef2d44 P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:40.300855 1191 tablet_bootstrap.cc:492] T cf7b8bd6801743f49e9b9473747447c5 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:40.300992 1191 ts_tablet_manager.cc:1397] T cf7b8bd6801743f49e9b9473747447c5 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.048s user 0.004s sys 0.000s
I20250113 02:51:40.301826 1191 raft_consensus.cc:357] T cf7b8bd6801743f49e9b9473747447c5 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:40.302089 1191 raft_consensus.cc:383] T cf7b8bd6801743f49e9b9473747447c5 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:40.302153 1191 raft_consensus.cc:738] T cf7b8bd6801743f49e9b9473747447c5 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
W20250113 02:51:40.303593 995 consensus_peers.cc:487] T bc533518faa24dffa3306ff092e8d507 P 4216a7f853d642b7bc3e5cff458f3aca -> Peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Couldn't send request to peer 1186fa869eb14812832fa6562531f196. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bc533518faa24dffa3306ff092e8d507. This is attempt 1: this message will repeat every 5th retry.
I20250113 02:51:40.304780 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 348c21d783ef4103a37409ef8634eb35
I20250113 02:51:40.306789 890 tablet_service.cc:1514] Processing DeleteTablet for tablet 348c21d783ef4103a37409ef8634eb35 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:40 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:40.307396 738 tablet_service.cc:1514] Processing DeleteTablet for tablet 348c21d783ef4103a37409ef8634eb35 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:40 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:40.302393 1191 consensus_queue.cc:260] T cf7b8bd6801743f49e9b9473747447c5 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:40.308080 1031 tablet_service.cc:1514] Processing DeleteTablet for tablet 59fadc693f554854bc2874f8db7c64c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:40 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:40.310952 1031 tablet_service.cc:1514] Processing DeleteTablet for tablet 348c21d783ef4103a37409ef8634eb35 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:40 UTC) from {username='slave'} at 127.0.0.1:60230
W20250113 02:51:40.311348 440 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet eb4a985261f4452db4de74891b7bf65a (table default.Table_e38f2c9d348948238d245b93b82b0dce [id=7caed6f6020f49a5898a2261f653070d], current schema version=4) failed: Not found: failed to reset TS proxy: No leader found: Tablet eb4a985261f4452db4de74891b7bf65a consensus state has no leader
I20250113 02:51:40.311276 1191 ts_tablet_manager.cc:1428] T cf7b8bd6801743f49e9b9473747447c5 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.010s user 0.002s sys 0.001s
I20250113 02:51:40.312250 1191 tablet_bootstrap.cc:492] T ca75e7cae4f6423191a17bb5a8f150a9 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:40.313812 1191 tablet_bootstrap.cc:654] T ca75e7cae4f6423191a17bb5a8f150a9 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:40.314263 449 catalog_manager.cc:5618] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095) does not have the latest schema for tablet f1237f7b04af4ebbb39b03aa3c78a499 (table default.Table_4c5e2e1008b54019a243ffe1ec72c020 [id=0817677719d045fd83bb85890f11df25]). Expected version 2 got 0
I20250113 02:51:40.314666 653 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 348c21d783ef4103a37409ef8634eb35
I20250113 02:51:40.315346 890 tablet_service.cc:1514] Processing DeleteTablet for tablet 348c21d783ef4103a37409ef8634eb35 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:40 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:40.316071 1031 tablet_service.cc:1514] Processing DeleteTablet for tablet 348c21d783ef4103a37409ef8634eb35 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:40 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:40.316824 890 tablet_service.cc:1467] Processing CreateTablet for tablet a86210c2f74c41458621f0683bed3ce3 (DEFAULT_TABLE table=default.Table_c89db70b70a64f88a0203c8c66867911 [id=fb02a240886448c6811de4ae0d192d3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:40.317271 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a86210c2f74c41458621f0683bed3ce3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:40.319062 738 tablet_service.cc:1514] Processing DeleteTablet for tablet 348c21d783ef4103a37409ef8634eb35 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:40 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:40.324623 738 tablet_service.cc:1467] Processing CreateTablet for tablet a86210c2f74c41458621f0683bed3ce3 (DEFAULT_TABLE table=default.Table_c89db70b70a64f88a0203c8c66867911 [id=fb02a240886448c6811de4ae0d192d3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:40.324748 890 tablet_service.cc:1467] Processing CreateTablet for tablet 5f0ab4954da04bf38e8c49d29b137061 (DEFAULT_TABLE table=default.Table_2e04cc940d734e3bbe72ae90690357d6 [id=cb5111f2024c49dab0bb6727e3e0b2a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:40.325021 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a86210c2f74c41458621f0683bed3ce3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:40.325134 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f0ab4954da04bf38e8c49d29b137061. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:40.338478 1031 tablet_service.cc:1467] Processing CreateTablet for tablet a86210c2f74c41458621f0683bed3ce3 (DEFAULT_TABLE table=default.Table_c89db70b70a64f88a0203c8c66867911 [id=fb02a240886448c6811de4ae0d192d3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:40.338867 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a86210c2f74c41458621f0683bed3ce3. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:40.342746 448 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f59dc0fa43dd4eb5932a1ad8a5a08349 consensus state has no leader
I20250113 02:51:40.344022 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_ff5dca8686a64b858a2ab711d109d4a9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:40.344372 452 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet 0757a3bb04344874bda487041a0fa002
W20250113 02:51:40.344569 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ff5dca8686a64b858a2ab711d109d4a9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:40.346530 1198 ts_tablet_manager.cc:1918] T 0644f5600c5b47e29f3fa73060ed96e3 P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250113 02:51:40.346650 1198 log.cc:1198] T 0644f5600c5b47e29f3fa73060ed96e3 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/0644f5600c5b47e29f3fa73060ed96e3
I20250113 02:51:40.347044 1198 ts_tablet_manager.cc:1939] T 0644f5600c5b47e29f3fa73060ed96e3 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:40.344098 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_5625ac97f9f84d3e8379eb9536bf94c5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:40.348336 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5625ac97f9f84d3e8379eb9536bf94c5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:40.348273 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_8d0905ada0b74b8fba4964abaf95a2b3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:40.348753 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8d0905ada0b74b8fba4964abaf95a2b3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:40.348943 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_0c7d032c4488498d9796defce01fc237"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:40.349393 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0c7d032c4488498d9796defce01fc237 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:40.356014 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet c368b5b0c3a940c4b4efb64c946375c3
I20250113 02:51:40.356489 447 catalog_manager.cc:6596] ReplaceTablet: tablet c368b5b0c3a940c4b4efb64c946375c3 deleted and replaced by tablet 17f02be32ddb44c089266fd9f0378f77
I20250113 02:51:40.357668 895 tablet_service.cc:1514] Processing DeleteTablet for tablet c368b5b0c3a940c4b4efb64c946375c3 with delete_type TABLET_DATA_DELETED (replaced by tablet 17f02be32ddb44c089266fd9f0378f77) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:40.357831 1040 tablet_service.cc:1514] Processing DeleteTablet for tablet c368b5b0c3a940c4b4efb64c946375c3 with delete_type TABLET_DATA_DELETED (replaced by tablet 17f02be32ddb44c089266fd9f0378f77) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:40.358611 738 tablet_service.cc:1514] Processing DeleteTablet for tablet c368b5b0c3a940c4b4efb64c946375c3 with delete_type TABLET_DATA_DELETED (replaced by tablet 17f02be32ddb44c089266fd9f0378f77) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:40.358898 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_0fea98afc97f400a8fee7d8258d7b37f" } new_table_name: "default.Table_c4e11d686918496c9fa1a5c9ddde90ae" modify_external_catalogs: true
I20250113 02:51:40.361263 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_5851d43a71e641058f8d820b078e9aca"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:40.361764 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5851d43a71e641058f8d820b078e9aca in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:40.364971 2306 raft_consensus.cc:491] T bc6a7322c0c34adb85457750526d8955 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:40.365118 2306 raft_consensus.cc:513] T bc6a7322c0c34adb85457750526d8955 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:40.365741 2306 leader_election.cc:290] T bc6a7322c0c34adb85457750526d8955 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:40.366410 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc6a7322c0c34adb85457750526d8955" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
I20250113 02:51:40.366633 772 raft_consensus.cc:2463] T bc6a7322c0c34adb85457750526d8955 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 0.
I20250113 02:51:40.366743 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc6a7322c0c34adb85457750526d8955" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:40.367527 995 leader_election.cc:304] T bc6a7322c0c34adb85457750526d8955 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters:
I20250113 02:51:40.367564 416 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskXJrxug/build/asan/bin/kudu pid 1706
W20250113 02:51:40.367841 998 leader_election.cc:343] T bc6a7322c0c34adb85457750526d8955 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:40.370000 455 master_service.cc:946] ReplaceTablet: received request to replace tablet eceed0f52c9f46bbaa78ca773a7afde5 from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:40.371420 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_7f3e4c1c46cd4df1b77899c2f674068a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:40.371909 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7f3e4c1c46cd4df1b77899c2f674068a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:40.376047 2306 raft_consensus.cc:2798] T bc6a7322c0c34adb85457750526d8955 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 02:51:40.376135 2306 raft_consensus.cc:491] T bc6a7322c0c34adb85457750526d8955 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 02:51:40.376196 2306 raft_consensus.cc:3054] T bc6a7322c0c34adb85457750526d8955 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:40.377540 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet e4998b86d32040ec959ff6fd94b1e9eb
I20250113 02:51:40.380278 1040 tablet_service.cc:1514] Processing DeleteTablet for tablet e4998b86d32040ec959ff6fd94b1e9eb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:40 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:40.380658 1196 ts_tablet_manager.cc:1918] T 42cf1e708644427f9d4dcd41ae2bcb37 P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:40.380928 895 tablet_service.cc:1514] Processing DeleteTablet for tablet e4998b86d32040ec959ff6fd94b1e9eb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:40 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:40.381088 1196 log.cc:1198] T 42cf1e708644427f9d4dcd41ae2bcb37 P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/42cf1e708644427f9d4dcd41ae2bcb37
I20250113 02:51:40.381433 1196 ts_tablet_manager.cc:1939] T 42cf1e708644427f9d4dcd41ae2bcb37 P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:40.381791 1196 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37366 (request call id 1076) took 1673 ms. Trace:
I20250113 02:51:40.381805 738 tablet_service.cc:1514] Processing DeleteTablet for tablet e4998b86d32040ec959ff6fd94b1e9eb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:40 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:40.381909 1196 rpcz_store.cc:276] 0113 02:51:38.708535 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:38.708587 (+ 52us) service_pool.cc:225] Handling call
0113 02:51:40.177935 (+1469348us) ts_tablet_manager.cc:1145] Deleting tablet 42cf1e708644427f9d4dcd41ae2bcb37
0113 02:51:40.177943 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:40.204183 (+ 26240us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:40.380548 (+176365us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:40.381778 (+ 1230us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1469172}
I20250113 02:51:40.382330 1196 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37366 (request call id 1080) took 1617 ms. Trace:
I20250113 02:51:40.382409 1196 rpcz_store.cc:276] 0113 02:51:38.764715 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:38.764768 (+ 53us) service_pool.cc:225] Handling call
0113 02:51:40.382201 (+1617433us) ts_tablet_manager.cc:1145] Deleting tablet 49c2e648a49f4034a521f38e7d98edc4
0113 02:51:40.382210 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:40.382313 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1617249}
I20250113 02:51:40.382711 1196 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37366 (request call id 1081) took 1617 ms. Trace:
I20250113 02:51:40.382783 1196 rpcz_store.cc:276] 0113 02:51:38.764822 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:38.764977 (+ 155us) service_pool.cc:225] Handling call
0113 02:51:40.382627 (+1617650us) ts_tablet_manager.cc:1145] Deleting tablet 368e878e89b54694823863a49074c11e
0113 02:51:40.382634 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:40.382698 (+ 64us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1617506}
I20250113 02:51:40.383128 1196 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37366 (request call id 1082) took 1618 ms. Trace:
I20250113 02:51:40.383198 1196 rpcz_store.cc:276] 0113 02:51:38.764928 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:38.765152 (+ 224us) service_pool.cc:225] Handling call
0113 02:51:40.383030 (+1617878us) ts_tablet_manager.cc:1145] Deleting tablet 71befed8b5c74b41b199919b82325891
0113 02:51:40.383041 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:40.383115 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1617757}
I20250113 02:51:40.383507 1196 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37366 (request call id 1079) took 1631 ms. Trace:
I20250113 02:51:40.383558 1196 rpcz_store.cc:276] 0113 02:51:38.752344 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:38.806636 (+ 54292us) service_pool.cc:225] Handling call
0113 02:51:40.383419 (+1576783us) ts_tablet_manager.cc:1145] Deleting tablet 13f9711b16ae4e36be00db07d9bde7fc
0113 02:51:40.383429 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:40.383495 (+ 66us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1576557}
I20250113 02:51:40.383801 1196 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37366 (request call id 1089) took 1550 ms. Trace:
I20250113 02:51:40.383867 1196 rpcz_store.cc:276] 0113 02:51:38.833513 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:38.833949 (+ 436us) service_pool.cc:225] Handling call
0113 02:51:40.383720 (+1549771us) ts_tablet_manager.cc:1145] Deleting tablet a6d0f9a0a7104b8fb3d22aac976b7d99
0113 02:51:40.383728 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:40.383789 (+ 61us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1549580}
I20250113 02:51:40.385221 1040 tablet_service.cc:1467] Processing CreateTablet for tablet 5f0ab4954da04bf38e8c49d29b137061 (DEFAULT_TABLE table=default.Table_2e04cc940d734e3bbe72ae90690357d6 [id=cb5111f2024c49dab0bb6727e3e0b2a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:40.385600 1040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f0ab4954da04bf38e8c49d29b137061. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:40.388513 1026 tablet_service.cc:1467] Processing CreateTablet for tablet e3a1e65ec9874cc7b83485683aafff77 (DEFAULT_TABLE table=default.Table_9887bd371c8046b49c4ae84f97f16802 [id=ede56dd0d041496bbbad872b3213fe4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:40.388893 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e3a1e65ec9874cc7b83485683aafff77. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:40.388859 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet 42cf1e708644427f9d4dcd41ae2bcb37 (table default.Table_54963a1a7ff24d5a99644441a755f5de [id=acf16114671d4a3eb7ed4a0ffc984fef]) successfully deleted
W20250113 02:51:40.389079 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 49c2e648a49f4034a521f38e7d98edc4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 49c2e648a49f4034a521f38e7d98edc4 already in progress: creating tablet
W20250113 02:51:40.389292 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 368e878e89b54694823863a49074c11e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 368e878e89b54694823863a49074c11e already in progress: creating tablet
I20250113 02:51:40.389258 1030 tablet_service.cc:1467] Processing CreateTablet for tablet 1e135eb5a19d45c3bfea54b8cf1bb898 (DEFAULT_TABLE table=default.Table_f6e2816287cf46e1b05c86ba38199184 [id=1420544bb23548ebb22272f02582bcd0]), partition=RANGE (key) PARTITION UNBOUNDED
W20250113 02:51:40.389457 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 71befed8b5c74b41b199919b82325891 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 71befed8b5c74b41b199919b82325891 already in progress: creating tablet
W20250113 02:51:40.389604 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 13f9711b16ae4e36be00db07d9bde7fc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 13f9711b16ae4e36be00db07d9bde7fc already in progress: creating tablet
I20250113 02:51:40.389670 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e135eb5a19d45c3bfea54b8cf1bb898. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:40.390429 752 tablet_service.cc:1467] Processing CreateTablet for tablet e3a1e65ec9874cc7b83485683aafff77 (DEFAULT_TABLE table=default.Table_9887bd371c8046b49c4ae84f97f16802 [id=ede56dd0d041496bbbad872b3213fe4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:40.390815 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e3a1e65ec9874cc7b83485683aafff77. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:40.392215 1038 tablet_service.cc:1467] Processing CreateTablet for tablet bc3e356b0af24f6791dcd91c9856e3c8 (DEFAULT_TABLE table=default.Table_5bbda7be1ad04c32864d6d49a342e41f [id=4728241476c24922a4c3ae39f0b18130]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:40.392216 743 tablet_service.cc:1467] Processing CreateTablet for tablet bc3e356b0af24f6791dcd91c9856e3c8 (DEFAULT_TABLE table=default.Table_5bbda7be1ad04c32864d6d49a342e41f [id=4728241476c24922a4c3ae39f0b18130]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:40.392517 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc3e356b0af24f6791dcd91c9856e3c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:40.392517 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc3e356b0af24f6791dcd91c9856e3c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:40.395653 1034 tablet_service.cc:1467] Processing CreateTablet for tablet a0c14501ad0449aca28909bc14428dad (DEFAULT_TABLE table=default.Table_8c80aefb21ab41c6a1e42a722943ce86 [id=1b0cb6645c264262ba238efa61853b94]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:40.395936 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a0c14501ad0449aca28909bc14428dad. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:40.396210 754 tablet_service.cc:1467] Processing CreateTablet for tablet fdb011b208214f928496addf4ed9a4e1 (DEFAULT_TABLE table=default.Table_830a393895fb4abc877fe4c60f6ed04a [id=64db6e24960a4d1587eb76145f1efa91]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:40.396508 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fdb011b208214f928496addf4ed9a4e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:40.399081 1024 tablet_service.cc:1467] Processing CreateTablet for tablet fdb011b208214f928496addf4ed9a4e1 (DEFAULT_TABLE table=default.Table_830a393895fb4abc877fe4c60f6ed04a [id=64db6e24960a4d1587eb76145f1efa91]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:40.399389 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fdb011b208214f928496addf4ed9a4e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:40.401911 746 tablet_service.cc:1467] Processing CreateTablet for tablet 1e135eb5a19d45c3bfea54b8cf1bb898 (DEFAULT_TABLE table=default.Table_f6e2816287cf46e1b05c86ba38199184 [id=1420544bb23548ebb22272f02582bcd0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:40.402213 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e135eb5a19d45c3bfea54b8cf1bb898. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:40.405357 1195 ts_tablet_manager.cc:1918] T ba8980ad3a364c888a825affbcbc9b1f P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:40.405951 1195 log.cc:1198] T ba8980ad3a364c888a825affbcbc9b1f P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/ba8980ad3a364c888a825affbcbc9b1f
I20250113 02:51:40.406617 1195 ts_tablet_manager.cc:1939] T ba8980ad3a364c888a825affbcbc9b1f P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:40.408944 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1075) took 3806 ms. Trace:
W20250113 02:51:40.409236 998 consensus_peers.cc:487] T 44d2c7d5334c476fac60fdd57e2c02b6 P 4216a7f853d642b7bc3e5cff458f3aca -> Peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Couldn't send request to peer df69f48656b44bb890f76587ee69cc0f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250113 02:51:40.410370 1195 rpcz_store.cc:276] 0113 02:51:36.602907 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:36.602990 (+ 83us) service_pool.cc:225] Handling call
0113 02:51:40.294189 (+3691199us) ts_tablet_manager.cc:1145] Deleting tablet ba8980ad3a364c888a825affbcbc9b1f
0113 02:51:40.294199 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:40.304619 (+ 10420us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:40.307312 (+ 2693us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:40.408926 (+101614us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3558,"tablet-delete.queue_time_us":3691037}
I20250113 02:51:40.411458 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet ba8980ad3a364c888a825affbcbc9b1f (table default.Table_459ff6639996470cb94e898c9ef4880e [id=a1d8c84954c747adb2aa43726d7b5334]) successfully deleted
I20250113 02:51:40.412290 738 tablet_service.cc:1467] Processing CreateTablet for tablet 5f0ab4954da04bf38e8c49d29b137061 (DEFAULT_TABLE table=default.Table_2e04cc940d734e3bbe72ae90690357d6 [id=cb5111f2024c49dab0bb6727e3e0b2a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:40.412635 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f0ab4954da04bf38e8c49d29b137061. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:40.416958 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_1e8aa807c9e642c8950cff0dccb6e83a" } new_table_name: "default.Table_15b7a5fb45ce401296bb81b3c95ea9ad" modify_external_catalogs: true
I20250113 02:51:40.417294 1042 tablet_service.cc:1467] Processing CreateTablet for tablet 21ff2fc65c1c4a5799498fb3982beea2 (DEFAULT_TABLE table=default.Table_b4e63f7e8868448ebd839ba0352acd2c [id=57a1c4b9790b41b382fc968b3c9a5cc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:40.417646 1042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 21ff2fc65c1c4a5799498fb3982beea2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:40.418381 741 tablet_service.cc:1467] Processing CreateTablet for tablet a0c14501ad0449aca28909bc14428dad (DEFAULT_TABLE table=default.Table_8c80aefb21ab41c6a1e42a722943ce86 [id=1b0cb6645c264262ba238efa61853b94]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:40.418704 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a0c14501ad0449aca28909bc14428dad. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:40.418907 448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_8bca38c8f95a4ea8aaec77a9e6c57f96" } new_table_name: "default.Table_1ebb0d3cf33347f79b69a2951a6718cc" modify_external_catalogs: true
I20250113 02:51:40.418874 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_3e43331bf36845f9aa983823e4853d04"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:40.419375 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3e43331bf36845f9aa983823e4853d04 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:40.420348 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1068) took 3861 ms. Trace:
I20250113 02:51:40.420425 1195 rpcz_store.cc:276] 0113 02:51:36.558877 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:36.709521 (+150644us) service_pool.cc:225] Handling call
0113 02:51:40.420228 (+3710707us) ts_tablet_manager.cc:1145] Deleting tablet ff5e40047f9e43fbb980b5fedfb36ade
0113 02:51:40.420236 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:40.420330 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3710508}
W20250113 02:51:40.420817 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet ff5e40047f9e43fbb980b5fedfb36ade because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ff5e40047f9e43fbb980b5fedfb36ade already in progress: creating tablet
I20250113 02:51:40.421532 1195 tablet_replica.cc:331] T 788ab82626424b6e833e31761a15b9b2 P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:40.421725 1195 raft_consensus.cc:2238] T 788ab82626424b6e833e31761a15b9b2 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:40.421860 1195 raft_consensus.cc:2267] T 788ab82626424b6e833e31761a15b9b2 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:40.422386 1195 ts_tablet_manager.cc:1905] T 788ab82626424b6e833e31761a15b9b2 P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:40.422322 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_69621fd76ebb45e5803d1a54f9023251" } new_table_name: "default.Table_845dec17042f495a9b4454f197771269" modify_external_catalogs: true
I20250113 02:51:40.422858 1191 tablet_bootstrap.cc:492] T ca75e7cae4f6423191a17bb5a8f150a9 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:40.422969 1191 ts_tablet_manager.cc:1397] T ca75e7cae4f6423191a17bb5a8f150a9 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.111s user 0.004s sys 0.001s
I20250113 02:51:40.423725 1191 raft_consensus.cc:357] T ca75e7cae4f6423191a17bb5a8f150a9 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:40.423913 1191 raft_consensus.cc:383] T ca75e7cae4f6423191a17bb5a8f150a9 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:40.423987 1191 raft_consensus.cc:738] T ca75e7cae4f6423191a17bb5a8f150a9 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:40.400641 895 tablet_service.cc:1467] Processing CreateTablet for tablet e3a1e65ec9874cc7b83485683aafff77 (DEFAULT_TABLE table=default.Table_9887bd371c8046b49c4ae84f97f16802 [id=ede56dd0d041496bbbad872b3213fe4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:40.400843 886 tablet_service.cc:1467] Processing CreateTablet for tablet bc3e356b0af24f6791dcd91c9856e3c8 (DEFAULT_TABLE table=default.Table_5bbda7be1ad04c32864d6d49a342e41f [id=4728241476c24922a4c3ae39f0b18130]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:40.401153 899 tablet_service.cc:1467] Processing CreateTablet for tablet fdb011b208214f928496addf4ed9a4e1 (DEFAULT_TABLE table=default.Table_830a393895fb4abc877fe4c60f6ed04a [id=64db6e24960a4d1587eb76145f1efa91]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:40.401434 888 tablet_service.cc:1467] Processing CreateTablet for tablet 1e135eb5a19d45c3bfea54b8cf1bb898 (DEFAULT_TABLE table=default.Table_f6e2816287cf46e1b05c86ba38199184 [id=1420544bb23548ebb22272f02582bcd0]), partition=RANGE (key) PARTITION UNBOUNDED
W20250113 02:51:40.425127 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet a6d0f9a0a7104b8fb3d22aac976b7d99 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a6d0f9a0a7104b8fb3d22aac976b7d99 already in progress: creating tablet
I20250113 02:51:40.425129 751 tablet_service.cc:1467] Processing CreateTablet for tablet 21ff2fc65c1c4a5799498fb3982beea2 (DEFAULT_TABLE table=default.Table_b4e63f7e8868448ebd839ba0352acd2c [id=57a1c4b9790b41b382fc968b3c9a5cc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:40.425441 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 21ff2fc65c1c4a5799498fb3982beea2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:40.425431 891 tablet_service.cc:1467] Processing CreateTablet for tablet 21ff2fc65c1c4a5799498fb3982beea2 (DEFAULT_TABLE table=default.Table_b4e63f7e8868448ebd839ba0352acd2c [id=57a1c4b9790b41b382fc968b3c9a5cc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:40.425482 1035 tablet_service.cc:1467] Processing CreateTablet for tablet 8e4b74a1ef824f40a40c0b215b2b5711 (DEFAULT_TABLE table=default.Table_7a15797b8bd04c2e911c0803d28cecf6 [id=47f3af6540554b3a8f10c98110e808b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:40.425506 890 tablet_service.cc:1467] Processing CreateTablet for tablet a0c14501ad0449aca28909bc14428dad (DEFAULT_TABLE table=default.Table_8c80aefb21ab41c6a1e42a722943ce86 [id=1b0cb6645c264262ba238efa61853b94]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:40.425804 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e4b74a1ef824f40a40c0b215b2b5711. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:40.425796 883 tablet_service.cc:1467] Processing CreateTablet for tablet 8e4b74a1ef824f40a40c0b215b2b5711 (DEFAULT_TABLE table=default.Table_7a15797b8bd04c2e911c0803d28cecf6 [id=47f3af6540554b3a8f10c98110e808b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:40.426759 1037 tablet_service.cc:1467] Processing CreateTablet for tablet 7ea95eda6470427790a5b8ec1f7dd29c (DEFAULT_TABLE table=default.Table_af07bf28ba6943afa90fcc5287e7d8b1 [id=bda1cf60414f44fba0de3d8a94744790]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:40.427063 1037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ea95eda6470427790a5b8ec1f7dd29c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:40.430279 742 tablet_service.cc:1467] Processing CreateTablet for tablet 8e4b74a1ef824f40a40c0b215b2b5711 (DEFAULT_TABLE table=default.Table_7a15797b8bd04c2e911c0803d28cecf6 [id=47f3af6540554b3a8f10c98110e808b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:40.430284 882 tablet_service.cc:1467] Processing CreateTablet for tablet 7ea95eda6470427790a5b8ec1f7dd29c (DEFAULT_TABLE table=default.Table_af07bf28ba6943afa90fcc5287e7d8b1 [id=bda1cf60414f44fba0de3d8a94744790]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:40.430611 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e4b74a1ef824f40a40c0b215b2b5711. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:40.436295 1191 consensus_queue.cc:260] T ca75e7cae4f6423191a17bb5a8f150a9 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:40.437615 881 tablet_service.cc:1467] Processing CreateTablet for tablet 90db69ccf0604f1b8069f7955e69a1ed (DEFAULT_TABLE table=default.Table_79688f9ecca44fccb291229417508b84 [id=d8edd4271809448f9e61b87b41f9dcaf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:40.439797 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e3a1e65ec9874cc7b83485683aafff77. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:40.438023 742 tablet_service.cc:1467] Processing CreateTablet for tablet 7ea95eda6470427790a5b8ec1f7dd29c (DEFAULT_TABLE table=default.Table_af07bf28ba6943afa90fcc5287e7d8b1 [id=bda1cf60414f44fba0de3d8a94744790]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:40.440284 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ea95eda6470427790a5b8ec1f7dd29c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:40.440337 2306 raft_consensus.cc:513] T bc6a7322c0c34adb85457750526d8955 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:40.347908 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1129) took 2252 ms. Trace:
I20250113 02:51:40.441342 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc6a7322c0c34adb85457750526d8955" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196"
I20250113 02:51:40.441305 1198 rpcz_store.cc:276] 0113 02:51:38.095263 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:38.096022 (+ 759us) service_pool.cc:225] Handling call
0113 02:51:40.212508 (+2116486us) ts_tablet_manager.cc:1145] Deleting tablet 0644f5600c5b47e29f3fa73060ed96e3
0113 02:51:40.214324 (+ 1816us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:40.262023 (+ 47699us) spinlock_profiling.cc:243] Waited 39.5 ms on lock 0x616000202654. stack: 00007f0bad094fd6 00007f0bad094d90 00007f0bac07448d 000000000096d544 000000000096d50d 000000000096ceda 00007f0bb8937ed0 00007f0bb9c2c463 00007f0bbc6d0eb7 00007f0bbc703584 00007f0bbc6e4690 00007f0bbc6e441d 0000000000a71ecc 00007f0bad0dbb64 00007f0bad0df18b 00007f0bad0df008
0113 02:51:40.291458 (+ 29435us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:40.346414 (+ 54956us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:40.347885 (+ 1471us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":90795264,"tablet-delete.queue_time_us":2116272}
I20250113 02:51:40.441504 772 raft_consensus.cc:3054] T bc6a7322c0c34adb85457750526d8955 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:40.441800 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1131) took 2225 ms. Trace:
I20250113 02:51:40.441874 1198 rpcz_store.cc:276] 0113 02:51:38.216066 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:38.216126 (+ 60us) service_pool.cc:225] Handling call
0113 02:51:40.441675 (+2225549us) ts_tablet_manager.cc:1145] Deleting tablet ed9e6ba0729b483fa6599dd6dde016e7
0113 02:51:40.441687 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:40.441783 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2225359}
I20250113 02:51:40.442112 1198 tablet_replica.cc:331] T dd566a49d3ff4d3889b80d8d2ccbee0e P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:40.442303 1198 raft_consensus.cc:2238] T dd566a49d3ff4d3889b80d8d2ccbee0e P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:40.442445 1198 raft_consensus.cc:2267] T dd566a49d3ff4d3889b80d8d2ccbee0e P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:40.442950 1198 ts_tablet_manager.cc:1905] T dd566a49d3ff4d3889b80d8d2ccbee0e P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:40.443851 772 raft_consensus.cc:2463] T bc6a7322c0c34adb85457750526d8955 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 1.
I20250113 02:51:40.444298 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet 0644f5600c5b47e29f3fa73060ed96e3 (table default.Table_cd499f4a8487400690a9ab957c606b7f [id=be5f659ed070438bb6d0e61a3820eb58]) successfully deleted
W20250113 02:51:40.444567 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet ed9e6ba0729b483fa6599dd6dde016e7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ed9e6ba0729b483fa6599dd6dde016e7 already in progress: creating tablet
I20250113 02:51:40.445744 995 leader_election.cc:304] T bc6a7322c0c34adb85457750526d8955 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters:
I20250113 02:51:40.445987 2368 raft_consensus.cc:2798] T bc6a7322c0c34adb85457750526d8955 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Leader election won for term 1
I20250113 02:51:40.446103 2368 raft_consensus.cc:695] T bc6a7322c0c34adb85457750526d8955 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: Becoming Leader. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Running, Role: LEADER
I20250113 02:51:40.446323 2368 consensus_queue.cc:237] T bc6a7322c0c34adb85457750526d8955 P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:40.453137 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_d5096557ae6445e6bcd3b6a9b3299e8b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:40.453671 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d5096557ae6445e6bcd3b6a9b3299e8b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:40.458371 1191 ts_tablet_manager.cc:1428] T ca75e7cae4f6423191a17bb5a8f150a9 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.035s user 0.002s sys 0.000s
I20250113 02:51:40.459455 2306 leader_election.cc:290] T bc6a7322c0c34adb85457750526d8955 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Requested vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:40.462004 885 tablet_service.cc:1467] Processing CreateTablet for tablet d00e90d8ba0a4f36898efcc2e129c130 (DEFAULT_TABLE table=default.Table_5e1a59c4550b4cc5b58b5b624b5d71b5 [id=306cf9b8548d49ad9c371794207f8b92]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:40.465466 889 tablet_service.cc:1467] Processing CreateTablet for tablet 3163078bf56c42b2bd6f6b91885c6400 (DEFAULT_TABLE table=default.Table_5dd36060f1944c05a472dc98020c553f [id=4bc2dd7ad7ee4bab808ececd0da88512]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:40.465945 1035 tablet_service.cc:1467] Processing CreateTablet for tablet 90db69ccf0604f1b8069f7955e69a1ed (DEFAULT_TABLE table=default.Table_79688f9ecca44fccb291229417508b84 [id=d8edd4271809448f9e61b87b41f9dcaf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:40.466333 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90db69ccf0604f1b8069f7955e69a1ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:40.468317 1042 tablet_service.cc:1467] Processing CreateTablet for tablet 3163078bf56c42b2bd6f6b91885c6400 (DEFAULT_TABLE table=default.Table_5dd36060f1944c05a472dc98020c553f [id=4bc2dd7ad7ee4bab808ececd0da88512]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:40.468688 1042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3163078bf56c42b2bd6f6b91885c6400. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:40.469242 884 tablet_service.cc:1467] Processing CreateTablet for tablet a8c2dc1bc45d43db929ccd2859459c89 (DEFAULT_TABLE table=default.Table_0a0352107b6c4b01a7abdce96fbb3c48 [id=1213cce16aca4331bd0d473c0a520344]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:40.469393 746 tablet_service.cc:1467] Processing CreateTablet for tablet 90db69ccf0604f1b8069f7955e69a1ed (DEFAULT_TABLE table=default.Table_79688f9ecca44fccb291229417508b84 [id=d8edd4271809448f9e61b87b41f9dcaf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:40.469669 741 tablet_service.cc:1467] Processing CreateTablet for tablet d00e90d8ba0a4f36898efcc2e129c130 (DEFAULT_TABLE table=default.Table_5e1a59c4550b4cc5b58b5b624b5d71b5 [id=306cf9b8548d49ad9c371794207f8b92]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:40.470062 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d00e90d8ba0a4f36898efcc2e129c130. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:40.471341 1026 tablet_service.cc:1467] Processing CreateTablet for tablet a8c2dc1bc45d43db929ccd2859459c89 (DEFAULT_TABLE table=default.Table_0a0352107b6c4b01a7abdce96fbb3c48 [id=1213cce16aca4331bd0d473c0a520344]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:40.471668 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8c2dc1bc45d43db929ccd2859459c89. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:40.471779 2306 raft_consensus.cc:491] T bbc7832c50114d6f890fec45a3d994e9 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:40.471875 2306 raft_consensus.cc:513] T bbc7832c50114d6f890fec45a3d994e9 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:40.473368 2306 leader_election.cc:290] T bbc7832c50114d6f890fec45a3d994e9 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
I20250113 02:51:40.473495 894 tablet_service.cc:1467] Processing CreateTablet for tablet 298eb28ce7da4818acea3060a6ef427c (DEFAULT_TABLE table=default.Table_593fc8a8736e4d4aaeda0d72bbb30337 [id=f1f94d2e63fc40e9b252333c00b87bba]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:40.473899 743 tablet_service.cc:1467] Processing CreateTablet for tablet a8c2dc1bc45d43db929ccd2859459c89 (DEFAULT_TABLE table=default.Table_0a0352107b6c4b01a7abdce96fbb3c48 [id=1213cce16aca4331bd0d473c0a520344]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:40.474354 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8c2dc1bc45d43db929ccd2859459c89. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:40.474650 995 leader_election.cc:343] T bbc7832c50114d6f890fec45a3d994e9 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Not found: Tablet not found: bbc7832c50114d6f890fec45a3d994e9
I20250113 02:51:40.474785 2413 raft_consensus.cc:491] T 79aaab73797c474bae06831fa308b066 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:40.474900 2413 raft_consensus.cc:513] T 79aaab73797c474bae06831fa308b066 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:40.475415 1031 tablet_service.cc:1467] Processing CreateTablet for tablet d00e90d8ba0a4f36898efcc2e129c130 (DEFAULT_TABLE table=default.Table_5e1a59c4550b4cc5b58b5b624b5d71b5 [id=306cf9b8548d49ad9c371794207f8b92]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:40.475700 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d00e90d8ba0a4f36898efcc2e129c130. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:40.476956 1191 tablet_bootstrap.cc:492] T 0839a35fcab04f3c9d03e256553e01c8 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:40.478386 1040 tablet_service.cc:1467] Processing CreateTablet for tablet 298eb28ce7da4818acea3060a6ef427c (DEFAULT_TABLE table=default.Table_593fc8a8736e4d4aaeda0d72bbb30337 [id=f1f94d2e63fc40e9b252333c00b87bba]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:40.478767 1040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 298eb28ce7da4818acea3060a6ef427c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:40.479928 752 tablet_service.cc:1467] Processing CreateTablet for tablet 3163078bf56c42b2bd6f6b91885c6400 (DEFAULT_TABLE table=default.Table_5dd36060f1944c05a472dc98020c553f [id=4bc2dd7ad7ee4bab808ececd0da88512]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:40.413062 1194 tablet_bootstrap.cc:492] T 999fecb3fc65466cad21cc1ff5ef2d44 P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:40.487607 1194 ts_tablet_manager.cc:1397] T 999fecb3fc65466cad21cc1ff5ef2d44 P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.216s user 0.004s sys 0.006s
I20250113 02:51:40.488049 1980 generic_service.cc:196] Checking for leaks (request via RPC)
I20250113 02:51:40.489442 887 tablet_service.cc:1467] Processing CreateTablet for tablet e11387e33dbf4d759f4943eed1d77917 (DEFAULT_TABLE table=default.Table_c32890b1c89b40158e226670a74c04c8 [id=41f97448164d4137896c83b0246dc272]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:40.488559 1194 raft_consensus.cc:357] T 999fecb3fc65466cad21cc1ff5ef2d44 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:40.489778 1194 raft_consensus.cc:383] T 999fecb3fc65466cad21cc1ff5ef2d44 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:40.489833 1194 raft_consensus.cc:738] T 999fecb3fc65466cad21cc1ff5ef2d44 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:40.490074 1194 consensus_queue.cc:260] T 999fecb3fc65466cad21cc1ff5ef2d44 P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:40.490713 1194 ts_tablet_manager.cc:1428] T 999fecb3fc65466cad21cc1ff5ef2d44 P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250113 02:51:40.491209 2413 leader_election.cc:290] T 79aaab73797c474bae06831fa308b066 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:40.491309 751 tablet_service.cc:1467] Processing CreateTablet for tablet e11387e33dbf4d759f4943eed1d77917 (DEFAULT_TABLE table=default.Table_c32890b1c89b40158e226670a74c04c8 [id=41f97448164d4137896c83b0246dc272]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:40.491693 898 tablet_service.cc:1467] Processing CreateTablet for tablet ad0b1e35cc824daf8927f79ed36b0616 (DEFAULT_TABLE table=default.Table_ca7c8c6ee46346229bf5a7696f4b8165 [id=222c932068564a94b2c182127ff284e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:40.491791 1058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79aaab73797c474bae06831fa308b066" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
W20250113 02:51:40.492460 709 leader_election.cc:343] T 79aaab73797c474bae06831fa308b066 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:40.493886 754 tablet_service.cc:1467] Processing CreateTablet for tablet 298eb28ce7da4818acea3060a6ef427c (DEFAULT_TABLE table=default.Table_593fc8a8736e4d4aaeda0d72bbb30337 [id=f1f94d2e63fc40e9b252333c00b87bba]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:40.494275 1026 tablet_service.cc:1467] Processing CreateTablet for tablet e11387e33dbf4d759f4943eed1d77917 (DEFAULT_TABLE table=default.Table_c32890b1c89b40158e226670a74c04c8 [id=41f97448164d4137896c83b0246dc272]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:40.497313 892 tablet_service.cc:1467] Processing CreateTablet for tablet f85bc8e8042e40f4909a0c3e1337405e (DEFAULT_TABLE table=default.Table_b7f2f1b2cac147dd9ee0a9af3186aaf9 [id=04813e37d7d247ca9d6a861b06b6557b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:40.498987 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3163078bf56c42b2bd6f6b91885c6400. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:40.499814 2413 raft_consensus.cc:491] T f1237f7b04af4ebbb39b03aa3c78a499 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:40.499936 2413 raft_consensus.cc:513] T f1237f7b04af4ebbb39b03aa3c78a499 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:40.500711 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e11387e33dbf4d759f4943eed1d77917. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:40.500718 880 tablet_service.cc:1467] Processing CreateTablet for tablet ffb27f71b1464e18977f46d954194bef (DEFAULT_TABLE table=default.Table_5bc5d0b055904e50b74e0981675f4abb [id=80cd7e75d4e041b593c9307e47b532f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:40.501338 1058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1237f7b04af4ebbb39b03aa3c78a499" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
W20250113 02:51:40.501905 709 leader_election.cc:343] T f1237f7b04af4ebbb39b03aa3c78a499 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:40.500226 1035 tablet_service.cc:1467] Processing CreateTablet for tablet ad0b1e35cc824daf8927f79ed36b0616 (DEFAULT_TABLE table=default.Table_ca7c8c6ee46346229bf5a7696f4b8165 [id=222c932068564a94b2c182127ff284e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:40.502737 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ad0b1e35cc824daf8927f79ed36b0616. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:40.503001 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79aaab73797c474bae06831fa308b066" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:40.503080 912 raft_consensus.cc:3054] T fb18c16fb3ba44ec8a190b88d3bbd1b1 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:40.504033 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbc7832c50114d6f890fec45a3d994e9" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:40.504321 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc6a7322c0c34adb85457750526d8955" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f"
W20250113 02:51:40.506678 711 leader_election.cc:343] T 79aaab73797c474bae06831fa308b066 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:40.506779 711 leader_election.cc:304] T 79aaab73797c474bae06831fa308b066 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:40.508524 2413 leader_election.cc:290] T f1237f7b04af4ebbb39b03aa3c78a499 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:40.510609 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1237f7b04af4ebbb39b03aa3c78a499" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:40.511183 711 leader_election.cc:343] T f1237f7b04af4ebbb39b03aa3c78a499 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:40.511296 711 leader_election.cc:304] T f1237f7b04af4ebbb39b03aa3c78a499 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:40.511682 2413 raft_consensus.cc:2743] T 79aaab73797c474bae06831fa308b066 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:40.511713 1042 tablet_service.cc:1467] Processing CreateTablet for tablet f85bc8e8042e40f4909a0c3e1337405e (DEFAULT_TABLE table=default.Table_b7f2f1b2cac147dd9ee0a9af3186aaf9 [id=04813e37d7d247ca9d6a861b06b6557b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:40.511819 2413 raft_consensus.cc:2743] T f1237f7b04af4ebbb39b03aa3c78a499 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:40.512050 1042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f85bc8e8042e40f4909a0c3e1337405e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:40.513674 1196 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37366 (request call id 1090) took 1679 ms. Trace:
I20250113 02:51:40.513772 1196 rpcz_store.cc:276] 0113 02:51:38.834487 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:38.834592 (+ 105us) service_pool.cc:225] Handling call
0113 02:51:40.424306 (+1589714us) ts_tablet_manager.cc:1145] Deleting tablet f4acbe59400247678d2441b1b844d976
0113 02:51:40.502734 (+ 78428us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:40.513655 (+ 10921us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1589553}
W20250113 02:51:40.514267 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet f4acbe59400247678d2441b1b844d976 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f4acbe59400247678d2441b1b844d976 already in progress: creating tablet
I20250113 02:51:40.515084 2427 tablet.cc:1721] T fb18c16fb3ba44ec8a190b88d3bbd1b1 P df69f48656b44bb890f76587ee69cc0f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250113 02:51:40.520757 1192 tablet_bootstrap.cc:492] T b33025a462e74f2aa26cc3779f36615b P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:40.521020 1192 ts_tablet_manager.cc:1397] T b33025a462e74f2aa26cc3779f36615b P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.308s user 0.002s sys 0.018s
I20250113 02:51:40.524200 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a0c14501ad0449aca28909bc14428dad. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:40.524403 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 298eb28ce7da4818acea3060a6ef427c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:40.525681 1192 raft_consensus.cc:357] T b33025a462e74f2aa26cc3779f36615b P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:40.525911 1192 raft_consensus.cc:383] T b33025a462e74f2aa26cc3779f36615b P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:40.525974 1192 raft_consensus.cc:738] T b33025a462e74f2aa26cc3779f36615b P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:40.526192 1192 consensus_queue.cc:260] T b33025a462e74f2aa26cc3779f36615b P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:40.526876 1192 ts_tablet_manager.cc:1428] T b33025a462e74f2aa26cc3779f36615b P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20250113 02:51:40.527386 1196 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37366 (request call id 1087) took 1697 ms. Trace:
I20250113 02:51:40.527463 1196 rpcz_store.cc:276] 0113 02:51:38.830304 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:38.856049 (+ 25745us) service_pool.cc:225] Handling call
0113 02:51:40.527277 (+1671228us) ts_tablet_manager.cc:1145] Deleting tablet 9fad58cccf1744b0a476523e0af0036e
0113 02:51:40.527286 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:40.527370 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1629654}
W20250113 02:51:40.527931 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 9fad58cccf1744b0a476523e0af0036e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9fad58cccf1744b0a476523e0af0036e already in progress: creating tablet
I20250113 02:51:40.528086 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8c2dc1bc45d43db929ccd2859459c89. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:40.528386 1196 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37366 (request call id 1094) took 1647 ms. Trace:
I20250113 02:51:40.528455 1196 rpcz_store.cc:276] 0113 02:51:38.881143 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:38.881241 (+ 98us) service_pool.cc:225] Handling call
0113 02:51:40.528286 (+1647045us) ts_tablet_manager.cc:1145] Deleting tablet 50c4d4783e6d4a55bd3bd355613e1ea0
0113 02:51:40.528295 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:40.528370 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1627209}
W20250113 02:51:40.528854 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 50c4d4783e6d4a55bd3bd355613e1ea0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 50c4d4783e6d4a55bd3bd355613e1ea0 already in progress: creating tablet
I20250113 02:51:40.529923 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f85bc8e8042e40f4909a0c3e1337405e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:40.531388 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e11387e33dbf4d759f4943eed1d77917. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:40.532078 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e4b74a1ef824f40a40c0b215b2b5711. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:40.532177 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 298eb28ce7da4818acea3060a6ef427c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:40.532716 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90db69ccf0604f1b8069f7955e69a1ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:40.533751 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d00e90d8ba0a4f36898efcc2e129c130. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:40.534109 450 catalog_manager.cc:5618] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095) does not have the latest schema for tablet f1237f7b04af4ebbb39b03aa3c78a499 (table default.Table_4c5e2e1008b54019a243ffe1ec72c020 [id=0817677719d045fd83bb85890f11df25]). Expected version 2 got 0
I20250113 02:51:40.536077 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ea95eda6470427790a5b8ec1f7dd29c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:40.536254 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90db69ccf0604f1b8069f7955e69a1ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:40.536803 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fdb011b208214f928496addf4ed9a4e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:40.536932 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e11387e33dbf4d759f4943eed1d77917. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:40.537092 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ad0b1e35cc824daf8927f79ed36b0616. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:40.537285 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e135eb5a19d45c3bfea54b8cf1bb898. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:40.537421 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 21ff2fc65c1c4a5799498fb3982beea2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:40.537927 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc3e356b0af24f6791dcd91c9856e3c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:40.538242 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ffb27f71b1464e18977f46d954194bef. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:40.538573 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3163078bf56c42b2bd6f6b91885c6400. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:40.539215 736 tablet_service.cc:1467] Processing CreateTablet for tablet ad0b1e35cc824daf8927f79ed36b0616 (DEFAULT_TABLE table=default.Table_ca7c8c6ee46346229bf5a7696f4b8165 [id=222c932068564a94b2c182127ff284e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:40.539549 736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ad0b1e35cc824daf8927f79ed36b0616. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:40.512735 998 leader_election.cc:343] T bbc7832c50114d6f890fec45a3d994e9 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:40.541085 998 leader_election.cc:304] T bbc7832c50114d6f890fec45a3d994e9 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4216a7f853d642b7bc3e5cff458f3aca; no voters: 1186fa869eb14812832fa6562531f196, df69f48656b44bb890f76587ee69cc0f
W20250113 02:51:40.541579 998 leader_election.cc:343] T bc6a7322c0c34adb85457750526d8955 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:40.542094 2363 raft_consensus.cc:2743] T bbc7832c50114d6f890fec45a3d994e9 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:40.546840 1192 tablet_bootstrap.cc:492] T b815003a27f64809baaa2d43d6293cef P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:40.549274 1030 tablet_service.cc:1467] Processing CreateTablet for tablet ffb27f71b1464e18977f46d954194bef (DEFAULT_TABLE table=default.Table_5bc5d0b055904e50b74e0981675f4abb [id=80cd7e75d4e041b593c9307e47b532f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:40.549614 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ffb27f71b1464e18977f46d954194bef. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:40.550854 1196 tablet_replica.cc:331] T b8b6ffe690fc459db78b2b3bc12671dd P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:40.551930 1196 raft_consensus.cc:2238] T b8b6ffe690fc459db78b2b3bc12671dd P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:40.552143 1196 raft_consensus.cc:2267] T b8b6ffe690fc459db78b2b3bc12671dd P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:40.553050 1196 ts_tablet_manager.cc:1905] T b8b6ffe690fc459db78b2b3bc12671dd P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:40.554682 748 tablet_service.cc:1514] Processing DeleteTablet for tablet cf7b8bd6801743f49e9b9473747447c5 with delete_type TABLET_DATA_DELETED (replaced by tablet a8aedbc8d7bb4183bb1ab9902f07bea7) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:40.556878 1040 tablet_service.cc:1467] Processing CreateTablet for tablet 4e59953509884a0f82b276b232bb00fb (DEFAULT_TABLE table=default.Table_2bcd0f2a378c4d2aaefbbbfa2c144431 [id=818f5552fdaf4a54b3d10b920a04e7b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:40.557289 897 tablet_service.cc:1467] Processing CreateTablet for tablet d2b61dc5590a4e43af7bd9fecfb0dc52 (DEFAULT_TABLE table=default.Table_9de04b3793d248e6be61bb3b1e7feb77 [id=2017303fb0674ba6ba6d8142f7c32582]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:40.558455 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d2b61dc5590a4e43af7bd9fecfb0dc52. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:40.558382 744 tablet_service.cc:1467] Processing CreateTablet for tablet 94307d22b7f842ca89844768e4a2a2ab (DEFAULT_TABLE table=default.Table_6dc54733221e4e7b8d4b5752f841c953 [id=e41630b8ebcc4ca39ef178569ecd1205]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:40.558743 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 94307d22b7f842ca89844768e4a2a2ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:40.559514 896 tablet_service.cc:1467] Processing CreateTablet for tablet 94307d22b7f842ca89844768e4a2a2ab (DEFAULT_TABLE table=default.Table_6dc54733221e4e7b8d4b5752f841c953 [id=e41630b8ebcc4ca39ef178569ecd1205]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:40.559820 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 94307d22b7f842ca89844768e4a2a2ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:40.560647 742 tablet_service.cc:1467] Processing CreateTablet for tablet f85bc8e8042e40f4909a0c3e1337405e (DEFAULT_TABLE table=default.Table_b7f2f1b2cac147dd9ee0a9af3186aaf9 [id=04813e37d7d247ca9d6a861b06b6557b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:40.561031 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f85bc8e8042e40f4909a0c3e1337405e. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:40.562196 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a6503d0e37cf4675b5d721975c1245b7 consensus state has no leader
I20250113 02:51:40.563274 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_ed460a21a2184db893d2e76e67cdb974"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:40.563751 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ed460a21a2184db893d2e76e67cdb974 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:40.564100 447 catalog_manager.cc:5526] T bc6a7322c0c34adb85457750526d8955 P 4216a7f853d642b7bc3e5cff458f3aca reported cstate change: term changed from 0 to 1, leader changed from <none> to 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3). New cstate: current_term: 1 leader_uuid: "4216a7f853d642b7bc3e5cff458f3aca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } health_report { overall_health: HEALTHY } } }
I20250113 02:51:40.567049 738 tablet_service.cc:1467] Processing CreateTablet for tablet ffb27f71b1464e18977f46d954194bef (DEFAULT_TABLE table=default.Table_5bc5d0b055904e50b74e0981675f4abb [id=80cd7e75d4e041b593c9307e47b532f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:40.567567 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ffb27f71b1464e18977f46d954194bef. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:40.568142 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet eceed0f52c9f46bbaa78ca773a7afde5
I20250113 02:51:40.568555 455 catalog_manager.cc:6596] ReplaceTablet: tablet eceed0f52c9f46bbaa78ca773a7afde5 deleted and replaced by tablet 7d90e1689b164943b09f6cda5a6ae5cb
I20250113 02:51:40.568851 455 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_3a224acfb9184dd295d9e7321324a60e" } modify_external_catalogs: true
I20250113 02:51:40.568985 455 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_3a224acfb9184dd295d9e7321324a60e" } modify_external_catalogs: true
I20250113 02:51:40.569903 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_6dc54733221e4e7b8d4b5752f841c953" } modify_external_catalogs: true
W20250113 02:51:40.569967 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 543149069ea24d00a875e0a1aee51902 consensus state has no leader
I20250113 02:51:40.570046 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_6dc54733221e4e7b8d4b5752f841c953" } modify_external_catalogs: true
I20250113 02:51:40.570362 453 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_8c80aefb21ab41c6a1e42a722943ce86" } modify_external_catalogs: true
I20250113 02:51:40.570528 453 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_8c80aefb21ab41c6a1e42a722943ce86" } modify_external_catalogs: true
I20250113 02:51:40.573952 2449 raft_consensus.cc:491] T 1d15c7a01f854f0aaabcd30ec3166e31 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:40.574148 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_2d947c38688144d9a7534584d059e5b8" } new_table_name: "default.Table_be2b756d1ad24981880b0debe778986a" modify_external_catalogs: true
I20250113 02:51:40.578106 1037 tablet_service.cc:1467] Processing CreateTablet for tablet d2b61dc5590a4e43af7bd9fecfb0dc52 (DEFAULT_TABLE table=default.Table_9de04b3793d248e6be61bb3b1e7feb77 [id=2017303fb0674ba6ba6d8142f7c32582]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:40.578738 748 tablet_service.cc:1467] Processing CreateTablet for tablet 4e59953509884a0f82b276b232bb00fb (DEFAULT_TABLE table=default.Table_2bcd0f2a378c4d2aaefbbbfa2c144431 [id=818f5552fdaf4a54b3d10b920a04e7b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:40.579149 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e59953509884a0f82b276b232bb00fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:40.585703 747 tablet_service.cc:1467] Processing CreateTablet for tablet d2b61dc5590a4e43af7bd9fecfb0dc52 (DEFAULT_TABLE table=default.Table_9de04b3793d248e6be61bb3b1e7feb77 [id=2017303fb0674ba6ba6d8142f7c32582]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:40.587055 1037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d2b61dc5590a4e43af7bd9fecfb0dc52. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:40.588049 893 tablet_service.cc:1467] Processing CreateTablet for tablet 1c153e47a7aa4bf5ae61dd19a91930b9 (DEFAULT_TABLE table=default.Table_302b76de6d984d2586c6a6e182f85156 [id=095952e818d24cb08c5e1ff3cc5c0a8f]), partition=RANGE (key) PARTITION UNBOUNDED
W20250113 02:51:40.590278 448 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 448d3daf897247f8bd53ffbcdc16a22c consensus state has no leader
I20250113 02:51:40.578106 1038 tablet_service.cc:1467] Processing CreateTablet for tablet 94307d22b7f842ca89844768e4a2a2ab (DEFAULT_TABLE table=default.Table_6dc54733221e4e7b8d4b5752f841c953 [id=e41630b8ebcc4ca39ef178569ecd1205]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:40.590624 448 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_1f3f594aac1e4cd2bcec522b01f7e746" } modify_external_catalogs: true
I20250113 02:51:40.590817 448 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_1f3f594aac1e4cd2bcec522b01f7e746" } modify_external_catalogs: true
I20250113 02:51:40.590935 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 94307d22b7f842ca89844768e4a2a2ab. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:40.591754 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 01a455eda0ee488fb3d45a6a09c43833 consensus state has no leader
I20250113 02:51:40.592072 1024 tablet_service.cc:1467] Processing CreateTablet for tablet 65139dcfe89249ec95f5425b294f04d8 (DEFAULT_TABLE table=default.Table_f1d37dd0db604e66abe5918b55426fa2 [id=a7a3b5ef19d94090a39ab850588951bb]), partition=RANGE (key) PARTITION UNBOUNDED
W20250113 02:51:40.592201 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet bf72471311854e01ac4b95a35e770d76 consensus state has no leader
I20250113 02:51:40.592263 1040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e59953509884a0f82b276b232bb00fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:40.592100 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_9458e2e0d9a446378600ea1ef1696bd5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:40.592604 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9458e2e0d9a446378600ea1ef1696bd5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:40.592518 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_c7dd7ce595fa467a916cd31cf34526c8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:40.592911 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c7dd7ce595fa467a916cd31cf34526c8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:40.593057 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65139dcfe89249ec95f5425b294f04d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:40.593053 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_bf4806a5a68e4fcb98205b6fed9d0b5e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:40.593466 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bf4806a5a68e4fcb98205b6fed9d0b5e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:40.594501 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 42a8e01bb71f49b09c83a8f5c905f38d
I20250113 02:51:40.595053 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_dcdc0cc411724eba84ac9af713fb17e3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:40.595347 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dcdc0cc411724eba84ac9af713fb17e3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:40.596309 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 94307d22b7f842ca89844768e4a2a2ab
I20250113 02:51:40.596493 447 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_fda8e0268b174c6f81a15f80e24825b0" } modify_external_catalogs: true
I20250113 02:51:40.596609 447 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_fda8e0268b174c6f81a15f80e24825b0" } modify_external_catalogs: true
I20250113 02:51:40.597560 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_8c672e59aa494348abcd33a8e2dc2000"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:40.597915 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8c672e59aa494348abcd33a8e2dc2000 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:40.598353 1034 tablet_service.cc:1467] Processing CreateTablet for tablet 1c153e47a7aa4bf5ae61dd19a91930b9 (DEFAULT_TABLE table=default.Table_302b76de6d984d2586c6a6e182f85156 [id=095952e818d24cb08c5e1ff3cc5c0a8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:40.598671 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c153e47a7aa4bf5ae61dd19a91930b9. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:40.599124 711 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:38467, user_credentials={real_user=slave}} blocked reactor thread for 87938.3us
I20250113 02:51:40.601591 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d2b61dc5590a4e43af7bd9fecfb0dc52. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:40.607254 1036 tablet_service.cc:1467] Processing CreateTablet for tablet 559c2f8ea91d4e81bf303b3a0c61c143 (DEFAULT_TABLE table=default.Table_babc6b8af3474518b339d149d6a2ccb9 [id=5b0c091b36cb4f9a96bcd45ecae4c659]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:40.607697 1036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 559c2f8ea91d4e81bf303b3a0c61c143. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:40.608184 739 tablet_service.cc:1467] Processing CreateTablet for tablet 1c153e47a7aa4bf5ae61dd19a91930b9 (DEFAULT_TABLE table=default.Table_302b76de6d984d2586c6a6e182f85156 [id=095952e818d24cb08c5e1ff3cc5c0a8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:40.608350 1026 tablet_service.cc:1467] Processing CreateTablet for tablet 223fddf3cbb041399d6eb0153a348f4f (DEFAULT_TABLE table=default.Table_4d8efcc9d773454bb7158a4d396ec5bc [id=2c08a2745b5049fa935cfef62c1570e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:40.608428 740 tablet_service.cc:1467] Processing CreateTablet for tablet 223fddf3cbb041399d6eb0153a348f4f (DEFAULT_TABLE table=default.Table_4d8efcc9d773454bb7158a4d396ec5bc [id=2c08a2745b5049fa935cfef62c1570e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:40.608530 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c153e47a7aa4bf5ae61dd19a91930b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:40.608662 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 223fddf3cbb041399d6eb0153a348f4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:40.608724 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 223fddf3cbb041399d6eb0153a348f4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:40.608433 755 tablet_service.cc:1467] Processing CreateTablet for tablet 0efdd95748e3469f9eb815e63b5a98b7 (DEFAULT_TABLE table=default.Table_ba9a270799304a45a458f11c1c84cf0c [id=b54c7e182fad4e7a9b949540422309ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:40.609373 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0efdd95748e3469f9eb815e63b5a98b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:40.611290 750 tablet_service.cc:1467] Processing CreateTablet for tablet 65139dcfe89249ec95f5425b294f04d8 (DEFAULT_TABLE table=default.Table_f1d37dd0db604e66abe5918b55426fa2 [id=a7a3b5ef19d94090a39ab850588951bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:40.611605 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65139dcfe89249ec95f5425b294f04d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:40.612694 1031 tablet_service.cc:1467] Processing CreateTablet for tablet 0efdd95748e3469f9eb815e63b5a98b7 (DEFAULT_TABLE table=default.Table_ba9a270799304a45a458f11c1c84cf0c [id=b54c7e182fad4e7a9b949540422309ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:40.614123 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0efdd95748e3469f9eb815e63b5a98b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:40.617089 1191 tablet_bootstrap.cc:654] T 0839a35fcab04f3c9d03e256553e01c8 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:40.619601 745 tablet_service.cc:1467] Processing CreateTablet for tablet 559c2f8ea91d4e81bf303b3a0c61c143 (DEFAULT_TABLE table=default.Table_babc6b8af3474518b339d149d6a2ccb9 [id=5b0c091b36cb4f9a96bcd45ecae4c659]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:40.619585 737 tablet_service.cc:1467] Processing CreateTablet for tablet 543149069ea24d00a875e0a1aee51902 (DEFAULT_TABLE table=default.Table_1e8aa807c9e642c8950cff0dccb6e83a [id=fb8bb33c586f4278a5ce021c3f492477]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:40.619585 753 tablet_service.cc:1467] Processing CreateTablet for tablet 4f90d18b45c74f5bab35dd8642d43304 (DEFAULT_TABLE table=default.Table_d6974c08d3a44b0d93d7eb4404b475b5 [id=69188b43c03b416b9b5495b89eecc088]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:40.620074 737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 543149069ea24d00a875e0a1aee51902. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:40.620507 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f90d18b45c74f5bab35dd8642d43304. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:40.620575 1023 tablet_service.cc:1467] Processing CreateTablet for tablet 0a2cc037c96b4d8cbe41d22e3893646c (DEFAULT_TABLE table=default.Table_ac9027839dca4defb2176e380dac44d9 [id=4fb8080da4a940adb75d551a42934e4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:40.620939 1039 tablet_service.cc:1467] Processing CreateTablet for tablet 4f90d18b45c74f5bab35dd8642d43304 (DEFAULT_TABLE table=default.Table_d6974c08d3a44b0d93d7eb4404b475b5 [id=69188b43c03b416b9b5495b89eecc088]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:40.621239 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f90d18b45c74f5bab35dd8642d43304. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:40.621986 1035 tablet_service.cc:1467] Processing CreateTablet for tablet 543149069ea24d00a875e0a1aee51902 (DEFAULT_TABLE table=default.Table_1e8aa807c9e642c8950cff0dccb6e83a [id=fb8bb33c586f4278a5ce021c3f492477]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:40.622287 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 543149069ea24d00a875e0a1aee51902. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:40.623531 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0a2cc037c96b4d8cbe41d22e3893646c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:40.624186 749 tablet_service.cc:1467] Processing CreateTablet for tablet 0a2cc037c96b4d8cbe41d22e3893646c (DEFAULT_TABLE table=default.Table_ac9027839dca4defb2176e380dac44d9 [id=4fb8080da4a940adb75d551a42934e4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:40.624512 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0a2cc037c96b4d8cbe41d22e3893646c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:40.624581 1033 tablet_service.cc:1467] Processing CreateTablet for tablet 29adf4ad7b454c60b2689e6f7d50cbc8 (DEFAULT_TABLE table=default.Table_3b79620cd21744a28c4292aaebc54307 [id=889b5ac813d34e40b88e2ad9bdcdcbe6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:40.624959 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 29adf4ad7b454c60b2689e6f7d50cbc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:40.625658 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet eceed0f52c9f46bbaa78ca773a7afde5 with delete_type TABLET_DATA_DELETED (replaced by tablet 7d90e1689b164943b09f6cda5a6ae5cb) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:40.626609 1192 tablet_bootstrap.cc:654] T b815003a27f64809baaa2d43d6293cef P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:40.627540 1041 tablet_service.cc:1467] Processing CreateTablet for tablet f3f6113ee3d94bb98dbe95d21b1478ef (DEFAULT_TABLE table=default.Table_a306bb67a3c44ddb86b707c1660cf369 [id=fb6fa034fd8c4bb291e73e2e88e5ebb5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:40.627821 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet a0c14501ad0449aca28909bc14428dad
I20250113 02:51:40.627913 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3f6113ee3d94bb98dbe95d21b1478ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:40.629282 448 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet 0136684e4cbe428db182017844e3909d
I20250113 02:51:40.629753 1025 tablet_service.cc:1467] Processing CreateTablet for tablet 04eb898506aa49f78f8ad18fe4b1c99f (DEFAULT_TABLE table=default.Table_df54b4888fd444f0ad01bc7f7b13c336 [id=d8b9ecfcfed845e1b817bc5191136987]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:40.630065 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 04eb898506aa49f78f8ad18fe4b1c99f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:40.630916 1032 tablet_service.cc:1467] Processing CreateTablet for tablet 636f2cf30a3f4e509d39f99c6828639c (DEFAULT_TABLE table=default.Table_ae7b078c404f4196ac5289edb6ac5b3f [id=8ff9f8850f374206826a9a8e1403bb72]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:40.631302 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 636f2cf30a3f4e509d39f99c6828639c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:40.631863 1029 tablet_service.cc:1467] Processing CreateTablet for tablet 46aac43a1af14867a82639554ca59c1f (DEFAULT_TABLE table=default.Table_370f637005ce4fef8bac91353769e3ec [id=442ae0ed20e54a13babc17448f4fa272]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:40.632289 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46aac43a1af14867a82639554ca59c1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:40.633414 1028 tablet_service.cc:1467] Processing CreateTablet for tablet 7e13b8ad06964f13868c80d5bae43d35 (DEFAULT_TABLE table=default.Table_781f7956cbc4450ab623dc03830f909f [id=1ea55f15f82d48759efa4da17e314cd7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:40.634552 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c153e47a7aa4bf5ae61dd19a91930b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:40.556257 895 tablet_service.cc:1467] Processing CreateTablet for tablet 4e59953509884a0f82b276b232bb00fb (DEFAULT_TABLE table=default.Table_2bcd0f2a378c4d2aaefbbbfa2c144431 [id=818f5552fdaf4a54b3d10b920a04e7b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:40.636022 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e59953509884a0f82b276b232bb00fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:40.636994 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7e13b8ad06964f13868c80d5bae43d35. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:40.637486 1027 tablet_service.cc:1467] Processing CreateTablet for tablet 348c21d783ef4103a37409ef8634eb35 (DEFAULT_TABLE table=default.Table_d70d62ca846e4071af60ad816ded897f [id=f69b424f93094afdb00b9010440fc4b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:40.637818 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 348c21d783ef4103a37409ef8634eb35. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:40.574098 2449 raft_consensus.cc:513] T 1d15c7a01f854f0aaabcd30ec3166e31 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:40.640890 2449 leader_election.cc:290] T 1d15c7a01f854f0aaabcd30ec3166e31 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
W20250113 02:51:40.641860 851 leader_election.cc:343] T 1d15c7a01f854f0aaabcd30ec3166e31 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Not found: Tablet not found: 1d15c7a01f854f0aaabcd30ec3166e31
I20250113 02:51:40.642191 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 6001369444894b48b51a48fb60ddc52a
I20250113 02:51:40.623600 2473 raft_consensus.cc:491] T cf7b8bd6801743f49e9b9473747447c5 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:40.647189 1034 tablet_service.cc:1467] Processing CreateTablet for tablet 71122b6a1c0145b4a674815f3cbf6e92 (DEFAULT_TABLE table=default.Table_2d9099633e344b5886574d99e90fe2b7 [id=60494a64215247ba9dc8aa0bf5d06948]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:40.647553 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 71122b6a1c0145b4a674815f3cbf6e92. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:40.653296 2474 raft_consensus.cc:491] T 788ab82626424b6e833e31761a15b9b2 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:40.653429 2474 raft_consensus.cc:513] T 788ab82626424b6e833e31761a15b9b2 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:40.653774 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 559c2f8ea91d4e81bf303b3a0c61c143. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:40.654232 2473 raft_consensus.cc:513] T cf7b8bd6801743f49e9b9473747447c5 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:40.655972 2474 leader_election.cc:290] T 788ab82626424b6e833e31761a15b9b2 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
I20250113 02:51:40.655366 1036 tablet_service.cc:1467] Processing CreateTablet for tablet b7247150e20348619fda21ebbdd305c8 (DEFAULT_TABLE table=default.Table_4648c0331016457faa041613f30e8a95 [id=0ce517b3d4664eb4a3360a647ccc150f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:40.656677 1036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b7247150e20348619fda21ebbdd305c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:40.661897 2449 raft_consensus.cc:491] T 3025935f9743442eb3e6f2db22eb52ad P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:40.662050 2449 raft_consensus.cc:513] T 3025935f9743442eb3e6f2db22eb52ad P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
W20250113 02:51:40.662467 852 leader_election.cc:343] T 1d15c7a01f854f0aaabcd30ec3166e31 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Not found: Tablet not found: 1d15c7a01f854f0aaabcd30ec3166e31
I20250113 02:51:40.662580 852 leader_election.cc:304] T 1d15c7a01f854f0aaabcd30ec3166e31 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df69f48656b44bb890f76587ee69cc0f; no voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca
W20250113 02:51:40.663941 851 leader_election.cc:343] T 3025935f9743442eb3e6f2db22eb52ad P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Not found: Tablet not found: 3025935f9743442eb3e6f2db22eb52ad
I20250113 02:51:40.667481 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "788ab82626424b6e833e31761a15b9b2" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:40.668138 892 tablet_service.cc:1467] Processing CreateTablet for tablet 65139dcfe89249ec95f5425b294f04d8 (DEFAULT_TABLE table=default.Table_f1d37dd0db604e66abe5918b55426fa2 [id=a7a3b5ef19d94090a39ab850588951bb]), partition=RANGE (key) PARTITION UNBOUNDED
W20250113 02:51:40.668388 998 leader_election.cc:343] T 788ab82626424b6e833e31761a15b9b2 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
W20250113 02:51:40.668581 998 consensus_peers.cc:487] T b5c1882125b5491ea79a29a6dac6771e P 4216a7f853d642b7bc3e5cff458f3aca -> Peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Couldn't send request to peer df69f48656b44bb890f76587ee69cc0f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250113 02:51:40.669394 1026 tablet_service.cc:1467] Processing CreateTablet for tablet 2c60925ff0f64925bebba9a88eea9c99 (DEFAULT_TABLE table=default.Table_4f8f4a48900d4aa7bd05738d54c4f309 [id=b161888f490a482aa8ed428ae8a9a176]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:40.669807 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c60925ff0f64925bebba9a88eea9c99. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:40.671135 1198 ts_tablet_manager.cc:1918] T dd566a49d3ff4d3889b80d8d2ccbee0e P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:40.671236 1198 log.cc:1198] T dd566a49d3ff4d3889b80d8d2ccbee0e P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/dd566a49d3ff4d3889b80d8d2ccbee0e
I20250113 02:51:40.671509 1198 ts_tablet_manager.cc:1939] T dd566a49d3ff4d3889b80d8d2ccbee0e P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:40.672271 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1132) took 2450 ms. Trace:
I20250113 02:51:40.672403 1198 rpcz_store.cc:276] 0113 02:51:38.222035 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:38.222094 (+ 59us) service_pool.cc:225] Handling call
0113 02:51:40.442074 (+2219980us) ts_tablet_manager.cc:1145] Deleting tablet dd566a49d3ff4d3889b80d8d2ccbee0e
0113 02:51:40.442082 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:40.486670 (+ 44588us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:40.664629 (+177959us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:40.672225 (+ 7596us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2219778}
I20250113 02:51:40.672780 1198 tablet_replica.cc:331] T bc533518faa24dffa3306ff092e8d507 P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:40.673007 1198 raft_consensus.cc:2238] T bc533518faa24dffa3306ff092e8d507 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: Raft consensus shutting down.
I20250113 02:51:40.673362 1198 pending_rounds.cc:62] T bc533518faa24dffa3306ff092e8d507 P 4216a7f853d642b7bc3e5cff458f3aca: Trying to abort 1 pending ops.
I20250113 02:51:40.674902 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet dd566a49d3ff4d3889b80d8d2ccbee0e (table default.Table_34e676c9d2fb4f7ba8a20c74bc14eccf [id=0e2055968e5a492492305d4513f0edba]) successfully deleted
I20250113 02:51:40.676342 1194 tablet_bootstrap.cc:492] T eebad045e7bb43fa9ca9c9af8b3b7ea5 P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:40.676451 1030 tablet_service.cc:1467] Processing CreateTablet for tablet 1c3c3b7cf89e4f22b8a5dd7ba4f8f5ec (DEFAULT_TABLE table=default.Table_921e31e7ce184317873cbfa17f6f32ac [id=11be84de8ed648e18e40d9d55534401c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:40.676479 883 tablet_service.cc:1467] Processing CreateTablet for tablet 0efdd95748e3469f9eb815e63b5a98b7 (DEFAULT_TABLE table=default.Table_ba9a270799304a45a458f11c1c84cf0c [id=b54c7e182fad4e7a9b949540422309ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:40.677744 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c3c3b7cf89e4f22b8a5dd7ba4f8f5ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:40.678593 1058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf7b8bd6801743f49e9b9473747447c5" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
W20250113 02:51:40.679177 709 leader_election.cc:343] T cf7b8bd6801743f49e9b9473747447c5 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:40.679821 1037 tablet_service.cc:1467] Processing CreateTablet for tablet 292f2a0651144e569c71d4296be837d3 (DEFAULT_TABLE table=default.Table_6c3417d70cd4422993f46368566b2d51 [id=8ccebc7fe83241ecbca9eb444bcc7bc0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:40.680294 1037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 292f2a0651144e569c71d4296be837d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:40.678309 2473 leader_election.cc:290] T cf7b8bd6801743f49e9b9473747447c5 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:40.673408 1198 pending_rounds.cc:69] T bc533518faa24dffa3306ff092e8d507 P 4216a7f853d642b7bc3e5cff458f3aca: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113673519392489472 op_type: NO_OP noop_request { }
I20250113 02:51:40.688977 2473 raft_consensus.cc:491] T 0752fde3a30a4947a2a688c56ceadab5 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4216a7f853d642b7bc3e5cff458f3aca)
I20250113 02:51:40.689236 1038 tablet_service.cc:1467] Processing CreateTablet for tablet d1f22bb035f447568a13abf2b5c70ce3 (DEFAULT_TABLE table=default.Table_fe974dd3772d4afd880bc574c573a649 [id=cc2cd3b6418e4c74b50b42b72536ff5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:40.692646 1027 tablet_service.cc:1467] Processing CreateTablet for tablet 80d42e6aea134575b3bd95dee82e5416 (DEFAULT_TABLE table=default.Table_8eeeda2b6ca04de78b01bb716651d49c [id=9360dcb5f4b64ec68a75f1ae3b43894c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:40.693063 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 80d42e6aea134575b3bd95dee82e5416. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:40.688992 1198 raft_consensus.cc:2883] T bc533518faa24dffa3306ff092e8d507 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250113 02:51:40.693689 1198 raft_consensus.cc:2267] T bc533518faa24dffa3306ff092e8d507 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:40.694171 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d1f22bb035f447568a13abf2b5c70ce3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:40.695225 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65139dcfe89249ec95f5425b294f04d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:40.695878 2449 leader_election.cc:290] T 3025935f9743442eb3e6f2db22eb52ad P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:40.696067 2449 raft_consensus.cc:2743] T 1d15c7a01f854f0aaabcd30ec3166e31 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:40.696175 1198 ts_tablet_manager.cc:1905] T bc533518faa24dffa3306ff092e8d507 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:40.696768 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0efdd95748e3469f9eb815e63b5a98b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:40.702466 1024 tablet_service.cc:1467] Processing CreateTablet for tablet ccc9ae0461cd40a38cb3a90412df7e82 (DEFAULT_TABLE table=default.Table_fa2ea67525994579bd81a459396f4281 [id=cfce13a8e9e1420dadc977703b49e107]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:40.702407 2473 raft_consensus.cc:513] T 0752fde3a30a4947a2a688c56ceadab5 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:40.702783 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ccc9ae0461cd40a38cb3a90412df7e82. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:40.705196 1028 tablet_service.cc:1467] Processing CreateTablet for tablet 284ca2fe0c6a4b79a3bc6d096a90ccf1 (DEFAULT_TABLE table=default.Table_f830fa2aaadc44bbb73b8eb61545aabc [id=3a485245494241b5a902dc44980c7c8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:40.706378 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 284ca2fe0c6a4b79a3bc6d096a90ccf1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:40.707176 2473 leader_election.cc:290] T 0752fde3a30a4947a2a688c56ceadab5 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
W20250113 02:51:40.712507 709 leader_election.cc:343] T 0752fde3a30a4947a2a688c56ceadab5 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Not found: Tablet not found: 0752fde3a30a4947a2a688c56ceadab5
I20250113 02:51:40.725090 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet 42a8e01bb71f49b09c83a8f5c905f38d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:40 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:40.725306 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet 94307d22b7f842ca89844768e4a2a2ab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:40 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:40.725440 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet a0c14501ad0449aca28909bc14428dad with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:40 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:40.725914 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet 6001369444894b48b51a48fb60ddc52a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:40 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:40.726061 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet b33025a462e74f2aa26cc3779f36615b with delete_type TABLET_DATA_DELETED (replaced by tablet f9269967dea94fc3a337546ad0842e11) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:40.731387 1195 ts_tablet_manager.cc:1918] T 788ab82626424b6e833e31761a15b9b2 P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:40.731499 1195 log.cc:1198] T 788ab82626424b6e833e31761a15b9b2 P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/788ab82626424b6e833e31761a15b9b2
I20250113 02:51:40.713174 884 tablet_service.cc:1467] Processing CreateTablet for tablet 223fddf3cbb041399d6eb0153a348f4f (DEFAULT_TABLE table=default.Table_4d8efcc9d773454bb7158a4d396ec5bc [id=2c08a2745b5049fa935cfef62c1570e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:40.738348 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 223fddf3cbb041399d6eb0153a348f4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:40.739856 2473 raft_consensus.cc:491] T cfa963a6df094ce0b9c54408f75ca6b2 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:40.709106 882 tablet_service.cc:1514] Processing DeleteTablet for tablet 999fecb3fc65466cad21cc1ff5ef2d44 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:23 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:40.744418 882 tablet_service.cc:1467] Processing CreateTablet for tablet 543149069ea24d00a875e0a1aee51902 (DEFAULT_TABLE table=default.Table_1e8aa807c9e642c8950cff0dccb6e83a [id=fb8bb33c586f4278a5ce021c3f492477]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:40.748775 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 543149069ea24d00a875e0a1aee51902. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:40.719228 899 tablet_service.cc:1467] Processing CreateTablet for tablet 559c2f8ea91d4e81bf303b3a0c61c143 (DEFAULT_TABLE table=default.Table_babc6b8af3474518b339d149d6a2ccb9 [id=5b0c091b36cb4f9a96bcd45ecae4c659]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:40.751454 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 559c2f8ea91d4e81bf303b3a0c61c143. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:40.740010 2473 raft_consensus.cc:513] T cfa963a6df094ce0b9c54408f75ca6b2 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:40.760561 2473 leader_election.cc:290] T cfa963a6df094ce0b9c54408f75ca6b2 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:40.764508 752 tablet_service.cc:1467] Processing CreateTablet for tablet f3f6113ee3d94bb98dbe95d21b1478ef (DEFAULT_TABLE table=default.Table_a306bb67a3c44ddb86b707c1660cf369 [id=fb6fa034fd8c4bb291e73e2e88e5ebb5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:40.764865 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3f6113ee3d94bb98dbe95d21b1478ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:40.740732 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "788ab82626424b6e833e31761a15b9b2" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
W20250113 02:51:40.768911 995 leader_election.cc:343] T 788ab82626424b6e833e31761a15b9b2 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Illegal state: Tablet not RUNNING: STOPPED
I20250113 02:51:40.769027 995 leader_election.cc:304] T 788ab82626424b6e833e31761a15b9b2 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4216a7f853d642b7bc3e5cff458f3aca; no voters: 1186fa869eb14812832fa6562531f196, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:40.769490 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_50cc6a4316724447bdeefc5aefe989a6" } new_table_name: "default.Table_a2a4fd46d33f4ea6ae4f3a7883f4ff1c" modify_external_catalogs: true
I20250113 02:51:40.770402 1058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfa963a6df094ce0b9c54408f75ca6b2" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:40.771188 2363 consensus_queue.cc:1035] T b5c1882125b5491ea79a29a6dac6771e P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 02:51:40.771562 2363 raft_consensus.cc:2743] T 788ab82626424b6e833e31761a15b9b2 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:40.771595 898 tablet_service.cc:1467] Processing CreateTablet for tablet 0a2cc037c96b4d8cbe41d22e3893646c (DEFAULT_TABLE table=default.Table_ac9027839dca4defb2176e380dac44d9 [id=4fb8080da4a940adb75d551a42934e4b]), partition=RANGE (key) PARTITION UNBOUNDED
W20250113 02:51:40.772874 709 leader_election.cc:343] T cfa963a6df094ce0b9c54408f75ca6b2 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:40.773937 447 master_service.cc:946] ReplaceTablet: received request to replace tablet a894199f21aa4dba8a03583ea14a9092 from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:40.774638 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_32fe238c17574c1ea5721f507c0bdcc9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:40.775123 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_32fe238c17574c1ea5721f507c0bdcc9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:40.776242 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_0894b27199a241e594a6f314094d0010" } new_table_name: "default.Table_1ebcecf319bc4ba6856c60c6d6b5fe8c" modify_external_catalogs: true
I20250113 02:51:40.777796 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0a2cc037c96b4d8cbe41d22e3893646c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:40.779844 738 tablet_service.cc:1467] Processing CreateTablet for tablet 29adf4ad7b454c60b2689e6f7d50cbc8 (DEFAULT_TABLE table=default.Table_3b79620cd21744a28c4292aaebc54307 [id=889b5ac813d34e40b88e2ad9bdcdcbe6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:40.780915 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 29adf4ad7b454c60b2689e6f7d50cbc8. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:40.781356 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 119baabad10a488082f86f2a52ac3779 consensus state has no leader
I20250113 02:51:40.783239 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet a894199f21aa4dba8a03583ea14a9092
I20250113 02:51:40.783632 447 catalog_manager.cc:6596] ReplaceTablet: tablet a894199f21aa4dba8a03583ea14a9092 deleted and replaced by tablet f6f4df3bde434667a923be19aa39817a
I20250113 02:51:40.784850 744 tablet_service.cc:1467] Processing CreateTablet for tablet 46aac43a1af14867a82639554ca59c1f (DEFAULT_TABLE table=default.Table_370f637005ce4fef8bac91353769e3ec [id=442ae0ed20e54a13babc17448f4fa272]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:40.785244 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46aac43a1af14867a82639554ca59c1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:40.788264 1033 tablet_service.cc:1514] Processing DeleteTablet for tablet a894199f21aa4dba8a03583ea14a9092 with delete_type TABLET_DATA_DELETED (replaced by tablet f6f4df3bde434667a923be19aa39817a) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:40.788661 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_64fa660da06b4c7cb2c76054b3a4ceba"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:40.789194 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_64fa660da06b4c7cb2c76054b3a4ceba in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:40.802390 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_d5a79bb9160448269ae65f45fcd592a9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:40.802621 750 tablet_service.cc:1514] Processing DeleteTablet for tablet eceed0f52c9f46bbaa78ca773a7afde5 with delete_type TABLET_DATA_DELETED (replaced by tablet 7d90e1689b164943b09f6cda5a6ae5cb) from {username='slave'} at 127.0.0.1:55398
W20250113 02:51:40.802848 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9fcd4b72b66c4373aa3e8168ab438b56 consensus state has no leader
W20250113 02:51:40.802897 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d5a79bb9160448269ae65f45fcd592a9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:40.804029 453 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_f8a9bb5154f746afa795d4996851b7b5" } modify_external_catalogs: true
I20250113 02:51:40.804162 453 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_f8a9bb5154f746afa795d4996851b7b5" } modify_external_catalogs: true
I20250113 02:51:40.805146 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_183e07b4f9aa438e8a1c37ef385f57d6" } new_table_name: "default.Table_00582a64ee964822ba060ccbffeb352f" modify_external_catalogs: true
I20250113 02:51:40.806363 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_0e134d21b63b463ea086ed3ac6921486"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:40.806828 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0e134d21b63b463ea086ed3ac6921486 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:40.807852 448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_8c672e59aa494348abcd33a8e2dc2000" } new_table_name: "default.Table_bdab3234b9d44a7fb534b8479b322f4f" modify_external_catalogs: true
I20250113 02:51:40.811206 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_52010496bf7f4c85a99f61e12fb965a9" } modify_external_catalogs: true
I20250113 02:51:40.811354 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_52010496bf7f4c85a99f61e12fb965a9" } modify_external_catalogs: true
I20250113 02:51:40.812925 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_b4f66c4bb40c4135ac5516b28550f0d5" } new_table_name: "default.Table_4ddc6e899be347f7bc0912e45206255c" modify_external_catalogs: true
I20250113 02:51:40.816547 1042 tablet_service.cc:1467] Processing CreateTablet for tablet 3c4224468d8d42d2ba21cdb266a9c8c3 (DEFAULT_TABLE table=default.Table_4d27f00185134500b7a8461f7efe9961 [id=cc3c6b8e215a42ad8ae2f26b4e793e47]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:40.816999 1042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c4224468d8d42d2ba21cdb266a9c8c3. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:40.817631 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6b7716e67cab449ab3ed4e58eae20c9b consensus state has no leader
I20250113 02:51:40.818010 1037 tablet_service.cc:1467] Processing CreateTablet for tablet 3d40b9c159c44a5cba16cd4d262df0c2 (DEFAULT_TABLE table=default.Table_9175390f220844ad984ccaa8df33cecf [id=1f913bf2d3c841a198ce7be091a3642a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:40.818301 1037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3d40b9c159c44a5cba16cd4d262df0c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:40.820858 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 8f2b4df7c6a74896ade95a490ea9b54c
I20250113 02:51:40.816808 754 tablet_service.cc:1467] Processing CreateTablet for tablet 348c21d783ef4103a37409ef8634eb35 (DEFAULT_TABLE table=default.Table_d70d62ca846e4071af60ad816ded897f [id=f69b424f93094afdb00b9010440fc4b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:40.821883 2479 tablet.cc:1721] T 0b119704bc774ca9b4155c9619d4765b P 4216a7f853d642b7bc3e5cff458f3aca: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20250113 02:51:40.822260 448 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d3619576c8f94583b330fe1828cea2e0 has no consensus state
I20250113 02:51:40.822434 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 348c21d783ef4103a37409ef8634eb35. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:40.823889 1031 tablet_service.cc:1467] Processing CreateTablet for tablet 4b75150005e045f08148e1437586999d (DEFAULT_TABLE table=default.Table_ff5dca8686a64b858a2ab711d109d4a9 [id=4c43f64ec2a340e1bfc7f5923b3acc2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:40.824257 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b75150005e045f08148e1437586999d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:40.827579 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 3c623bd0fbbf4922aedda031a88b79eb
I20250113 02:51:40.828682 750 tablet_service.cc:1467] Processing CreateTablet for tablet 636f2cf30a3f4e509d39f99c6828639c (DEFAULT_TABLE table=default.Table_ae7b078c404f4196ac5289edb6ac5b3f [id=8ff9f8850f374206826a9a8e1403bb72]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:40.823534 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f2b4df7c6a74896ade95a490ea9b54c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:40 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:40.829090 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 636f2cf30a3f4e509d39f99c6828639c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:40.829541 1032 tablet_service.cc:1467] Processing CreateTablet for tablet 4dab7d43f5314715a0021cf7da82c285 (DEFAULT_TABLE table=default.Table_0e6eae4ff7724f6aa85277439a83f3b1 [id=e984030bfc2d450995f2ec90c463670b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:40.829859 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4dab7d43f5314715a0021cf7da82c285. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:40.831727 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c623bd0fbbf4922aedda031a88b79eb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:40 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:40.832479 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_255be37d0d3e4cfca070e8137a82f90b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:40.833104 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_255be37d0d3e4cfca070e8137a82f90b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:40.834457 448 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_33d7dda5ebe341bc8ef01b7edb3cd1c6" } modify_external_catalogs: true
I20250113 02:51:40.834599 448 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_33d7dda5ebe341bc8ef01b7edb3cd1c6" } modify_external_catalogs: true
I20250113 02:51:40.834903 1192 tablet_bootstrap.cc:492] T b815003a27f64809baaa2d43d6293cef P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:40.835024 1192 ts_tablet_manager.cc:1397] T b815003a27f64809baaa2d43d6293cef P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.308s user 0.014s sys 0.027s
I20250113 02:51:40.837899 1025 tablet_service.cc:1467] Processing CreateTablet for tablet 0f091eb311d94956bb398b6c5efce284 (DEFAULT_TABLE table=default.Table_5576eb7eae61401bbb7065ffbd571846 [id=92e0e4b4d89f4bba8d8e1e4841e76108]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:40.838342 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f091eb311d94956bb398b6c5efce284. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:40.839404 883 tablet_service.cc:1467] Processing CreateTablet for tablet f3f6113ee3d94bb98dbe95d21b1478ef (DEFAULT_TABLE table=default.Table_a306bb67a3c44ddb86b707c1660cf369 [id=fb6fa034fd8c4bb291e73e2e88e5ebb5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:40.839725 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3f6113ee3d94bb98dbe95d21b1478ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:40.840093 1036 tablet_service.cc:1467] Processing CreateTablet for tablet 476c89cc585b4e00947f5d1e8d0229f6 (DEFAULT_TABLE table=default.Table_f7a044df83094686b983b6a11fd518cd [id=bc64fd48c0b14d318a9e2bdf1025cf46]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:40.840991 890 tablet_service.cc:1467] Processing CreateTablet for tablet 29adf4ad7b454c60b2689e6f7d50cbc8 (DEFAULT_TABLE table=default.Table_3b79620cd21744a28c4292aaebc54307 [id=889b5ac813d34e40b88e2ad9bdcdcbe6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:40.841600 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_da027dfe87ee40ada76b095abebd0e7e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:40.842049 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_da027dfe87ee40ada76b095abebd0e7e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:40.842545 1035 tablet_service.cc:1467] Processing CreateTablet for tablet 3a89e991ebee4b4f8c639ea95e1d1ea3 (DEFAULT_TABLE table=default.Table_dc9db684670a46229bcca7df696bb04c [id=d7a6380bf81d4946b180bbbad5f870b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:40.845469 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_98fee465d761490c85252dbb021e3f96"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:40.845954 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_98fee465d761490c85252dbb021e3f96 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:40.846119 1028 tablet_service.cc:1467] Processing CreateTablet for tablet 8be8227851a740d890e46b3a3fa4a2ce (DEFAULT_TABLE table=default.Table_344e7583f72042fb926d0f2e1aa2ce87 [id=fffdac188fdf437589acc52fa0b0dab7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:40.847596 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_588573aeb9464e8088f6759b1ca47c54"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:40.848021 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_588573aeb9464e8088f6759b1ca47c54 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:40.848167 1198 ts_tablet_manager.cc:1918] T bc533518faa24dffa3306ff092e8d507 P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250113 02:51:40.848284 1198 log.cc:1198] T bc533518faa24dffa3306ff092e8d507 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/bc533518faa24dffa3306ff092e8d507
I20250113 02:51:40.848625 1198 ts_tablet_manager.cc:1939] T bc533518faa24dffa3306ff092e8d507 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:40.849294 455 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_d7d1f5a34e5f4c05b474b0b61f912cd9" } modify_external_catalogs: true
I20250113 02:51:40.849433 455 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_d7d1f5a34e5f4c05b474b0b61f912cd9" } modify_external_catalogs: true
I20250113 02:51:40.850000 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_c1e2aac17dd047708ca98085186252a7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:40.850409 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c1e2aac17dd047708ca98085186252a7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:40.853155 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_be2b756d1ad24981880b0debe778986a" } modify_external_catalogs: true
I20250113 02:51:40.853288 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_be2b756d1ad24981880b0debe778986a" } modify_external_catalogs: true
I20250113 02:51:40.855342 1033 tablet_service.cc:1467] Processing CreateTablet for tablet 5581f9bacd6b4019be6ebbe5b447e0e5 (DEFAULT_TABLE table=default.Table_f1f79d9ce8f24b5cb56d1ce5082196c6 [id=0d1e9f75d89145d1949af1a60cd866a4]), partition=RANGE (key) PARTITION UNBOUNDED
W20250113 02:51:40.858654 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 141a8eb1a30f41549786117f9e2d9d5f consensus state has no leader
I20250113 02:51:40.860488 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_969a9705d160408cb78f3c7029fa4023"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:40.860924 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_969a9705d160408cb78f3c7029fa4023 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:40.861447 1036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 476c89cc585b4e00947f5d1e8d0229f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:40.796991 880 tablet_service.cc:1467] Processing CreateTablet for tablet 4f90d18b45c74f5bab35dd8642d43304 (DEFAULT_TABLE table=default.Table_d6974c08d3a44b0d93d7eb4404b475b5 [id=69188b43c03b416b9b5495b89eecc088]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:40.862767 1192 raft_consensus.cc:357] T b815003a27f64809baaa2d43d6293cef P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:40.863168 1192 raft_consensus.cc:383] T b815003a27f64809baaa2d43d6293cef P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:40.863286 1192 raft_consensus.cc:738] T b815003a27f64809baaa2d43d6293cef P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:40.864985 1034 tablet_service.cc:1467] Processing CreateTablet for tablet 1bb7c8c9217143e79659da0650d1bf11 (DEFAULT_TABLE table=default.Table_0e07e7c736e74a51bd66a147c524027f [id=bc63054e26c2493084812e45ebec909e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:40.866130 1030 tablet_service.cc:1467] Processing CreateTablet for tablet 045ff4572f3347fc890b439d90cb0bbe (DEFAULT_TABLE table=default.Table_5a9dd13ac839443f862131a444ab842c [id=29faf5f637cf4e8cbff1d3c502dc01b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:40.863579 1192 consensus_queue.cc:260] T b815003a27f64809baaa2d43d6293cef P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:40.866972 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1133) took 2508 ms. Trace:
I20250113 02:51:40.867082 1198 rpcz_store.cc:276] 0113 02:51:38.358770 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:38.358838 (+ 68us) service_pool.cc:225] Handling call
0113 02:51:40.672724 (+2313886us) ts_tablet_manager.cc:1145] Deleting tablet bc533518faa24dffa3306ff092e8d507
0113 02:51:40.672737 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:40.768976 (+ 96239us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:40.848093 (+ 79117us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:40.866949 (+ 18856us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2313656}
I20250113 02:51:40.867707 1198 tablet_replica.cc:331] T 655b63ff61ff4f4aa08e0449df39352a P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:40.867856 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet bc533518faa24dffa3306ff092e8d507 (table default.Table_03babf194cc448f296512a09b2e3fe3e [id=64b42698782f41a589577af20b33501e]) successfully deleted
I20250113 02:51:40.867933 1198 raft_consensus.cc:2238] T 655b63ff61ff4f4aa08e0449df39352a P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:40.868566 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 045ff4572f3347fc890b439d90cb0bbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:40.870303 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 29adf4ad7b454c60b2689e6f7d50cbc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:40.872354 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5581f9bacd6b4019be6ebbe5b447e0e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:40.872542 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a89e991ebee4b4f8c639ea95e1d1ea3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:40.872808 1198 raft_consensus.cc:2267] T 655b63ff61ff4f4aa08e0449df39352a P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:40.873091 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1bb7c8c9217143e79659da0650d1bf11. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:40.873410 1198 ts_tablet_manager.cc:1905] T 655b63ff61ff4f4aa08e0449df39352a P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:40.874428 1024 tablet_service.cc:1467] Processing CreateTablet for tablet d114f02268794e4fa8c464e0442b8eff (DEFAULT_TABLE table=default.Table_5851d43a71e641058f8d820b078e9aca [id=4695f75c633f4513a6381aa3d293f0cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:40.874730 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d114f02268794e4fa8c464e0442b8eff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:40.875135 884 tablet_service.cc:1514] Processing DeleteTablet for tablet eceed0f52c9f46bbaa78ca773a7afde5 with delete_type TABLET_DATA_DELETED (replaced by tablet 7d90e1689b164943b09f6cda5a6ae5cb) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:40.875316 1194 tablet_bootstrap.cc:654] T eebad045e7bb43fa9ca9c9af8b3b7ea5 P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:40.875479 884 tablet_service.cc:1467] Processing CreateTablet for tablet 46aac43a1af14867a82639554ca59c1f (DEFAULT_TABLE table=default.Table_370f637005ce4fef8bac91353769e3ec [id=442ae0ed20e54a13babc17448f4fa272]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:40.875818 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46aac43a1af14867a82639554ca59c1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:40.878458 1192 ts_tablet_manager.cc:1428] T b815003a27f64809baaa2d43d6293cef P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.026s user 0.001s sys 0.001s
I20250113 02:51:40.880896 897 tablet_service.cc:1467] Processing CreateTablet for tablet 636f2cf30a3f4e509d39f99c6828639c (DEFAULT_TABLE table=default.Table_ae7b078c404f4196ac5289edb6ac5b3f [id=8ff9f8850f374206826a9a8e1403bb72]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:40.881240 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 636f2cf30a3f4e509d39f99c6828639c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:40.882439 1192 tablet_bootstrap.cc:492] T 82505f1389184d0ca12ec0e2d6084d22 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:40.884019 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8be8227851a740d890e46b3a3fa4a2ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:40.886505 886 tablet_service.cc:1467] Processing CreateTablet for tablet 348c21d783ef4103a37409ef8634eb35 (DEFAULT_TABLE table=default.Table_d70d62ca846e4071af60ad816ded897f [id=f69b424f93094afdb00b9010440fc4b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:40.886827 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 348c21d783ef4103a37409ef8634eb35. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:40.888193 885 tablet_service.cc:1467] Processing CreateTablet for tablet 04eb898506aa49f78f8ad18fe4b1c99f (DEFAULT_TABLE table=default.Table_df54b4888fd444f0ad01bc7f7b13c336 [id=d8b9ecfcfed845e1b817bc5191136987]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:40.888502 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 04eb898506aa49f78f8ad18fe4b1c99f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:40.888612 451 catalog_manager.cc:5618] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587) does not have the latest schema for tablet b815003a27f64809baaa2d43d6293cef (table default.Table_c2e8fd7d2cc14fa39c0abfece783c6c9 [id=42e19138c08f422a92ba9621e9609957]). Expected version 1 got 0
I20250113 02:51:40.899808 887 tablet_service.cc:1467] Processing CreateTablet for tablet 7e13b8ad06964f13868c80d5bae43d35 (DEFAULT_TABLE table=default.Table_781f7956cbc4450ab623dc03830f909f [id=1ea55f15f82d48759efa4da17e314cd7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:40.905056 1196 ts_tablet_manager.cc:1918] T b8b6ffe690fc459db78b2b3bc12671dd P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:40.905159 1196 log.cc:1198] T b8b6ffe690fc459db78b2b3bc12671dd P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/b8b6ffe690fc459db78b2b3bc12671dd
I20250113 02:51:40.905434 1196 ts_tablet_manager.cc:1939] T b8b6ffe690fc459db78b2b3bc12671dd P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
W20250113 02:51:40.908737 995 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:40095, user_credentials={real_user=slave}} blocked reactor thread for 139826us
I20250113 02:51:40.916237 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7e13b8ad06964f13868c80d5bae43d35. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:40.921480 1198 ts_tablet_manager.cc:1918] T 655b63ff61ff4f4aa08e0449df39352a P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:40.921586 1198 log.cc:1198] T 655b63ff61ff4f4aa08e0449df39352a P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/655b63ff61ff4f4aa08e0449df39352a
I20250113 02:51:40.921856 1198 ts_tablet_manager.cc:1939] T 655b63ff61ff4f4aa08e0449df39352a P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:40.924055 895 tablet_service.cc:1467] Processing CreateTablet for tablet 71122b6a1c0145b4a674815f3cbf6e92 (DEFAULT_TABLE table=default.Table_2d9099633e344b5886574d99e90fe2b7 [id=60494a64215247ba9dc8aa0bf5d06948]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:40.924474 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 71122b6a1c0145b4a674815f3cbf6e92. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:40.924732 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1135) took 2482 ms. Trace:
I20250113 02:51:40.926648 1028 tablet_service.cc:1467] Processing CreateTablet for tablet 5979002528af42659ccd8e419cd8ccae (DEFAULT_TABLE table=default.Table_e076d93131bb4c78bbc1548986736de4 [id=6fbe4ba8f7bf4c36b331df31efdb8eef]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:40.928294 1198 rpcz_store.cc:276] 0113 02:51:38.442579 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:38.442632 (+ 53us) service_pool.cc:225] Handling call
0113 02:51:40.867389 (+2424757us) ts_tablet_manager.cc:1145] Deleting tablet 655b63ff61ff4f4aa08e0449df39352a
0113 02:51:40.867661 (+ 272us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:40.898302 (+ 30641us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:40.921414 (+ 23112us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:40.924712 (+ 3298us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2139,"tablet-delete.queue_time_us":2424604}
I20250113 02:51:40.928161 891 tablet_service.cc:1467] Processing CreateTablet for tablet b7247150e20348619fda21ebbdd305c8 (DEFAULT_TABLE table=default.Table_4648c0331016457faa041613f30e8a95 [id=0ce517b3d4664eb4a3360a647ccc150f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:40.929158 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b7247150e20348619fda21ebbdd305c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:40.930013 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1139) took 2176 ms. Trace:
I20250113 02:51:40.929481 2480 consensus_queue.cc:1035] T 44d2c7d5334c476fac60fdd57e2c02b6 P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 02:51:40.928624 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5979002528af42659ccd8e419cd8ccae. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:40.931066 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet 655b63ff61ff4f4aa08e0449df39352a (table default.Table_c6c64517b49e4843ba63567a871d22bd [id=ec4bdc424404427589852603bb6d8112]) successfully deleted
I20250113 02:51:40.931463 1198 rpcz_store.cc:276] 0113 02:51:38.753377 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:38.753434 (+ 57us) service_pool.cc:225] Handling call
0113 02:51:40.929896 (+2176462us) ts_tablet_manager.cc:1145] Deleting tablet 13f9711b16ae4e36be00db07d9bde7fc
0113 02:51:40.929905 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:40.929999 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2176313}
I20250113 02:51:40.931875 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1140) took 2165 ms. Trace:
W20250113 02:51:40.932103 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 13f9711b16ae4e36be00db07d9bde7fc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 13f9711b16ae4e36be00db07d9bde7fc already in progress: creating tablet
I20250113 02:51:40.932255 896 tablet_service.cc:1467] Processing CreateTablet for tablet 2c60925ff0f64925bebba9a88eea9c99 (DEFAULT_TABLE table=default.Table_4f8f4a48900d4aa7bd05738d54c4f309 [id=b161888f490a482aa8ed428ae8a9a176]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:40.932626 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c60925ff0f64925bebba9a88eea9c99. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:40.934700 888 tablet_service.cc:1467] Processing CreateTablet for tablet 1c3c3b7cf89e4f22b8a5dd7ba4f8f5ec (DEFAULT_TABLE table=default.Table_921e31e7ce184317873cbfa17f6f32ac [id=11be84de8ed648e18e40d9d55534401c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:40.935076 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c3c3b7cf89e4f22b8a5dd7ba4f8f5ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:40.932054 1198 rpcz_store.cc:276] 0113 02:51:38.766774 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:38.766832 (+ 58us) service_pool.cc:225] Handling call
0113 02:51:40.931778 (+2164946us) ts_tablet_manager.cc:1145] Deleting tablet 49c2e648a49f4034a521f38e7d98edc4
0113 02:51:40.931786 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:40.931860 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2164726}
I20250113 02:51:40.935860 1192 tablet_bootstrap.cc:654] T 82505f1389184d0ca12ec0e2d6084d22 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
W20250113 02:51:40.936125 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 49c2e648a49f4034a521f38e7d98edc4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 49c2e648a49f4034a521f38e7d98edc4 already in progress: creating tablet
I20250113 02:51:40.927230 1032 tablet_service.cc:1467] Processing CreateTablet for tablet e6f5331ec0044ea7b4134fb2c3d12ebb (DEFAULT_TABLE table=default.Table_7f3e4c1c46cd4df1b77899c2f674068a [id=de907be87311474ea04a275eb44efdc0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:40.936882 1034 tablet_service.cc:1467] Processing CreateTablet for tablet 366e5c5ee708458ba443dd69d65c3988 (DEFAULT_TABLE table=default.Table_40c1445967024038bf8a25c42bf346ad [id=ea4fc227dfb145a2b4a660062410e193]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:40.937626 1030 tablet_service.cc:1467] Processing CreateTablet for tablet 978601b734ea4c44a210347a263e163f (DEFAULT_TABLE table=default.Table_5625ac97f9f84d3e8379eb9536bf94c5 [id=a3c26f79c92e4ce3a8f6a3f7d85b50a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:40.940148 741 tablet_service.cc:1467] Processing CreateTablet for tablet 04eb898506aa49f78f8ad18fe4b1c99f (DEFAULT_TABLE table=default.Table_df54b4888fd444f0ad01bc7f7b13c336 [id=d8b9ecfcfed845e1b817bc5191136987]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:40.940500 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 04eb898506aa49f78f8ad18fe4b1c99f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:40.936815 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1141) took 2169 ms. Trace:
I20250113 02:51:40.944952 1198 rpcz_store.cc:276] 0113 02:51:38.766875 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:38.767727 (+ 852us) service_pool.cc:225] Handling call
0113 02:51:40.936708 (+2168981us) ts_tablet_manager.cc:1145] Deleting tablet 368e878e89b54694823863a49074c11e
0113 02:51:40.936717 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:40.936798 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2168785}
W20250113 02:51:40.945453 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 368e878e89b54694823863a49074c11e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 368e878e89b54694823863a49074c11e already in progress: creating tablet
I20250113 02:51:40.945837 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e6f5331ec0044ea7b4134fb2c3d12ebb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:40.946693 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1142) took 2179 ms. Trace:
I20250113 02:51:40.946830 1198 rpcz_store.cc:276] 0113 02:51:38.766988 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:38.767791 (+ 803us) service_pool.cc:225] Handling call
0113 02:51:40.946579 (+2178788us) ts_tablet_manager.cc:1145] Deleting tablet 71befed8b5c74b41b199919b82325891
0113 02:51:40.946588 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:40.946677 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2178626}
W20250113 02:51:40.947364 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 71befed8b5c74b41b199919b82325891 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 71befed8b5c74b41b199919b82325891 already in progress: creating tablet
I20250113 02:51:40.950471 451 catalog_manager.cc:5618] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587) does not have the latest schema for tablet b815003a27f64809baaa2d43d6293cef (table default.Table_c2e8fd7d2cc14fa39c0abfece783c6c9 [id=42e19138c08f422a92ba9621e9609957]). Expected version 1 got 0
I20250113 02:51:40.952483 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet aef2ffd67def48eea501a98fbc9dc43b
I20250113 02:51:40.961894 736 tablet_service.cc:1467] Processing CreateTablet for tablet 7e13b8ad06964f13868c80d5bae43d35 (DEFAULT_TABLE table=default.Table_781f7956cbc4450ab623dc03830f909f [id=1ea55f15f82d48759efa4da17e314cd7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:40.962038 881 tablet_service.cc:1467] Processing CreateTablet for tablet 292f2a0651144e569c71d4296be837d3 (DEFAULT_TABLE table=default.Table_6c3417d70cd4422993f46368566b2d51 [id=8ccebc7fe83241ecbca9eb444bcc7bc0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:40.962280 736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7e13b8ad06964f13868c80d5bae43d35. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:40.962364 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 292f2a0651144e569c71d4296be837d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:40.963639 1191 tablet_bootstrap.cc:492] T 0839a35fcab04f3c9d03e256553e01c8 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:40.963768 1191 ts_tablet_manager.cc:1397] T 0839a35fcab04f3c9d03e256553e01c8 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.487s user 0.006s sys 0.013s
I20250113 02:51:40.966734 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f90d18b45c74f5bab35dd8642d43304. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:40.975116 1196 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37366 (request call id 1093) took 2107 ms. Trace:
I20250113 02:51:40.975675 893 tablet_service.cc:1467] Processing CreateTablet for tablet d1f22bb035f447568a13abf2b5c70ce3 (DEFAULT_TABLE table=default.Table_fe974dd3772d4afd880bc574c573a649 [id=cc2cd3b6418e4c74b50b42b72536ff5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:40.976449 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d1f22bb035f447568a13abf2b5c70ce3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:40.981628 888 tablet_service.cc:1467] Processing CreateTablet for tablet 80d42e6aea134575b3bd95dee82e5416 (DEFAULT_TABLE table=default.Table_8eeeda2b6ca04de78b01bb716651d49c [id=9360dcb5f4b64ec68a75f1ae3b43894c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:40.987147 743 tablet_service.cc:1467] Processing CreateTablet for tablet 2c60925ff0f64925bebba9a88eea9c99 (DEFAULT_TABLE table=default.Table_4f8f4a48900d4aa7bd05738d54c4f309 [id=b161888f490a482aa8ed428ae8a9a176]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:40.987479 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c60925ff0f64925bebba9a88eea9c99. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:40.985211 882 tablet_service.cc:1467] Processing CreateTablet for tablet ccc9ae0461cd40a38cb3a90412df7e82 (DEFAULT_TABLE table=default.Table_fa2ea67525994579bd81a459396f4281 [id=cfce13a8e9e1420dadc977703b49e107]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:40.987641 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 80d42e6aea134575b3bd95dee82e5416. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:40.988018 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ccc9ae0461cd40a38cb3a90412df7e82. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:40.990021 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_626d5cb595994368aa592eec7021b889" } new_table_name: "default.Table_521a6fff244247279ed3ca15ee49dc50" modify_external_catalogs: true
I20250113 02:51:40.993027 747 tablet_service.cc:1467] Processing CreateTablet for tablet b7247150e20348619fda21ebbdd305c8 (DEFAULT_TABLE table=default.Table_4648c0331016457faa041613f30e8a95 [id=0ce517b3d4664eb4a3360a647ccc150f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:40.985494 892 tablet_service.cc:1467] Processing CreateTablet for tablet 284ca2fe0c6a4b79a3bc6d096a90ccf1 (DEFAULT_TABLE table=default.Table_f830fa2aaadc44bbb73b8eb61545aabc [id=3a485245494241b5a902dc44980c7c8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:40.993557 748 tablet_service.cc:1467] Processing CreateTablet for tablet 71122b6a1c0145b4a674815f3cbf6e92 (DEFAULT_TABLE table=default.Table_2d9099633e344b5886574d99e90fe2b7 [id=60494a64215247ba9dc8aa0bf5d06948]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:40.993984 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 284ca2fe0c6a4b79a3bc6d096a90ccf1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:40.995787 746 tablet_service.cc:1467] Processing CreateTablet for tablet 1c3c3b7cf89e4f22b8a5dd7ba4f8f5ec (DEFAULT_TABLE table=default.Table_921e31e7ce184317873cbfa17f6f32ac [id=11be84de8ed648e18e40d9d55534401c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:40.994238 886 tablet_service.cc:1514] Processing DeleteTablet for tablet 42a8e01bb71f49b09c83a8f5c905f38d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:40 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:40.997543 886 tablet_service.cc:1514] Processing DeleteTablet for tablet 94307d22b7f842ca89844768e4a2a2ab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:40 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:40.997649 886 tablet_service.cc:1514] Processing DeleteTablet for tablet a0c14501ad0449aca28909bc14428dad with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:40 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:41.000087 890 tablet_service.cc:1467] Processing CreateTablet for tablet 4b75150005e045f08148e1437586999d (DEFAULT_TABLE table=default.Table_ff5dca8686a64b858a2ab711d109d4a9 [id=4c43f64ec2a340e1bfc7f5923b3acc2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:41.000411 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b75150005e045f08148e1437586999d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:41.001881 2473 raft_consensus.cc:491] T 4c759831773849f2bc52e484ffa0cf49 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:41.002017 2473 raft_consensus.cc:513] T 4c759831773849f2bc52e484ffa0cf49 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:41.002621 2473 leader_election.cc:290] T 4c759831773849f2bc52e484ffa0cf49 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:41.003605 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_96649abef65740ad8698fabc7e9d4a63"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:41.004542 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_96649abef65740ad8698fabc7e9d4a63 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:41.005322 883 tablet_service.cc:1514] Processing DeleteTablet for tablet a894199f21aa4dba8a03583ea14a9092 with delete_type TABLET_DATA_DELETED (replaced by tablet f6f4df3bde434667a923be19aa39817a) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:41.005630 883 tablet_service.cc:1467] Processing CreateTablet for tablet 3c4224468d8d42d2ba21cdb266a9c8c3 (DEFAULT_TABLE table=default.Table_4d27f00185134500b7a8461f7efe9961 [id=cc3c6b8e215a42ad8ae2f26b4e793e47]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:41.005908 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c4224468d8d42d2ba21cdb266a9c8c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:40.997738 886 tablet_service.cc:1514] Processing DeleteTablet for tablet 6001369444894b48b51a48fb60ddc52a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:40 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:41.008638 886 tablet_service.cc:1467] Processing CreateTablet for tablet 3d40b9c159c44a5cba16cd4d262df0c2 (DEFAULT_TABLE table=default.Table_9175390f220844ad984ccaa8df33cecf [id=1f913bf2d3c841a198ce7be091a3642a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:41.009624 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_82c1ba1d2d8048c7b50167f1eb6ec54a" } new_table_name: "default.Table_df3bf4dfbe3f49879d25c00df782686c" modify_external_catalogs: true
I20250113 02:51:41.012655 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_4bb227dc715148c9862b2d999509533a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:41.013087 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4bb227dc715148c9862b2d999509533a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:41.014240 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_851855dfd1934ca38a5714adabc71199"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:41.014665 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_851855dfd1934ca38a5714adabc71199 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:41.014983 1191 raft_consensus.cc:357] T 0839a35fcab04f3c9d03e256553e01c8 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:41.015157 1191 raft_consensus.cc:383] T 0839a35fcab04f3c9d03e256553e01c8 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:41.015198 1191 raft_consensus.cc:738] T 0839a35fcab04f3c9d03e256553e01c8 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:41.015360 1191 consensus_queue.cc:260] T 0839a35fcab04f3c9d03e256553e01c8 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:41.015806 1191 ts_tablet_manager.cc:1428] T 0839a35fcab04f3c9d03e256553e01c8 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.052s user 0.000s sys 0.001s
I20250113 02:51:41.011394 894 tablet_service.cc:1467] Processing CreateTablet for tablet 4dab7d43f5314715a0021cf7da82c285 (DEFAULT_TABLE table=default.Table_0e6eae4ff7724f6aa85277439a83f3b1 [id=e984030bfc2d450995f2ec90c463670b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:40.975334 1196 rpcz_store.cc:276] 0113 02:51:38.868060 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:38.911409 (+ 43349us) service_pool.cc:225] Handling call
0113 02:51:40.529078 (+1617669us) ts_tablet_manager.cc:1145] Deleting tablet b8b6ffe690fc459db78b2b3bc12671dd
0113 02:51:40.550253 (+ 21175us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:40.839437 (+289184us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:40.904985 (+ 65548us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:40.966531 (+ 61546us) spinlock_profiling.cc:243] Waited 10.8 ms on lock 0x61800000fd18. stack: 00007f0b1ea7cfd6 00007f0b1ea7cd90 00007f0b1da5c48d 000000000096d544 000000000096d50d 000000000096ceda 00007f0b1ea650fd 00007f0b1ea6616e 00007f0b31e99c7d 00007f0b31e15d43 00007f0b2e0b9323 00007f0b2e0eb584 00007f0b2e0cc690 00007f0b2e0cc41d 0000000000a71ecc 00007f0b1eac3b64
0113 02:51:40.975093 (+ 8562us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":24926976,"tablet-delete.queue_time_us":1606029}
I20250113 02:51:41.022290 1196 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37366 (request call id 1098) took 2095 ms. Trace:
I20250113 02:51:41.022382 1196 rpcz_store.cc:276] 0113 02:51:38.927136 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:38.927199 (+ 63us) service_pool.cc:225] Handling call
0113 02:51:41.022139 (+2094940us) ts_tablet_manager.cc:1145] Deleting tablet 17fc320eb01f4ac2b9361aeab5b452fc
0113 02:51:41.022150 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:41.022270 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2094727}
I20250113 02:51:41.022689 1196 tablet_replica.cc:331] T 4b07a985897641c3ac48021c2a4daedb P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:41.022918 1196 raft_consensus.cc:2238] T 4b07a985897641c3ac48021c2a4daedb P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:41.023078 1196 raft_consensus.cc:2267] T 4b07a985897641c3ac48021c2a4daedb P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:41.023654 1196 ts_tablet_manager.cc:1905] T 4b07a985897641c3ac48021c2a4daedb P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:41.023932 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4dab7d43f5314715a0021cf7da82c285. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:41.030004 884 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c623bd0fbbf4922aedda031a88b79eb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:40 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:41.030412 884 tablet_service.cc:1467] Processing CreateTablet for tablet 476c89cc585b4e00947f5d1e8d0229f6 (DEFAULT_TABLE table=default.Table_f7a044df83094686b983b6a11fd518cd [id=bc64fd48c0b14d318a9e2bdf1025cf46]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:41.030771 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 476c89cc585b4e00947f5d1e8d0229f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:41.031438 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3d40b9c159c44a5cba16cd4d262df0c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:41.036909 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 438d1d2300e044f39a15586f335935ba
I20250113 02:51:41.044106 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c3c3b7cf89e4f22b8a5dd7ba4f8f5ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:41.045075 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 71122b6a1c0145b4a674815f3cbf6e92. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:41.021486 899 tablet_service.cc:1467] Processing CreateTablet for tablet 0f091eb311d94956bb398b6c5efce284 (DEFAULT_TABLE table=default.Table_5576eb7eae61401bbb7065ffbd571846 [id=92e0e4b4d89f4bba8d8e1e4841e76108]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:41.041234 896 tablet_service.cc:1467] Processing CreateTablet for tablet 3a89e991ebee4b4f8c639ea95e1d1ea3 (DEFAULT_TABLE table=default.Table_dc9db684670a46229bcca7df696bb04c [id=d7a6380bf81d4946b180bbbad5f870b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:41.065164 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a89e991ebee4b4f8c639ea95e1d1ea3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:41.067077 1196 ts_tablet_manager.cc:1918] T 4b07a985897641c3ac48021c2a4daedb P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:41.070372 895 tablet_service.cc:1467] Processing CreateTablet for tablet 045ff4572f3347fc890b439d90cb0bbe (DEFAULT_TABLE table=default.Table_5a9dd13ac839443f862131a444ab842c [id=29faf5f637cf4e8cbff1d3c502dc01b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:41.070715 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 045ff4572f3347fc890b439d90cb0bbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:41.071004 1196 log.cc:1198] T 4b07a985897641c3ac48021c2a4daedb P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/4b07a985897641c3ac48021c2a4daedb
I20250113 02:51:41.072427 1196 ts_tablet_manager.cc:1939] T 4b07a985897641c3ac48021c2a4daedb P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:41.047834 898 tablet_service.cc:1467] Processing CreateTablet for tablet 8be8227851a740d890e46b3a3fa4a2ce (DEFAULT_TABLE table=default.Table_344e7583f72042fb926d0f2e1aa2ce87 [id=fffdac188fdf437589acc52fa0b0dab7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:41.048497 885 tablet_service.cc:1467] Processing CreateTablet for tablet 1bb7c8c9217143e79659da0650d1bf11 (DEFAULT_TABLE table=default.Table_0e07e7c736e74a51bd66a147c524027f [id=bc63054e26c2493084812e45ebec909e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:41.073603 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8be8227851a740d890e46b3a3fa4a2ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:41.073796 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1bb7c8c9217143e79659da0650d1bf11. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:41.021086 897 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f2b4df7c6a74896ade95a490ea9b54c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:40 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:41.078809 897 tablet_service.cc:1467] Processing CreateTablet for tablet d114f02268794e4fa8c464e0442b8eff (DEFAULT_TABLE table=default.Table_5851d43a71e641058f8d820b078e9aca [id=4695f75c633f4513a6381aa3d293f0cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:41.079185 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d114f02268794e4fa8c464e0442b8eff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:41.059556 882 tablet_service.cc:1467] Processing CreateTablet for tablet 5581f9bacd6b4019be6ebbe5b447e0e5 (DEFAULT_TABLE table=default.Table_f1f79d9ce8f24b5cb56d1ce5082196c6 [id=0d1e9f75d89145d1949af1a60cd866a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:41.081876 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5581f9bacd6b4019be6ebbe5b447e0e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:41.084981 1196 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37366 (request call id 1101) took 2071 ms. Trace:
I20250113 02:51:41.087530 887 tablet_service.cc:1467] Processing CreateTablet for tablet e6f5331ec0044ea7b4134fb2c3d12ebb (DEFAULT_TABLE table=default.Table_7f3e4c1c46cd4df1b77899c2f674068a [id=de907be87311474ea04a275eb44efdc0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:41.090351 1196 rpcz_store.cc:276] 0113 02:51:39.013954 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:39.014014 (+ 60us) service_pool.cc:225] Handling call
0113 02:51:41.022637 (+2008623us) ts_tablet_manager.cc:1145] Deleting tablet 4b07a985897641c3ac48021c2a4daedb
0113 02:51:41.022649 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:41.028431 (+ 5782us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:41.066979 (+ 38548us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:41.084961 (+ 17982us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1987146}
I20250113 02:51:41.087685 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f091eb311d94956bb398b6c5efce284. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:41.092890 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e6f5331ec0044ea7b4134fb2c3d12ebb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:41.094053 753 tablet_service.cc:1467] Processing CreateTablet for tablet d1f22bb035f447568a13abf2b5c70ce3 (DEFAULT_TABLE table=default.Table_fe974dd3772d4afd880bc574c573a649 [id=cc2cd3b6418e4c74b50b42b72536ff5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:41.094167 1196 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37366 (request call id 1111) took 1879 ms. Trace:
I20250113 02:51:41.094450 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d1f22bb035f447568a13abf2b5c70ce3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:41.096434 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b7247150e20348619fda21ebbdd305c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:41.095878 1196 rpcz_store.cc:276] 0113 02:51:39.214985 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:39.260276 (+ 45291us) service_pool.cc:225] Handling call
0113 02:51:41.092417 (+1832141us) ts_tablet_manager.cc:1145] Deleting tablet 960a7078181f4621a5ff4f3eb3050de3
0113 02:51:41.092428 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:41.094150 (+ 1722us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1831896}
I20250113 02:51:41.098121 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 01a455eda0ee488fb3d45a6a09c43833
I20250113 02:51:41.097774 1196 tablet_replica.cc:331] T f129bb3fbbfd492d98822be0a6626e44 P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:41.099344 1196 raft_consensus.cc:2238] T f129bb3fbbfd492d98822be0a6626e44 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:41.100915 751 tablet_service.cc:1467] Processing CreateTablet for tablet 292f2a0651144e569c71d4296be837d3 (DEFAULT_TABLE table=default.Table_6c3417d70cd4422993f46368566b2d51 [id=8ccebc7fe83241ecbca9eb444bcc7bc0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:41.101233 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 292f2a0651144e569c71d4296be837d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:41.101981 1196 raft_consensus.cc:2267] T f129bb3fbbfd492d98822be0a6626e44 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:41.104557 742 tablet_service.cc:1467] Processing CreateTablet for tablet ccc9ae0461cd40a38cb3a90412df7e82 (DEFAULT_TABLE table=default.Table_fa2ea67525994579bd81a459396f4281 [id=cfce13a8e9e1420dadc977703b49e107]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:41.104826 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ccc9ae0461cd40a38cb3a90412df7e82. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:41.104911 754 tablet_service.cc:1467] Processing CreateTablet for tablet 284ca2fe0c6a4b79a3bc6d096a90ccf1 (DEFAULT_TABLE table=default.Table_f830fa2aaadc44bbb73b8eb61545aabc [id=3a485245494241b5a902dc44980c7c8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:41.105180 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 284ca2fe0c6a4b79a3bc6d096a90ccf1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:41.109076 736 tablet_service.cc:1467] Processing CreateTablet for tablet 80d42e6aea134575b3bd95dee82e5416 (DEFAULT_TABLE table=default.Table_8eeeda2b6ca04de78b01bb716651d49c [id=9360dcb5f4b64ec68a75f1ae3b43894c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:41.109827 1191 tablet_bootstrap.cc:492] T 16a132384b0e4264a73d09a3ea133893 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:41.111706 1191 tablet_bootstrap.cc:654] T 16a132384b0e4264a73d09a3ea133893 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:41.112438 739 tablet_service.cc:1514] Processing DeleteTablet for tablet 42a8e01bb71f49b09c83a8f5c905f38d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:40 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:41.112692 739 tablet_service.cc:1514] Processing DeleteTablet for tablet 94307d22b7f842ca89844768e4a2a2ab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:40 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:41.112890 739 tablet_service.cc:1514] Processing DeleteTablet for tablet a0c14501ad0449aca28909bc14428dad with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:40 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:41.113098 739 tablet_service.cc:1514] Processing DeleteTablet for tablet 6001369444894b48b51a48fb60ddc52a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:40 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:41.113284 739 tablet_service.cc:1514] Processing DeleteTablet for tablet a894199f21aa4dba8a03583ea14a9092 with delete_type TABLET_DATA_DELETED (replaced by tablet f6f4df3bde434667a923be19aa39817a) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:41.113673 739 tablet_service.cc:1467] Processing CreateTablet for tablet 4b75150005e045f08148e1437586999d (DEFAULT_TABLE table=default.Table_ff5dca8686a64b858a2ab711d109d4a9 [id=4c43f64ec2a340e1bfc7f5923b3acc2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:41.114131 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b75150005e045f08148e1437586999d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:41.114936 1196 ts_tablet_manager.cc:1905] T f129bb3fbbfd492d98822be0a6626e44 P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:41.116751 736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 80d42e6aea134575b3bd95dee82e5416. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:41.116868 752 tablet_service.cc:1467] Processing CreateTablet for tablet 3c4224468d8d42d2ba21cdb266a9c8c3 (DEFAULT_TABLE table=default.Table_4d27f00185134500b7a8461f7efe9961 [id=cc3c6b8e215a42ad8ae2f26b4e793e47]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:41.117219 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c4224468d8d42d2ba21cdb266a9c8c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:41.118405 880 tablet_service.cc:1467] Processing CreateTablet for tablet 5979002528af42659ccd8e419cd8ccae (DEFAULT_TABLE table=default.Table_e076d93131bb4c78bbc1548986736de4 [id=6fbe4ba8f7bf4c36b331df31efdb8eef]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:41.119141 744 tablet_service.cc:1467] Processing CreateTablet for tablet 3d40b9c159c44a5cba16cd4d262df0c2 (DEFAULT_TABLE table=default.Table_9175390f220844ad984ccaa8df33cecf [id=1f913bf2d3c841a198ce7be091a3642a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:41.119514 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3d40b9c159c44a5cba16cd4d262df0c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:41.121433 745 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f2b4df7c6a74896ade95a490ea9b54c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:40 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:41.121758 745 tablet_service.cc:1467] Processing CreateTablet for tablet 4dab7d43f5314715a0021cf7da82c285 (DEFAULT_TABLE table=default.Table_0e6eae4ff7724f6aa85277439a83f3b1 [id=e984030bfc2d450995f2ec90c463670b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:41.122138 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4dab7d43f5314715a0021cf7da82c285. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:41.122373 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c623bd0fbbf4922aedda031a88b79eb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:40 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:41.122671 749 tablet_service.cc:1467] Processing CreateTablet for tablet 0f091eb311d94956bb398b6c5efce284 (DEFAULT_TABLE table=default.Table_5576eb7eae61401bbb7065ffbd571846 [id=92e0e4b4d89f4bba8d8e1e4841e76108]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:41.122959 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f091eb311d94956bb398b6c5efce284. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:41.123135 885 tablet_service.cc:1467] Processing CreateTablet for tablet 366e5c5ee708458ba443dd69d65c3988 (DEFAULT_TABLE table=default.Table_40c1445967024038bf8a25c42bf346ad [id=ea4fc227dfb145a2b4a660062410e193]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:41.123466 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 366e5c5ee708458ba443dd69d65c3988. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:41.124378 891 tablet_service.cc:1467] Processing CreateTablet for tablet 978601b734ea4c44a210347a263e163f (DEFAULT_TABLE table=default.Table_5625ac97f9f84d3e8379eb9536bf94c5 [id=a3c26f79c92e4ce3a8f6a3f7d85b50a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:41.125052 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 978601b734ea4c44a210347a263e163f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:41.126894 737 tablet_service.cc:1467] Processing CreateTablet for tablet 476c89cc585b4e00947f5d1e8d0229f6 (DEFAULT_TABLE table=default.Table_f7a044df83094686b983b6a11fd518cd [id=bc64fd48c0b14d318a9e2bdf1025cf46]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:41.127238 737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 476c89cc585b4e00947f5d1e8d0229f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:41.127385 755 tablet_service.cc:1467] Processing CreateTablet for tablet 8be8227851a740d890e46b3a3fa4a2ce (DEFAULT_TABLE table=default.Table_344e7583f72042fb926d0f2e1aa2ce87 [id=fffdac188fdf437589acc52fa0b0dab7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:41.127655 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8be8227851a740d890e46b3a3fa4a2ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:41.128386 740 tablet_service.cc:1467] Processing CreateTablet for tablet 5581f9bacd6b4019be6ebbe5b447e0e5 (DEFAULT_TABLE table=default.Table_f1f79d9ce8f24b5cb56d1ce5082196c6 [id=0d1e9f75d89145d1949af1a60cd866a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:41.128660 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5581f9bacd6b4019be6ebbe5b447e0e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:41.132882 738 tablet_service.cc:1467] Processing CreateTablet for tablet 82865847f4bb4937a63499bd834f683d (DEFAULT_TABLE table=default.Table_0c7d032c4488498d9796defce01fc237 [id=7188d7f2f1dd44179c48efbc97f0a875]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:41.133214 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82865847f4bb4937a63499bd834f683d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:41.135700 737 tablet_service.cc:1467] Processing CreateTablet for tablet 1bb7c8c9217143e79659da0650d1bf11 (DEFAULT_TABLE table=default.Table_0e07e7c736e74a51bd66a147c524027f [id=bc63054e26c2493084812e45ebec909e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:41.136978 737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1bb7c8c9217143e79659da0650d1bf11. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:41.139840 882 tablet_service.cc:1514] Processing DeleteTablet for tablet aef2ffd67def48eea501a98fbc9dc43b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:40 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:41.142707 882 tablet_service.cc:1467] Processing CreateTablet for tablet c6d1dc31a8af4086ad069fba6fd22d26 (DEFAULT_TABLE table=default.Table_8d0905ada0b74b8fba4964abaf95a2b3 [id=ac3e10b97c9e4b46ba6c969d0c7bdc41]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:41.147264 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c6d1dc31a8af4086ad069fba6fd22d26. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:41.165624 2449 raft_consensus.cc:491] T 999fecb3fc65466cad21cc1ff5ef2d44 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:41.165762 2449 raft_consensus.cc:513] T 999fecb3fc65466cad21cc1ff5ef2d44 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:41.165831 888 tablet_service.cc:1467] Processing CreateTablet for tablet 17f02be32ddb44c089266fd9f0378f77 (DEFAULT_TABLE table=default.Table_a7b637f0a6a244fdb2172e2015816fce [id=6666a614063942c99a78b95d35add037]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:41.166167 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 17f02be32ddb44c089266fd9f0378f77. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:41.168246 751 tablet_service.cc:1467] Processing CreateTablet for tablet 3a89e991ebee4b4f8c639ea95e1d1ea3 (DEFAULT_TABLE table=default.Table_dc9db684670a46229bcca7df696bb04c [id=d7a6380bf81d4946b180bbbad5f870b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:41.168727 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a89e991ebee4b4f8c639ea95e1d1ea3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:41.172878 448 catalog_manager.cc:6071] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: default.Table_b95a9b1d12284c9aac701a8ef9c93482 [id=843aabcff7074acd98e5801b21282faa] alter complete (version 1)
I20250113 02:51:41.183771 899 tablet_service.cc:1467] Processing CreateTablet for tablet 7d90e1689b164943b09f6cda5a6ae5cb (DEFAULT_TABLE table=default.Table_6a0195b054a043269e133f6ed8dd5979 [id=4b2bd999368c4b628c70cb3ee8c2b1f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:41.148710 741 tablet_service.cc:1467] Processing CreateTablet for tablet d114f02268794e4fa8c464e0442b8eff (DEFAULT_TABLE table=default.Table_5851d43a71e641058f8d820b078e9aca [id=4695f75c633f4513a6381aa3d293f0cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:41.184131 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d90e1689b164943b09f6cda5a6ae5cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:41.184284 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d114f02268794e4fa8c464e0442b8eff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:41.183240 736 tablet_service.cc:1467] Processing CreateTablet for tablet e7da56c011094384903a3d9a5d9d6969 (DEFAULT_TABLE table=default.Table_03ffbd822433478e8576aa5297fc07ae [id=fe8b587daf714f5c9cffa75f9c3ddb67]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:41.184582 736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e7da56c011094384903a3d9a5d9d6969. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:41.184890 883 tablet_service.cc:1467] Processing CreateTablet for tablet e7da56c011094384903a3d9a5d9d6969 (DEFAULT_TABLE table=default.Table_03ffbd822433478e8576aa5297fc07ae [id=fe8b587daf714f5c9cffa75f9c3ddb67]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:41.185155 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e7da56c011094384903a3d9a5d9d6969. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:41.187640 747 tablet_service.cc:1514] Processing DeleteTablet for tablet 438d1d2300e044f39a15586f335935ba with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:40 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:41.188916 2449 leader_election.cc:290] T 999fecb3fc65466cad21cc1ff5ef2d44 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
W20250113 02:51:41.189576 851 leader_election.cc:343] T 999fecb3fc65466cad21cc1ff5ef2d44 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Not found: Tablet not found: 999fecb3fc65466cad21cc1ff5ef2d44
I20250113 02:51:41.153378 748 tablet_service.cc:1467] Processing CreateTablet for tablet 5979002528af42659ccd8e419cd8ccae (DEFAULT_TABLE table=default.Table_e076d93131bb4c78bbc1548986736de4 [id=6fbe4ba8f7bf4c36b331df31efdb8eef]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:41.192365 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5979002528af42659ccd8e419cd8ccae. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:41.168960 746 tablet_service.cc:1514] Processing DeleteTablet for tablet aef2ffd67def48eea501a98fbc9dc43b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:40 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:41.151499 742 tablet_service.cc:1467] Processing CreateTablet for tablet e6f5331ec0044ea7b4134fb2c3d12ebb (DEFAULT_TABLE table=default.Table_7f3e4c1c46cd4df1b77899c2f674068a [id=de907be87311474ea04a275eb44efdc0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:41.178412 754 tablet_service.cc:1467] Processing CreateTablet for tablet 17f02be32ddb44c089266fd9f0378f77 (DEFAULT_TABLE table=default.Table_a7b637f0a6a244fdb2172e2015816fce [id=6666a614063942c99a78b95d35add037]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:41.143872 744 tablet_service.cc:1467] Processing CreateTablet for tablet 045ff4572f3347fc890b439d90cb0bbe (DEFAULT_TABLE table=default.Table_5a9dd13ac839443f862131a444ab842c [id=29faf5f637cf4e8cbff1d3c502dc01b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:41.196527 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e6f5331ec0044ea7b4134fb2c3d12ebb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:41.196775 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 17f02be32ddb44c089266fd9f0378f77. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:41.197038 889 consensus_meta.cc:287] T 3163078bf56c42b2bd6f6b91885c6400 P df69f48656b44bb890f76587ee69cc0f: Time spent flushing consensus metadata: real 0.543s user 0.017s sys 0.002s
I20250113 02:51:41.197297 2481 tablet.cc:1721] T 0b119704bc774ca9b4155c9619d4765b P 1186fa869eb14812832fa6562531f196: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250113 02:51:40.952656 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 978601b734ea4c44a210347a263e163f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:41.197609 889 tablet_service.cc:1467] Processing CreateTablet for tablet 82865847f4bb4937a63499bd834f683d (DEFAULT_TABLE table=default.Table_0c7d032c4488498d9796defce01fc237 [id=7188d7f2f1dd44179c48efbc97f0a875]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:40.959775 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet aef2ffd67def48eea501a98fbc9dc43b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:40 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:40.960945 1024 tablet_service.cc:1467] Processing CreateTablet for tablet c6d1dc31a8af4086ad069fba6fd22d26 (DEFAULT_TABLE table=default.Table_8d0905ada0b74b8fba4964abaf95a2b3 [id=ac3e10b97c9e4b46ba6c969d0c7bdc41]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:41.198451 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c6d1dc31a8af4086ad069fba6fd22d26. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:40.962388 1192 tablet_bootstrap.cc:492] T 82505f1389184d0ca12ec0e2d6084d22 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:41.204111 1192 ts_tablet_manager.cc:1397] T 82505f1389184d0ca12ec0e2d6084d22 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.322s user 0.002s sys 0.001s
I20250113 02:51:40.950812 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1147) took 2119 ms. Trace:
I20250113 02:51:41.204807 1192 raft_consensus.cc:357] T 82505f1389184d0ca12ec0e2d6084d22 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:41.204996 1192 raft_consensus.cc:383] T 82505f1389184d0ca12ec0e2d6084d22 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:41.205046 1192 raft_consensus.cc:738] T 82505f1389184d0ca12ec0e2d6084d22 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:41.204865 1198 rpcz_store.cc:276] 0113 02:51:38.831852 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:38.831908 (+ 56us) service_pool.cc:225] Handling call
0113 02:51:40.950711 (+2118803us) ts_tablet_manager.cc:1145] Deleting tablet 9fad58cccf1744b0a476523e0af0036e
0113 02:51:40.950720 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:40.950795 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2118596}
I20250113 02:51:41.205227 1192 consensus_queue.cc:260] T 82505f1389184d0ca12ec0e2d6084d22 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
W20250113 02:51:41.205569 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a58588b1befe48a8a1418b788a227519 consensus state has no leader
I20250113 02:51:41.205713 1192 ts_tablet_manager.cc:1428] T 82505f1389184d0ca12ec0e2d6084d22 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250113 02:51:41.205957 1192 tablet_bootstrap.cc:492] T 42dbd37e20fa47318bc6497423dbe565 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
W20250113 02:51:41.206269 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 9fad58cccf1744b0a476523e0af0036e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9fad58cccf1744b0a476523e0af0036e already in progress: creating tablet
I20250113 02:51:41.207322 1192 tablet_bootstrap.cc:654] T 42dbd37e20fa47318bc6497423dbe565 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:40.962591 1041 tablet_service.cc:1467] Processing CreateTablet for tablet 17f02be32ddb44c089266fd9f0378f77 (DEFAULT_TABLE table=default.Table_a7b637f0a6a244fdb2172e2015816fce [id=6666a614063942c99a78b95d35add037]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:41.208391 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 17f02be32ddb44c089266fd9f0378f77. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:40.966167 1031 tablet_service.cc:1467] Processing CreateTablet for tablet e7da56c011094384903a3d9a5d9d6969 (DEFAULT_TABLE table=default.Table_03ffbd822433478e8576aa5297fc07ae [id=fe8b587daf714f5c9cffa75f9c3ddb67]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:40.967113 1025 tablet_service.cc:1467] Processing CreateTablet for tablet 82865847f4bb4937a63499bd834f683d (DEFAULT_TABLE table=default.Table_0c7d032c4488498d9796defce01fc237 [id=7188d7f2f1dd44179c48efbc97f0a875]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:41.003294 1058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c759831773849f2bc52e484ffa0cf49" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:41.038657 1036 tablet_service.cc:1514] Processing DeleteTablet for tablet 438d1d2300e044f39a15586f335935ba with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:40 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:41.099213 1039 tablet_service.cc:1514] Processing DeleteTablet for tablet 01a455eda0ee488fb3d45a6a09c43833 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:40 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:40.964116 1033 tablet_service.cc:1467] Processing CreateTablet for tablet 7d90e1689b164943b09f6cda5a6ae5cb (DEFAULT_TABLE table=default.Table_6a0195b054a043269e133f6ed8dd5979 [id=4b2bd999368c4b628c70cb3ee8c2b1f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:41.212985 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e7da56c011094384903a3d9a5d9d6969. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:41.216581 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d90e1689b164943b09f6cda5a6ae5cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:41.220222 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82865847f4bb4937a63499bd834f683d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:41.220319 1194 tablet_bootstrap.cc:492] T eebad045e7bb43fa9ca9c9af8b3b7ea5 P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:41.220443 1194 ts_tablet_manager.cc:1397] T eebad045e7bb43fa9ca9c9af8b3b7ea5 P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.664s user 0.006s sys 0.000s
I20250113 02:51:41.229662 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 366e5c5ee708458ba443dd69d65c3988. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:41.229746 1192 tablet_bootstrap.cc:492] T 42dbd37e20fa47318bc6497423dbe565 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:41.230067 1192 ts_tablet_manager.cc:1397] T 42dbd37e20fa47318bc6497423dbe565 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.024s user 0.001s sys 0.001s
I20250113 02:51:41.230805 1192 raft_consensus.cc:357] T 42dbd37e20fa47318bc6497423dbe565 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:41.231094 1192 raft_consensus.cc:383] T 42dbd37e20fa47318bc6497423dbe565 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:41.231199 1192 raft_consensus.cc:738] T 42dbd37e20fa47318bc6497423dbe565 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:41.231457 1192 consensus_queue.cc:260] T 42dbd37e20fa47318bc6497423dbe565 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:41.232084 1192 ts_tablet_manager.cc:1428] T 42dbd37e20fa47318bc6497423dbe565 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250113 02:51:41.232391 1192 tablet_bootstrap.cc:492] T 153b8f90b5a847b88944f683d306f579 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:41.234556 1192 tablet_bootstrap.cc:654] T 153b8f90b5a847b88944f683d306f579 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:41.234786 1041 tablet_service.cc:1514] Processing DeleteTablet for tablet 82505f1389184d0ca12ec0e2d6084d22 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:27 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:41.241665 1196 ts_tablet_manager.cc:1918] T f129bb3fbbfd492d98822be0a6626e44 P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:41.243435 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5979002528af42659ccd8e419cd8ccae. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:41.244325 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82865847f4bb4937a63499bd834f683d. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:41.244971 852 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.582s user 0.015s sys 0.006s
I20250113 02:51:41.247813 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1149) took 2413 ms. Trace:
I20250113 02:51:41.248538 1198 rpcz_store.cc:276] 0113 02:51:38.834206 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:38.834287 (+ 81us) service_pool.cc:225] Handling call
0113 02:51:41.247655 (+2413368us) ts_tablet_manager.cc:1145] Deleting tablet a6d0f9a0a7104b8fb3d22aac976b7d99
0113 02:51:41.247670 (+ 15us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:41.247788 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2413091}
W20250113 02:51:41.252480 852 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.590s user 0.015s sys 0.006s
I20250113 02:51:41.253309 886 tablet_service.cc:1514] Processing DeleteTablet for tablet 01a455eda0ee488fb3d45a6a09c43833 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:40 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:41.247888 1041 tablet_service.cc:1514] Processing DeleteTablet for tablet 42dbd37e20fa47318bc6497423dbe565 with delete_type TABLET_DATA_DELETED (replaced by tablet 9174df876fe64adc805fba0a36fb4d38) from {username='slave'} at 127.0.0.1:60230
W20250113 02:51:41.253659 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet a6d0f9a0a7104b8fb3d22aac976b7d99 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a6d0f9a0a7104b8fb3d22aac976b7d99 already in progress: creating tablet
I20250113 02:51:41.253456 892 tablet_service.cc:1514] Processing DeleteTablet for tablet 438d1d2300e044f39a15586f335935ba with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:40 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:41.254427 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1150) took 2419 ms. Trace:
I20250113 02:51:41.178092 750 tablet_service.cc:1467] Processing CreateTablet for tablet c6d1dc31a8af4086ad069fba6fd22d26 (DEFAULT_TABLE table=default.Table_8d0905ada0b74b8fba4964abaf95a2b3 [id=ac3e10b97c9e4b46ba6c969d0c7bdc41]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:41.155004 743 tablet_service.cc:1467] Processing CreateTablet for tablet 366e5c5ee708458ba443dd69d65c3988 (DEFAULT_TABLE table=default.Table_40c1445967024038bf8a25c42bf346ad [id=ea4fc227dfb145a2b4a660062410e193]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:41.200062 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 045ff4572f3347fc890b439d90cb0bbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:41.255501 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 366e5c5ee708458ba443dd69d65c3988. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:41.180433 740 tablet_service.cc:1467] Processing CreateTablet for tablet 7d90e1689b164943b09f6cda5a6ae5cb (DEFAULT_TABLE table=default.Table_6a0195b054a043269e133f6ed8dd5979 [id=4b2bd999368c4b628c70cb3ee8c2b1f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:41.258502 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d90e1689b164943b09f6cda5a6ae5cb. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:41.259877 852 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.3:42587, user_credentials={real_user=slave}} blocked reactor thread for 597363us
W20250113 02:51:41.212878 709 leader_election.cc:343] T 4c759831773849f2bc52e484ffa0cf49 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
W20250113 02:51:41.263499 709 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.3:42587, user_credentials={real_user=slave}} blocked reactor thread for 50633.6us
I20250113 02:51:41.262907 1194 raft_consensus.cc:357] T eebad045e7bb43fa9ca9c9af8b3b7ea5 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:41.264864 1194 raft_consensus.cc:383] T eebad045e7bb43fa9ca9c9af8b3b7ea5 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:41.264925 1194 raft_consensus.cc:738] T eebad045e7bb43fa9ca9c9af8b3b7ea5 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:41.266096 1198 rpcz_store.cc:276] 0113 02:51:38.835095 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:38.835146 (+ 51us) service_pool.cc:225] Handling call
0113 02:51:41.254273 (+2419127us) ts_tablet_manager.cc:1145] Deleting tablet f4acbe59400247678d2441b1b844d976
0113 02:51:41.254288 (+ 15us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:41.254404 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2418862}
I20250113 02:51:41.266536 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c6d1dc31a8af4086ad069fba6fd22d26. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:41.267484 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet f4acbe59400247678d2441b1b844d976 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f4acbe59400247678d2441b1b844d976 already in progress: creating tablet
I20250113 02:51:41.272291 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1153) took 2390 ms. Trace:
I20250113 02:51:41.273272 752 tablet_service.cc:1467] Processing CreateTablet for tablet 978601b734ea4c44a210347a263e163f (DEFAULT_TABLE table=default.Table_5625ac97f9f84d3e8379eb9536bf94c5 [id=a3c26f79c92e4ce3a8f6a3f7d85b50a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:41.274297 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 978601b734ea4c44a210347a263e163f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:41.274430 1198 rpcz_store.cc:276] 0113 02:51:38.882212 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:38.885300 (+ 3088us) service_pool.cc:225] Handling call
0113 02:51:41.272154 (+2386854us) ts_tablet_manager.cc:1145] Deleting tablet 50c4d4783e6d4a55bd3bd355613e1ea0
0113 02:51:41.272166 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:41.272271 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2386651}
W20250113 02:51:41.277376 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 50c4d4783e6d4a55bd3bd355613e1ea0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 50c4d4783e6d4a55bd3bd355613e1ea0 already in progress: creating tablet
I20250113 02:51:41.280611 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1157) took 2355 ms. Trace:
I20250113 02:51:41.287082 1198 rpcz_store.cc:276] 0113 02:51:38.925508 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:38.925550 (+ 42us) service_pool.cc:225] Handling call
0113 02:51:41.278590 (+2353040us) ts_tablet_manager.cc:1145] Deleting tablet 17fc320eb01f4ac2b9361aeab5b452fc
0113 02:51:41.278602 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:41.280589 (+ 1987us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2352858}
I20250113 02:51:41.288327 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet b8b6ffe690fc459db78b2b3bc12671dd (table default.Table_b77c072e465c474db9349f1a3b31c323 [id=d7fe00879d3143c1bf0ff603f9284750]) successfully deleted
I20250113 02:51:41.291579 1198 tablet_replica.cc:331] T 30ca8a716bfc4ab789b216e17567785e P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:41.296705 1198 raft_consensus.cc:2238] T 30ca8a716bfc4ab789b216e17567785e P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:41.296736 1194 consensus_queue.cc:260] T eebad045e7bb43fa9ca9c9af8b3b7ea5 P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
W20250113 02:51:41.294526 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 17fc320eb01f4ac2b9361aeab5b452fc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 17fc320eb01f4ac2b9361aeab5b452fc already in progress: creating tablet
I20250113 02:51:41.297374 1194 ts_tablet_manager.cc:1428] T eebad045e7bb43fa9ca9c9af8b3b7ea5 P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.077s user 0.001s sys 0.001s
I20250113 02:51:41.297669 1194 tablet_bootstrap.cc:492] T 0916b41c171a419791797559fbc2807d P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:41.191623 745 tablet_service.cc:1514] Processing DeleteTablet for tablet 01a455eda0ee488fb3d45a6a09c43833 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:40 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:41.298072 1198 raft_consensus.cc:2267] T 30ca8a716bfc4ab789b216e17567785e P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus is shut down!
W20250113 02:51:41.297683 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 17fc320eb01f4ac2b9361aeab5b452fc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 17fc320eb01f4ac2b9361aeab5b452fc already in progress: creating tablet
I20250113 02:51:41.299132 1194 tablet_bootstrap.cc:654] T 0916b41c171a419791797559fbc2807d P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:41.299719 1198 ts_tablet_manager.cc:1905] T 30ca8a716bfc4ab789b216e17567785e P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:41.303534 2479 tablet.cc:1721] T 6e1284c8aaba44b5810b0d06d36757eb P 4216a7f853d642b7bc3e5cff458f3aca: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250113 02:51:41.308745 1033 tablet_service.cc:1467] Processing CreateTablet for tablet 711544894da147458c6a342883d7b83b (DEFAULT_TABLE table=default.Table_bf4806a5a68e4fcb98205b6fed9d0b5e [id=8bfd0e28c2ac404fb624b628675e5d84]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:41.309651 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 711544894da147458c6a342883d7b83b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:41.310920 1041 tablet_service.cc:1467] Processing CreateTablet for tablet 6ba5a61a6aae4a23812503cc858f8957 (DEFAULT_TABLE table=default.Table_0e134d21b63b463ea086ed3ac6921486 [id=d15996f669224fe1a2d856b6c687931e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:41.311308 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ba5a61a6aae4a23812503cc858f8957. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:41.312512 440 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 0269663b35f148c4a2a93cfa6132d0f8 (table default.Table_634c9d690a0d42a1ac0653a55e6b98d0 [id=47fa61abd6f14b16b52a694c72fb6eb2], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 0269663b35f148c4a2a93cfa6132d0f8 consensus state has no leader
I20250113 02:51:41.320806 1033 tablet_service.cc:1467] Processing CreateTablet for tablet 09a311cab8f04f01831bd18f95995b32 (DEFAULT_TABLE table=default.Table_c7dd7ce595fa467a916cd31cf34526c8 [id=30c66da7133c45f69df20177100d4b22]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:41.321188 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 09a311cab8f04f01831bd18f95995b32. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:41.321698 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet 4b07a985897641c3ac48021c2a4daedb (table default.Table_066f136aea8141fa91862f7285fbaac2 [id=fa29f9bbea034fafad86ad86aa5775eb]) successfully deleted
W20250113 02:51:41.323360 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 960a7078181f4621a5ff4f3eb3050de3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 960a7078181f4621a5ff4f3eb3050de3 already in progress: creating tablet
I20250113 02:51:41.325716 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_5625ac97f9f84d3e8379eb9536bf94c5" } new_table_name: "default.Table_ff383fd2575643ee95b0b9d10541cf69" modify_external_catalogs: true
I20250113 02:51:41.333065 2479 tablet.cc:1721] T 06f36d09bbb1443bafce906907795404 P 4216a7f853d642b7bc3e5cff458f3aca: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
W20250113 02:51:41.333818 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 978601b734ea4c44a210347a263e163f consensus state has no leader
I20250113 02:51:41.336465 449 catalog_manager.cc:6071] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: default.Table_bfc5031f68f64006aceefe2ee986223a [id=aa4679fc0d7148e3bef21342b3a0ea22] alter complete (version 2)
I20250113 02:51:41.343014 2473 raft_consensus.cc:491] T ca75e7cae4f6423191a17bb5a8f150a9 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:41.343205 2473 raft_consensus.cc:513] T ca75e7cae4f6423191a17bb5a8f150a9 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:41.348867 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf7b8bd6801743f49e9b9473747447c5" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:41.350108 1032 tablet_service.cc:1467] Processing CreateTablet for tablet 192a226332e64766a0e367c6fc1c1673 (DEFAULT_TABLE table=default.Table_64fa660da06b4c7cb2c76054b3a4ceba [id=7360924dbe7547e1b6e7a2f6f7404642]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:41.353067 2473 leader_election.cc:290] T ca75e7cae4f6423191a17bb5a8f150a9 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:41.353799 1058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca75e7cae4f6423191a17bb5a8f150a9" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
W20250113 02:51:41.354780 709 leader_election.cc:343] T ca75e7cae4f6423191a17bb5a8f150a9 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:41.357262 449 catalog_manager.cc:6071] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: default.Table_521a6fff244247279ed3ca15ee49dc50 [id=423021f304964246bd29e4daaf470154] alter complete (version 5)
I20250113 02:51:41.359346 1025 tablet_service.cc:1467] Processing CreateTablet for tablet b98da6f197764465a1e10aaadcddf3a4 (DEFAULT_TABLE table=default.Table_d5a79bb9160448269ae65f45fcd592a9 [id=c25a4dff6adb47c9a6445dc72a0d8546]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:41.359805 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b98da6f197764465a1e10aaadcddf3a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:41.356675 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfa963a6df094ce0b9c54408f75ca6b2" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:41.363541 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c759831773849f2bc52e484ffa0cf49" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:41.373147 1025 tablet_service.cc:1467] Processing CreateTablet for tablet 0e3231aafc4e49aabaef5f34db272659 (DEFAULT_TABLE table=default.Table_255be37d0d3e4cfca070e8137a82f90b [id=3c9c08212c044e63b9b0263ca43270ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:41.373528 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e3231aafc4e49aabaef5f34db272659. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:41.370244 745 tablet_service.cc:1467] Processing CreateTablet for tablet 09a311cab8f04f01831bd18f95995b32 (DEFAULT_TABLE table=default.Table_c7dd7ce595fa467a916cd31cf34526c8 [id=30c66da7133c45f69df20177100d4b22]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:41.375716 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 09a311cab8f04f01831bd18f95995b32. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:41.378731 739 tablet_service.cc:1467] Processing CreateTablet for tablet 192a226332e64766a0e367c6fc1c1673 (DEFAULT_TABLE table=default.Table_64fa660da06b4c7cb2c76054b3a4ceba [id=7360924dbe7547e1b6e7a2f6f7404642]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:41.381078 1031 tablet_service.cc:1467] Processing CreateTablet for tablet ca6f77f97a2343f49dccb54361fb0cea (DEFAULT_TABLE table=default.Table_dcdc0cc411724eba84ac9af713fb17e3 [id=912e959477af4c90a1e1773c7edd69e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:41.381453 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca6f77f97a2343f49dccb54361fb0cea. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:41.383775 2473 raft_consensus.cc:491] T dc680b5680b84abbab219296bc7ee707 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:41.383896 2473 raft_consensus.cc:513] T dc680b5680b84abbab219296bc7ee707 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:41.384719 2473 leader_election.cc:290] T dc680b5680b84abbab219296bc7ee707 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:41.385546 1058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc680b5680b84abbab219296bc7ee707" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
W20250113 02:51:41.386124 709 leader_election.cc:343] T dc680b5680b84abbab219296bc7ee707 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:41.387189 1194 tablet_bootstrap.cc:492] T 0916b41c171a419791797559fbc2807d P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:41.387362 1194 ts_tablet_manager.cc:1397] T 0916b41c171a419791797559fbc2807d P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.090s user 0.007s sys 0.007s
I20250113 02:51:41.388139 740 tablet_service.cc:1467] Processing CreateTablet for tablet 0e3231aafc4e49aabaef5f34db272659 (DEFAULT_TABLE table=default.Table_255be37d0d3e4cfca070e8137a82f90b [id=3c9c08212c044e63b9b0263ca43270ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:41.388168 1194 raft_consensus.cc:357] T 0916b41c171a419791797559fbc2807d P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:41.388345 1194 raft_consensus.cc:383] T 0916b41c171a419791797559fbc2807d P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:41.388388 1194 raft_consensus.cc:738] T 0916b41c171a419791797559fbc2807d P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:41.388562 1194 consensus_queue.cc:260] T 0916b41c171a419791797559fbc2807d P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:41.388705 744 tablet_service.cc:1467] Processing CreateTablet for tablet ca6f77f97a2343f49dccb54361fb0cea (DEFAULT_TABLE table=default.Table_dcdc0cc411724eba84ac9af713fb17e3 [id=912e959477af4c90a1e1773c7edd69e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:41.389127 1194 ts_tablet_manager.cc:1428] T 0916b41c171a419791797559fbc2807d P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250113 02:51:41.389226 2480 raft_consensus.cc:491] T b33025a462e74f2aa26cc3779f36615b P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:41.389382 1194 tablet_bootstrap.cc:492] T 7527e6e0b65e4a43b7f48899a34cdf7f P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:41.389366 2480 raft_consensus.cc:513] T b33025a462e74f2aa26cc3779f36615b P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:41.390077 2480 leader_election.cc:290] T b33025a462e74f2aa26cc3779f36615b P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:41.391073 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca75e7cae4f6423191a17bb5a8f150a9" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:41.391119 1031 tablet_service.cc:1467] Processing CreateTablet for tablet 99843ce7f1bf41da9b7cb5bd7ac2e875 (DEFAULT_TABLE table=default.Table_ed460a21a2184db893d2e76e67cdb974 [id=fb3d90554f7943099cd541f610084cbf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:41.391420 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 99843ce7f1bf41da9b7cb5bd7ac2e875. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:41.391748 711 leader_election.cc:343] T 0752fde3a30a4947a2a688c56ceadab5 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Not found: Tablet not found: 0752fde3a30a4947a2a688c56ceadab5
I20250113 02:51:41.391875 711 leader_election.cc:304] T 0752fde3a30a4947a2a688c56ceadab5 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:41.392812 2473 raft_consensus.cc:2743] T 0752fde3a30a4947a2a688c56ceadab5 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250113 02:51:41.393151 2498 tablet.cc:1721] T 6e1284c8aaba44b5810b0d06d36757eb P df69f48656b44bb890f76587ee69cc0f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250113 02:51:41.394920 738 tablet_service.cc:1467] Processing CreateTablet for tablet b98da6f197764465a1e10aaadcddf3a4 (DEFAULT_TABLE table=default.Table_d5a79bb9160448269ae65f45fcd592a9 [id=c25a4dff6adb47c9a6445dc72a0d8546]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:41.395956 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b33025a462e74f2aa26cc3779f36615b" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:41.396517 998 leader_election.cc:343] T b33025a462e74f2aa26cc3779f36615b P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:41.396811 1039 tablet_service.cc:1467] Processing CreateTablet for tablet 12707a38496c464a859fa294cf6bbe36 (DEFAULT_TABLE table=default.Table_3e43331bf36845f9aa983823e4853d04 [id=6e23db586a4740a38379b905fdf40c13]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:41.397145 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 12707a38496c464a859fa294cf6bbe36. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:41.398648 1194 tablet_bootstrap.cc:654] T 7527e6e0b65e4a43b7f48899a34cdf7f P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:41.401196 1192 tablet_bootstrap.cc:492] T 153b8f90b5a847b88944f683d306f579 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:41.401422 1192 ts_tablet_manager.cc:1397] T 153b8f90b5a847b88944f683d306f579 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.169s user 0.012s sys 0.010s
I20250113 02:51:41.402355 1192 raft_consensus.cc:357] T 153b8f90b5a847b88944f683d306f579 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:41.402635 1192 raft_consensus.cc:383] T 153b8f90b5a847b88944f683d306f579 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:41.402737 1192 raft_consensus.cc:738] T 153b8f90b5a847b88944f683d306f579 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:41.402853 2479 tablet.cc:1716] T 6e1284c8aaba44b5810b0d06d36757eb P 4216a7f853d642b7bc3e5cff458f3aca: Skipping requested alter to schema version 2, tablet already version 2
I20250113 02:51:41.403049 1192 consensus_queue.cc:260] T 153b8f90b5a847b88944f683d306f579 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:41.403661 1192 ts_tablet_manager.cc:1428] T 153b8f90b5a847b88944f683d306f579 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250113 02:51:41.403941 1192 tablet_bootstrap.cc:492] T ba8980ad3a364c888a825affbcbc9b1f P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:41.407724 1196 log.cc:1198] T f129bb3fbbfd492d98822be0a6626e44 P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/f129bb3fbbfd492d98822be0a6626e44
I20250113 02:51:41.408944 1036 tablet_service.cc:1467] Processing CreateTablet for tablet 86c8c07f90d2470ebb2b3f1d16daf4cd (DEFAULT_TABLE table=default.Table_32fe238c17574c1ea5721f507c0bdcc9 [id=394f1a3b62704790a2a3cee6d5a06f82]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:41.409154 1194 tablet_bootstrap.cc:492] T 7527e6e0b65e4a43b7f48899a34cdf7f P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:41.409271 1194 ts_tablet_manager.cc:1397] T 7527e6e0b65e4a43b7f48899a34cdf7f P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.000s
I20250113 02:51:41.409286 1036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 86c8c07f90d2470ebb2b3f1d16daf4cd. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:41.409664 852 leader_election.cc:343] T 3025935f9743442eb3e6f2db22eb52ad P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Not found: Tablet not found: 3025935f9743442eb3e6f2db22eb52ad
I20250113 02:51:41.410883 1030 tablet_service.cc:1467] Processing CreateTablet for tablet d3619576c8f94583b330fe1828cea2e0 (DEFAULT_TABLE table=default.Table_bdab3234b9d44a7fb534b8479b322f4f [id=d80ff8c583684dc59f8960b4a6fe8810]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:41.411339 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d3619576c8f94583b330fe1828cea2e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:41.411761 852 leader_election.cc:304] T 3025935f9743442eb3e6f2db22eb52ad P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df69f48656b44bb890f76587ee69cc0f; no voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca
I20250113 02:51:41.412425 2476 raft_consensus.cc:2743] T 3025935f9743442eb3e6f2db22eb52ad P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250113 02:51:41.412842 852 leader_election.cc:343] T 999fecb3fc65466cad21cc1ff5ef2d44 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Not found: Tablet not found: 999fecb3fc65466cad21cc1ff5ef2d44
W20250113 02:51:41.406628 774 thread.cc:635] apply [worker] (thread pool) Time spent creating pthread: real 0.637s user 0.011s sys 0.021s
I20250113 02:51:41.414395 852 leader_election.cc:304] T 999fecb3fc65466cad21cc1ff5ef2d44 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df69f48656b44bb890f76587ee69cc0f; no voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca
I20250113 02:51:41.414916 2476 raft_consensus.cc:2743] T 999fecb3fc65466cad21cc1ff5ef2d44 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:41.415148 1024 tablet_service.cc:1467] Processing CreateTablet for tablet b01dcbb7e28c4c1fbc389f829cead09e (DEFAULT_TABLE table=default.Table_9458e2e0d9a446378600ea1ef1696bd5 [id=1d5ff280114343d691657c188fb6a111]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:41.415409 1035 tablet_service.cc:1467] Processing CreateTablet for tablet b946dd2d428a43d399bb75a43d843a28 (DEFAULT_TABLE table=default.Table_d5096557ae6445e6bcd3b6a9b3299e8b [id=b56ce78e920f464fbeab248484d6f2f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:41.415589 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b01dcbb7e28c4c1fbc389f829cead09e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:41.415828 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b946dd2d428a43d399bb75a43d843a28. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:41.419109 421 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.104.2:38467, user_credentials={real_user=slave}} blocked reactor thread for 99226.6us
I20250113 02:51:41.420257 1028 tablet_service.cc:1467] Processing CreateTablet for tablet f6f4df3bde434667a923be19aa39817a (DEFAULT_TABLE table=default.Table_4078d475fb2e4e52ad184b9318ec9c7d [id=6e6f2eaf51294f6a8a1228cc66b89575]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:41.420598 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f6f4df3bde434667a923be19aa39817a. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:41.421980 998 consensus_peers.cc:487] T bc6a7322c0c34adb85457750526d8955 P 4216a7f853d642b7bc3e5cff458f3aca -> Peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Couldn't send request to peer df69f48656b44bb890f76587ee69cc0f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250113 02:51:41.423034 774 thread.cc:601] apply [worker] (thread pool) Time spent starting thread: real 0.647s user 0.011s sys 0.021s
I20250113 02:51:41.425330 1192 tablet_bootstrap.cc:654] T ba8980ad3a364c888a825affbcbc9b1f P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:41.425911 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 192a226332e64766a0e367c6fc1c1673. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:41.426164 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b98da6f197764465a1e10aaadcddf3a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:41.426355 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e3231aafc4e49aabaef5f34db272659. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:41.426847 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca6f77f97a2343f49dccb54361fb0cea. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:41.427731 1195 ts_tablet_manager.cc:1939] T 788ab82626424b6e833e31761a15b9b2 P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:41.434909 1196 ts_tablet_manager.cc:1939] T f129bb3fbbfd492d98822be0a6626e44 P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:41.435513 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc680b5680b84abbab219296bc7ee707" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:41.442436 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1081) took 4648 ms. Trace:
I20250113 02:51:41.443225 1196 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37366 (request call id 1116) took 2143 ms. Trace:
I20250113 02:51:41.443354 1196 rpcz_store.cc:276] 0113 02:51:39.300096 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:39.313173 (+ 13077us) service_pool.cc:225] Handling call
0113 02:51:41.097534 (+1784361us) ts_tablet_manager.cc:1145] Deleting tablet f129bb3fbbfd492d98822be0a6626e44
0113 02:51:41.097542 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:41.157314 (+ 59772us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:41.241340 (+ 84026us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:41.443201 (+201861us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1781147}
I20250113 02:51:41.443914 1196 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37366 (request call id 1133) took 1923 ms. Trace:
I20250113 02:51:41.444049 1196 rpcz_store.cc:276] 0113 02:51:39.520644 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:39.524051 (+ 3407us) service_pool.cc:225] Handling call
0113 02:51:41.443763 (+1919712us) ts_tablet_manager.cc:1145] Deleting tablet 128f74fdc98f4135b39dbf23f1d987b0
0113 02:51:41.443775 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:41.443896 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1919464}
I20250113 02:51:41.444443 1196 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37366 (request call id 1134) took 1923 ms. Trace:
I20250113 02:51:41.444527 1196 rpcz_store.cc:276] 0113 02:51:39.520741 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:39.524330 (+ 3589us) service_pool.cc:225] Handling call
0113 02:51:41.444328 (+1919998us) ts_tablet_manager.cc:1145] Deleting tablet e5ed7016af184b86be4c234f42e2a6ef
0113 02:51:41.444337 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:41.444428 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1919869}
I20250113 02:51:41.444849 1196 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37366 (request call id 1132) took 1924 ms. Trace:
I20250113 02:51:41.444926 1196 rpcz_store.cc:276] 0113 02:51:39.520548 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:39.520616 (+ 68us) service_pool.cc:225] Handling call
0113 02:51:41.444752 (+1924136us) ts_tablet_manager.cc:1145] Deleting tablet 16f547e3cec54baf86b96ad40e3afb6a
0113 02:51:41.444761 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:41.444834 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1867067}
I20250113 02:51:41.445269 1196 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37366 (request call id 1158) took 1475 ms. Trace:
I20250113 02:51:41.445353 1196 rpcz_store.cc:276] 0113 02:51:39.970209 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:39.970256 (+ 47us) service_pool.cc:225] Handling call
0113 02:51:41.445183 (+1474927us) ts_tablet_manager.cc:1145] Deleting tablet 045167a6122a4560b298aae2358a485c
0113 02:51:41.445192 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:41.445256 (+ 64us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1474701}
I20250113 02:51:41.445670 1196 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37366 (request call id 1157) took 1475 ms. Trace:
I20250113 02:51:41.445744 1196 rpcz_store.cc:276] 0113 02:51:39.970100 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:40.016442 (+ 46342us) service_pool.cc:225] Handling call
0113 02:51:41.445576 (+1429134us) ts_tablet_manager.cc:1145] Deleting tablet 61c79029dfb74f01aaac0f7f828880d0
0113 02:51:41.445585 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:41.445657 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1428734}
I20250113 02:51:41.446062 1196 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37366 (request call id 1159) took 1366 ms. Trace:
I20250113 02:51:41.448225 1192 tablet_bootstrap.cc:492] T ba8980ad3a364c888a825affbcbc9b1f P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:41.448421 1192 ts_tablet_manager.cc:1397] T ba8980ad3a364c888a825affbcbc9b1f P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.045s user 0.004s sys 0.004s
I20250113 02:51:41.449194 1192 raft_consensus.cc:357] T ba8980ad3a364c888a825affbcbc9b1f P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:41.449463 1192 raft_consensus.cc:383] T ba8980ad3a364c888a825affbcbc9b1f P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:41.449563 1192 raft_consensus.cc:738] T ba8980ad3a364c888a825affbcbc9b1f P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:41.449838 1192 consensus_queue.cc:260] T ba8980ad3a364c888a825affbcbc9b1f P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:41.450598 1192 ts_tablet_manager.cc:1428] T ba8980ad3a364c888a825affbcbc9b1f P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250113 02:51:41.453331 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 192a226332e64766a0e367c6fc1c1673. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:41.457234 1195 rpcz_store.cc:276] 0113 02:51:36.794233 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:36.795699 (+ 1466us) service_pool.cc:225] Handling call
0113 02:51:40.421483 (+3625784us) ts_tablet_manager.cc:1145] Deleting tablet 788ab82626424b6e833e31761a15b9b2
0113 02:51:40.421493 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:40.437653 (+ 16160us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:40.731307 (+293654us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:41.442407 (+711100us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3625578}
I20250113 02:51:41.457798 755 tablet_service.cc:1467] Processing CreateTablet for tablet b946dd2d428a43d399bb75a43d843a28 (DEFAULT_TABLE table=default.Table_d5096557ae6445e6bcd3b6a9b3299e8b [id=b56ce78e920f464fbeab248484d6f2f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:41.458153 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b946dd2d428a43d399bb75a43d843a28. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:41.459259 447 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_3cfdcce7afbf48ee8cf67f1a4c839612" } modify_external_catalogs: true
I20250113 02:51:41.459311 1194 raft_consensus.cc:357] T 7527e6e0b65e4a43b7f48899a34cdf7f P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:41.459496 447 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_3cfdcce7afbf48ee8cf67f1a4c839612" } modify_external_catalogs: true
I20250113 02:51:41.459549 1194 raft_consensus.cc:383] T 7527e6e0b65e4a43b7f48899a34cdf7f P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:41.459606 1194 raft_consensus.cc:738] T 7527e6e0b65e4a43b7f48899a34cdf7f P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:41.459831 1194 consensus_queue.cc:260] T 7527e6e0b65e4a43b7f48899a34cdf7f P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:41.460366 1194 ts_tablet_manager.cc:1428] T 7527e6e0b65e4a43b7f48899a34cdf7f P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.051s user 0.002s sys 0.000s
I20250113 02:51:41.460593 1194 tablet_bootstrap.cc:492] T 33c665e149dd48e8868acd935de581bf P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:41.463245 1194 tablet_bootstrap.cc:654] T 33c665e149dd48e8868acd935de581bf P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:41.463717 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet 788ab82626424b6e833e31761a15b9b2 (table default.Table_9494f6e5f96141dbaec6769ed51d39b2 [id=0f07147c73f24decb09399a1dd188e1f]) successfully deleted
I20250113 02:51:41.464601 2480 consensus_queue.cc:1035] T bc6a7322c0c34adb85457750526d8955 P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 02:51:41.466073 2499 tablet.cc:1721] T 06f36d09bbb1443bafce906907795404 P df69f48656b44bb890f76587ee69cc0f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20250113 02:51:41.469375 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet c8966b3b93144a30813bfdaf04f9c43a
I20250113 02:51:41.473466 1194 tablet_bootstrap.cc:492] T 33c665e149dd48e8868acd935de581bf P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:41.473604 1194 ts_tablet_manager.cc:1397] T 33c665e149dd48e8868acd935de581bf P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.000s
I20250113 02:51:41.474404 1194 raft_consensus.cc:357] T 33c665e149dd48e8868acd935de581bf P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:41.474596 1194 raft_consensus.cc:383] T 33c665e149dd48e8868acd935de581bf P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:41.474654 1194 raft_consensus.cc:738] T 33c665e149dd48e8868acd935de581bf P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:41.455610 753 tablet_service.cc:1467] Processing CreateTablet for tablet d3619576c8f94583b330fe1828cea2e0 (DEFAULT_TABLE table=default.Table_bdab3234b9d44a7fb534b8479b322f4f [id=d80ff8c583684dc59f8960b4a6fe8810]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:41.477116 741 tablet_service.cc:1467] Processing CreateTablet for tablet 86c8c07f90d2470ebb2b3f1d16daf4cd (DEFAULT_TABLE table=default.Table_32fe238c17574c1ea5721f507c0bdcc9 [id=394f1a3b62704790a2a3cee6d5a06f82]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:41.477301 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d3619576c8f94583b330fe1828cea2e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:41.477559 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 86c8c07f90d2470ebb2b3f1d16daf4cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:41.456254 1196 rpcz_store.cc:276] 0113 02:51:40.110305 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:40.110367 (+ 62us) service_pool.cc:225] Handling call
0113 02:51:41.445955 (+1335588us) ts_tablet_manager.cc:1145] Deleting tablet 432bcbc7ba6a44ba98cfe588f8292e32
0113 02:51:41.445964 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:41.446047 (+ 83us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1335385}
I20250113 02:51:41.480412 1196 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37366 (request call id 1160) took 1370 ms. Trace:
I20250113 02:51:41.480491 1196 rpcz_store.cc:276] 0113 02:51:40.110424 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:40.110575 (+ 151us) service_pool.cc:225] Handling call
0113 02:51:41.480293 (+1369718us) ts_tablet_manager.cc:1145] Deleting tablet 797d91d4aae240bd82b00021717772ae
0113 02:51:41.480303 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:41.480397 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1369560}
I20250113 02:51:41.481050 1196 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37366 (request call id 1161) took 1370 ms. Trace:
I20250113 02:51:41.481123 1196 rpcz_store.cc:276] 0113 02:51:40.110509 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:40.110628 (+ 119us) service_pool.cc:225] Handling call
0113 02:51:41.480908 (+1370280us) ts_tablet_manager.cc:1145] Deleting tablet 67fe2f52d98048dab9f37dd820b7f712
0113 02:51:41.480916 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:41.480976 (+ 60us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1369664}
I20250113 02:51:41.481498 1196 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37366 (request call id 1162) took 1336 ms. Trace:
I20250113 02:51:41.481520 1041 tablet_service.cc:1514] Processing DeleteTablet for tablet c8966b3b93144a30813bfdaf04f9c43a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:41 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:41.481562 1196 rpcz_store.cc:276] 0113 02:51:40.144627 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:40.144683 (+ 56us) service_pool.cc:225] Handling call
0113 02:51:41.481416 (+1336733us) ts_tablet_manager.cc:1145] Deleting tablet efe5f5ffd2c8423686b4007ded447466
0113 02:51:41.481424 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:41.481486 (+ 62us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1336557}
I20250113 02:51:41.481866 1196 tablet_replica.cc:331] T 1d15c7a01f854f0aaabcd30ec3166e31 P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:41.482049 1196 raft_consensus.cc:2238] T 1d15c7a01f854f0aaabcd30ec3166e31 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:41.482180 1196 raft_consensus.cc:2267] T 1d15c7a01f854f0aaabcd30ec3166e31 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:41.482630 1196 ts_tablet_manager.cc:1905] T 1d15c7a01f854f0aaabcd30ec3166e31 P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:41.482861 2498 tablet.cc:1716] T 6e1284c8aaba44b5810b0d06d36757eb P df69f48656b44bb890f76587ee69cc0f: Skipping requested alter to schema version 2, tablet already version 2
I20250113 02:51:41.489010 447 catalog_manager.cc:5618] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587) does not have the latest schema for tablet b5c1882125b5491ea79a29a6dac6771e (table default.Table_99031e0341fd4ad8a3bb04a6e222a1d0 [id=3237c1be7e1c48918ff095162d191dc1]). Expected version 3 got 0
I20250113 02:51:41.491111 1030 tablet_service.cc:1514] Processing DeleteTablet for tablet ba8980ad3a364c888a825affbcbc9b1f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:36 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:41.491866 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet 153b8f90b5a847b88944f683d306f579 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:24 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:41.492647 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_d7372f791f52447593557efc3f8226ff" } new_table_name: "default.Table_5eb345a0620d4026ba17f399692ff6a3" modify_external_catalogs: true
W20250113 02:51:41.499940 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fb715f9fbf0e4e3d935b0510e9872257 consensus state has no leader
I20250113 02:51:41.436710 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b33025a462e74f2aa26cc3779f36615b" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
I20250113 02:51:41.504062 774 raft_consensus.cc:2463] T b33025a462e74f2aa26cc3779f36615b P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 0.
I20250113 02:51:41.506410 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_302b50fc67d44fa79ef59d7cc8fbc408"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:41.506948 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_302b50fc67d44fa79ef59d7cc8fbc408 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:41.448889 751 tablet_service.cc:1467] Processing CreateTablet for tablet 711544894da147458c6a342883d7b83b (DEFAULT_TABLE table=default.Table_bf4806a5a68e4fcb98205b6fed9d0b5e [id=8bfd0e28c2ac404fb624b628675e5d84]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:41.511062 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 711544894da147458c6a342883d7b83b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:41.445631 743 tablet_service.cc:1467] Processing CreateTablet for tablet 6ba5a61a6aae4a23812503cc858f8957 (DEFAULT_TABLE table=default.Table_0e134d21b63b463ea086ed3ac6921486 [id=d15996f669224fe1a2d856b6c687931e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:41.512396 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ba5a61a6aae4a23812503cc858f8957. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:41.515825 995 leader_election.cc:304] T b33025a462e74f2aa26cc3779f36615b P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters: df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:41.516294 2487 raft_consensus.cc:2798] T b33025a462e74f2aa26cc3779f36615b P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 02:51:41.516372 2487 raft_consensus.cc:491] T b33025a462e74f2aa26cc3779f36615b P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 02:51:41.516448 2487 raft_consensus.cc:3054] T b33025a462e74f2aa26cc3779f36615b P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:41.449628 752 tablet_service.cc:1467] Processing CreateTablet for tablet 99843ce7f1bf41da9b7cb5bd7ac2e875 (DEFAULT_TABLE table=default.Table_ed460a21a2184db893d2e76e67cdb974 [id=fb3d90554f7943099cd541f610084cbf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:41.450707 754 tablet_service.cc:1467] Processing CreateTablet for tablet f6f4df3bde434667a923be19aa39817a (DEFAULT_TABLE table=default.Table_4078d475fb2e4e52ad184b9318ec9c7d [id=6e6f2eaf51294f6a8a1228cc66b89575]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:41.517133 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f6f4df3bde434667a923be19aa39817a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:41.519093 2487 raft_consensus.cc:513] T b33025a462e74f2aa26cc3779f36615b P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:41.520586 2487 leader_election.cc:290] T b33025a462e74f2aa26cc3779f36615b P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Requested vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:41.521375 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b33025a462e74f2aa26cc3779f36615b" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f"
W20250113 02:51:41.521870 998 leader_election.cc:343] T b33025a462e74f2aa26cc3779f36615b P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:41.522148 2479 tablet.cc:1721] T 44d2c7d5334c476fac60fdd57e2c02b6 P 4216a7f853d642b7bc3e5cff458f3aca: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250113 02:51:41.522756 1025 tablet_service.cc:1467] Processing CreateTablet for tablet 079ef7df7b164466b3fc24f58caec611 (DEFAULT_TABLE table=default.Table_da027dfe87ee40ada76b095abebd0e7e [id=dff2d4a62b8b42dca4e6cd8d999d57d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:41.523491 1030 tablet_service.cc:1467] Processing CreateTablet for tablet 568fae43734b4d3f81802e01251c6a7a (DEFAULT_TABLE table=default.Table_588573aeb9464e8088f6759b1ca47c54 [id=0d5f79f41c8b4eae9ee13e055d577525]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:41.523897 1034 tablet_service.cc:1467] Processing CreateTablet for tablet f6763009f1c14ed2b3b311a2da07ec9f (DEFAULT_TABLE table=default.Table_c1e2aac17dd047708ca98085186252a7 [id=c26157fba3744bad819883e90511bf19]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:41.456099 749 tablet_service.cc:1467] Processing CreateTablet for tablet b01dcbb7e28c4c1fbc389f829cead09e (DEFAULT_TABLE table=default.Table_9458e2e0d9a446378600ea1ef1696bd5 [id=1d5ff280114343d691657c188fb6a111]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:41.524448 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b01dcbb7e28c4c1fbc389f829cead09e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:41.526118 1024 tablet_service.cc:1467] Processing CreateTablet for tablet b2a3477d01c444ef8e346d28ede09267 (DEFAULT_TABLE table=default.Table_4bb227dc715148c9862b2d999509533a [id=48cada9c7be54533b989bc4d360b43e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:41.526547 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 99843ce7f1bf41da9b7cb5bd7ac2e875. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:41.530484 2306 raft_consensus.cc:491] T b815003a27f64809baaa2d43d6293cef P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:41.530611 2306 raft_consensus.cc:513] T b815003a27f64809baaa2d43d6293cef P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:41.531236 2306 leader_election.cc:290] T b815003a27f64809baaa2d43d6293cef P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:41.531868 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b815003a27f64809baaa2d43d6293cef" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:41.531853 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b815003a27f64809baaa2d43d6293cef" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
I20250113 02:51:41.532063 2481 tablet.cc:1721] T 44d2c7d5334c476fac60fdd57e2c02b6 P 1186fa869eb14812832fa6562531f196: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250113 02:51:41.532127 773 raft_consensus.cc:2463] T b815003a27f64809baaa2d43d6293cef P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 0.
W20250113 02:51:41.532362 998 leader_election.cc:343] T b815003a27f64809baaa2d43d6293cef P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:41.532732 995 leader_election.cc:304] T b815003a27f64809baaa2d43d6293cef P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters: df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:41.533365 1191 tablet_bootstrap.cc:492] T 16a132384b0e4264a73d09a3ea133893 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:41.533614 2306 raft_consensus.cc:2798] T b815003a27f64809baaa2d43d6293cef P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 02:51:41.533694 2306 raft_consensus.cc:491] T b815003a27f64809baaa2d43d6293cef P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 02:51:41.533771 2306 raft_consensus.cc:3054] T b815003a27f64809baaa2d43d6293cef P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:41.534693 1191 ts_tablet_manager.cc:1397] T 16a132384b0e4264a73d09a3ea133893 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.425s user 0.006s sys 0.007s
I20250113 02:51:41.536444 1035 tablet_service.cc:1467] Processing CreateTablet for tablet b2dc876471544cc19d7bddcc003f694f (DEFAULT_TABLE table=default.Table_96649abef65740ad8698fabc7e9d4a63 [id=37df785bff71410297d0a94e7f788210]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:41.536696 1028 tablet_service.cc:1467] Processing CreateTablet for tablet 7203461deb504fbe95d62e67b86b134e (DEFAULT_TABLE table=default.Table_98fee465d761490c85252dbb021e3f96 [id=f2a79d79b71c4f8baba1a72c78b8b116]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:41.536788 1036 tablet_service.cc:1467] Processing CreateTablet for tablet b96279fa76124046986e0e692ebee6c4 (DEFAULT_TABLE table=default.Table_969a9705d160408cb78f3c7029fa4023 [id=a3ee14433570445a9313d6e75456c4ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:41.538451 2306 raft_consensus.cc:513] T b815003a27f64809baaa2d43d6293cef P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:41.539712 2306 leader_election.cc:290] T b815003a27f64809baaa2d43d6293cef P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Requested vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:41.540370 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b815003a27f64809baaa2d43d6293cef" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f"
I20250113 02:51:41.540588 753 tablet_service.cc:1514] Processing DeleteTablet for tablet c8966b3b93144a30813bfdaf04f9c43a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:41 UTC) from {username='slave'} at 127.0.0.1:55398
W20250113 02:51:41.540884 998 leader_election.cc:343] T b815003a27f64809baaa2d43d6293cef P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:41.446884 742 tablet_service.cc:1467] Processing CreateTablet for tablet 12707a38496c464a859fa294cf6bbe36 (DEFAULT_TABLE table=default.Table_3e43331bf36845f9aa983823e4853d04 [id=6e23db586a4740a38379b905fdf40c13]), partition=RANGE (key) PARTITION UNBOUNDED
W20250113 02:51:41.507844 711 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.2:38467, user_credentials={real_user=slave}} blocked reactor thread for 115536us
W20250113 02:51:41.541570 711 leader_election.cc:343] T cf7b8bd6801743f49e9b9473747447c5 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:41.541757 711 leader_election.cc:304] T cf7b8bd6801743f49e9b9473747447c5 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:41.541649 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 12707a38496c464a859fa294cf6bbe36. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:41.541924 1041 tablet_service.cc:1467] Processing CreateTablet for tablet dfb0fee94d734f72ba354812f0e111aa (DEFAULT_TABLE table=default.Table_851855dfd1934ca38a5714adabc71199 [id=b1215be81ae44fb99bbc5dda27e605ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:41.544284 1196 ts_tablet_manager.cc:1918] T 1d15c7a01f854f0aaabcd30ec3166e31 P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:41.544420 1196 log.cc:1198] T 1d15c7a01f854f0aaabcd30ec3166e31 P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/1d15c7a01f854f0aaabcd30ec3166e31
I20250113 02:51:41.544756 1196 ts_tablet_manager.cc:1939] T 1d15c7a01f854f0aaabcd30ec3166e31 P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:41.544759 2481 tablet.cc:1721] T 06f36d09bbb1443bafce906907795404 P 1186fa869eb14812832fa6562531f196: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20250113 02:51:41.549126 1192 tablet_bootstrap.cc:492] T dc680b5680b84abbab219296bc7ee707 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:41.551318 1192 tablet_bootstrap.cc:654] T dc680b5680b84abbab219296bc7ee707 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:41.553557 449 catalog_manager.cc:5618] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095) does not have the latest schema for tablet b5c1882125b5491ea79a29a6dac6771e (table default.Table_99031e0341fd4ad8a3bb04a6e222a1d0 [id=3237c1be7e1c48918ff095162d191dc1]). Expected version 3 got 0
I20250113 02:51:41.553768 449 catalog_manager.cc:5618] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095) does not have the latest schema for tablet 44d2c7d5334c476fac60fdd57e2c02b6 (table default.Table_6daec99b05744d0caed86e51e86f24c2 [id=2931f217041c46cc8ae52bd6ce8cfd17]). Expected version 1 got 0
I20250113 02:51:41.554741 2473 raft_consensus.cc:2743] T cf7b8bd6801743f49e9b9473747447c5 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250113 02:51:41.555730 711 leader_election.cc:343] T cfa963a6df094ce0b9c54408f75ca6b2 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:41.555904 711 leader_election.cc:304] T cfa963a6df094ce0b9c54408f75ca6b2 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:41.556203 2473 raft_consensus.cc:2743] T cfa963a6df094ce0b9c54408f75ca6b2 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:41.556664 754 tablet_service.cc:1467] Processing CreateTablet for tablet 568fae43734b4d3f81802e01251c6a7a (DEFAULT_TABLE table=default.Table_588573aeb9464e8088f6759b1ca47c54 [id=0d5f79f41c8b4eae9ee13e055d577525]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:41.556856 736 tablet_service.cc:1467] Processing CreateTablet for tablet b2dc876471544cc19d7bddcc003f694f (DEFAULT_TABLE table=default.Table_96649abef65740ad8698fabc7e9d4a63 [id=37df785bff71410297d0a94e7f788210]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:41.557087 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 568fae43734b4d3f81802e01251c6a7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:41.557292 736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b2dc876471544cc19d7bddcc003f694f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:41.560443 737 tablet_service.cc:1467] Processing CreateTablet for tablet 7203461deb504fbe95d62e67b86b134e (DEFAULT_TABLE table=default.Table_98fee465d761490c85252dbb021e3f96 [id=f2a79d79b71c4f8baba1a72c78b8b116]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:41.560751 737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7203461deb504fbe95d62e67b86b134e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:41.561591 750 tablet_service.cc:1467] Processing CreateTablet for tablet f6763009f1c14ed2b3b311a2da07ec9f (DEFAULT_TABLE table=default.Table_c1e2aac17dd047708ca98085186252a7 [id=c26157fba3744bad819883e90511bf19]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:41.561898 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f6763009f1c14ed2b3b311a2da07ec9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:41.563151 1192 tablet_bootstrap.cc:492] T dc680b5680b84abbab219296bc7ee707 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:41.563264 1192 ts_tablet_manager.cc:1397] T dc680b5680b84abbab219296bc7ee707 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.006s
I20250113 02:51:41.564028 1192 raft_consensus.cc:357] T dc680b5680b84abbab219296bc7ee707 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:41.564258 1192 raft_consensus.cc:383] T dc680b5680b84abbab219296bc7ee707 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:41.564316 1192 raft_consensus.cc:738] T dc680b5680b84abbab219296bc7ee707 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
W20250113 02:51:41.564533 711 leader_election.cc:343] T 4c759831773849f2bc52e484ffa0cf49 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:41.564637 711 leader_election.cc:304] T 4c759831773849f2bc52e484ffa0cf49 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:41.564589 1192 consensus_queue.cc:260] T dc680b5680b84abbab219296bc7ee707 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:41.564847 2473 raft_consensus.cc:2743] T 4c759831773849f2bc52e484ffa0cf49 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:41.565166 1192 ts_tablet_manager.cc:1428] T dc680b5680b84abbab219296bc7ee707 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20250113 02:51:41.565171 711 leader_election.cc:343] T ca75e7cae4f6423191a17bb5a8f150a9 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:41.565265 711 leader_election.cc:304] T ca75e7cae4f6423191a17bb5a8f150a9 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:41.565476 2473 raft_consensus.cc:2743] T ca75e7cae4f6423191a17bb5a8f150a9 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:41.566094 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b815003a27f64809baaa2d43d6293cef" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196"
I20250113 02:51:41.566331 775 raft_consensus.cc:3054] T b815003a27f64809baaa2d43d6293cef P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:41.566598 744 tablet_service.cc:1467] Processing CreateTablet for tablet b2a3477d01c444ef8e346d28ede09267 (DEFAULT_TABLE table=default.Table_4bb227dc715148c9862b2d999509533a [id=48cada9c7be54533b989bc4d360b43e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:41.566912 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b2a3477d01c444ef8e346d28ede09267. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:41.567090 711 leader_election.cc:343] T dc680b5680b84abbab219296bc7ee707 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:41.567231 711 leader_election.cc:304] T dc680b5680b84abbab219296bc7ee707 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:41.567328 740 tablet_service.cc:1467] Processing CreateTablet for tablet 079ef7df7b164466b3fc24f58caec611 (DEFAULT_TABLE table=default.Table_da027dfe87ee40ada76b095abebd0e7e [id=dff2d4a62b8b42dca4e6cd8d999d57d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:41.567488 2473 raft_consensus.cc:2743] T dc680b5680b84abbab219296bc7ee707 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:41.567621 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 079ef7df7b164466b3fc24f58caec611. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:41.568064 2473 raft_consensus.cc:491] T 438604e8859f422f9caccb61c5b5b125 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:41.568167 2473 raft_consensus.cc:513] T 438604e8859f422f9caccb61c5b5b125 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:41.568292 453 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_0a0f9ff8173c404bac451d9168bf9426" } modify_external_catalogs: true
I20250113 02:51:41.568475 453 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_0a0f9ff8173c404bac451d9168bf9426" } modify_external_catalogs: true
I20250113 02:51:41.569742 746 tablet_service.cc:1467] Processing CreateTablet for tablet b96279fa76124046986e0e692ebee6c4 (DEFAULT_TABLE table=default.Table_969a9705d160408cb78f3c7029fa4023 [id=a3ee14433570445a9313d6e75456c4ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:41.458309 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1090) took 4334 ms. Trace:
I20250113 02:51:41.570751 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_ca4412c5b1a24d739bcdad8874cc84e9" } new_table_name: "default.Table_cf16887c855544b1987f7539a58537fb" modify_external_catalogs: true
I20250113 02:51:41.573427 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_414756feb23c4db3a86bfd310703a889"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:41.574000 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_414756feb23c4db3a86bfd310703a889 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:41.569741 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b33025a462e74f2aa26cc3779f36615b" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196"
I20250113 02:51:41.574465 774 raft_consensus.cc:3054] T b33025a462e74f2aa26cc3779f36615b P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:41.570341 1195 rpcz_store.cc:276] 0113 02:51:37.124195 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:37.125143 (+ 948us) service_pool.cc:225] Handling call
0113 02:51:41.458063 (+4332920us) ts_tablet_manager.cc:1145] Deleting tablet cb141fb2e9894757b3ae423242ad93d2
0113 02:51:41.458072 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:41.458262 (+ 190us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4332454}
W20250113 02:51:41.575263 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet cb141fb2e9894757b3ae423242ad93d2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cb141fb2e9894757b3ae423242ad93d2 already in progress: creating tablet
I20250113 02:51:41.575589 448 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_7c964c948c4f44298d9bae9fc5e608c6" } modify_external_catalogs: true
I20250113 02:51:41.575749 448 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_7c964c948c4f44298d9bae9fc5e608c6" } modify_external_catalogs: true
W20250113 02:51:41.576087 421 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.104.2:38467, user_credentials={real_user=slave}} blocked reactor thread for 72696.3us
W20250113 02:51:41.576388 416 external_mini_cluster.cc:1677] Unable to check leaks on /tmp/dist-test-taskXJrxug/build/asan/bin/kudu pid 1706: Timed out: CheckLeaks RPC to 127.0.104.60:42067 timed out after 1.000s (SENT)
I20250113 02:51:41.576401 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_cc25a0009bc34b2fbf160fe03db677ba"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:41.576537 416 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskXJrxug/build/asan/bin/kudu with pid 1706
W20250113 02:51:41.576798 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cc25a0009bc34b2fbf160fe03db677ba in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:41.578583 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_58234d02d4164492995fe080423737ec"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:41.581089 1196 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37366 (request call id 1163) took 1345 ms. Trace:
I20250113 02:51:41.581173 1196 rpcz_store.cc:276] 0113 02:51:40.235407 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:40.235580 (+ 173us) service_pool.cc:225] Handling call
0113 02:51:41.481827 (+1246247us) ts_tablet_manager.cc:1145] Deleting tablet 1d15c7a01f854f0aaabcd30ec3166e31
0113 02:51:41.481834 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:41.540800 (+ 58966us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:41.544172 (+ 3372us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:41.581067 (+ 36895us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1246056}
I20250113 02:51:41.581470 1196 tablet_replica.cc:331] T 3025935f9743442eb3e6f2db22eb52ad P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:41.581624 1196 raft_consensus.cc:2238] T 3025935f9743442eb3e6f2db22eb52ad P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:41.581769 1196 raft_consensus.cc:2267] T 3025935f9743442eb3e6f2db22eb52ad P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:41.582264 1196 ts_tablet_manager.cc:1905] T 3025935f9743442eb3e6f2db22eb52ad P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:41.582810 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1091) took 4456 ms. Trace:
I20250113 02:51:41.582890 1195 rpcz_store.cc:276] 0113 02:51:37.126041 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:37.150544 (+ 24503us) service_pool.cc:225] Handling call
0113 02:51:41.582672 (+4432128us) ts_tablet_manager.cc:1145] Deleting tablet 60fac96c00d64ec193b0c706ec64bfaf
0113 02:51:41.582683 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:41.582788 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4431957}
W20250113 02:51:41.583340 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 60fac96c00d64ec193b0c706ec64bfaf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 60fac96c00d64ec193b0c706ec64bfaf already in progress: creating tablet
I20250113 02:51:41.584223 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_bb6e88b17f3d497386f8098edb3719b9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:41.584702 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bb6e88b17f3d497386f8098edb3719b9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:41.585742 454 master_service.cc:946] ReplaceTablet: received request to replace tablet 8ee0b2e06dc74faab48912dd16fe1537 from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:41.586009 455 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_6c9405a4b90a4010b3033c83c9c8a8c4" } modify_external_catalogs: true
I20250113 02:51:41.586129 455 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_6c9405a4b90a4010b3033c83c9c8a8c4" } modify_external_catalogs: true
I20250113 02:51:41.587109 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 079ef7df7b164466b3fc24f58caec611. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:41.587311 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 568fae43734b4d3f81802e01251c6a7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:41.592777 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f6763009f1c14ed2b3b311a2da07ec9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:41.599395 2473 leader_election.cc:290] T 438604e8859f422f9caccb61c5b5b125 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:41.599782 1058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "438604e8859f422f9caccb61c5b5b125" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
W20250113 02:51:41.600312 709 leader_election.cc:343] T 438604e8859f422f9caccb61c5b5b125 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:41.601889 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b96279fa76124046986e0e692ebee6c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:41.602074 1196 ts_tablet_manager.cc:1918] T 3025935f9743442eb3e6f2db22eb52ad P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:41.602180 1196 log.cc:1198] T 3025935f9743442eb3e6f2db22eb52ad P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/3025935f9743442eb3e6f2db22eb52ad
I20250113 02:51:41.602463 1196 ts_tablet_manager.cc:1939] T 3025935f9743442eb3e6f2db22eb52ad P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:41.602794 1196 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37366 (request call id 1164) took 1345 ms. Trace:
I20250113 02:51:41.602882 1196 rpcz_store.cc:276] 0113 02:51:40.256847 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:40.256902 (+ 55us) service_pool.cc:225] Handling call
0113 02:51:41.581426 (+1324524us) ts_tablet_manager.cc:1145] Deleting tablet 3025935f9743442eb3e6f2db22eb52ad
0113 02:51:41.581435 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:41.585437 (+ 4002us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:41.601954 (+ 16517us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:41.602774 (+ 820us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1324323}
I20250113 02:51:41.604224 1196 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37366 (request call id 1165) took 1319 ms. Trace:
I20250113 02:51:41.604308 1196 rpcz_store.cc:276] 0113 02:51:40.284551 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:40.284612 (+ 61us) service_pool.cc:225] Handling call
0113 02:51:41.604101 (+1319489us) ts_tablet_manager.cc:1145] Deleting tablet 368e5f00861b40ffb76e79376245b885
0113 02:51:41.604112 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:41.604207 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1305639}
I20250113 02:51:41.609656 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b2dc876471544cc19d7bddcc003f694f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:41.612207 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7203461deb504fbe95d62e67b86b134e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:41.615374 1036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b96279fa76124046986e0e692ebee6c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:41.617395 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b2a3477d01c444ef8e346d28ede09267. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:41.621605 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c7da2c406e64426faf123c4bddfaf213 consensus state has no leader
I20250113 02:51:41.624315 1196 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37366 (request call id 1166) took 1330 ms. Trace:
I20250113 02:51:41.624406 1196 rpcz_store.cc:276] 0113 02:51:40.294363 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:40.294425 (+ 62us) service_pool.cc:225] Handling call
0113 02:51:41.624164 (+1329739us) ts_tablet_manager.cc:1145] Deleting tablet 59fadc693f554854bc2874f8db7c64c8
0113 02:51:41.624176 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:41.624291 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1325567}
I20250113 02:51:41.624928 1196 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37366 (request call id 1167) took 1318 ms. Trace:
W20250113 02:51:41.625308 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_58234d02d4164492995fe080423737ec in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:41.625007 1196 rpcz_store.cc:276] 0113 02:51:40.306690 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:40.306743 (+ 53us) service_pool.cc:225] Handling call
0113 02:51:41.624810 (+1318067us) ts_tablet_manager.cc:1145] Deleting tablet 348c21d783ef4103a37409ef8634eb35
0113 02:51:41.624818 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:41.624912 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1317897}
I20250113 02:51:41.630208 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "438604e8859f422f9caccb61c5b5b125" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:41.625538 1194 consensus_queue.cc:260] T 33c665e149dd48e8868acd935de581bf P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:41.634567 1192 tablet_bootstrap.cc:492] T 438604e8859f422f9caccb61c5b5b125 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:41.634526 775 raft_consensus.cc:2463] T b815003a27f64809baaa2d43d6293cef P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 1.
I20250113 02:51:41.635432 1194 ts_tablet_manager.cc:1428] T 33c665e149dd48e8868acd935de581bf P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.162s user 0.019s sys 0.002s
I20250113 02:51:41.635753 1194 tablet_bootstrap.cc:492] T a92f9b1c9c1a4cbdbf0cc7b2a3f2275b P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:41.636473 1192 tablet_bootstrap.cc:654] T 438604e8859f422f9caccb61c5b5b125 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:41.637518 1194 tablet_bootstrap.cc:654] T a92f9b1c9c1a4cbdbf0cc7b2a3f2275b P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:41.637449 1191 raft_consensus.cc:357] T 16a132384b0e4264a73d09a3ea133893 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:41.637673 995 leader_election.cc:304] T b815003a27f64809baaa2d43d6293cef P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters: df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:41.637740 1191 raft_consensus.cc:383] T 16a132384b0e4264a73d09a3ea133893 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:41.637926 2306 raft_consensus.cc:2798] T b815003a27f64809baaa2d43d6293cef P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Leader election won for term 1
I20250113 02:51:41.638046 2306 raft_consensus.cc:695] T b815003a27f64809baaa2d43d6293cef P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: Becoming Leader. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Running, Role: LEADER
I20250113 02:51:41.638288 2306 consensus_queue.cc:237] T b815003a27f64809baaa2d43d6293cef P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:41.638427 1196 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37366 (request call id 1168) took 1323 ms. Trace:
I20250113 02:51:41.638500 1196 rpcz_store.cc:276] 0113 02:51:40.315240 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:40.315298 (+ 58us) service_pool.cc:225] Handling call
0113 02:51:41.638302 (+1323004us) ts_tablet_manager.cc:1145] Deleting tablet 348c21d783ef4103a37409ef8634eb35
0113 02:51:41.638311 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:41.638409 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1322815}
I20250113 02:51:41.638948 1196 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37366 (request call id 1171) took 1281 ms. Trace:
I20250113 02:51:41.639030 1196 rpcz_store.cc:276] 0113 02:51:40.357516 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:40.357615 (+ 99us) service_pool.cc:225] Handling call
0113 02:51:41.638838 (+1281223us) ts_tablet_manager.cc:1145] Deleting tablet c368b5b0c3a940c4b4efb64c946375c3
0113 02:51:41.638846 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:41.638932 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1281034}
I20250113 02:51:41.639422 1196 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37366 (request call id 1172) took 1258 ms. Trace:
I20250113 02:51:41.639484 1196 rpcz_store.cc:276] 0113 02:51:40.380806 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:40.380869 (+ 63us) service_pool.cc:225] Handling call
0113 02:51:41.639332 (+1258463us) ts_tablet_manager.cc:1145] Deleting tablet e4998b86d32040ec959ff6fd94b1e9eb
0113 02:51:41.639342 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:41.639409 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1258268}
I20250113 02:51:41.639811 1196 tablet_replica.cc:331] T 999fecb3fc65466cad21cc1ff5ef2d44 P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:41.640014 1196 raft_consensus.cc:2238] T 999fecb3fc65466cad21cc1ff5ef2d44 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:41.640187 1196 raft_consensus.cc:2267] T 999fecb3fc65466cad21cc1ff5ef2d44 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:41.640702 1196 ts_tablet_manager.cc:1905] T 999fecb3fc65466cad21cc1ff5ef2d44 P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:41.640908 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_27eb308f2dc84858b3d1a2fa4b42c3f6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:41.641476 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_27eb308f2dc84858b3d1a2fa4b42c3f6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:41.642692 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_7ebd2aec3eb74fc29dccfd5d399dc4ea"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:41.643100 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7ebd2aec3eb74fc29dccfd5d399dc4ea in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:41.641607 2473 raft_consensus.cc:491] T 0839a35fcab04f3c9d03e256553e01c8 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:41.644037 2473 raft_consensus.cc:513] T 0839a35fcab04f3c9d03e256553e01c8 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:41.644381 449 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_25657b231b2d43b096996e80f8a35274" } modify_external_catalogs: true
I20250113 02:51:41.644526 449 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_25657b231b2d43b096996e80f8a35274" } modify_external_catalogs: true
I20250113 02:51:41.645268 1058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0839a35fcab04f3c9d03e256553e01c8" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
W20250113 02:51:41.645848 709 leader_election.cc:343] T 0839a35fcab04f3c9d03e256553e01c8 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:41.646003 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_fa2ea67525994579bd81a459396f4281" } new_table_name: "default.Table_accbd6c5b320439695fe35a2a7c09f40" modify_external_catalogs: true
I20250113 02:51:41.646200 2473 leader_election.cc:290] T 0839a35fcab04f3c9d03e256553e01c8 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:41.646588 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dfb0fee94d734f72ba354812f0e111aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:41.646684 1191 raft_consensus.cc:738] T 16a132384b0e4264a73d09a3ea133893 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:41.647141 1191 consensus_queue.cc:260] T 16a132384b0e4264a73d09a3ea133893 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:41.662165 774 raft_consensus.cc:2463] T b33025a462e74f2aa26cc3779f36615b P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 1.
I20250113 02:51:41.662770 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0839a35fcab04f3c9d03e256553e01c8" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:41.665855 1192 tablet_bootstrap.cc:492] T 438604e8859f422f9caccb61c5b5b125 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:41.665992 1192 ts_tablet_manager.cc:1397] T 438604e8859f422f9caccb61c5b5b125 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.031s user 0.007s sys 0.000s
I20250113 02:51:41.666729 1192 raft_consensus.cc:357] T 438604e8859f422f9caccb61c5b5b125 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:41.666903 1192 raft_consensus.cc:383] T 438604e8859f422f9caccb61c5b5b125 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:41.666947 1192 raft_consensus.cc:738] T 438604e8859f422f9caccb61c5b5b125 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:41.667143 1192 consensus_queue.cc:260] T 438604e8859f422f9caccb61c5b5b125 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:41.667753 1192 ts_tablet_manager.cc:1428] T 438604e8859f422f9caccb61c5b5b125 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250113 02:51:41.668067 1192 tablet_bootstrap.cc:492] T 0c408921a09a463eb464e08edcf042b3 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:41.669713 1192 tablet_bootstrap.cc:654] T 0c408921a09a463eb464e08edcf042b3 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:41.670442 882 tablet_service.cc:1467] Processing CreateTablet for tablet 6ba5a61a6aae4a23812503cc858f8957 (DEFAULT_TABLE table=default.Table_0e134d21b63b463ea086ed3ac6921486 [id=d15996f669224fe1a2d856b6c687931e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:41.670907 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ba5a61a6aae4a23812503cc858f8957. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:41.671914 995 leader_election.cc:304] T b33025a462e74f2aa26cc3779f36615b P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters: df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:41.672396 2487 raft_consensus.cc:2798] T b33025a462e74f2aa26cc3779f36615b P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Leader election won for term 1
I20250113 02:51:41.672498 2487 raft_consensus.cc:695] T b33025a462e74f2aa26cc3779f36615b P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: Becoming Leader. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Running, Role: LEADER
I20250113 02:51:41.672703 2487 consensus_queue.cc:237] T b33025a462e74f2aa26cc3779f36615b P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:41.662750 1191 ts_tablet_manager.cc:1428] T 16a132384b0e4264a73d09a3ea133893 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.128s user 0.001s sys 0.001s
I20250113 02:51:41.677153 1191 tablet_bootstrap.cc:492] T f105f8b32a654e4ca154e8fc0cb96091 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:41.665094 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1099) took 4478 ms. Trace:
I20250113 02:51:41.677434 1195 rpcz_store.cc:276] 0113 02:51:37.186445 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:37.187715 (+ 1270us) service_pool.cc:225] Handling call
0113 02:51:41.664961 (+4477246us) ts_tablet_manager.cc:1145] Deleting tablet fe3d35d9aaf64188aaa3c1ac38d279d8
0113 02:51:41.664971 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:41.665073 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4476887}
W20250113 02:51:41.677868 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet fe3d35d9aaf64188aaa3c1ac38d279d8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fe3d35d9aaf64188aaa3c1ac38d279d8 already in progress: creating tablet
I20250113 02:51:41.678519 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1102) took 4283 ms. Trace:
I20250113 02:51:41.678643 1195 rpcz_store.cc:276] 0113 02:51:37.395534 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:37.395579 (+ 45us) service_pool.cc:225] Handling call
0113 02:51:41.678359 (+4282780us) ts_tablet_manager.cc:1145] Deleting tablet e51b750440a24570b734bb51e7a6f923
0113 02:51:41.678374 (+ 15us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:41.678497 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4282598}
W20250113 02:51:41.679085 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet e51b750440a24570b734bb51e7a6f923 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e51b750440a24570b734bb51e7a6f923 already in progress: creating tablet
I20250113 02:51:41.679493 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1101) took 4284 ms. Trace:
I20250113 02:51:41.679627 1195 rpcz_store.cc:276] 0113 02:51:37.395405 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:37.395590 (+ 185us) service_pool.cc:225] Handling call
0113 02:51:41.679365 (+4283775us) ts_tablet_manager.cc:1145] Deleting tablet 93c0ce9f457e431193a2ceb0ad4d4af6
0113 02:51:41.679377 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:41.679474 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4282609}
W20250113 02:51:41.680119 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 93c0ce9f457e431193a2ceb0ad4d4af6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 93c0ce9f457e431193a2ceb0ad4d4af6 already in progress: creating tablet
I20250113 02:51:41.680541 1195 tablet_replica.cc:331] T 0752fde3a30a4947a2a688c56ceadab5 P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:41.680873 896 tablet_service.cc:1467] Processing CreateTablet for tablet 09a311cab8f04f01831bd18f95995b32 (DEFAULT_TABLE table=default.Table_c7dd7ce595fa467a916cd31cf34526c8 [id=30c66da7133c45f69df20177100d4b22]), partition=RANGE (key) PARTITION UNBOUNDED
W20250113 02:51:41.681144 711 leader_election.cc:343] T 438604e8859f422f9caccb61c5b5b125 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:41.681237 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 09a311cab8f04f01831bd18f95995b32. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:41.681272 711 leader_election.cc:304] T 438604e8859f422f9caccb61c5b5b125 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:41.681412 1195 raft_consensus.cc:2238] T 0752fde3a30a4947a2a688c56ceadab5 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:41.681536 2473 raft_consensus.cc:2743] T 438604e8859f422f9caccb61c5b5b125 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:41.681599 1195 raft_consensus.cc:2267] T 0752fde3a30a4947a2a688c56ceadab5 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:41.682793 1195 ts_tablet_manager.cc:1905] T 0752fde3a30a4947a2a688c56ceadab5 P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250113 02:51:41.687464 711 leader_election.cc:343] T 0839a35fcab04f3c9d03e256553e01c8 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:41.687574 711 leader_election.cc:304] T 0839a35fcab04f3c9d03e256553e01c8 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:41.688014 2473 raft_consensus.cc:2743] T 0839a35fcab04f3c9d03e256553e01c8 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:41.688092 882 tablet_service.cc:1467] Processing CreateTablet for tablet 711544894da147458c6a342883d7b83b (DEFAULT_TABLE table=default.Table_bf4806a5a68e4fcb98205b6fed9d0b5e [id=8bfd0e28c2ac404fb624b628675e5d84]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:41.688436 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 711544894da147458c6a342883d7b83b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:41.689167 1194 tablet_bootstrap.cc:492] T a92f9b1c9c1a4cbdbf0cc7b2a3f2275b P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:41.689244 1192 tablet_bootstrap.cc:492] T 0c408921a09a463eb464e08edcf042b3 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:41.689383 1192 ts_tablet_manager.cc:1397] T 0c408921a09a463eb464e08edcf042b3 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.000s
I20250113 02:51:41.689424 1194 ts_tablet_manager.cc:1397] T a92f9b1c9c1a4cbdbf0cc7b2a3f2275b P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.054s user 0.003s sys 0.000s
I20250113 02:51:41.690189 1192 raft_consensus.cc:357] T 0c408921a09a463eb464e08edcf042b3 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:41.690400 1192 raft_consensus.cc:383] T 0c408921a09a463eb464e08edcf042b3 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:41.690459 1192 raft_consensus.cc:738] T 0c408921a09a463eb464e08edcf042b3 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:41.690265 1194 raft_consensus.cc:357] T a92f9b1c9c1a4cbdbf0cc7b2a3f2275b P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:41.690608 1194 raft_consensus.cc:383] T a92f9b1c9c1a4cbdbf0cc7b2a3f2275b P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:41.690694 1194 raft_consensus.cc:738] T a92f9b1c9c1a4cbdbf0cc7b2a3f2275b P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:41.690670 1192 consensus_queue.cc:260] T 0c408921a09a463eb464e08edcf042b3 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:41.690958 1194 consensus_queue.cc:260] T a92f9b1c9c1a4cbdbf0cc7b2a3f2275b P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:41.691273 1192 ts_tablet_manager.cc:1428] T 0c408921a09a463eb464e08edcf042b3 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250113 02:51:41.691555 1192 tablet_bootstrap.cc:492] T 3d85539cf0074a03b167d3279e940079 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:41.691744 1194 ts_tablet_manager.cc:1428] T a92f9b1c9c1a4cbdbf0cc7b2a3f2275b P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250113 02:51:41.692065 1194 tablet_bootstrap.cc:492] T e70658c90555463498d3fc2e2f13ea7f P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:41.692369 748 tablet_service.cc:1467] Processing CreateTablet for tablet dfb0fee94d734f72ba354812f0e111aa (DEFAULT_TABLE table=default.Table_851855dfd1934ca38a5714adabc71199 [id=b1215be81ae44fb99bbc5dda27e605ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:41.693012 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dfb0fee94d734f72ba354812f0e111aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:41.693313 1192 tablet_bootstrap.cc:654] T 3d85539cf0074a03b167d3279e940079 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:41.693359 1194 tablet_bootstrap.cc:654] T e70658c90555463498d3fc2e2f13ea7f P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:41.702407 1192 tablet_bootstrap.cc:492] T 3d85539cf0074a03b167d3279e940079 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:41.702555 1192 ts_tablet_manager.cc:1397] T 3d85539cf0074a03b167d3279e940079 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20250113 02:51:41.703358 1192 raft_consensus.cc:357] T 3d85539cf0074a03b167d3279e940079 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:41.703554 1192 raft_consensus.cc:383] T 3d85539cf0074a03b167d3279e940079 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:41.703609 1192 raft_consensus.cc:738] T 3d85539cf0074a03b167d3279e940079 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:41.703856 1192 consensus_queue.cc:260] T 3d85539cf0074a03b167d3279e940079 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:41.704036 1196 ts_tablet_manager.cc:1918] T 999fecb3fc65466cad21cc1ff5ef2d44 P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:41.704159 1196 log.cc:1198] T 999fecb3fc65466cad21cc1ff5ef2d44 P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/999fecb3fc65466cad21cc1ff5ef2d44
I20250113 02:51:41.704465 1196 ts_tablet_manager.cc:1939] T 999fecb3fc65466cad21cc1ff5ef2d44 P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:41.704522 1192 ts_tablet_manager.cc:1428] T 3d85539cf0074a03b167d3279e940079 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250113 02:51:41.704767 1192 tablet_bootstrap.cc:492] T 64aa8390b1b1475c8743b6491d40674c P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:41.704792 1196 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37366 (request call id 1196) took 1138 ms. Trace:
I20250113 02:51:41.704876 1196 rpcz_store.cc:276] 0113 02:51:40.566326 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:40.709053 (+142727us) service_pool.cc:225] Handling call
0113 02:51:41.639769 (+930716us) ts_tablet_manager.cc:1145] Deleting tablet 999fecb3fc65466cad21cc1ff5ef2d44
0113 02:51:41.639777 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:41.643631 (+ 3854us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:41.703905 (+ 60274us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:41.704771 (+ 866us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":895684}
I20250113 02:51:41.706308 1192 tablet_bootstrap.cc:654] T 64aa8390b1b1475c8743b6491d40674c P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:41.708283 1196 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37366 (request call id 1204) took 1042 ms. Trace:
I20250113 02:51:41.708423 896 tablet_service.cc:1467] Processing CreateTablet for tablet 192a226332e64766a0e367c6fc1c1673 (DEFAULT_TABLE table=default.Table_64fa660da06b4c7cb2c76054b3a4ceba [id=7360924dbe7547e1b6e7a2f6f7404642]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:41.708911 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 192a226332e64766a0e367c6fc1c1673. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:41.708381 1196 rpcz_store.cc:276] 0113 02:51:40.665967 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:40.875088 (+209121us) service_pool.cc:225] Handling call
0113 02:51:41.708117 (+833029us) ts_tablet_manager.cc:1145] Deleting tablet eceed0f52c9f46bbaa78ca773a7afde5
0113 02:51:41.708131 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:41.708260 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":832851}
I20250113 02:51:41.709883 1196 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37366 (request call id 1219) took 1042 ms. Trace:
I20250113 02:51:41.709982 1196 rpcz_store.cc:276] 0113 02:51:40.667047 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:40.993814 (+326767us) service_pool.cc:225] Handling call
0113 02:51:41.709764 (+715950us) ts_tablet_manager.cc:1145] Deleting tablet 42a8e01bb71f49b09c83a8f5c905f38d
0113 02:51:41.709775 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:41.709865 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":712271}
I20250113 02:51:41.710425 1196 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37366 (request call id 1220) took 1043 ms. Trace:
I20250113 02:51:41.710503 1196 rpcz_store.cc:276] 0113 02:51:40.667112 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:40.997507 (+330395us) service_pool.cc:225] Handling call
0113 02:51:41.710334 (+712827us) ts_tablet_manager.cc:1145] Deleting tablet 94307d22b7f842ca89844768e4a2a2ab
0113 02:51:41.710341 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:41.710411 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":712713}
I20250113 02:51:41.710822 1192 tablet_bootstrap.cc:492] T 64aa8390b1b1475c8743b6491d40674c P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:41.710870 1196 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37366 (request call id 1221) took 1043 ms. Trace:
I20250113 02:51:41.710944 1196 rpcz_store.cc:276] 0113 02:51:40.667183 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:40.997628 (+330445us) service_pool.cc:225] Handling call
0113 02:51:41.710786 (+713158us) ts_tablet_manager.cc:1145] Deleting tablet a0c14501ad0449aca28909bc14428dad
0113 02:51:41.710794 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:41.710858 (+ 64us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":713071}
I20250113 02:51:41.711522 1192 ts_tablet_manager.cc:1397] T 64aa8390b1b1475c8743b6491d40674c P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.002s
I20250113 02:51:41.712281 1192 raft_consensus.cc:357] T 64aa8390b1b1475c8743b6491d40674c P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:41.712487 1192 raft_consensus.cc:383] T 64aa8390b1b1475c8743b6491d40674c P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:41.712548 1192 raft_consensus.cc:738] T 64aa8390b1b1475c8743b6491d40674c P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:41.712612 1196 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37366 (request call id 1222) took 1045 ms. Trace:
I20250113 02:51:41.712687 1196 rpcz_store.cc:276] 0113 02:51:40.667248 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:40.997719 (+330471us) service_pool.cc:225] Handling call
0113 02:51:41.712519 (+714800us) ts_tablet_manager.cc:1145] Deleting tablet 6001369444894b48b51a48fb60ddc52a
0113 02:51:41.712527 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:41.712598 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":704110}
I20250113 02:51:41.712774 1192 consensus_queue.cc:260] T 64aa8390b1b1475c8743b6491d40674c P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:41.712875 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_bea21867cb0743d98a95f39f6fc76cc2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:41.713426 1192 ts_tablet_manager.cc:1428] T 64aa8390b1b1475c8743b6491d40674c P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
W20250113 02:51:41.713487 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bea21867cb0743d98a95f39f6fc76cc2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:41.713655 1192 tablet_bootstrap.cc:492] T 1d2fdc75d00f4e749d322ac6cd5fc410 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:41.714789 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_507a4e0b009b4d6ca0fce33fde33c90c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:41.715118 1192 tablet_bootstrap.cc:654] T 1d2fdc75d00f4e749d322ac6cd5fc410 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
W20250113 02:51:41.715190 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_507a4e0b009b4d6ca0fce33fde33c90c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:41.715358 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_a7b637f0a6a244fdb2172e2015816fce" } new_table_name: "default.Table_c3ba3f6047274ab885a4c853b06e1417" modify_external_catalogs: true
I20250113 02:51:41.716619 1196 tablet_replica.cc:331] T 438d1d2300e044f39a15586f335935ba P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:41.716804 1196 raft_consensus.cc:2238] T 438d1d2300e044f39a15586f335935ba P df69f48656b44bb890f76587ee69cc0f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:41.716979 1196 raft_consensus.cc:2267] T 438d1d2300e044f39a15586f335935ba P df69f48656b44bb890f76587ee69cc0f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:41.712558 446 catalog_manager.cc:5618] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467) does not have the latest schema for tablet 7527e6e0b65e4a43b7f48899a34cdf7f (table default.Table_c8b93363989c4e109540af6bef426f73 [id=b7846a8ebee84005b1b5d259560a26bd]). Expected version 1 got 0
I20250113 02:51:41.717355 1194 tablet_bootstrap.cc:492] T e70658c90555463498d3fc2e2f13ea7f P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:41.719046 1192 tablet_bootstrap.cc:492] T 1d2fdc75d00f4e749d322ac6cd5fc410 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:41.719168 1192 ts_tablet_manager.cc:1397] T 1d2fdc75d00f4e749d322ac6cd5fc410 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.003s
I20250113 02:51:41.719336 1194 ts_tablet_manager.cc:1397] T e70658c90555463498d3fc2e2f13ea7f P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.027s user 0.003s sys 0.000s
I20250113 02:51:41.717522 1196 ts_tablet_manager.cc:1905] T 438d1d2300e044f39a15586f335935ba P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:41.719842 1192 raft_consensus.cc:357] T 1d2fdc75d00f4e749d322ac6cd5fc410 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:41.720050 1192 raft_consensus.cc:383] T 1d2fdc75d00f4e749d322ac6cd5fc410 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:41.720134 1192 raft_consensus.cc:738] T 1d2fdc75d00f4e749d322ac6cd5fc410 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:41.720386 1192 consensus_queue.cc:260] T 1d2fdc75d00f4e749d322ac6cd5fc410 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:41.720876 888 tablet_service.cc:1467] Processing CreateTablet for tablet b98da6f197764465a1e10aaadcddf3a4 (DEFAULT_TABLE table=default.Table_d5a79bb9160448269ae65f45fcd592a9 [id=c25a4dff6adb47c9a6445dc72a0d8546]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:41.721210 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b98da6f197764465a1e10aaadcddf3a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:41.721489 1192 ts_tablet_manager.cc:1428] T 1d2fdc75d00f4e749d322ac6cd5fc410 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250113 02:51:41.721742 1192 tablet_bootstrap.cc:492] T acd83024592e431898e9675c17f50d4f P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:41.723104 881 tablet_service.cc:1514] Processing DeleteTablet for tablet c8966b3b93144a30813bfdaf04f9c43a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:41 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:41.723747 1192 tablet_bootstrap.cc:654] T acd83024592e431898e9675c17f50d4f P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:41.724217 898 tablet_service.cc:1467] Processing CreateTablet for tablet 0e3231aafc4e49aabaef5f34db272659 (DEFAULT_TABLE table=default.Table_255be37d0d3e4cfca070e8137a82f90b [id=3c9c08212c044e63b9b0263ca43270ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:41.724525 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e3231aafc4e49aabaef5f34db272659. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:41.724490 1194 raft_consensus.cc:357] T e70658c90555463498d3fc2e2f13ea7f P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:41.724648 1194 raft_consensus.cc:383] T e70658c90555463498d3fc2e2f13ea7f P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:41.724692 1194 raft_consensus.cc:738] T e70658c90555463498d3fc2e2f13ea7f P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:41.724876 1194 consensus_queue.cc:260] T e70658c90555463498d3fc2e2f13ea7f P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:41.725077 881 tablet_service.cc:1467] Processing CreateTablet for tablet dfb0fee94d734f72ba354812f0e111aa (DEFAULT_TABLE table=default.Table_851855dfd1934ca38a5714adabc71199 [id=b1215be81ae44fb99bbc5dda27e605ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:41.725490 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dfb0fee94d734f72ba354812f0e111aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:41.727646 880 tablet_service.cc:1467] Processing CreateTablet for tablet ca6f77f97a2343f49dccb54361fb0cea (DEFAULT_TABLE table=default.Table_dcdc0cc411724eba84ac9af713fb17e3 [id=912e959477af4c90a1e1773c7edd69e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:41.727931 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca6f77f97a2343f49dccb54361fb0cea. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:41.728299 1194 ts_tablet_manager.cc:1428] T e70658c90555463498d3fc2e2f13ea7f P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.008s user 0.001s sys 0.000s
I20250113 02:51:41.728552 1194 tablet_bootstrap.cc:492] T 7213e3284b4d4cb581d9fb946a9f87e9 P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:41.730167 897 tablet_service.cc:1467] Processing CreateTablet for tablet b2a3477d01c444ef8e346d28ede09267 (DEFAULT_TABLE table=default.Table_4bb227dc715148c9862b2d999509533a [id=48cada9c7be54533b989bc4d360b43e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:41.730595 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b2a3477d01c444ef8e346d28ede09267. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:41.730929 899 tablet_service.cc:1467] Processing CreateTablet for tablet 99843ce7f1bf41da9b7cb5bd7ac2e875 (DEFAULT_TABLE table=default.Table_ed460a21a2184db893d2e76e67cdb974 [id=fb3d90554f7943099cd541f610084cbf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:41.731225 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 99843ce7f1bf41da9b7cb5bd7ac2e875. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:41.731614 1192 tablet_bootstrap.cc:492] T acd83024592e431898e9675c17f50d4f P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:41.731734 1192 ts_tablet_manager.cc:1397] T acd83024592e431898e9675c17f50d4f P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.007s
I20250113 02:51:41.732463 1192 raft_consensus.cc:357] T acd83024592e431898e9675c17f50d4f P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:41.732632 1192 raft_consensus.cc:383] T acd83024592e431898e9675c17f50d4f P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:41.732687 1192 raft_consensus.cc:738] T acd83024592e431898e9675c17f50d4f P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:41.732906 1192 consensus_queue.cc:260] T acd83024592e431898e9675c17f50d4f P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:41.733835 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_cef3f1e1be374b87a39f67fb569234e8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:41.734257 887 tablet_service.cc:1467] Processing CreateTablet for tablet 568fae43734b4d3f81802e01251c6a7a (DEFAULT_TABLE table=default.Table_588573aeb9464e8088f6759b1ca47c54 [id=0d5f79f41c8b4eae9ee13e055d577525]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:41.734283 885 tablet_service.cc:1467] Processing CreateTablet for tablet 12707a38496c464a859fa294cf6bbe36 (DEFAULT_TABLE table=default.Table_3e43331bf36845f9aa983823e4853d04 [id=6e23db586a4740a38379b905fdf40c13]), partition=RANGE (key) PARTITION UNBOUNDED
W20250113 02:51:41.734381 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cef3f1e1be374b87a39f67fb569234e8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:41.734602 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 568fae43734b4d3f81802e01251c6a7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:41.734637 891 tablet_service.cc:1467] Processing CreateTablet for tablet 86c8c07f90d2470ebb2b3f1d16daf4cd (DEFAULT_TABLE table=default.Table_32fe238c17574c1ea5721f507c0bdcc9 [id=394f1a3b62704790a2a3cee6d5a06f82]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:41.734964 883 tablet_service.cc:1467] Processing CreateTablet for tablet d3619576c8f94583b330fe1828cea2e0 (DEFAULT_TABLE table=default.Table_bdab3234b9d44a7fb534b8479b322f4f [id=d80ff8c583684dc59f8960b4a6fe8810]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:41.735294 889 tablet_service.cc:1467] Processing CreateTablet for tablet b01dcbb7e28c4c1fbc389f829cead09e (DEFAULT_TABLE table=default.Table_9458e2e0d9a446378600ea1ef1696bd5 [id=1d5ff280114343d691657c188fb6a111]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:41.735595 895 tablet_service.cc:1467] Processing CreateTablet for tablet b946dd2d428a43d399bb75a43d843a28 (DEFAULT_TABLE table=default.Table_d5096557ae6445e6bcd3b6a9b3299e8b [id=b56ce78e920f464fbeab248484d6f2f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:41.735901 893 tablet_service.cc:1467] Processing CreateTablet for tablet f6f4df3bde434667a923be19aa39817a (DEFAULT_TABLE table=default.Table_4078d475fb2e4e52ad184b9318ec9c7d [id=6e6f2eaf51294f6a8a1228cc66b89575]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:41.736582 1192 ts_tablet_manager.cc:1428] T acd83024592e431898e9675c17f50d4f P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250113 02:51:41.736882 1192 tablet_bootstrap.cc:492] T c13e9a51c1e74b44b352bade78390ba5 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:41.737802 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 86c8c07f90d2470ebb2b3f1d16daf4cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:41.738489 1192 tablet_bootstrap.cc:654] T c13e9a51c1e74b44b352bade78390ba5 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:41.740175 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d3619576c8f94583b330fe1828cea2e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:41.741072 2306 raft_consensus.cc:491] T ba8980ad3a364c888a825affbcbc9b1f P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:41.741189 2306 raft_consensus.cc:513] T ba8980ad3a364c888a825affbcbc9b1f P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:41.741746 2306 leader_election.cc:290] T ba8980ad3a364c888a825affbcbc9b1f P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:41.744343 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b01dcbb7e28c4c1fbc389f829cead09e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:41.748215 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b946dd2d428a43d399bb75a43d843a28. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:41.751120 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ccc9ae0461cd40a38cb3a90412df7e82 consensus state has no leader
I20250113 02:51:41.752126 451 master_service.cc:946] ReplaceTablet: received request to replace tablet 283fc7f1c74e480c8a63c74d3ae9eb7d from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:41.752291 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f6f4df3bde434667a923be19aa39817a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:41.753151 1194 tablet_bootstrap.cc:654] T 7213e3284b4d4cb581d9fb946a9f87e9 P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:41.753620 452 master_service.cc:946] ReplaceTablet: received request to replace tablet d8bb60fdd06946739b75b255303d9d24 from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:41.756116 2487 raft_consensus.cc:491] T 82505f1389184d0ca12ec0e2d6084d22 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:41.756243 2487 raft_consensus.cc:513] T 82505f1389184d0ca12ec0e2d6084d22 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:41.756850 2487 leader_election.cc:290] T 82505f1389184d0ca12ec0e2d6084d22 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:41.757555 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82505f1389184d0ca12ec0e2d6084d22" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
I20250113 02:51:41.757754 772 raft_consensus.cc:2463] T 82505f1389184d0ca12ec0e2d6084d22 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 0.
I20250113 02:51:41.758152 995 leader_election.cc:304] T 82505f1389184d0ca12ec0e2d6084d22 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters:
I20250113 02:51:41.758450 2487 raft_consensus.cc:2798] T 82505f1389184d0ca12ec0e2d6084d22 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 02:51:41.758457 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 12707a38496c464a859fa294cf6bbe36. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:41.758520 2487 raft_consensus.cc:491] T 82505f1389184d0ca12ec0e2d6084d22 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 02:51:41.758579 2487 raft_consensus.cc:3054] T 82505f1389184d0ca12ec0e2d6084d22 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:41.761247 2487 raft_consensus.cc:513] T 82505f1389184d0ca12ec0e2d6084d22 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:41.762187 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82505f1389184d0ca12ec0e2d6084d22" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196"
I20250113 02:51:41.762374 772 raft_consensus.cc:3054] T 82505f1389184d0ca12ec0e2d6084d22 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:41.762990 2487 leader_election.cc:290] T 82505f1389184d0ca12ec0e2d6084d22 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Requested vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:41.763911 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba8980ad3a364c888a825affbcbc9b1f" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:41.764400 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82505f1389184d0ca12ec0e2d6084d22" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:41.766547 998 leader_election.cc:343] T ba8980ad3a364c888a825affbcbc9b1f P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:41.766882 1196 ts_tablet_manager.cc:1918] T 438d1d2300e044f39a15586f335935ba P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250113 02:51:41.766978 1196 log.cc:1198] T 438d1d2300e044f39a15586f335935ba P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/438d1d2300e044f39a15586f335935ba
I20250113 02:51:41.767342 1196 ts_tablet_manager.cc:1939] T 438d1d2300e044f39a15586f335935ba P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
W20250113 02:51:41.768337 998 leader_election.cc:343] T 82505f1389184d0ca12ec0e2d6084d22 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:41.768769 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82505f1389184d0ca12ec0e2d6084d22" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f"
W20250113 02:51:41.771324 998 leader_election.cc:343] T 82505f1389184d0ca12ec0e2d6084d22 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:41.772969 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 316cb2863ed645158a8f72e3cb0a94fd
I20250113 02:51:41.773831 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet 316cb2863ed645158a8f72e3cb0a94fd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:41 UTC) from {username='slave'} at 127.0.0.1:60230
W20250113 02:51:41.776604 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 17f02be32ddb44c089266fd9f0378f77 consensus state has no leader
I20250113 02:51:41.782425 2476 raft_consensus.cc:491] T a92f9b1c9c1a4cbdbf0cc7b2a3f2275b P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:41.782569 2476 raft_consensus.cc:513] T a92f9b1c9c1a4cbdbf0cc7b2a3f2275b P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:41.783128 2476 leader_election.cc:290] T a92f9b1c9c1a4cbdbf0cc7b2a3f2275b P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
W20250113 02:51:41.784377 852 leader_election.cc:343] T a92f9b1c9c1a4cbdbf0cc7b2a3f2275b P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Not found: Tablet not found: a92f9b1c9c1a4cbdbf0cc7b2a3f2275b
I20250113 02:51:41.785101 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 4cc15487be474354b995c4e415276a18
I20250113 02:51:41.785797 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet 4cc15487be474354b995c4e415276a18 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:41 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:41.785997 738 tablet_service.cc:1514] Processing DeleteTablet for tablet 4cc15487be474354b995c4e415276a18 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:41 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:41.787081 1194 tablet_bootstrap.cc:492] T 7213e3284b4d4cb581d9fb946a9f87e9 P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:41.787273 1194 ts_tablet_manager.cc:1397] T 7213e3284b4d4cb581d9fb946a9f87e9 P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.059s user 0.005s sys 0.002s
I20250113 02:51:41.788014 1194 raft_consensus.cc:357] T 7213e3284b4d4cb581d9fb946a9f87e9 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:41.788251 1194 raft_consensus.cc:383] T 7213e3284b4d4cb581d9fb946a9f87e9 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:41.788352 1194 raft_consensus.cc:738] T 7213e3284b4d4cb581d9fb946a9f87e9 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:41.788647 2476 raft_consensus.cc:491] T 0916b41c171a419791797559fbc2807d P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:41.788591 1194 consensus_queue.cc:260] T 7213e3284b4d4cb581d9fb946a9f87e9 P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:41.788750 2476 raft_consensus.cc:513] T 0916b41c171a419791797559fbc2807d P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:41.789324 2476 leader_election.cc:290] T 0916b41c171a419791797559fbc2807d P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
W20250113 02:51:41.790877 852 leader_election.cc:343] T 0916b41c171a419791797559fbc2807d P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Not found: Tablet not found: 0916b41c171a419791797559fbc2807d
I20250113 02:51:41.794159 1194 ts_tablet_manager.cc:1428] T 7213e3284b4d4cb581d9fb946a9f87e9 P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20250113 02:51:41.794471 1194 tablet_bootstrap.cc:492] T 0b119704bc774ca9b4155c9619d4765b P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:41.795802 1194 tablet_bootstrap.cc:654] T 0b119704bc774ca9b4155c9619d4765b P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:41.801155 1198 ts_tablet_manager.cc:1918] T 30ca8a716bfc4ab789b216e17567785e P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:41.801438 1198 log.cc:1198] T 30ca8a716bfc4ab789b216e17567785e P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/30ca8a716bfc4ab789b216e17567785e
I20250113 02:51:41.804657 1034 tablet_service.cc:1467] Processing CreateTablet for tablet 1511a0ec646549239f6cfe0fc2bdf9f0 (DEFAULT_TABLE table=default.Table_302b50fc67d44fa79ef59d7cc8fbc408 [id=e3422649c46443b1b91579cfcd6070b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:41.805153 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1511a0ec646549239f6cfe0fc2bdf9f0. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:41.806262 995 leader_election.cc:343] T ba8980ad3a364c888a825affbcbc9b1f P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Not found: Tablet not found: ba8980ad3a364c888a825affbcbc9b1f
I20250113 02:51:41.806432 995 leader_election.cc:304] T ba8980ad3a364c888a825affbcbc9b1f P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4216a7f853d642b7bc3e5cff458f3aca; no voters: 1186fa869eb14812832fa6562531f196, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:41.806733 2487 raft_consensus.cc:2743] T ba8980ad3a364c888a825affbcbc9b1f P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:41.814208 2473 raft_consensus.cc:491] T 0c408921a09a463eb464e08edcf042b3 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:41.814366 2473 raft_consensus.cc:513] T 0c408921a09a463eb464e08edcf042b3 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:41.815411 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c408921a09a463eb464e08edcf042b3" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:41.815898 711 leader_election.cc:343] T 0c408921a09a463eb464e08edcf042b3 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:41.823849 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 85d13f8abbec484d82c4d0b86480531b
I20250113 02:51:41.824976 2476 raft_consensus.cc:491] T 33c665e149dd48e8868acd935de581bf P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:41.825142 2476 raft_consensus.cc:513] T 33c665e149dd48e8868acd935de581bf P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:41.825570 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet 85d13f8abbec484d82c4d0b86480531b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:41 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:41.825913 2476 leader_election.cc:290] T 33c665e149dd48e8868acd935de581bf P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
W20250113 02:51:41.827268 852 leader_election.cc:343] T 33c665e149dd48e8868acd935de581bf P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Not found: Tablet not found: 33c665e149dd48e8868acd935de581bf
I20250113 02:51:41.834802 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 8ee0b2e06dc74faab48912dd16fe1537
I20250113 02:51:41.835309 454 catalog_manager.cc:6596] ReplaceTablet: tablet 8ee0b2e06dc74faab48912dd16fe1537 deleted and replaced by tablet 3f65c4ebb7df4637bc580fffd094a87a
I20250113 02:51:41.836136 739 tablet_service.cc:1514] Processing DeleteTablet for tablet 85d13f8abbec484d82c4d0b86480531b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:41 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:41.837143 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ee0b2e06dc74faab48912dd16fe1537 with delete_type TABLET_DATA_DELETED (replaced by tablet 3f65c4ebb7df4637bc580fffd094a87a) from {username='slave'} at 127.0.0.1:60230
W20250113 02:51:41.845016 851 leader_election.cc:343] T a92f9b1c9c1a4cbdbf0cc7b2a3f2275b P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Not found: Tablet not found: a92f9b1c9c1a4cbdbf0cc7b2a3f2275b
I20250113 02:51:41.845151 851 leader_election.cc:304] T a92f9b1c9c1a4cbdbf0cc7b2a3f2275b P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df69f48656b44bb890f76587ee69cc0f; no voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca
W20250113 02:51:41.845682 851 leader_election.cc:343] T 33c665e149dd48e8868acd935de581bf P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Not found: Tablet not found: 33c665e149dd48e8868acd935de581bf
I20250113 02:51:41.845777 851 leader_election.cc:304] T 33c665e149dd48e8868acd935de581bf P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df69f48656b44bb890f76587ee69cc0f; no voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca
I20250113 02:51:41.846607 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 9b9e7deda7334ff4a3429f70d9b3a7fc
I20250113 02:51:41.848548 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b9e7deda7334ff4a3429f70d9b3a7fc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:41 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:41.849047 1058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c408921a09a463eb464e08edcf042b3" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:41.849277 1058 raft_consensus.cc:2463] T 0c408921a09a463eb464e08edcf042b3 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1186fa869eb14812832fa6562531f196 in term 0.
I20250113 02:51:41.851192 2529 raft_consensus.cc:2743] T a92f9b1c9c1a4cbdbf0cc7b2a3f2275b P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:41.851625 2529 raft_consensus.cc:2743] T 33c665e149dd48e8868acd935de581bf P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250113 02:51:41.861461 851 leader_election.cc:343] T 0916b41c171a419791797559fbc2807d P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Not found: Tablet not found: 0916b41c171a419791797559fbc2807d
I20250113 02:51:41.861584 851 leader_election.cc:304] T 0916b41c171a419791797559fbc2807d P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df69f48656b44bb890f76587ee69cc0f; no voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca
I20250113 02:51:41.861930 2476 raft_consensus.cc:2743] T 0916b41c171a419791797559fbc2807d P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:41.862783 1194 tablet_bootstrap.cc:492] T 0b119704bc774ca9b4155c9619d4765b P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:41.862921 1194 ts_tablet_manager.cc:1397] T 0b119704bc774ca9b4155c9619d4765b P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.069s user 0.011s sys 0.013s
I20250113 02:51:41.863554 1194 raft_consensus.cc:357] T 0b119704bc774ca9b4155c9619d4765b P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:41.863734 1194 raft_consensus.cc:383] T 0b119704bc774ca9b4155c9619d4765b P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:41.863780 1194 raft_consensus.cc:738] T 0b119704bc774ca9b4155c9619d4765b P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:41.864037 1194 consensus_queue.cc:260] T 0b119704bc774ca9b4155c9619d4765b P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:41.867453 449 catalog_manager.cc:5618] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467) does not have the latest schema for tablet 0b119704bc774ca9b4155c9619d4765b (table default.Table_b95a9b1d12284c9aac701a8ef9c93482 [id=843aabcff7074acd98e5801b21282faa]). Expected version 1 got 0
I20250113 02:51:41.836809 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ee0b2e06dc74faab48912dd16fe1537 with delete_type TABLET_DATA_DELETED (replaced by tablet 3f65c4ebb7df4637bc580fffd094a87a) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:41.867726 1194 ts_tablet_manager.cc:1428] T 0b119704bc774ca9b4155c9619d4765b P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250113 02:51:41.846617 1191 tablet_bootstrap.cc:654] T f105f8b32a654e4ca154e8fc0cb96091 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:41.868021 1194 tablet_bootstrap.cc:492] T 4eb5f1944cb44fd5a40ca92a6a3d7214 P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:41.814924 2473 leader_election.cc:290] T 0c408921a09a463eb464e08edcf042b3 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:41.847800 738 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b9e7deda7334ff4a3429f70d9b3a7fc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:41 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:41.848058 751 tablet_service.cc:1467] Processing CreateTablet for tablet 1511a0ec646549239f6cfe0fc2bdf9f0 (DEFAULT_TABLE table=default.Table_302b50fc67d44fa79ef59d7cc8fbc408 [id=e3422649c46443b1b91579cfcd6070b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:41.848212 2501 raft_consensus.cc:491] T 378d59d314a84ddb84cf2ab041c51cf1 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4216a7f853d642b7bc3e5cff458f3aca)
I20250113 02:51:41.869076 2501 raft_consensus.cc:513] T 378d59d314a84ddb84cf2ab041c51cf1 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:41.869624 1194 tablet_bootstrap.cc:654] T 4eb5f1944cb44fd5a40ca92a6a3d7214 P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:41.869801 2501 leader_election.cc:290] T 378d59d314a84ddb84cf2ab041c51cf1 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:41.848340 2481 tablet.cc:1721] T 6e1284c8aaba44b5810b0d06d36757eb P 1186fa869eb14812832fa6562531f196: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250113 02:51:41.850724 709 leader_election.cc:304] T 0c408921a09a463eb464e08edcf042b3 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters: df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:41.871183 913 raft_consensus.cc:3054] T 7527e6e0b65e4a43b7f48899a34cdf7f P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:41.871313 2473 raft_consensus.cc:2798] T 0c408921a09a463eb464e08edcf042b3 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 02:51:41.871407 2473 raft_consensus.cc:491] T 0c408921a09a463eb464e08edcf042b3 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 02:51:41.871480 2473 raft_consensus.cc:3054] T 0c408921a09a463eb464e08edcf042b3 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:41.875228 2499 tablet.cc:1721] T 7527e6e0b65e4a43b7f48899a34cdf7f P df69f48656b44bb890f76587ee69cc0f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250113 02:51:41.860059 2508 raft_consensus.cc:491] T 1d2fdc75d00f4e749d322ac6cd5fc410 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:41.859000 1195 ts_tablet_manager.cc:1918] T 0752fde3a30a4947a2a688c56ceadab5 P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250113 02:51:41.861042 754 tablet_service.cc:1514] Processing DeleteTablet for tablet 316cb2863ed645158a8f72e3cb0a94fd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:41 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:41.876113 2508 raft_consensus.cc:513] T 1d2fdc75d00f4e749d322ac6cd5fc410 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:41.868980 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1511a0ec646549239f6cfe0fc2bdf9f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:41.876718 2508 leader_election.cc:290] T 1d2fdc75d00f4e749d322ac6cd5fc410 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
W20250113 02:51:41.877645 711 leader_election.cc:343] T 378d59d314a84ddb84cf2ab041c51cf1 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Not found: Tablet not found: 378d59d314a84ddb84cf2ab041c51cf1
I20250113 02:51:41.879710 1195 log.cc:1198] T 0752fde3a30a4947a2a688c56ceadab5 P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/0752fde3a30a4947a2a688c56ceadab5
I20250113 02:51:41.880332 449 catalog_manager.cc:5618] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467) does not have the latest schema for tablet 0b119704bc774ca9b4155c9619d4765b (table default.Table_b95a9b1d12284c9aac701a8ef9c93482 [id=843aabcff7074acd98e5801b21282faa]). Expected version 1 got 0
I20250113 02:51:41.880357 1195 ts_tablet_manager.cc:1939] T 0752fde3a30a4947a2a688c56ceadab5 P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:41.880518 449 catalog_manager.cc:5618] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467) does not have the latest schema for tablet 7527e6e0b65e4a43b7f48899a34cdf7f (table default.Table_c8b93363989c4e109540af6bef426f73 [id=b7846a8ebee84005b1b5d259560a26bd]). Expected version 1 got 0
I20250113 02:51:41.883328 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d2fdc75d00f4e749d322ac6cd5fc410" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:41.883555 1059 raft_consensus.cc:2463] T 1d2fdc75d00f4e749d322ac6cd5fc410 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1186fa869eb14812832fa6562531f196 in term 0.
W20250113 02:51:41.884397 709 leader_election.cc:343] T 378d59d314a84ddb84cf2ab041c51cf1 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Not found: Tablet not found: 378d59d314a84ddb84cf2ab041c51cf1
I20250113 02:51:41.884464 2481 tablet.cc:1716] T 6e1284c8aaba44b5810b0d06d36757eb P 1186fa869eb14812832fa6562531f196: Skipping requested alter to schema version 2, tablet already version 2
I20250113 02:51:41.884516 709 leader_election.cc:304] T 378d59d314a84ddb84cf2ab041c51cf1 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:41.884902 709 leader_election.cc:304] T 1d2fdc75d00f4e749d322ac6cd5fc410 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters:
I20250113 02:51:41.885145 2501 raft_consensus.cc:2743] T 378d59d314a84ddb84cf2ab041c51cf1 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250113 02:51:41.885314 2501 raft_consensus.cc:2798] T 1d2fdc75d00f4e749d322ac6cd5fc410 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 02:51:41.885377 2501 raft_consensus.cc:491] T 1d2fdc75d00f4e749d322ac6cd5fc410 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 02:51:41.885438 2501 raft_consensus.cc:3054] T 1d2fdc75d00f4e749d322ac6cd5fc410 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:41.885735 2476 raft_consensus.cc:491] T eebad045e7bb43fa9ca9c9af8b3b7ea5 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:41.885879 2476 raft_consensus.cc:513] T eebad045e7bb43fa9ca9c9af8b3b7ea5 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:41.886511 2476 leader_election.cc:290] T eebad045e7bb43fa9ca9c9af8b3b7ea5 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:41.886713 893 tablet_service.cc:1467] Processing CreateTablet for tablet f6763009f1c14ed2b3b311a2da07ec9f (DEFAULT_TABLE table=default.Table_c1e2aac17dd047708ca98085186252a7 [id=c26157fba3744bad819883e90511bf19]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:41.887378 1194 tablet_bootstrap.cc:492] T 4eb5f1944cb44fd5a40ca92a6a3d7214 P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
W20250113 02:51:41.887501 851 leader_election.cc:343] T eebad045e7bb43fa9ca9c9af8b3b7ea5 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Not found: Tablet not found: eebad045e7bb43fa9ca9c9af8b3b7ea5
I20250113 02:51:41.887569 1194 ts_tablet_manager.cc:1397] T 4eb5f1944cb44fd5a40ca92a6a3d7214 P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.002s
W20250113 02:51:41.887687 852 leader_election.cc:343] T eebad045e7bb43fa9ca9c9af8b3b7ea5 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Not found: Tablet not found: eebad045e7bb43fa9ca9c9af8b3b7ea5
I20250113 02:51:41.887771 852 leader_election.cc:304] T eebad045e7bb43fa9ca9c9af8b3b7ea5 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df69f48656b44bb890f76587ee69cc0f; no voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca
I20250113 02:51:41.888053 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f6763009f1c14ed2b3b311a2da07ec9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:41.888128 2476 raft_consensus.cc:2743] T eebad045e7bb43fa9ca9c9af8b3b7ea5 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:41.888239 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d2fdc75d00f4e749d322ac6cd5fc410" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:41.888698 711 leader_election.cc:343] T 1d2fdc75d00f4e749d322ac6cd5fc410 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:41.888907 1194 raft_consensus.cc:357] T 4eb5f1944cb44fd5a40ca92a6a3d7214 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:41.889185 1194 raft_consensus.cc:383] T 4eb5f1944cb44fd5a40ca92a6a3d7214 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:41.889290 1194 raft_consensus.cc:738] T 4eb5f1944cb44fd5a40ca92a6a3d7214 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:41.889556 1194 consensus_queue.cc:260] T 4eb5f1944cb44fd5a40ca92a6a3d7214 P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:41.890164 1194 ts_tablet_manager.cc:1428] T 4eb5f1944cb44fd5a40ca92a6a3d7214 P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250113 02:51:41.890901 1194 tablet_bootstrap.cc:492] T f21eb3fa0f414816a29e3d4560cdc0fe P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:41.892906 1194 tablet_bootstrap.cc:654] T f21eb3fa0f414816a29e3d4560cdc0fe P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:41.909148 897 tablet_service.cc:1514] Processing DeleteTablet for tablet eebad045e7bb43fa9ca9c9af8b3b7ea5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:21 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:41.909453 1196 tablet_replica.cc:331] T eebad045e7bb43fa9ca9c9af8b3b7ea5 P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:41.909775 1194 tablet_bootstrap.cc:492] T f21eb3fa0f414816a29e3d4560cdc0fe P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:41.909937 883 tablet_service.cc:1514] Processing DeleteTablet for tablet 0916b41c171a419791797559fbc2807d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:23 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:41.909713 887 tablet_service.cc:1467] Processing CreateTablet for tablet 7203461deb504fbe95d62e67b86b134e (DEFAULT_TABLE table=default.Table_98fee465d761490c85252dbb021e3f96 [id=f2a79d79b71c4f8baba1a72c78b8b116]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:41.910293 885 tablet_service.cc:1467] Processing CreateTablet for tablet b2dc876471544cc19d7bddcc003f694f (DEFAULT_TABLE table=default.Table_96649abef65740ad8698fabc7e9d4a63 [id=37df785bff71410297d0a94e7f788210]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:41.910396 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7203461deb504fbe95d62e67b86b134e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:41.910774 889 tablet_service.cc:1467] Processing CreateTablet for tablet b96279fa76124046986e0e692ebee6c4 (DEFAULT_TABLE table=default.Table_969a9705d160408cb78f3c7029fa4023 [id=a3ee14433570445a9313d6e75456c4ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:41.911042 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b2dc876471544cc19d7bddcc003f694f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:41.912011 1194 ts_tablet_manager.cc:1397] T f21eb3fa0f414816a29e3d4560cdc0fe P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.021s user 0.006s sys 0.004s
I20250113 02:51:41.912587 2508 raft_consensus.cc:491] T 153b8f90b5a847b88944f683d306f579 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:41.912640 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b96279fa76124046986e0e692ebee6c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:41.912750 2508 raft_consensus.cc:513] T 153b8f90b5a847b88944f683d306f579 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:41.912912 1194 raft_consensus.cc:357] T f21eb3fa0f414816a29e3d4560cdc0fe P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:41.913125 1194 raft_consensus.cc:383] T f21eb3fa0f414816a29e3d4560cdc0fe P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:41.913179 1194 raft_consensus.cc:738] T f21eb3fa0f414816a29e3d4560cdc0fe P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:41.909723 1196 raft_consensus.cc:2238] T eebad045e7bb43fa9ca9c9af8b3b7ea5 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:41.913367 1196 raft_consensus.cc:2267] T eebad045e7bb43fa9ca9c9af8b3b7ea5 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:41.913507 2508 leader_election.cc:290] T 153b8f90b5a847b88944f683d306f579 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:41.913887 1196 ts_tablet_manager.cc:1905] T eebad045e7bb43fa9ca9c9af8b3b7ea5 P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:41.914229 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "153b8f90b5a847b88944f683d306f579" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:41.914538 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "153b8f90b5a847b88944f683d306f579" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:41.915127 711 leader_election.cc:343] T 153b8f90b5a847b88944f683d306f579 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:41.914516 1059 raft_consensus.cc:2463] T 153b8f90b5a847b88944f683d306f579 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1186fa869eb14812832fa6562531f196 in term 0.
I20250113 02:51:41.916538 709 leader_election.cc:304] T 153b8f90b5a847b88944f683d306f579 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters: df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:41.917071 2508 raft_consensus.cc:2798] T 153b8f90b5a847b88944f683d306f579 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 02:51:41.917155 2508 raft_consensus.cc:491] T 153b8f90b5a847b88944f683d306f579 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 02:51:41.917218 2508 raft_consensus.cc:3054] T 153b8f90b5a847b88944f683d306f579 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:41.913378 1194 consensus_queue.cc:260] T f21eb3fa0f414816a29e3d4560cdc0fe P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:41.918561 1194 ts_tablet_manager.cc:1428] T f21eb3fa0f414816a29e3d4560cdc0fe P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20250113 02:51:41.930300 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet f129bb3fbbfd492d98822be0a6626e44 (table default.Table_c3ba3f6047274ab885a4c853b06e1417 [id=6666a614063942c99a78b95d35add037]) successfully deleted
W20250113 02:51:41.930586 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 128f74fdc98f4135b39dbf23f1d987b0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 128f74fdc98f4135b39dbf23f1d987b0 already in progress: creating tablet
W20250113 02:51:41.930783 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet e5ed7016af184b86be4c234f42e2a6ef because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e5ed7016af184b86be4c234f42e2a6ef already in progress: creating tablet
W20250113 02:51:41.930953 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 16f547e3cec54baf86b96ad40e3afb6a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 16f547e3cec54baf86b96ad40e3afb6a already in progress: creating tablet
W20250113 02:51:41.931144 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 045167a6122a4560b298aae2358a485c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 045167a6122a4560b298aae2358a485c already in progress: creating tablet
W20250113 02:51:41.931337 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 61c79029dfb74f01aaac0f7f828880d0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 61c79029dfb74f01aaac0f7f828880d0 already in progress: creating tablet
W20250113 02:51:41.931529 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 432bcbc7ba6a44ba98cfe588f8292e32 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 432bcbc7ba6a44ba98cfe588f8292e32 already in progress: creating tablet
W20250113 02:51:41.931733 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 797d91d4aae240bd82b00021717772ae because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 797d91d4aae240bd82b00021717772ae already in progress: creating tablet
I20250113 02:51:41.958957 2306 raft_consensus.cc:491] T 64aa8390b1b1475c8743b6491d40674c P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:41.959183 2306 raft_consensus.cc:513] T 64aa8390b1b1475c8743b6491d40674c P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:41.960130 2306 leader_election.cc:290] T 64aa8390b1b1475c8743b6491d40674c P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:41.960702 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64aa8390b1b1475c8743b6491d40674c" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:41.960729 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64aa8390b1b1475c8743b6491d40674c" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
I20250113 02:51:41.960937 773 raft_consensus.cc:2463] T 64aa8390b1b1475c8743b6491d40674c P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 0.
W20250113 02:51:41.961356 998 leader_election.cc:343] T 64aa8390b1b1475c8743b6491d40674c P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:41.961592 995 leader_election.cc:304] T 64aa8390b1b1475c8743b6491d40674c P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters: df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:41.962064 2306 raft_consensus.cc:2798] T 64aa8390b1b1475c8743b6491d40674c P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 02:51:41.962157 2306 raft_consensus.cc:491] T 64aa8390b1b1475c8743b6491d40674c P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 02:51:41.962236 2306 raft_consensus.cc:3054] T 64aa8390b1b1475c8743b6491d40674c P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:41.965016 2306 raft_consensus.cc:513] T 64aa8390b1b1475c8743b6491d40674c P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:41.966084 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64aa8390b1b1475c8743b6491d40674c" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196"
I20250113 02:51:41.966266 773 raft_consensus.cc:3054] T 64aa8390b1b1475c8743b6491d40674c P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:41.967101 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64aa8390b1b1475c8743b6491d40674c" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f"
W20250113 02:51:41.967646 998 leader_election.cc:343] T 64aa8390b1b1475c8743b6491d40674c P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:41.967841 2306 leader_election.cc:290] T 64aa8390b1b1475c8743b6491d40674c P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Requested vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:41.970798 449 catalog_manager.cc:5618] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095) does not have the latest schema for tablet b5c1882125b5491ea79a29a6dac6771e (table default.Table_99031e0341fd4ad8a3bb04a6e222a1d0 [id=3237c1be7e1c48918ff095162d191dc1]). Expected version 3 got 0
I20250113 02:51:41.980976 1191 tablet_bootstrap.cc:492] T f105f8b32a654e4ca154e8fc0cb96091 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:41.981271 1191 ts_tablet_manager.cc:1397] T f105f8b32a654e4ca154e8fc0cb96091 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.304s user 0.005s sys 0.005s
I20250113 02:51:41.982364 1191 raft_consensus.cc:357] T f105f8b32a654e4ca154e8fc0cb96091 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:41.983613 1191 raft_consensus.cc:383] T f105f8b32a654e4ca154e8fc0cb96091 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:41.983783 1191 raft_consensus.cc:738] T f105f8b32a654e4ca154e8fc0cb96091 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:41.984167 1191 consensus_queue.cc:260] T f105f8b32a654e4ca154e8fc0cb96091 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:41.984841 1191 ts_tablet_manager.cc:1428] T f105f8b32a654e4ca154e8fc0cb96091 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250113 02:51:41.985242 1191 tablet_bootstrap.cc:492] T 8cc78bf7d6324b7ba10a1b1045923f2b P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:41.987176 1191 tablet_bootstrap.cc:654] T 8cc78bf7d6324b7ba10a1b1045923f2b P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:41.988420 882 tablet_service.cc:1467] Processing CreateTablet for tablet 079ef7df7b164466b3fc24f58caec611 (DEFAULT_TABLE table=default.Table_da027dfe87ee40ada76b095abebd0e7e [id=dff2d4a62b8b42dca4e6cd8d999d57d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:41.993232 2480 raft_consensus.cc:491] T 3d85539cf0074a03b167d3279e940079 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:41.993395 2480 raft_consensus.cc:513] T 3d85539cf0074a03b167d3279e940079 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:41.994174 2480 leader_election.cc:290] T 3d85539cf0074a03b167d3279e940079 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:41.994786 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d85539cf0074a03b167d3279e940079" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:42.001039 772 raft_consensus.cc:2463] T 82505f1389184d0ca12ec0e2d6084d22 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 1.
I20250113 02:51:42.002077 1194 tablet_bootstrap.cc:492] T df87e25053a74990959f85288a85217b P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:42.004757 1194 tablet_bootstrap.cc:654] T df87e25053a74990959f85288a85217b P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
W20250113 02:51:42.004920 998 leader_election.cc:343] T 3d85539cf0074a03b167d3279e940079 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:42.005638 995 leader_election.cc:304] T 82505f1389184d0ca12ec0e2d6084d22 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters: df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:42.005609 454 catalog_manager.cc:5526] T b815003a27f64809baaa2d43d6293cef P 4216a7f853d642b7bc3e5cff458f3aca reported cstate change: term changed from 0 to 1, leader changed from <none> to 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3). New cstate: current_term: 1 leader_uuid: "4216a7f853d642b7bc3e5cff458f3aca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } health_report { overall_health: HEALTHY } } }
I20250113 02:51:42.005995 454 catalog_manager.cc:5618] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587) does not have the latest schema for tablet b815003a27f64809baaa2d43d6293cef (table default.Table_c2e8fd7d2cc14fa39c0abfece783c6c9 [id=42e19138c08f422a92ba9621e9609957]). Expected version 1 got 0
I20250113 02:51:42.006142 2306 raft_consensus.cc:2798] T 82505f1389184d0ca12ec0e2d6084d22 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Leader election won for term 1
I20250113 02:51:42.006247 2306 raft_consensus.cc:695] T 82505f1389184d0ca12ec0e2d6084d22 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: Becoming Leader. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Running, Role: LEADER
I20250113 02:51:42.006451 2306 consensus_queue.cc:237] T 82505f1389184d0ca12ec0e2d6084d22 P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:42.006980 449 catalog_manager.cc:6071] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: default.Table_6daec99b05744d0caed86e51e86f24c2 [id=2931f217041c46cc8ae52bd6ce8cfd17] alter complete (version 1)
I20250113 02:51:42.010681 2473 raft_consensus.cc:513] T 0c408921a09a463eb464e08edcf042b3 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:42.011322 2473 leader_election.cc:290] T 0c408921a09a463eb464e08edcf042b3 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 election: Requested vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:42.011904 2480 raft_consensus.cc:491] T acd83024592e431898e9675c17f50d4f P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:42.012049 2480 raft_consensus.cc:513] T acd83024592e431898e9675c17f50d4f P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:42.012157 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c408921a09a463eb464e08edcf042b3" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca"
I20250113 02:51:42.012303 1059 raft_consensus.cc:3054] T 0c408921a09a463eb464e08edcf042b3 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:42.012681 2480 leader_election.cc:290] T acd83024592e431898e9675c17f50d4f P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
I20250113 02:51:42.013301 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acd83024592e431898e9675c17f50d4f" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:42.013923 998 leader_election.cc:343] T acd83024592e431898e9675c17f50d4f P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:42.014493 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acd83024592e431898e9675c17f50d4f" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
I20250113 02:51:42.014672 772 raft_consensus.cc:2463] T acd83024592e431898e9675c17f50d4f P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 0.
I20250113 02:51:42.014798 1059 raft_consensus.cc:2463] T 0c408921a09a463eb464e08edcf042b3 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1186fa869eb14812832fa6562531f196 in term 1.
I20250113 02:51:42.011929 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c408921a09a463eb464e08edcf042b3" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f"
I20250113 02:51:42.015353 709 leader_election.cc:304] T 0c408921a09a463eb464e08edcf042b3 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters:
W20250113 02:51:42.015558 711 leader_election.cc:343] T 0c408921a09a463eb464e08edcf042b3 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:42.015990 995 leader_election.cc:304] T acd83024592e431898e9675c17f50d4f P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters: df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:42.017033 2474 raft_consensus.cc:2798] T acd83024592e431898e9675c17f50d4f P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 02:51:42.017128 2474 raft_consensus.cc:491] T acd83024592e431898e9675c17f50d4f P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 02:51:42.017234 2474 raft_consensus.cc:3054] T acd83024592e431898e9675c17f50d4f P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:42.018836 2473 raft_consensus.cc:2798] T 0c408921a09a463eb464e08edcf042b3 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Leader election won for term 1
I20250113 02:51:42.018957 2473 raft_consensus.cc:695] T 0c408921a09a463eb464e08edcf042b3 P 1186fa869eb14812832fa6562531f196 [term 1 LEADER]: Becoming Leader. State: Replica: 1186fa869eb14812832fa6562531f196, State: Running, Role: LEADER
I20250113 02:51:42.019176 2473 consensus_queue.cc:237] T 0c408921a09a463eb464e08edcf042b3 P 1186fa869eb14812832fa6562531f196 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:42.019573 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d2fdc75d00f4e749d322ac6cd5fc410 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:27 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:42.022637 2501 raft_consensus.cc:513] T 1d2fdc75d00f4e749d322ac6cd5fc410 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:42.019835 2474 raft_consensus.cc:513] T acd83024592e431898e9675c17f50d4f P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:42.020339 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet acd83024592e431898e9675c17f50d4f with delete_type TABLET_DATA_DELETED (replaced by tablet 18716752335043e89038f01b4d65d711) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:42.023403 2501 leader_election.cc:290] T 1d2fdc75d00f4e749d322ac6cd5fc410 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 election: Requested vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:42.023746 2474 leader_election.cc:290] T acd83024592e431898e9675c17f50d4f P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Requested vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
I20250113 02:51:42.020838 1030 tablet_service.cc:1514] Processing DeleteTablet for tablet 64aa8390b1b1475c8743b6491d40674c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:29 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:42.021337 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c408921a09a463eb464e08edcf042b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:24 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:42.021828 1036 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d85539cf0074a03b167d3279e940079 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:29 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:42.026126 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet 438604e8859f422f9caccb61c5b5b125 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:24 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:42.026304 2508 raft_consensus.cc:513] T 153b8f90b5a847b88944f683d306f579 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:42.026932 2508 leader_election.cc:290] T 153b8f90b5a847b88944f683d306f579 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 election: Requested vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:42.028030 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d85539cf0074a03b167d3279e940079" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
I20250113 02:51:42.028210 769 raft_consensus.cc:2463] T 3d85539cf0074a03b167d3279e940079 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 0.
I20250113 02:51:42.029091 1032 tablet_service.cc:1514] Processing DeleteTablet for tablet dc680b5680b84abbab219296bc7ee707 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:23 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:42.029444 773 raft_consensus.cc:2463] T 64aa8390b1b1475c8743b6491d40674c P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 1.
I20250113 02:51:42.029771 1032 tablet_service.cc:1467] Processing CreateTablet for tablet 4fa23e653b27441d8128d001ea96fdff (DEFAULT_TABLE table=default.Table_507a4e0b009b4d6ca0fce33fde33c90c [id=a3d333ed6f744fa188283bcb939fd853]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:42.030176 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4fa23e653b27441d8128d001ea96fdff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:42.032617 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1103) took 4621 ms. Trace:
I20250113 02:51:42.032694 1195 rpcz_store.cc:276] 0113 02:51:37.411275 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:37.411491 (+ 216us) service_pool.cc:225] Handling call
0113 02:51:41.680497 (+4269006us) ts_tablet_manager.cc:1145] Deleting tablet 0752fde3a30a4947a2a688c56ceadab5
0113 02:51:41.680505 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:41.852731 (+172226us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:41.858145 (+ 5414us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:42.032598 (+174453us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":512,"tablet-delete.queue_time_us":4268854}
I20250113 02:51:42.032639 1024 tablet_service.cc:1467] Processing CreateTablet for tablet fd700105d7e248d2a3dd1f2898b2ec09 (DEFAULT_TABLE table=default.Table_58234d02d4164492995fe080423737ec [id=5e904f72b23046e199580ec2428ddac7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:42.033052 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1104) took 4621 ms. Trace:
I20250113 02:51:42.033198 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd700105d7e248d2a3dd1f2898b2ec09. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:42.033136 1195 rpcz_store.cc:276] 0113 02:51:37.411437 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:37.411483 (+ 46us) service_pool.cc:225] Handling call
0113 02:51:42.032939 (+4621456us) ts_tablet_manager.cc:1145] Deleting tablet 5d2d14548b734b1fbe57d68c0ea3e2f8
0113 02:51:42.032947 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:42.033037 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4621215}
I20250113 02:51:42.033432 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1107) took 4526 ms. Trace:
I20250113 02:51:42.033490 1195 rpcz_store.cc:276] 0113 02:51:37.506511 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:37.506576 (+ 65us) service_pool.cc:225] Handling call
0113 02:51:42.033350 (+4526774us) ts_tablet_manager.cc:1145] Deleting tablet a1d6cbfb61564e929efab798d4681409
0113 02:51:42.033357 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:42.033421 (+ 64us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4526569}
I20250113 02:51:42.033671 1195 tablet_replica.cc:331] T 0c408921a09a463eb464e08edcf042b3 P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:42.035286 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet 0752fde3a30a4947a2a688c56ceadab5 (table default.Table_da3e49fa1e654145a05858466ec2260f [id=94704429b49b46e38a48258a450d2705]) successfully deleted
W20250113 02:51:42.035893 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 5d2d14548b734b1fbe57d68c0ea3e2f8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5d2d14548b734b1fbe57d68c0ea3e2f8 already in progress: creating tablet
W20250113 02:51:42.036139 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet a1d6cbfb61564e929efab798d4681409 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a1d6cbfb61564e929efab798d4681409 already in progress: creating tablet
I20250113 02:51:42.036806 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "153b8f90b5a847b88944f683d306f579" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f"
I20250113 02:51:42.037160 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d2fdc75d00f4e749d322ac6cd5fc410" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f"
I20250113 02:51:42.038113 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acd83024592e431898e9675c17f50d4f" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f"
W20250113 02:51:42.038832 711 leader_election.cc:343] T 153b8f90b5a847b88944f683d306f579 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
W20250113 02:51:42.039012 711 leader_election.cc:343] T 1d2fdc75d00f4e749d322ac6cd5fc410 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:42.039481 995 leader_election.cc:304] T 64aa8390b1b1475c8743b6491d40674c P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters: df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:42.039916 995 leader_election.cc:304] T 3d85539cf0074a03b167d3279e940079 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters: df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:42.040279 2368 raft_consensus.cc:2798] T 64aa8390b1b1475c8743b6491d40674c P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Leader election won for term 1
I20250113 02:51:42.040436 2368 raft_consensus.cc:695] T 64aa8390b1b1475c8743b6491d40674c P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: Becoming Leader. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Running, Role: LEADER
I20250113 02:51:42.040706 2368 consensus_queue.cc:237] T 64aa8390b1b1475c8743b6491d40674c P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:42.040994 1036 tablet_service.cc:1467] Processing CreateTablet for tablet dd833958fe24434e9c1560877de879ce (DEFAULT_TABLE table=default.Table_cc25a0009bc34b2fbf160fe03db677ba [id=ce4268a800db4c8d916032c44ae98029]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:42.041364 1036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd833958fe24434e9c1560877de879ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:42.041711 2363 raft_consensus.cc:2798] T 3d85539cf0074a03b167d3279e940079 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 02:51:42.041807 2363 raft_consensus.cc:491] T 3d85539cf0074a03b167d3279e940079 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 02:51:42.041893 2363 raft_consensus.cc:3054] T 3d85539cf0074a03b167d3279e940079 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:42.045285 1028 tablet_service.cc:1467] Processing CreateTablet for tablet 45443d40e80b4e71bc70716d3251510e (DEFAULT_TABLE table=default.Table_cef3f1e1be374b87a39f67fb569234e8 [id=03da3142c3764f21b29b7976bb35db39]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:42.045650 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 45443d40e80b4e71bc70716d3251510e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:42.046175 742 tablet_service.cc:1467] Processing CreateTablet for tablet 63001da570cd4695a776725dbdf991a1 (DEFAULT_TABLE table=default.Table_7ebd2aec3eb74fc29dccfd5d399dc4ea [id=a5c9cf6f78774e37903d9c11f673d1c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:42.046519 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 63001da570cd4695a776725dbdf991a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:42.048259 1025 tablet_service.cc:1467] Processing CreateTablet for tablet 05e6e2dd6b364b1590469e400159a6dd (DEFAULT_TABLE table=default.Table_bb6e88b17f3d497386f8098edb3719b9 [id=724c4b6388684355919a47b675619187]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:42.048323 1030 tablet_service.cc:1467] Processing CreateTablet for tablet 3f65c4ebb7df4637bc580fffd094a87a (DEFAULT_TABLE table=default.Table_035c14601d094e78bc0f136d22321b4d [id=1262f4cbebf8469ba8ea16d71e29b20e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:42.048584 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05e6e2dd6b364b1590469e400159a6dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:42.049021 1035 tablet_service.cc:1467] Processing CreateTablet for tablet 39fef46c51aa4c00acfdd2d31f5fd410 (DEFAULT_TABLE table=default.Table_414756feb23c4db3a86bfd310703a889 [id=8e253f8d77034748beaf1f16aa2fb374]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:42.049523 1039 tablet_service.cc:1467] Processing CreateTablet for tablet fbe1bedec31b47ad898acf7240a8c18a (DEFAULT_TABLE table=default.Table_bea21867cb0743d98a95f39f6fc76cc2 [id=fcb88e7484f44c15ac1a62127ba1236c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:42.050092 750 tablet_service.cc:1467] Processing CreateTablet for tablet dd833958fe24434e9c1560877de879ce (DEFAULT_TABLE table=default.Table_cc25a0009bc34b2fbf160fe03db677ba [id=ce4268a800db4c8d916032c44ae98029]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:42.050426 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd833958fe24434e9c1560877de879ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:42.052306 754 tablet_service.cc:1467] Processing CreateTablet for tablet 3f65c4ebb7df4637bc580fffd094a87a (DEFAULT_TABLE table=default.Table_035c14601d094e78bc0f136d22321b4d [id=1262f4cbebf8469ba8ea16d71e29b20e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:42.052619 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3f65c4ebb7df4637bc580fffd094a87a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:42.056042 749 tablet_service.cc:1467] Processing CreateTablet for tablet 39fef46c51aa4c00acfdd2d31f5fd410 (DEFAULT_TABLE table=default.Table_414756feb23c4db3a86bfd310703a889 [id=8e253f8d77034748beaf1f16aa2fb374]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:42.056187 738 tablet_service.cc:1467] Processing CreateTablet for tablet 08180ab0274a40fd955dda5b08197a08 (DEFAULT_TABLE table=default.Table_27eb308f2dc84858b3d1a2fa4b42c3f6 [id=1cfda70b239543dbbd14d5857a6bbf3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:42.056353 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 39fef46c51aa4c00acfdd2d31f5fd410. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:42.056656 745 tablet_service.cc:1467] Processing CreateTablet for tablet 05e6e2dd6b364b1590469e400159a6dd (DEFAULT_TABLE table=default.Table_bb6e88b17f3d497386f8098edb3719b9 [id=724c4b6388684355919a47b675619187]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:42.056942 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05e6e2dd6b364b1590469e400159a6dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:42.060230 740 tablet_service.cc:1467] Processing CreateTablet for tablet fbe1bedec31b47ad898acf7240a8c18a (DEFAULT_TABLE table=default.Table_bea21867cb0743d98a95f39f6fc76cc2 [id=fcb88e7484f44c15ac1a62127ba1236c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:42.060312 1034 tablet_service.cc:1467] Processing CreateTablet for tablet 08180ab0274a40fd955dda5b08197a08 (DEFAULT_TABLE table=default.Table_27eb308f2dc84858b3d1a2fa4b42c3f6 [id=1cfda70b239543dbbd14d5857a6bbf3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:42.060557 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fbe1bedec31b47ad898acf7240a8c18a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:42.060746 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3f65c4ebb7df4637bc580fffd094a87a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:42.063941 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fbe1bedec31b47ad898acf7240a8c18a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:42.064211 752 tablet_service.cc:1467] Processing CreateTablet for tablet fd700105d7e248d2a3dd1f2898b2ec09 (DEFAULT_TABLE table=default.Table_58234d02d4164492995fe080423737ec [id=5e904f72b23046e199580ec2428ddac7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:42.064570 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd700105d7e248d2a3dd1f2898b2ec09. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:42.064813 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 39fef46c51aa4c00acfdd2d31f5fd410. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:42.068096 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 08180ab0274a40fd955dda5b08197a08. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:42.068495 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 08180ab0274a40fd955dda5b08197a08. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:42.074580 751 tablet_service.cc:1467] Processing CreateTablet for tablet 4fa23e653b27441d8128d001ea96fdff (DEFAULT_TABLE table=default.Table_507a4e0b009b4d6ca0fce33fde33c90c [id=a3d333ed6f744fa188283bcb939fd853]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:42.074976 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4fa23e653b27441d8128d001ea96fdff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:42.076159 739 tablet_service.cc:1467] Processing CreateTablet for tablet 45443d40e80b4e71bc70716d3251510e (DEFAULT_TABLE table=default.Table_cef3f1e1be374b87a39f67fb569234e8 [id=03da3142c3764f21b29b7976bb35db39]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:42.076496 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 45443d40e80b4e71bc70716d3251510e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:42.080693 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acd83024592e431898e9675c17f50d4f" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196"
W20250113 02:51:42.082274 998 leader_election.cc:343] T acd83024592e431898e9675c17f50d4f P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:42.082818 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d2fdc75d00f4e749d322ac6cd5fc410" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca"
I20250113 02:51:42.082880 1058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "153b8f90b5a847b88944f683d306f579" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca"
I20250113 02:51:42.084527 769 raft_consensus.cc:3054] T acd83024592e431898e9675c17f50d4f P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:42.092506 2476 raft_consensus.cc:491] T 7213e3284b4d4cb581d9fb946a9f87e9 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:42.092672 2476 raft_consensus.cc:513] T 7213e3284b4d4cb581d9fb946a9f87e9 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:42.093287 2476 leader_election.cc:290] T 7213e3284b4d4cb581d9fb946a9f87e9 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587), 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
W20250113 02:51:42.095629 851 leader_election.cc:343] T 7213e3284b4d4cb581d9fb946a9f87e9 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Not found: Tablet not found: 7213e3284b4d4cb581d9fb946a9f87e9
I20250113 02:51:42.101724 1194 tablet_bootstrap.cc:492] T df87e25053a74990959f85288a85217b P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:42.101934 1194 ts_tablet_manager.cc:1397] T df87e25053a74990959f85288a85217b P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.100s user 0.006s sys 0.002s
W20250113 02:51:42.102499 852 leader_election.cc:343] T 7213e3284b4d4cb581d9fb946a9f87e9 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Not found: Tablet not found: 7213e3284b4d4cb581d9fb946a9f87e9
W20250113 02:51:41.931917 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 67fe2f52d98048dab9f37dd820b7f712 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 67fe2f52d98048dab9f37dd820b7f712 already in progress: creating tablet
I20250113 02:51:42.102695 852 leader_election.cc:304] T 7213e3284b4d4cb581d9fb946a9f87e9 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df69f48656b44bb890f76587ee69cc0f; no voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca
W20250113 02:51:42.102774 442 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.104.2:38467, user_credentials={real_user=slave}} blocked reactor thread for 170842us
I20250113 02:51:42.103130 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet 1d15c7a01f854f0aaabcd30ec3166e31 (table default.Table_21648ca05a3344848aeac2ad300f7400 [id=fe31ebf2077b43a285155bef4c983d75]) successfully deleted
I20250113 02:51:42.103191 2476 raft_consensus.cc:2743] T 7213e3284b4d4cb581d9fb946a9f87e9 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:42.103350 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet 3025935f9743442eb3e6f2db22eb52ad (table default.Table_854e525352dd47c889aaabfc312e33c7 [id=c0b628d230fd4eae83a20530cd998993]) successfully deleted
W20250113 02:51:42.103536 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 368e5f00861b40ffb76e79376245b885 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 368e5f00861b40ffb76e79376245b885 already in progress: creating tablet
I20250113 02:51:42.104611 1194 raft_consensus.cc:357] T df87e25053a74990959f85288a85217b P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:42.104945 1194 raft_consensus.cc:383] T df87e25053a74990959f85288a85217b P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:42.105051 1194 raft_consensus.cc:738] T df87e25053a74990959f85288a85217b P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:42.105446 2363 raft_consensus.cc:513] T 3d85539cf0074a03b167d3279e940079 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:42.105484 1194 consensus_queue.cc:260] T df87e25053a74990959f85288a85217b P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
W20250113 02:51:42.105708 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 59fadc693f554854bc2874f8db7c64c8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 59fadc693f554854bc2874f8db7c64c8 already in progress: creating tablet
W20250113 02:51:42.105908 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 348c21d783ef4103a37409ef8634eb35 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 348c21d783ef4103a37409ef8634eb35 already in progress: creating tablet
W20250113 02:51:42.106069 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 348c21d783ef4103a37409ef8634eb35 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 348c21d783ef4103a37409ef8634eb35 already in progress: creating tablet
W20250113 02:51:42.106236 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet c368b5b0c3a940c4b4efb64c946375c3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c368b5b0c3a940c4b4efb64c946375c3 already in progress: creating tablet
W20250113 02:51:42.106402 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet e4998b86d32040ec959ff6fd94b1e9eb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e4998b86d32040ec959ff6fd94b1e9eb already in progress: creating tablet
I20250113 02:51:42.106668 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d85539cf0074a03b167d3279e940079" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196"
I20250113 02:51:42.106773 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet 999fecb3fc65466cad21cc1ff5ef2d44 (table default.Table_991d8c72ce924153ad119a440b54fb1c [id=71e76dffaa6f4b0682422afed1749a64]) successfully deleted
I20250113 02:51:42.106838 773 raft_consensus.cc:3054] T 3d85539cf0074a03b167d3279e940079 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Advancing to term 1
W20250113 02:51:42.106945 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet eceed0f52c9f46bbaa78ca773a7afde5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet eceed0f52c9f46bbaa78ca773a7afde5 already in progress: creating tablet
W20250113 02:51:42.107100 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 42a8e01bb71f49b09c83a8f5c905f38d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 42a8e01bb71f49b09c83a8f5c905f38d already in progress: creating tablet
W20250113 02:51:42.107260 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 94307d22b7f842ca89844768e4a2a2ab because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 94307d22b7f842ca89844768e4a2a2ab already in progress: creating tablet
W20250113 02:51:42.107429 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet a0c14501ad0449aca28909bc14428dad because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a0c14501ad0449aca28909bc14428dad already in progress: creating tablet
W20250113 02:51:42.107581 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet a894199f21aa4dba8a03583ea14a9092 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a894199f21aa4dba8a03583ea14a9092 already in progress: creating tablet
W20250113 02:51:42.107740 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 6001369444894b48b51a48fb60ddc52a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6001369444894b48b51a48fb60ddc52a already in progress: creating tablet
W20250113 02:51:42.107911 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 3c623bd0fbbf4922aedda031a88b79eb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3c623bd0fbbf4922aedda031a88b79eb already in progress: creating tablet
W20250113 02:51:42.108103 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 8f2b4df7c6a74896ade95a490ea9b54c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8f2b4df7c6a74896ade95a490ea9b54c already in progress: creating tablet
W20250113 02:51:42.108276 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet aef2ffd67def48eea501a98fbc9dc43b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet aef2ffd67def48eea501a98fbc9dc43b already in progress: creating tablet
W20250113 02:51:42.108433 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 01a455eda0ee488fb3d45a6a09c43833 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 01a455eda0ee488fb3d45a6a09c43833 already in progress: creating tablet
I20250113 02:51:42.113497 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d85539cf0074a03b167d3279e940079" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f"
I20250113 02:51:42.115410 1059 raft_consensus.cc:3054] T 1d2fdc75d00f4e749d322ac6cd5fc410 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:42.116129 1194 ts_tablet_manager.cc:1428] T df87e25053a74990959f85288a85217b P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20250113 02:51:42.116708 2487 raft_consensus.cc:491] T 42dbd37e20fa47318bc6497423dbe565 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:42.117105 2474 raft_consensus.cc:491] T 438604e8859f422f9caccb61c5b5b125 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:42.117233 2474 raft_consensus.cc:513] T 438604e8859f422f9caccb61c5b5b125 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:42.117910 2363 leader_election.cc:290] T 3d85539cf0074a03b167d3279e940079 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Requested vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:42.118348 1058 raft_consensus.cc:3054] T 153b8f90b5a847b88944f683d306f579 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:42.115574 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet 438d1d2300e044f39a15586f335935ba (table default.Table_d7d1f5a34e5f4c05b474b0b61f912cd9 [id=2c7925ec0d8d4cd7b8761268d3962f15]) successfully deleted
W20250113 02:51:42.118992 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet c8966b3b93144a30813bfdaf04f9c43a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c8966b3b93144a30813bfdaf04f9c43a already in progress: creating tablet
I20250113 02:51:42.119380 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "438604e8859f422f9caccb61c5b5b125" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
I20250113 02:51:42.119587 772 raft_consensus.cc:2463] T 438604e8859f422f9caccb61c5b5b125 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 0.
W20250113 02:51:42.120211 998 leader_election.cc:343] T 3d85539cf0074a03b167d3279e940079 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:42.120949 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "438604e8859f422f9caccb61c5b5b125" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:42.121987 995 leader_election.cc:304] T 438604e8859f422f9caccb61c5b5b125 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters:
I20250113 02:51:42.122622 896 tablet_service.cc:1514] Processing DeleteTablet for tablet 33c665e149dd48e8868acd935de581bf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:20 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:42.123688 889 tablet_service.cc:1514] Processing DeleteTablet for tablet e70658c90555463498d3fc2e2f13ea7f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:23 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:42.125147 896 tablet_service.cc:1514] Processing DeleteTablet for tablet a92f9b1c9c1a4cbdbf0cc7b2a3f2275b with delete_type TABLET_DATA_DELETED (replaced by tablet bc6a7322c0c34adb85457750526d8955) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:42.126240 2363 raft_consensus.cc:2798] T 438604e8859f422f9caccb61c5b5b125 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 02:51:42.126358 1024 tablet_service.cc:1467] Processing CreateTablet for tablet 63001da570cd4695a776725dbdf991a1 (DEFAULT_TABLE table=default.Table_7ebd2aec3eb74fc29dccfd5d399dc4ea [id=a5c9cf6f78774e37903d9c11f673d1c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:42.126438 2474 leader_election.cc:290] T 438604e8859f422f9caccb61c5b5b125 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:42.126816 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 63001da570cd4695a776725dbdf991a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:42.126855 896 tablet_service.cc:1514] Processing DeleteTablet for tablet 316cb2863ed645158a8f72e3cb0a94fd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:41 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:42.128453 1194 tablet_bootstrap.cc:492] T 9438472464304da9a0c7658d4f0736b6 P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:42.125572 2487 raft_consensus.cc:513] T 42dbd37e20fa47318bc6497423dbe565 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:42.129524 896 tablet_service.cc:1514] Processing DeleteTablet for tablet 4cc15487be474354b995c4e415276a18 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:41 UTC) from {username='slave'} at 127.0.0.1:37366
W20250113 02:51:42.129904 998 leader_election.cc:343] T 438604e8859f422f9caccb61c5b5b125 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:42.130635 896 tablet_service.cc:1467] Processing CreateTablet for tablet 1511a0ec646549239f6cfe0fc2bdf9f0 (DEFAULT_TABLE table=default.Table_302b50fc67d44fa79ef59d7cc8fbc408 [id=e3422649c46443b1b91579cfcd6070b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:42.130676 889 tablet_service.cc:1514] Processing DeleteTablet for tablet 7213e3284b4d4cb581d9fb946a9f87e9 with delete_type TABLET_DATA_DELETED (replaced by tablet 7c443454994a47f2bfa7e2ec7ac576e0) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:42.130954 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42dbd37e20fa47318bc6497423dbe565" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
I20250113 02:51:42.131165 772 raft_consensus.cc:2463] T 42dbd37e20fa47318bc6497423dbe565 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 0.
I20250113 02:51:42.131407 1194 tablet_bootstrap.cc:654] T 9438472464304da9a0c7658d4f0736b6 P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:42.131660 889 tablet_service.cc:1514] Processing DeleteTablet for tablet 85d13f8abbec484d82c4d0b86480531b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:41 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:42.132033 2363 raft_consensus.cc:491] T 438604e8859f422f9caccb61c5b5b125 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 02:51:42.132164 2363 raft_consensus.cc:3054] T 438604e8859f422f9caccb61c5b5b125 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:42.132647 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42dbd37e20fa47318bc6497423dbe565" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:42.133265 995 leader_election.cc:304] T 42dbd37e20fa47318bc6497423dbe565 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters:
W20250113 02:51:42.133448 998 leader_election.cc:343] T 42dbd37e20fa47318bc6497423dbe565 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:42.133939 889 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ee0b2e06dc74faab48912dd16fe1537 with delete_type TABLET_DATA_DELETED (replaced by tablet 3f65c4ebb7df4637bc580fffd094a87a) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:42.134637 889 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b9e7deda7334ff4a3429f70d9b3a7fc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:41 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:42.136076 889 tablet_service.cc:1514] Processing DeleteTablet for tablet f21eb3fa0f414816a29e3d4560cdc0fe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:30 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:42.136770 889 tablet_service.cc:1467] Processing CreateTablet for tablet 4fa23e653b27441d8128d001ea96fdff (DEFAULT_TABLE table=default.Table_507a4e0b009b4d6ca0fce33fde33c90c [id=a3d333ed6f744fa188283bcb939fd853]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:42.137617 885 tablet_service.cc:1467] Processing CreateTablet for tablet fd700105d7e248d2a3dd1f2898b2ec09 (DEFAULT_TABLE table=default.Table_58234d02d4164492995fe080423737ec [id=5e904f72b23046e199580ec2428ddac7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:42.138108 2474 raft_consensus.cc:2798] T 42dbd37e20fa47318bc6497423dbe565 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 02:51:42.138763 887 tablet_service.cc:1467] Processing CreateTablet for tablet 63001da570cd4695a776725dbdf991a1 (DEFAULT_TABLE table=default.Table_7ebd2aec3eb74fc29dccfd5d399dc4ea [id=a5c9cf6f78774e37903d9c11f673d1c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:42.139281 2474 raft_consensus.cc:491] T 42dbd37e20fa47318bc6497423dbe565 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 02:51:42.139385 2474 raft_consensus.cc:3054] T 42dbd37e20fa47318bc6497423dbe565 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:42.139853 2487 leader_election.cc:290] T 42dbd37e20fa47318bc6497423dbe565 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:42.139880 883 tablet_service.cc:1467] Processing CreateTablet for tablet dd833958fe24434e9c1560877de879ce (DEFAULT_TABLE table=default.Table_cc25a0009bc34b2fbf160fe03db677ba [id=ce4268a800db4c8d916032c44ae98029]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:42.140797 893 tablet_service.cc:1467] Processing CreateTablet for tablet 45443d40e80b4e71bc70716d3251510e (DEFAULT_TABLE table=default.Table_cef3f1e1be374b87a39f67fb569234e8 [id=03da3142c3764f21b29b7976bb35db39]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:42.141162 897 tablet_service.cc:1467] Processing CreateTablet for tablet 3f65c4ebb7df4637bc580fffd094a87a (DEFAULT_TABLE table=default.Table_035c14601d094e78bc0f136d22321b4d [id=1262f4cbebf8469ba8ea16d71e29b20e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:42.141731 895 tablet_service.cc:1467] Processing CreateTablet for tablet 08180ab0274a40fd955dda5b08197a08 (DEFAULT_TABLE table=default.Table_27eb308f2dc84858b3d1a2fa4b42c3f6 [id=1cfda70b239543dbbd14d5857a6bbf3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:42.142067 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3f65c4ebb7df4637bc580fffd094a87a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:42.142351 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 08180ab0274a40fd955dda5b08197a08. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:42.142304 891 tablet_service.cc:1467] Processing CreateTablet for tablet 39fef46c51aa4c00acfdd2d31f5fd410 (DEFAULT_TABLE table=default.Table_414756feb23c4db3a86bfd310703a889 [id=8e253f8d77034748beaf1f16aa2fb374]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:42.142572 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 39fef46c51aa4c00acfdd2d31f5fd410. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:42.142791 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 079ef7df7b164466b3fc24f58caec611. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:42.143019 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd700105d7e248d2a3dd1f2898b2ec09. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:42.143214 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1511a0ec646549239f6cfe0fc2bdf9f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:42.143512 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 63001da570cd4695a776725dbdf991a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:42.143924 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4fa23e653b27441d8128d001ea96fdff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:42.144475 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd833958fe24434e9c1560877de879ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:42.146576 1059 raft_consensus.cc:2463] T 1d2fdc75d00f4e749d322ac6cd5fc410 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1186fa869eb14812832fa6562531f196 in term 1.
I20250113 02:51:42.147060 709 leader_election.cc:304] T 1d2fdc75d00f4e749d322ac6cd5fc410 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters: df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:42.147809 2508 raft_consensus.cc:2798] T 1d2fdc75d00f4e749d322ac6cd5fc410 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Leader election won for term 1
I20250113 02:51:42.148200 2508 raft_consensus.cc:695] T 1d2fdc75d00f4e749d322ac6cd5fc410 P 1186fa869eb14812832fa6562531f196 [term 1 LEADER]: Becoming Leader. State: Replica: 1186fa869eb14812832fa6562531f196, State: Running, Role: LEADER
I20250113 02:51:42.148546 2508 consensus_queue.cc:237] T 1d2fdc75d00f4e749d322ac6cd5fc410 P 1186fa869eb14812832fa6562531f196 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
W20250113 02:51:42.149456 439 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.104.60:42067 (error 108) [suppressed 1 similar messages]
I20250113 02:51:42.153839 881 tablet_service.cc:1467] Processing CreateTablet for tablet 05e6e2dd6b364b1590469e400159a6dd (DEFAULT_TABLE table=default.Table_bb6e88b17f3d497386f8098edb3719b9 [id=724c4b6388684355919a47b675619187]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:42.154280 880 tablet_service.cc:1514] Processing DeleteTablet for tablet df87e25053a74990959f85288a85217b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:34 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:42.154407 899 tablet_service.cc:1467] Processing CreateTablet for tablet fbe1bedec31b47ad898acf7240a8c18a (DEFAULT_TABLE table=default.Table_bea21867cb0743d98a95f39f6fc76cc2 [id=fcb88e7484f44c15ac1a62127ba1236c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:42.156082 2363 raft_consensus.cc:513] T 438604e8859f422f9caccb61c5b5b125 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:42.156167 454 master_service.cc:946] ReplaceTablet: received request to replace tablet 1c153e47a7aa4bf5ae61dd19a91930b9 from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:42.157079 2363 leader_election.cc:290] T 438604e8859f422f9caccb61c5b5b125 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Requested vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:42.159053 1058 raft_consensus.cc:2463] T 153b8f90b5a847b88944f683d306f579 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1186fa869eb14812832fa6562531f196 in term 1.
I20250113 02:51:42.161260 709 leader_election.cc:304] T 153b8f90b5a847b88944f683d306f579 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters: df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:42.161669 2501 raft_consensus.cc:2798] T 153b8f90b5a847b88944f683d306f579 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Leader election won for term 1
I20250113 02:51:42.161768 2501 raft_consensus.cc:695] T 153b8f90b5a847b88944f683d306f579 P 1186fa869eb14812832fa6562531f196 [term 1 LEADER]: Becoming Leader. State: Replica: 1186fa869eb14812832fa6562531f196, State: Running, Role: LEADER
I20250113 02:51:42.161975 2501 consensus_queue.cc:237] T 153b8f90b5a847b88944f683d306f579 P 1186fa869eb14812832fa6562531f196 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:42.163774 2474 raft_consensus.cc:513] T 42dbd37e20fa47318bc6497423dbe565 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:42.164017 1195 raft_consensus.cc:2238] T 0c408921a09a463eb464e08edcf042b3 P 1186fa869eb14812832fa6562531f196 [term 1 LEADER]: Raft consensus shutting down.
I20250113 02:51:42.164407 1195 pending_rounds.cc:62] T 0c408921a09a463eb464e08edcf042b3 P 1186fa869eb14812832fa6562531f196: Trying to abort 1 pending ops.
I20250113 02:51:42.164458 1195 pending_rounds.cc:69] T 0c408921a09a463eb464e08edcf042b3 P 1186fa869eb14812832fa6562531f196: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113673531472990208 op_type: NO_OP noop_request { }
I20250113 02:51:42.164543 1195 raft_consensus.cc:2883] T 0c408921a09a463eb464e08edcf042b3 P 1186fa869eb14812832fa6562531f196 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250113 02:51:42.164610 1195 raft_consensus.cc:2267] T 0c408921a09a463eb464e08edcf042b3 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:42.165038 2474 leader_election.cc:290] T 42dbd37e20fa47318bc6497423dbe565 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Requested vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:42.165148 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 45443d40e80b4e71bc70716d3251510e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:42.169454 769 raft_consensus.cc:2463] T acd83024592e431898e9675c17f50d4f P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 1.
I20250113 02:51:42.171367 995 leader_election.cc:304] T acd83024592e431898e9675c17f50d4f P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters: df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:42.171901 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42dbd37e20fa47318bc6497423dbe565" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196"
I20250113 02:51:42.172109 772 raft_consensus.cc:3054] T 42dbd37e20fa47318bc6497423dbe565 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:42.173025 2363 raft_consensus.cc:2798] T acd83024592e431898e9675c17f50d4f P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Leader election won for term 1
I20250113 02:51:42.173132 2363 raft_consensus.cc:695] T acd83024592e431898e9675c17f50d4f P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: Becoming Leader. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Running, Role: LEADER
I20250113 02:51:42.173372 2363 consensus_queue.cc:237] T acd83024592e431898e9675c17f50d4f P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:42.174571 773 raft_consensus.cc:2463] T 3d85539cf0074a03b167d3279e940079 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 1.
I20250113 02:51:42.174644 772 raft_consensus.cc:2463] T 42dbd37e20fa47318bc6497423dbe565 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 1.
I20250113 02:51:42.175107 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 1c153e47a7aa4bf5ae61dd19a91930b9
I20250113 02:51:42.175362 1192 tablet_bootstrap.cc:492] T c13e9a51c1e74b44b352bade78390ba5 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:42.175472 1192 ts_tablet_manager.cc:1397] T c13e9a51c1e74b44b352bade78390ba5 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.439s user 0.078s sys 0.084s
I20250113 02:51:42.175530 454 catalog_manager.cc:6596] ReplaceTablet: tablet 1c153e47a7aa4bf5ae61dd19a91930b9 deleted and replaced by tablet a5565bcb7761428fbf136ed8ce500642
I20250113 02:51:42.176218 1192 raft_consensus.cc:357] T c13e9a51c1e74b44b352bade78390ba5 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:42.181648 1192 raft_consensus.cc:383] T c13e9a51c1e74b44b352bade78390ba5 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:42.182312 1192 raft_consensus.cc:738] T c13e9a51c1e74b44b352bade78390ba5 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:42.183787 1192 consensus_queue.cc:260] T c13e9a51c1e74b44b352bade78390ba5 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:42.185581 2476 raft_consensus.cc:491] T 4eb5f1944cb44fd5a40ca92a6a3d7214 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:42.185706 2476 raft_consensus.cc:513] T 4eb5f1944cb44fd5a40ca92a6a3d7214 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:42.186311 2476 leader_election.cc:290] T 4eb5f1944cb44fd5a40ca92a6a3d7214 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587), 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
I20250113 02:51:42.187327 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4eb5f1944cb44fd5a40ca92a6a3d7214" candidate_uuid: "df69f48656b44bb890f76587ee69cc0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
I20250113 02:51:42.187319 1058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4eb5f1944cb44fd5a40ca92a6a3d7214" candidate_uuid: "df69f48656b44bb890f76587ee69cc0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:42.187186 1192 ts_tablet_manager.cc:1428] T c13e9a51c1e74b44b352bade78390ba5 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20250113 02:51:42.188374 852 leader_election.cc:304] T 4eb5f1944cb44fd5a40ca92a6a3d7214 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df69f48656b44bb890f76587ee69cc0f; no voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca
I20250113 02:51:42.188534 1192 tablet_bootstrap.cc:492] T 3f62f344c5a648328ca2de8ca1069c19 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:42.188815 2476 raft_consensus.cc:2743] T 4eb5f1944cb44fd5a40ca92a6a3d7214 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:42.189436 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05e6e2dd6b364b1590469e400159a6dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:42.189473 1196 ts_tablet_manager.cc:1918] T eebad045e7bb43fa9ca9c9af8b3b7ea5 P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:42.189572 1196 log.cc:1198] T eebad045e7bb43fa9ca9c9af8b3b7ea5 P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/eebad045e7bb43fa9ca9c9af8b3b7ea5
I20250113 02:51:42.189872 1196 ts_tablet_manager.cc:1939] T eebad045e7bb43fa9ca9c9af8b3b7ea5 P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:42.190023 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fbe1bedec31b47ad898acf7240a8c18a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:42.190280 1196 tablet_replica.cc:331] T 0916b41c171a419791797559fbc2807d P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:42.190488 1196 raft_consensus.cc:2238] T 0916b41c171a419791797559fbc2807d P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:42.190654 1196 raft_consensus.cc:2267] T 0916b41c171a419791797559fbc2807d P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:42.190734 1192 tablet_bootstrap.cc:654] T 3f62f344c5a648328ca2de8ca1069c19 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:42.191246 1196 ts_tablet_manager.cc:1905] T 0916b41c171a419791797559fbc2807d P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250113 02:51:42.192683 736 consensus_meta.cc:287] T b2dc876471544cc19d7bddcc003f694f P 1186fa869eb14812832fa6562531f196: Time spent flushing consensus metadata: real 0.627s user 0.022s sys 0.000s
I20250113 02:51:42.196622 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "438604e8859f422f9caccb61c5b5b125" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f"
I20250113 02:51:42.197325 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42dbd37e20fa47318bc6497423dbe565" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f"
W20250113 02:51:42.197731 998 leader_election.cc:343] T 438604e8859f422f9caccb61c5b5b125 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:42.198469 995 leader_election.cc:304] T 3d85539cf0074a03b167d3279e940079 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters: df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:42.198851 995 leader_election.cc:304] T 42dbd37e20fa47318bc6497423dbe565 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters:
W20250113 02:51:42.199280 998 leader_election.cc:343] T 42dbd37e20fa47318bc6497423dbe565 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:42.199565 2480 raft_consensus.cc:2798] T 3d85539cf0074a03b167d3279e940079 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Leader election won for term 1
I20250113 02:51:42.199673 2480 raft_consensus.cc:695] T 3d85539cf0074a03b167d3279e940079 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: Becoming Leader. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Running, Role: LEADER
I20250113 02:51:42.199828 2363 raft_consensus.cc:2798] T 42dbd37e20fa47318bc6497423dbe565 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Leader election won for term 1
I20250113 02:51:42.199954 2480 consensus_queue.cc:237] T 3d85539cf0074a03b167d3279e940079 P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:42.205035 1192 tablet_bootstrap.cc:492] T 3f62f344c5a648328ca2de8ca1069c19 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:42.205191 1192 ts_tablet_manager.cc:1397] T 3f62f344c5a648328ca2de8ca1069c19 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.000s
I20250113 02:51:42.206104 1192 raft_consensus.cc:357] T 3f62f344c5a648328ca2de8ca1069c19 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:42.206348 1192 raft_consensus.cc:383] T 3f62f344c5a648328ca2de8ca1069c19 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:42.206399 1192 raft_consensus.cc:738] T 3f62f344c5a648328ca2de8ca1069c19 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:42.206662 1192 consensus_queue.cc:260] T 3f62f344c5a648328ca2de8ca1069c19 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:42.209717 1195 ts_tablet_manager.cc:1905] T 0c408921a09a463eb464e08edcf042b3 P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250113 02:51:42.213631 851 connection.cc:537] client connection to 127.0.104.60:42067 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250113 02:51:42.216490 1195 ts_tablet_manager.cc:1918] T 0c408921a09a463eb464e08edcf042b3 P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250113 02:51:42.216697 1195 log.cc:1198] T 0c408921a09a463eb464e08edcf042b3 P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/0c408921a09a463eb464e08edcf042b3
I20250113 02:51:42.217279 1195 ts_tablet_manager.cc:1939] T 0c408921a09a463eb464e08edcf042b3 P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:42.218106 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1111) took 4683 ms. Trace:
I20250113 02:51:42.218246 1195 rpcz_store.cc:276] 0113 02:51:37.535042 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:37.535104 (+ 62us) service_pool.cc:225] Handling call
0113 02:51:42.033638 (+4498534us) ts_tablet_manager.cc:1145] Deleting tablet 0c408921a09a463eb464e08edcf042b3
0113 02:51:42.033645 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:42.164238 (+130593us) spinlock_profiling.cc:243] Waited 61.5 ms on lock 0x616000222754. stack: 00007f2b88d31fd6 00007f2b88d31d90 00007f2b87d1148d 000000000096d544 000000000096d50d 000000000096ceda 00007f2b945d4ed0 00007f2b958c9463 00007f2b9836deb7 00007f2b983a0584 00007f2b98381690 00007f2b9838141d 0000000000a71ecc 00007f2b88d78b64 00007f2b88d7c18b 00007f2b88d7c008
0113 02:51:42.212803 (+ 48565us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:42.216398 (+ 3595us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:42.218087 (+ 1689us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":141522176,"tablet-delete.queue_time_us":4498347}
I20250113 02:51:42.218900 1195 tablet_replica.cc:331] T dc680b5680b84abbab219296bc7ee707 P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:42.219144 1195 raft_consensus.cc:2238] T dc680b5680b84abbab219296bc7ee707 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:42.219717 1195 raft_consensus.cc:2267] T dc680b5680b84abbab219296bc7ee707 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:42.220413 1195 ts_tablet_manager.cc:1905] T dc680b5680b84abbab219296bc7ee707 P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:42.220504 1192 ts_tablet_manager.cc:1428] T 3f62f344c5a648328ca2de8ca1069c19 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.015s user 0.001s sys 0.001s
I20250113 02:51:42.220844 1192 tablet_bootstrap.cc:492] T 938cc1ccd8dd449996836188ba9e64ff P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:42.222241 1192 tablet_bootstrap.cc:654] T 938cc1ccd8dd449996836188ba9e64ff P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:42.225206 2363 raft_consensus.cc:695] T 42dbd37e20fa47318bc6497423dbe565 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: Becoming Leader. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Running, Role: LEADER
I20250113 02:51:42.225597 2363 consensus_queue.cc:237] T 42dbd37e20fa47318bc6497423dbe565 P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
W20250113 02:51:42.226734 995 connection.cc:537] client connection to 127.0.104.60:42067 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250113 02:51:42.237150 708 connection.cc:537] client connection to 127.0.104.60:42067 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250113 02:51:42.240756 2549 negotiation.cc:337] Failed RPC negotiation. Trace:
0113 02:51:42.214610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.60:42067 (local address 127.0.104.2:44827)
0113 02:51:42.240280 (+ 25670us) negotiation.cc:107] Waiting for socket to connect
0113 02:51:42.240409 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.60:42067: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":25511,"threads_started":1}
I20250113 02:51:42.242030 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "438604e8859f422f9caccb61c5b5b125" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196"
I20250113 02:51:42.243036 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c153e47a7aa4bf5ae61dd19a91930b9 with delete_type TABLET_DATA_DELETED (replaced by tablet a5565bcb7761428fbf136ed8ce500642) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:42.243307 769 raft_consensus.cc:3054] T 438604e8859f422f9caccb61c5b5b125 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Advancing to term 1
W20250113 02:51:42.243532 439 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.60:42067, user_credentials={real_user=slave}} blocked reactor thread for 94074us
I20250113 02:51:42.243777 454 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_642669d7ef62419bb9a386144ab4e234" } new_table_name: "default.Table_42916505fd384cb985886cab6cde5421" modify_external_catalogs: true
I20250113 02:51:42.245852 748 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c153e47a7aa4bf5ae61dd19a91930b9 with delete_type TABLET_DATA_DELETED (replaced by tablet a5565bcb7761428fbf136ed8ce500642) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:42.247706 449 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_6df0545f12364f96a61aafaa9e3ff8f5" } modify_external_catalogs: true
I20250113 02:51:42.247908 449 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_6df0545f12364f96a61aafaa9e3ff8f5" } modify_external_catalogs: true
W20250113 02:51:42.251314 454 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c4761d692adc405e88d45ca6e748140b consensus state has no leader
I20250113 02:51:42.252772 1194 tablet_bootstrap.cc:492] T 9438472464304da9a0c7658d4f0736b6 P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:42.253026 1194 ts_tablet_manager.cc:1397] T 9438472464304da9a0c7658d4f0736b6 P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.125s user 0.008s sys 0.002s
I20250113 02:51:42.253103 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_ceb8d423519441d595d613778ab75514" } new_table_name: "default.Table_6a2072d7a9e34312b6d3185b274624f6" modify_external_catalogs: true
I20250113 02:51:42.253922 1194 raft_consensus.cc:357] T 9438472464304da9a0c7658d4f0736b6 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:42.254097 1194 raft_consensus.cc:383] T 9438472464304da9a0c7658d4f0736b6 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:42.254165 1194 raft_consensus.cc:738] T 9438472464304da9a0c7658d4f0736b6 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:42.254405 1194 consensus_queue.cc:260] T 9438472464304da9a0c7658d4f0736b6 P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:42.258466 1194 ts_tablet_manager.cc:1428] T 9438472464304da9a0c7658d4f0736b6 P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250113 02:51:42.258860 1194 tablet_bootstrap.cc:492] T d8bb60fdd06946739b75b255303d9d24 P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:42.260584 1194 tablet_bootstrap.cc:654] T d8bb60fdd06946739b75b255303d9d24 P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
W20250113 02:51:42.262394 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c3bab84ac06c4b868475868edae3625a consensus state has no leader
I20250113 02:51:42.264215 913 raft_consensus.cc:3054] T 0b119704bc774ca9b4155c9619d4765b P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:42.269582 1195 ts_tablet_manager.cc:1918] T dc680b5680b84abbab219296bc7ee707 P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:42.269795 1195 log.cc:1198] T dc680b5680b84abbab219296bc7ee707 P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/dc680b5680b84abbab219296bc7ee707
I20250113 02:51:42.270133 1195 ts_tablet_manager.cc:1939] T dc680b5680b84abbab219296bc7ee707 P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:42.263599 454 catalog_manager.cc:5618] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467) does not have the latest schema for tablet 9438472464304da9a0c7658d4f0736b6 (table default.Table_acdc5be1660742d7a356474b86e8b35f [id=aaf6558a15124f2b8eb9f01d692b845d]). Expected version 2 got 0
I20250113 02:51:42.272771 1194 tablet_bootstrap.cc:492] T d8bb60fdd06946739b75b255303d9d24 P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:42.273242 1194 ts_tablet_manager.cc:1397] T d8bb60fdd06946739b75b255303d9d24 P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.000s
I20250113 02:51:42.274294 1194 raft_consensus.cc:357] T d8bb60fdd06946739b75b255303d9d24 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:42.274930 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1112) took 4739 ms. Trace:
I20250113 02:51:42.275453 1194 raft_consensus.cc:383] T d8bb60fdd06946739b75b255303d9d24 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:42.275645 1194 raft_consensus.cc:738] T d8bb60fdd06946739b75b255303d9d24 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:42.276052 1194 consensus_queue.cc:260] T d8bb60fdd06946739b75b255303d9d24 P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:42.279884 1198 ts_tablet_manager.cc:1939] T 30ca8a716bfc4ab789b216e17567785e P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:42.276624 1195 rpcz_store.cc:276] 0113 02:51:37.535143 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:37.535328 (+ 185us) service_pool.cc:225] Handling call
0113 02:51:42.218850 (+4683522us) ts_tablet_manager.cc:1145] Deleting tablet dc680b5680b84abbab219296bc7ee707
0113 02:51:42.218860 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:42.229198 (+ 10338us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:42.269473 (+ 40275us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:42.274900 (+ 5427us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4683411}
I20250113 02:51:42.280869 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1110) took 4745 ms. Trace:
I20250113 02:51:42.281180 1195 rpcz_store.cc:276] 0113 02:51:37.534956 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:37.535300 (+ 344us) service_pool.cc:225] Handling call
0113 02:51:42.280726 (+4745426us) ts_tablet_manager.cc:1145] Deleting tablet 097cc4bd230046e2ac8551ca23d8f3e8
0113 02:51:42.280737 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:42.280851 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4745216}
I20250113 02:51:42.281898 1195 tablet_replica.cc:331] T 153b8f90b5a847b88944f683d306f579 P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:42.282037 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_48b46990944b44dfae31af6b82c145e3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:42.282938 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_48b46990944b44dfae31af6b82c145e3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:42.283085 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1160) took 3322 ms. Trace:
I20250113 02:51:42.283272 1198 rpcz_store.cc:276] 0113 02:51:38.960243 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:38.960290 (+ 47us) service_pool.cc:225] Handling call
0113 02:51:41.291518 (+2331228us) ts_tablet_manager.cc:1145] Deleting tablet 30ca8a716bfc4ab789b216e17567785e
0113 02:51:41.291529 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:41.505112 (+213583us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:41.586529 (+ 81417us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:42.283064 (+696535us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2331023}
I20250113 02:51:42.285117 1198 tablet_replica.cc:331] T ee240d838d4a4bbba97f8ed61f69de78 P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:42.285347 1198 raft_consensus.cc:2238] T ee240d838d4a4bbba97f8ed61f69de78 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:42.285526 1198 raft_consensus.cc:2267] T ee240d838d4a4bbba97f8ed61f69de78 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:42.285893 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet 30ca8a716bfc4ab789b216e17567785e (table default.Table_aac49fd87c1f4c3f9fc59bd8afe6e48f [id=9eaf97b701fd4e79847dce3a9d259beb]) successfully deleted
I20250113 02:51:42.284363 1192 tablet_bootstrap.cc:492] T 938cc1ccd8dd449996836188ba9e64ff P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:42.286183 1192 ts_tablet_manager.cc:1397] T 938cc1ccd8dd449996836188ba9e64ff P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.065s user 0.003s sys 0.003s
I20250113 02:51:42.286880 1192 raft_consensus.cc:357] T 938cc1ccd8dd449996836188ba9e64ff P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:42.287110 1192 raft_consensus.cc:383] T 938cc1ccd8dd449996836188ba9e64ff P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:42.287195 1192 raft_consensus.cc:738] T 938cc1ccd8dd449996836188ba9e64ff P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:42.287465 1192 consensus_queue.cc:260] T 938cc1ccd8dd449996836188ba9e64ff P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:42.288017 1192 ts_tablet_manager.cc:1428] T 938cc1ccd8dd449996836188ba9e64ff P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250113 02:51:42.288337 1192 tablet_bootstrap.cc:492] T 6a4c6dc08df744e6856bdf99ab65c575 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:42.290275 1192 tablet_bootstrap.cc:654] T 6a4c6dc08df744e6856bdf99ab65c575 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:42.292830 2508 raft_consensus.cc:491] T aa5c172a69bf4ca79999f4b58df08de8 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4216a7f853d642b7bc3e5cff458f3aca)
I20250113 02:51:42.293076 2508 raft_consensus.cc:513] T aa5c172a69bf4ca79999f4b58df08de8 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
W20250113 02:51:42.294818 711 leader_election.cc:343] T aa5c172a69bf4ca79999f4b58df08de8 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Not found: Tablet not found: aa5c172a69bf4ca79999f4b58df08de8
I20250113 02:51:42.295681 2508 leader_election.cc:290] T aa5c172a69bf4ca79999f4b58df08de8 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
W20250113 02:51:42.296146 709 leader_election.cc:343] T aa5c172a69bf4ca79999f4b58df08de8 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Not found: Tablet not found: aa5c172a69bf4ca79999f4b58df08de8
I20250113 02:51:42.296270 709 leader_election.cc:304] T aa5c172a69bf4ca79999f4b58df08de8 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:42.296370 1194 ts_tablet_manager.cc:1428] T d8bb60fdd06946739b75b255303d9d24 P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.023s user 0.001s sys 0.000s
I20250113 02:51:42.296630 1194 tablet_bootstrap.cc:492] T 80d31d5ead804ddaa032969e401a8f7e P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:42.296716 2473 raft_consensus.cc:2743] T aa5c172a69bf4ca79999f4b58df08de8 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250113 02:51:42.298345 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_bbd8e889c0eb420fbda882e52e6beb0b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:42.298624 1194 tablet_bootstrap.cc:654] T 80d31d5ead804ddaa032969e401a8f7e P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
W20250113 02:51:42.299022 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bbd8e889c0eb420fbda882e52e6beb0b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:42.300289 2476 raft_consensus.cc:491] T f21eb3fa0f414816a29e3d4560cdc0fe P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:42.300426 2476 raft_consensus.cc:513] T f21eb3fa0f414816a29e3d4560cdc0fe P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:42.300954 2476 leader_election.cc:290] T f21eb3fa0f414816a29e3d4560cdc0fe P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587), 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
W20250113 02:51:42.301853 851 leader_election.cc:343] T f21eb3fa0f414816a29e3d4560cdc0fe P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Not found: Tablet not found: f21eb3fa0f414816a29e3d4560cdc0fe
I20250113 02:51:42.302964 1198 ts_tablet_manager.cc:1905] T ee240d838d4a4bbba97f8ed61f69de78 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:42.308559 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_4ab048b5ba254ac6af616103105d618e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:42.309412 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4ab048b5ba254ac6af616103105d618e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:42.313347 1191 tablet_bootstrap.cc:492] T 8cc78bf7d6324b7ba10a1b1045923f2b P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:42.313532 1191 ts_tablet_manager.cc:1397] T 8cc78bf7d6324b7ba10a1b1045923f2b P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.328s user 0.013s sys 0.011s
W20250113 02:51:42.314128 441 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet c3bab84ac06c4b868475868edae3625a (table default.Table_6a2072d7a9e34312b6d3185b274624f6 [id=9fe419f25ec04d11902957063d32cefa], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet c3bab84ac06c4b868475868edae3625a consensus state has no leader
I20250113 02:51:42.314262 1191 raft_consensus.cc:357] T 8cc78bf7d6324b7ba10a1b1045923f2b P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:42.314607 1191 raft_consensus.cc:383] T 8cc78bf7d6324b7ba10a1b1045923f2b P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:42.314698 1191 raft_consensus.cc:738] T 8cc78bf7d6324b7ba10a1b1045923f2b P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:42.314792 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_854e525352dd47c889aaabfc312e33c7" } new_table_name: "default.Table_9bc4411bb05b4a2395c5933b630de698" modify_external_catalogs: true
I20250113 02:51:42.314946 1191 consensus_queue.cc:260] T 8cc78bf7d6324b7ba10a1b1045923f2b P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:42.316655 1191 ts_tablet_manager.cc:1428] T 8cc78bf7d6324b7ba10a1b1045923f2b P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250113 02:51:42.317796 1191 tablet_bootstrap.cc:492] T 019eb21c661547cfa0b619d119eb024e P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:42.317940 1194 tablet_bootstrap.cc:492] T 80d31d5ead804ddaa032969e401a8f7e P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:42.318082 1194 ts_tablet_manager.cc:1397] T 80d31d5ead804ddaa032969e401a8f7e P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.000s
I20250113 02:51:42.318747 1194 raft_consensus.cc:357] T 80d31d5ead804ddaa032969e401a8f7e P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:42.318929 1194 raft_consensus.cc:383] T 80d31d5ead804ddaa032969e401a8f7e P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:42.318989 1194 raft_consensus.cc:738] T 80d31d5ead804ddaa032969e401a8f7e P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:42.319191 1194 consensus_queue.cc:260] T 80d31d5ead804ddaa032969e401a8f7e P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:42.323232 914 raft_consensus.cc:3054] T 9438472464304da9a0c7658d4f0736b6 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Advancing to term 1
W20250113 02:51:42.329668 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6c4868fe133043169238e2d67ea7276b consensus state has no leader
I20250113 02:51:42.330250 1191 tablet_bootstrap.cc:654] T 019eb21c661547cfa0b619d119eb024e P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:42.332181 1195 raft_consensus.cc:2238] T 153b8f90b5a847b88944f683d306f579 P 1186fa869eb14812832fa6562531f196 [term 1 LEADER]: Raft consensus shutting down.
I20250113 02:51:42.332598 1195 pending_rounds.cc:62] T 153b8f90b5a847b88944f683d306f579 P 1186fa869eb14812832fa6562531f196: Trying to abort 1 pending ops.
I20250113 02:51:42.332656 1192 tablet_bootstrap.cc:492] T 6a4c6dc08df744e6856bdf99ab65c575 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:42.332650 1195 pending_rounds.cc:69] T 153b8f90b5a847b88944f683d306f579 P 1186fa869eb14812832fa6562531f196: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113673532058226688 op_type: NO_OP noop_request { }
I20250113 02:51:42.332782 1192 ts_tablet_manager.cc:1397] T 6a4c6dc08df744e6856bdf99ab65c575 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.045s user 0.003s sys 0.000s
I20250113 02:51:42.332751 1195 raft_consensus.cc:2883] T 153b8f90b5a847b88944f683d306f579 P 1186fa869eb14812832fa6562531f196 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250113 02:51:42.332852 1195 raft_consensus.cc:2267] T 153b8f90b5a847b88944f683d306f579 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:42.333437 1192 raft_consensus.cc:357] T 6a4c6dc08df744e6856bdf99ab65c575 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:42.333675 1192 raft_consensus.cc:383] T 6a4c6dc08df744e6856bdf99ab65c575 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:42.333784 1192 raft_consensus.cc:738] T 6a4c6dc08df744e6856bdf99ab65c575 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:42.334038 1192 consensus_queue.cc:260] T 6a4c6dc08df744e6856bdf99ab65c575 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:42.334642 1192 ts_tablet_manager.cc:1428] T 6a4c6dc08df744e6856bdf99ab65c575 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250113 02:51:42.337205 1192 tablet_bootstrap.cc:492] T b38cd34d67834a3f9bf5b5636bb79217 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:42.339151 1192 tablet_bootstrap.cc:654] T b38cd34d67834a3f9bf5b5636bb79217 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:42.339955 1194 ts_tablet_manager.cc:1428] T 80d31d5ead804ddaa032969e401a8f7e P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.022s user 0.001s sys 0.001s
I20250113 02:51:42.344544 1194 tablet_bootstrap.cc:492] T 30ca8a716bfc4ab789b216e17567785e P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:42.346848 1194 tablet_bootstrap.cc:654] T 30ca8a716bfc4ab789b216e17567785e P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:42.348675 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_30951971e07445449dd042657deb91ec"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:42.349308 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_30951971e07445449dd042657deb91ec in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:42.354133 1191 tablet_bootstrap.cc:492] T 019eb21c661547cfa0b619d119eb024e P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:42.354314 1191 ts_tablet_manager.cc:1397] T 019eb21c661547cfa0b619d119eb024e P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.037s user 0.004s sys 0.004s
I20250113 02:51:42.355057 1191 raft_consensus.cc:357] T 019eb21c661547cfa0b619d119eb024e P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:42.355297 1191 raft_consensus.cc:383] T 019eb21c661547cfa0b619d119eb024e P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:42.355392 1191 raft_consensus.cc:738] T 019eb21c661547cfa0b619d119eb024e P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:42.355626 1191 consensus_queue.cc:260] T 019eb21c661547cfa0b619d119eb024e P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:42.356165 1191 ts_tablet_manager.cc:1428] T 019eb21c661547cfa0b619d119eb024e P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250113 02:51:42.359347 1191 tablet_bootstrap.cc:492] T e01e7f8db83d461bbfbb2e901419ce0b P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:42.343910 1198 ts_tablet_manager.cc:1918] T ee240d838d4a4bbba97f8ed61f69de78 P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:42.360347 1198 log.cc:1198] T ee240d838d4a4bbba97f8ed61f69de78 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/ee240d838d4a4bbba97f8ed61f69de78
I20250113 02:51:42.347564 2499 tablet.cc:1721] T 0b119704bc774ca9b4155c9619d4765b P df69f48656b44bb890f76587ee69cc0f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250113 02:51:42.348243 1196 ts_tablet_manager.cc:1918] T 0916b41c171a419791797559fbc2807d P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:42.357801 913 raft_consensus.cc:3054] T 4eb5f1944cb44fd5a40ca92a6a3d7214 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:42.360612 1196 log.cc:1198] T 0916b41c171a419791797559fbc2807d P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/0916b41c171a419791797559fbc2807d
I20250113 02:51:42.360970 1196 ts_tablet_manager.cc:1939] T 0916b41c171a419791797559fbc2807d P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:42.361757 1196 tablet_replica.cc:331] T 33c665e149dd48e8868acd935de581bf P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:42.362088 1196 raft_consensus.cc:2238] T 33c665e149dd48e8868acd935de581bf P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:42.362323 1196 raft_consensus.cc:2267] T 33c665e149dd48e8868acd935de581bf P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:42.363104 1196 ts_tablet_manager.cc:1905] T 33c665e149dd48e8868acd935de581bf P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:42.364445 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_cba6acff6d514b32be48c6da57f28ab4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:42.365029 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cba6acff6d514b32be48c6da57f28ab4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:42.367792 1191 tablet_bootstrap.cc:654] T e01e7f8db83d461bbfbb2e901419ce0b P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:42.370090 1198 ts_tablet_manager.cc:1939] T ee240d838d4a4bbba97f8ed61f69de78 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:42.373164 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1161) took 3412 ms. Trace:
I20250113 02:51:42.373538 2501 raft_consensus.cc:491] T 16a132384b0e4264a73d09a3ea133893 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:42.373764 2501 raft_consensus.cc:513] T 16a132384b0e4264a73d09a3ea133893 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:42.375008 1058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16a132384b0e4264a73d09a3ea133893" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:42.375335 2501 leader_election.cc:290] T 16a132384b0e4264a73d09a3ea133893 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:42.375439 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16a132384b0e4264a73d09a3ea133893" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:42.375697 709 leader_election.cc:343] T 16a132384b0e4264a73d09a3ea133893 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:42.375684 1198 rpcz_store.cc:276] 0113 02:51:38.960504 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:38.960773 (+ 269us) service_pool.cc:225] Handling call
0113 02:51:42.285074 (+3324301us) ts_tablet_manager.cc:1145] Deleting tablet ee240d838d4a4bbba97f8ed61f69de78
0113 02:51:42.285083 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:42.306566 (+ 21483us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:42.343808 (+ 37242us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:42.373129 (+ 29321us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3324151}
W20250113 02:51:42.375934 711 leader_election.cc:343] T 16a132384b0e4264a73d09a3ea133893 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:42.376057 711 leader_election.cc:304] T 16a132384b0e4264a73d09a3ea133893 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:42.376422 2501 raft_consensus.cc:2743] T 16a132384b0e4264a73d09a3ea133893 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:42.376637 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_51c45c1994534c5593ec0872e3f10a9d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:42.377242 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_51c45c1994534c5593ec0872e3f10a9d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:42.377295 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet ee240d838d4a4bbba97f8ed61f69de78 (table default.Table_1b71e9ea2fca48daa60e8d692eab8e49 [id=383148ed836144c1bc5403dae46aad50]) successfully deleted
I20250113 02:51:42.377892 1198 tablet_replica.cc:331] T 1014e51f1c184d729552a5b34292f232 P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:42.378211 1198 raft_consensus.cc:2238] T 1014e51f1c184d729552a5b34292f232 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:42.381027 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_e5d873c74354424aad0f76593065cacd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:42.381708 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e5d873c74354424aad0f76593065cacd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:42.378410 1198 raft_consensus.cc:2267] T 1014e51f1c184d729552a5b34292f232 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:42.381479 2363 raft_consensus.cc:491] T dc680b5680b84abbab219296bc7ee707 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:42.385840 2363 raft_consensus.cc:513] T dc680b5680b84abbab219296bc7ee707 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:42.386687 2363 leader_election.cc:290] T dc680b5680b84abbab219296bc7ee707 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:42.387307 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc680b5680b84abbab219296bc7ee707" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:42.387809 1198 ts_tablet_manager.cc:1905] T 1014e51f1c184d729552a5b34292f232 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250113 02:51:42.388239 852 leader_election.cc:343] T f21eb3fa0f414816a29e3d4560cdc0fe P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Not found: Tablet not found: f21eb3fa0f414816a29e3d4560cdc0fe
W20250113 02:51:42.388331 998 leader_election.cc:343] T dc680b5680b84abbab219296bc7ee707 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:42.388355 852 leader_election.cc:304] T f21eb3fa0f414816a29e3d4560cdc0fe P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df69f48656b44bb890f76587ee69cc0f; no voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca
I20250113 02:51:42.388705 2476 raft_consensus.cc:2743] T f21eb3fa0f414816a29e3d4560cdc0fe P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250113 02:51:42.389708 995 leader_election.cc:343] T dc680b5680b84abbab219296bc7ee707 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Not found: Tablet not found: dc680b5680b84abbab219296bc7ee707
I20250113 02:51:42.389873 995 leader_election.cc:304] T dc680b5680b84abbab219296bc7ee707 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4216a7f853d642b7bc3e5cff458f3aca; no voters: 1186fa869eb14812832fa6562531f196, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:42.390141 2474 raft_consensus.cc:2743] T dc680b5680b84abbab219296bc7ee707 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:42.392359 2476 raft_consensus.cc:491] T e70658c90555463498d3fc2e2f13ea7f P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:42.392495 2476 raft_consensus.cc:513] T e70658c90555463498d3fc2e2f13ea7f P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:42.393115 2476 leader_election.cc:290] T e70658c90555463498d3fc2e2f13ea7f P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
W20250113 02:51:42.394227 851 leader_election.cc:343] T e70658c90555463498d3fc2e2f13ea7f P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Not found: Tablet not found: e70658c90555463498d3fc2e2f13ea7f
I20250113 02:51:42.394666 1195 ts_tablet_manager.cc:1905] T 153b8f90b5a847b88944f683d306f579 P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250113 02:51:42.397454 852 leader_election.cc:343] T e70658c90555463498d3fc2e2f13ea7f P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Not found: Tablet not found: e70658c90555463498d3fc2e2f13ea7f
I20250113 02:51:42.397599 852 leader_election.cc:304] T e70658c90555463498d3fc2e2f13ea7f P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df69f48656b44bb890f76587ee69cc0f; no voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca
I20250113 02:51:42.398059 2476 raft_consensus.cc:2743] T e70658c90555463498d3fc2e2f13ea7f P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250113 02:51:42.346951 2552 negotiation.cc:337] Failed RPC negotiation. Trace:
0113 02:51:42.228200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.60:42067 (local address 127.0.104.3:59211)
0113 02:51:42.346612 (+118412us) negotiation.cc:107] Waiting for socket to connect
0113 02:51:42.346709 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.60:42067: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":118303,"threads_started":1}
I20250113 02:51:42.399340 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_3a930068998b48628b755bc9782bc464"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:42.399935 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3a930068998b48628b755bc9782bc464 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:42.406616 1195 ts_tablet_manager.cc:1918] T 153b8f90b5a847b88944f683d306f579 P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250113 02:51:42.406765 1196 ts_tablet_manager.cc:1918] T 33c665e149dd48e8868acd935de581bf P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:42.406842 1195 log.cc:1198] T 153b8f90b5a847b88944f683d306f579 P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/153b8f90b5a847b88944f683d306f579
I20250113 02:51:42.407280 1195 ts_tablet_manager.cc:1939] T 153b8f90b5a847b88944f683d306f579 P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:42.407298 1196 log.cc:1198] T 33c665e149dd48e8868acd935de581bf P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/33c665e149dd48e8868acd935de581bf
I20250113 02:51:42.407665 1196 ts_tablet_manager.cc:1939] T 33c665e149dd48e8868acd935de581bf P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:42.408311 1196 tablet_replica.cc:331] T e70658c90555463498d3fc2e2f13ea7f P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:42.408288 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1113) took 4873 ms. Trace:
I20250113 02:51:42.408561 1196 raft_consensus.cc:2238] T e70658c90555463498d3fc2e2f13ea7f P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:42.408406 1195 rpcz_store.cc:276] 0113 02:51:37.535250 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:37.535468 (+ 218us) service_pool.cc:225] Handling call
0113 02:51:42.281854 (+4746386us) ts_tablet_manager.cc:1145] Deleting tablet 153b8f90b5a847b88944f683d306f579
0113 02:51:42.281863 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:42.332410 (+ 50547us) spinlock_profiling.cc:243] Waited 49.9 ms on lock 0x6160001e4654. stack: 00007f2b88d31fd6 00007f2b88d31d90 00007f2b87d1148d 000000000096d544 000000000096d50d 000000000096ceda 00007f2b945d4ed0 00007f2b958c9463 00007f2b9836deb7 00007f2b983a0584 00007f2b98381690 00007f2b9838141d 0000000000a71ecc 00007f2b88d78b64 00007f2b88d7c18b 00007f2b88d7c008
0113 02:51:42.403369 (+ 70959us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:42.406494 (+ 3125us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:42.408248 (+ 1754us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":114734976,"tablet-delete.queue_time_us":4746294}
I20250113 02:51:42.408839 1195 tablet_replica.cc:331] T f8a6eebb6b6d4b63903e68750e181707 P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:42.408974 1196 raft_consensus.cc:2267] T e70658c90555463498d3fc2e2f13ea7f P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:42.409075 1195 raft_consensus.cc:2238] T f8a6eebb6b6d4b63903e68750e181707 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:42.409221 1195 raft_consensus.cc:2267] T f8a6eebb6b6d4b63903e68750e181707 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:42.409261 2499 tablet.cc:1721] T 9438472464304da9a0c7658d4f0736b6 P df69f48656b44bb890f76587ee69cc0f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250113 02:51:42.409700 1196 ts_tablet_manager.cc:1905] T e70658c90555463498d3fc2e2f13ea7f P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:42.409746 1195 ts_tablet_manager.cc:1905] T f8a6eebb6b6d4b63903e68750e181707 P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:42.417654 1196 ts_tablet_manager.cc:1918] T e70658c90555463498d3fc2e2f13ea7f P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:42.417845 1196 log.cc:1198] T e70658c90555463498d3fc2e2f13ea7f P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/e70658c90555463498d3fc2e2f13ea7f
I20250113 02:51:42.418183 1196 ts_tablet_manager.cc:1939] T e70658c90555463498d3fc2e2f13ea7f P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:42.418901 1196 tablet_replica.cc:331] T a92f9b1c9c1a4cbdbf0cc7b2a3f2275b P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:42.419394 1196 raft_consensus.cc:2238] T a92f9b1c9c1a4cbdbf0cc7b2a3f2275b P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:42.419870 1196 raft_consensus.cc:2267] T a92f9b1c9c1a4cbdbf0cc7b2a3f2275b P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:42.420012 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_8a3f2660341f4712aa264ffc3e3a9a0d" } new_table_name: "default.Table_d383e658f8654d41bf998481e5566ba3" modify_external_catalogs: true
I20250113 02:51:42.420848 1196 ts_tablet_manager.cc:1905] T a92f9b1c9c1a4cbdbf0cc7b2a3f2275b P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:42.425732 2368 consensus_queue.cc:1035] T b815003a27f64809baaa2d43d6293cef P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 02:51:42.425850 769 raft_consensus.cc:2463] T 438604e8859f422f9caccb61c5b5b125 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 1.
I20250113 02:51:42.427090 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_40a3924489b2446697bd846b8ab9d284"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:42.428036 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_40a3924489b2446697bd846b8ab9d284 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:42.430408 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_d65b7b94ce864ac6a57e6b151433a14d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:42.430859 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d65b7b94ce864ac6a57e6b151433a14d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:42.432406 446 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_f9e0115a922e4a7c825265fc1f7222f9" } modify_external_catalogs: true
I20250113 02:51:42.432590 446 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_f9e0115a922e4a7c825265fc1f7222f9" } modify_external_catalogs: true
I20250113 02:51:42.433432 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_b7f2f1b2cac147dd9ee0a9af3186aaf9" } new_table_name: "default.Table_478a962b89de45c7abd8df5a83c3e277" modify_external_catalogs: true
I20250113 02:51:42.434762 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_32fe238c17574c1ea5721f507c0bdcc9" } modify_external_catalogs: true
I20250113 02:51:42.434914 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_32fe238c17574c1ea5721f507c0bdcc9" } modify_external_catalogs: true
I20250113 02:51:42.436269 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_8c95a106e0684de587166acbc0de43e1" } modify_external_catalogs: true
I20250113 02:51:42.436404 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_8c95a106e0684de587166acbc0de43e1" } modify_external_catalogs: true
I20250113 02:51:42.436655 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet 0c408921a09a463eb464e08edcf042b3 (table default.Table_08def01561d44ec5a88f5b5a82d47e3b [id=c24af26d1a734152af598a5aa258cb06]) successfully deleted
I20250113 02:51:42.436851 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet dc680b5680b84abbab219296bc7ee707 (table default.Table_25a2c280183748df92b064c40756f5ac [id=a1ac1dc79bce47258f1dbf33837b782c]) successfully deleted
W20250113 02:51:42.437193 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 097cc4bd230046e2ac8551ca23d8f3e8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 097cc4bd230046e2ac8551ca23d8f3e8 already in progress: creating tablet
I20250113 02:51:42.437430 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet 153b8f90b5a847b88944f683d306f579 (table default.Table_a7c3697e25a94815ac03a3cb1a57bc0c [id=39d8412c30d24c44a01c24b3c9645a8d]) successfully deleted
W20250113 02:51:42.437783 439 consensus_peers.cc:487] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9 -> Peer 45e9d43ecac0401fb2c18fe5eaf089b5 (127.0.104.60:42067): Couldn't send request to peer 45e9d43ecac0401fb2c18fe5eaf089b5. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.60:42067: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250113 02:51:42.434653 878 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:37356 (request call id 267) took 1478 ms. Trace:
I20250113 02:51:42.437350 883 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c153e47a7aa4bf5ae61dd19a91930b9 with delete_type TABLET_DATA_DELETED (replaced by tablet a5565bcb7761428fbf136ed8ce500642) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:42.440111 878 rpcz_store.cc:276] 0113 02:51:40.956100 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:40.957733 (+ 1633us) service_pool.cc:225] Handling call
0113 02:51:41.432704 (+474971us) spinlock_profiling.cc:243] Waited 33 ms on lock 0x6180001c048c. stack: 00007f0b1ea7cfd6 00007f0b1ea7cd90 00007f0b1da5c48d 000000000096d544 000000000096d50d 000000000096ceda 00007f0b2b81f8ee 00007f0b2b61e1df 00007f0b2e02c948 00007f0b289878e6 00007f0b28987615 00007f0b257e430e 00007f0b257e31ee 00007f0b257e9425 00007f0b257e9d4f 00007f0b257e9b78
0113 02:51:42.434622 (+1001918us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":75807616}
I20250113 02:51:42.443504 995 leader_election.cc:304] T 438604e8859f422f9caccb61c5b5b125 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters: df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:42.443923 2368 raft_consensus.cc:2798] T 438604e8859f422f9caccb61c5b5b125 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Leader election won for term 1
I20250113 02:51:42.444061 2368 raft_consensus.cc:695] T 438604e8859f422f9caccb61c5b5b125 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: Becoming Leader. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Running, Role: LEADER
I20250113 02:51:42.444300 2368 consensus_queue.cc:237] T 438604e8859f422f9caccb61c5b5b125 P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
W20250113 02:51:42.449849 998 consensus_peers.cc:487] T b815003a27f64809baaa2d43d6293cef P 4216a7f853d642b7bc3e5cff458f3aca -> Peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Couldn't send request to peer df69f48656b44bb890f76587ee69cc0f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250113 02:51:42.453601 1196 ts_tablet_manager.cc:1918] T a92f9b1c9c1a4cbdbf0cc7b2a3f2275b P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:42.453725 1196 log.cc:1198] T a92f9b1c9c1a4cbdbf0cc7b2a3f2275b P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/a92f9b1c9c1a4cbdbf0cc7b2a3f2275b
I20250113 02:51:42.454030 1196 ts_tablet_manager.cc:1939] T a92f9b1c9c1a4cbdbf0cc7b2a3f2275b P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:42.458601 2501 raft_consensus.cc:491] T 019eb21c661547cfa0b619d119eb024e P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:42.458755 2501 raft_consensus.cc:513] T 019eb21c661547cfa0b619d119eb024e P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:42.459383 2501 leader_election.cc:290] T 019eb21c661547cfa0b619d119eb024e P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:42.459872 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "019eb21c661547cfa0b619d119eb024e" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:42.460039 1058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "019eb21c661547cfa0b619d119eb024e" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
W20250113 02:51:42.460588 709 leader_election.cc:343] T 019eb21c661547cfa0b619d119eb024e P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
W20250113 02:51:42.460722 711 leader_election.cc:343] T 019eb21c661547cfa0b619d119eb024e P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:42.460855 711 leader_election.cc:304] T 019eb21c661547cfa0b619d119eb024e P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:42.461462 2501 raft_consensus.cc:2743] T 019eb21c661547cfa0b619d119eb024e P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:42.462083 1196 tablet_replica.cc:331] T 7213e3284b4d4cb581d9fb946a9f87e9 P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:42.462262 1196 raft_consensus.cc:2238] T 7213e3284b4d4cb581d9fb946a9f87e9 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:42.462395 1196 raft_consensus.cc:2267] T 7213e3284b4d4cb581d9fb946a9f87e9 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:42.462852 1196 ts_tablet_manager.cc:1905] T 7213e3284b4d4cb581d9fb946a9f87e9 P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250113 02:51:42.464583 998 consensus_peers.cc:487] T b33025a462e74f2aa26cc3779f36615b P 4216a7f853d642b7bc3e5cff458f3aca -> Peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Couldn't send request to peer df69f48656b44bb890f76587ee69cc0f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250113 02:51:42.471412 1198 ts_tablet_manager.cc:1918] T 1014e51f1c184d729552a5b34292f232 P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:42.471627 1198 log.cc:1198] T 1014e51f1c184d729552a5b34292f232 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/1014e51f1c184d729552a5b34292f232
I20250113 02:51:42.472186 1198 ts_tablet_manager.cc:1939] T 1014e51f1c184d729552a5b34292f232 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:42.473258 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1159) took 3513 ms. Trace:
I20250113 02:51:42.474870 1198 rpcz_store.cc:276] 0113 02:51:38.960030 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:38.961340 (+ 1310us) service_pool.cc:225] Handling call
0113 02:51:42.377816 (+3416476us) ts_tablet_manager.cc:1145] Deleting tablet 1014e51f1c184d729552a5b34292f232
0113 02:51:42.377832 (+ 16us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:42.450336 (+ 72504us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:42.471311 (+ 20975us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:42.473218 (+ 1907us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3416308}
I20250113 02:51:42.479143 1196 ts_tablet_manager.cc:1918] T 7213e3284b4d4cb581d9fb946a9f87e9 P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:42.479256 1196 log.cc:1198] T 7213e3284b4d4cb581d9fb946a9f87e9 P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/7213e3284b4d4cb581d9fb946a9f87e9
I20250113 02:51:42.479534 1196 ts_tablet_manager.cc:1939] T 7213e3284b4d4cb581d9fb946a9f87e9 P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:42.480484 1196 tablet_replica.cc:331] T f21eb3fa0f414816a29e3d4560cdc0fe P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:42.485797 1196 raft_consensus.cc:2238] T f21eb3fa0f414816a29e3d4560cdc0fe P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:42.485987 1196 raft_consensus.cc:2267] T f21eb3fa0f414816a29e3d4560cdc0fe P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:42.486621 1196 ts_tablet_manager.cc:1905] T f21eb3fa0f414816a29e3d4560cdc0fe P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250113 02:51:42.487488 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0e2d8726a2ab489cae80fb1685d6951a consensus state has no leader
I20250113 02:51:42.495909 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet eebad045e7bb43fa9ca9c9af8b3b7ea5 (table default.Table_abd20980e42242719c1529668d68128b [id=043d63161d5e4454af0782a50d64d256]) successfully deleted
I20250113 02:51:42.497052 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet 0916b41c171a419791797559fbc2807d (table default.Table_8be7ebd7e1984b979fce6e7aac1f576e [id=8231428f86ba4a6388c55c3c06f6b194]) successfully deleted
I20250113 02:51:42.497318 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet 33c665e149dd48e8868acd935de581bf (table default.Table_ec91552e358547e99d908cef4bcb7c36 [id=0ebda43313f249d6b35f5beda3d5fc0e]) successfully deleted
I20250113 02:51:42.497491 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet e70658c90555463498d3fc2e2f13ea7f (table default.Table_2565c0551f9943a084ace8f7e463e59d [id=fe6f8d302c3e489d8bb832503fd05642]) successfully deleted
I20250113 02:51:42.497673 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet a92f9b1c9c1a4cbdbf0cc7b2a3f2275b (table default.Table_6185ed1a90a44576bc65ed60294c2537 [id=5634c185da26460f93cde9c65fbdb24a]) successfully deleted
W20250113 02:51:42.497898 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 316cb2863ed645158a8f72e3cb0a94fd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 316cb2863ed645158a8f72e3cb0a94fd already in progress: creating tablet
W20250113 02:51:42.498147 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 4cc15487be474354b995c4e415276a18 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4cc15487be474354b995c4e415276a18 already in progress: creating tablet
I20250113 02:51:42.498353 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet 7213e3284b4d4cb581d9fb946a9f87e9 (table default.Table_ff6df4afcf4d4f9a964d079ee24f02b5 [id=8a5901fc685345579a254a0f3af32051]) successfully deleted
W20250113 02:51:42.478104 998 consensus_peers.cc:487] T 438d1d2300e044f39a15586f335935ba P 4216a7f853d642b7bc3e5cff458f3aca -> Peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Couldn't send request to peer df69f48656b44bb890f76587ee69cc0f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 438d1d2300e044f39a15586f335935ba. This is attempt 1: this message will repeat every 5th retry.
W20250113 02:51:42.498553 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 85d13f8abbec484d82c4d0b86480531b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 85d13f8abbec484d82c4d0b86480531b already in progress: creating tablet
W20250113 02:51:42.498713 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 8ee0b2e06dc74faab48912dd16fe1537 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8ee0b2e06dc74faab48912dd16fe1537 already in progress: creating tablet
W20250113 02:51:42.498869 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 9b9e7deda7334ff4a3429f70d9b3a7fc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9b9e7deda7334ff4a3429f70d9b3a7fc already in progress: creating tablet
I20250113 02:51:42.499778 1196 ts_tablet_manager.cc:1918] T f21eb3fa0f414816a29e3d4560cdc0fe P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:42.499904 1196 log.cc:1198] T f21eb3fa0f414816a29e3d4560cdc0fe P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/f21eb3fa0f414816a29e3d4560cdc0fe
I20250113 02:51:42.499904 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet 1014e51f1c184d729552a5b34292f232 (table default.Table_2abef959c5c34459a8dfa3034e207995 [id=f7e15a7c1fe848d591ba3232281c45ed]) successfully deleted
I20250113 02:51:42.500244 1196 ts_tablet_manager.cc:1939] T f21eb3fa0f414816a29e3d4560cdc0fe P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:42.500232 1198 tablet_replica.cc:331] T 80d31d5ead804ddaa032969e401a8f7e P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:42.500468 1198 raft_consensus.cc:2238] T 80d31d5ead804ddaa032969e401a8f7e P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:42.500701 1198 raft_consensus.cc:2267] T 80d31d5ead804ddaa032969e401a8f7e P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:42.500710 453 master_service.cc:946] ReplaceTablet: received request to replace tablet 368e5f00861b40ffb76e79376245b885 from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:42.500989 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet f21eb3fa0f414816a29e3d4560cdc0fe (table default.Table_6a3d6ef326e74c4ab0a5ed1e1bec2ecc [id=02e7cb8431884f67989a39c2ffab28ad]) successfully deleted
I20250113 02:51:42.501904 1198 ts_tablet_manager.cc:1905] T 80d31d5ead804ddaa032969e401a8f7e P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:42.503197 1196 tablet_replica.cc:331] T df87e25053a74990959f85288a85217b P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:42.503391 1196 raft_consensus.cc:2238] T df87e25053a74990959f85288a85217b P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:42.503548 1196 raft_consensus.cc:2267] T df87e25053a74990959f85288a85217b P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:42.505822 1196 ts_tablet_manager.cc:1905] T df87e25053a74990959f85288a85217b P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:42.519299 1196 ts_tablet_manager.cc:1918] T df87e25053a74990959f85288a85217b P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:42.519430 1196 log.cc:1198] T df87e25053a74990959f85288a85217b P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/df87e25053a74990959f85288a85217b
I20250113 02:51:42.519698 1196 ts_tablet_manager.cc:1939] T df87e25053a74990959f85288a85217b P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:42.520280 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet df87e25053a74990959f85288a85217b (table default.Table_2f1a7bb2c4cd4ead95b175e1d3545f20 [id=b2c8431e06ac46be9840edf78b6c19a3]) successfully deleted
W20250113 02:51:42.524044 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 1c153e47a7aa4bf5ae61dd19a91930b9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1c153e47a7aa4bf5ae61dd19a91930b9 already in progress: creating tablet
I20250113 02:51:42.529294 1192 tablet_bootstrap.cc:492] T b38cd34d67834a3f9bf5b5636bb79217 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:42.529430 1192 ts_tablet_manager.cc:1397] T b38cd34d67834a3f9bf5b5636bb79217 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.192s user 0.006s sys 0.010s
I20250113 02:51:42.530295 1192 raft_consensus.cc:357] T b38cd34d67834a3f9bf5b5636bb79217 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:42.530535 1192 raft_consensus.cc:383] T b38cd34d67834a3f9bf5b5636bb79217 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:42.530597 1192 raft_consensus.cc:738] T b38cd34d67834a3f9bf5b5636bb79217 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:42.530819 1192 consensus_queue.cc:260] T b38cd34d67834a3f9bf5b5636bb79217 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:42.534153 1192 ts_tablet_manager.cc:1428] T b38cd34d67834a3f9bf5b5636bb79217 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250113 02:51:42.534471 1192 tablet_bootstrap.cc:492] T aa1ebdb92a5f4c92a5195b614689c100 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:42.538890 2547 raft_consensus.cc:491] T 3f62f344c5a648328ca2de8ca1069c19 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250113 02:51:42.540531 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f85bc8e8042e40f4909a0c3e1337405e consensus state has no leader
I20250113 02:51:42.540578 1192 tablet_bootstrap.cc:654] T aa1ebdb92a5f4c92a5195b614689c100 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:42.541226 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 86c8c07f90d2470ebb2b3f1d16daf4cd
I20250113 02:51:42.542055 883 tablet_service.cc:1514] Processing DeleteTablet for tablet 86c8c07f90d2470ebb2b3f1d16daf4cd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:42 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:42.542383 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet 86c8c07f90d2470ebb2b3f1d16daf4cd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:42 UTC) from {username='slave'} at 127.0.0.1:60230
W20250113 02:51:42.543087 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 86c8c07f90d2470ebb2b3f1d16daf4cd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 86c8c07f90d2470ebb2b3f1d16daf4cd already in progress: creating tablet
I20250113 02:51:42.539063 2547 raft_consensus.cc:513] T 3f62f344c5a648328ca2de8ca1069c19 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:42.544096 1194 tablet_bootstrap.cc:492] T 30ca8a716bfc4ab789b216e17567785e P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:42.544310 1194 ts_tablet_manager.cc:1397] T 30ca8a716bfc4ab789b216e17567785e P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.200s user 0.005s sys 0.012s
I20250113 02:51:42.544534 748 tablet_service.cc:1514] Processing DeleteTablet for tablet 86c8c07f90d2470ebb2b3f1d16daf4cd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:42 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:42.545209 1194 raft_consensus.cc:357] T 30ca8a716bfc4ab789b216e17567785e P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:42.545473 1194 raft_consensus.cc:383] T 30ca8a716bfc4ab789b216e17567785e P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:42.545531 1194 raft_consensus.cc:738] T 30ca8a716bfc4ab789b216e17567785e P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:42.545755 1194 consensus_queue.cc:260] T 30ca8a716bfc4ab789b216e17567785e P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:42.546334 1194 ts_tablet_manager.cc:1428] T 30ca8a716bfc4ab789b216e17567785e P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250113 02:51:42.546624 1194 tablet_bootstrap.cc:492] T 9e736211eb3c4536b9fd1d90d2d83838 P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:42.548236 2547 leader_election.cc:290] T 3f62f344c5a648328ca2de8ca1069c19 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:42.548619 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f62f344c5a648328ca2de8ca1069c19" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:42.549571 998 leader_election.cc:343] T 3f62f344c5a648328ca2de8ca1069c19 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:42.550597 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f62f344c5a648328ca2de8ca1069c19" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
I20250113 02:51:42.550843 769 raft_consensus.cc:2463] T 3f62f344c5a648328ca2de8ca1069c19 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 0.
I20250113 02:51:42.551402 995 leader_election.cc:304] T 3f62f344c5a648328ca2de8ca1069c19 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters: df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:42.551923 1194 tablet_bootstrap.cc:654] T 9e736211eb3c4536b9fd1d90d2d83838 P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:42.556218 447 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_49ba362618104736951a80e9988c5645" } modify_external_catalogs: true
I20250113 02:51:42.556433 447 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_49ba362618104736951a80e9988c5645" } modify_external_catalogs: true
I20250113 02:51:42.561038 448 catalog_manager.cc:5618] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467) does not have the latest schema for tablet 9438472464304da9a0c7658d4f0736b6 (table default.Table_acdc5be1660742d7a356474b86e8b35f [id=aaf6558a15124f2b8eb9f01d692b845d]). Expected version 2 got 0
I20250113 02:51:42.561220 448 catalog_manager.cc:5618] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467) does not have the latest schema for tablet 0b119704bc774ca9b4155c9619d4765b (table default.Table_b95a9b1d12284c9aac701a8ef9c93482 [id=843aabcff7074acd98e5801b21282faa]). Expected version 1 got 0
I20250113 02:51:42.561728 2547 raft_consensus.cc:2798] T 3f62f344c5a648328ca2de8ca1069c19 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 02:51:42.561831 2547 raft_consensus.cc:491] T 3f62f344c5a648328ca2de8ca1069c19 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 02:51:42.561911 2547 raft_consensus.cc:3054] T 3f62f344c5a648328ca2de8ca1069c19 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:42.564303 748 tablet_service.cc:1514] Processing DeleteTablet for tablet f105f8b32a654e4ca154e8fc0cb96091 with delete_type TABLET_DATA_DELETED (replaced by tablet 65e09181ee4c4061a7b5a5353be440cf) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:42.564864 748 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c408921a09a463eb464e08edcf042b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:24 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:42.565574 748 tablet_service.cc:1514] Processing DeleteTablet for tablet 16a132384b0e4264a73d09a3ea133893 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:31 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:42.567648 2547 raft_consensus.cc:513] T 3f62f344c5a648328ca2de8ca1069c19 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:42.568994 2547 leader_election.cc:290] T 3f62f344c5a648328ca2de8ca1069c19 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Requested vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:42.572431 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f62f344c5a648328ca2de8ca1069c19" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196"
I20250113 02:51:42.572611 772 raft_consensus.cc:3054] T 3f62f344c5a648328ca2de8ca1069c19 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:42.572297 1198 ts_tablet_manager.cc:1918] T 80d31d5ead804ddaa032969e401a8f7e P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:42.573117 1198 log.cc:1198] T 80d31d5ead804ddaa032969e401a8f7e P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/80d31d5ead804ddaa032969e401a8f7e
I20250113 02:51:42.573505 1198 ts_tablet_manager.cc:1939] T 80d31d5ead804ddaa032969e401a8f7e P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:42.574447 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1158) took 3614 ms. Trace:
I20250113 02:51:42.574548 1198 rpcz_store.cc:276] 0113 02:51:38.959694 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:38.961521 (+ 1827us) service_pool.cc:225] Handling call
0113 02:51:42.500168 (+3538647us) ts_tablet_manager.cc:1145] Deleting tablet 80d31d5ead804ddaa032969e401a8f7e
0113 02:51:42.500183 (+ 15us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:42.531934 (+ 31751us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:42.572141 (+ 40207us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:42.574423 (+ 2282us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3538530}
I20250113 02:51:42.575129 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet 80d31d5ead804ddaa032969e401a8f7e (table default.Table_2d201bcde7f04c76b66dd5c6a09ddbff [id=e0d56fb485af49c8bcf9152f95f89b2a]) successfully deleted
I20250113 02:51:42.575197 772 raft_consensus.cc:2463] T 3f62f344c5a648328ca2de8ca1069c19 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 1.
I20250113 02:51:42.575654 995 leader_election.cc:304] T 3f62f344c5a648328ca2de8ca1069c19 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters:
I20250113 02:51:42.575999 2547 raft_consensus.cc:2798] T 3f62f344c5a648328ca2de8ca1069c19 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Leader election won for term 1
I20250113 02:51:42.576490 2547 raft_consensus.cc:695] T 3f62f344c5a648328ca2de8ca1069c19 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: Becoming Leader. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Running, Role: LEADER
I20250113 02:51:42.576882 2547 consensus_queue.cc:237] T 3f62f344c5a648328ca2de8ca1069c19 P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:42.580305 1192 tablet_bootstrap.cc:492] T aa1ebdb92a5f4c92a5195b614689c100 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:42.580497 1192 ts_tablet_manager.cc:1397] T aa1ebdb92a5f4c92a5195b614689c100 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.046s user 0.006s sys 0.002s
I20250113 02:51:42.581271 1192 raft_consensus.cc:357] T aa1ebdb92a5f4c92a5195b614689c100 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:42.581523 1192 raft_consensus.cc:383] T aa1ebdb92a5f4c92a5195b614689c100 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:42.581614 1192 raft_consensus.cc:738] T aa1ebdb92a5f4c92a5195b614689c100 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:42.581864 1192 consensus_queue.cc:260] T aa1ebdb92a5f4c92a5195b614689c100 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:42.582424 1192 ts_tablet_manager.cc:1428] T aa1ebdb92a5f4c92a5195b614689c100 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250113 02:51:42.582719 1192 tablet_bootstrap.cc:492] T e570465a3d4046b1ac19a3a034fe91f1 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:42.584714 1192 tablet_bootstrap.cc:654] T e570465a3d4046b1ac19a3a034fe91f1 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:42.569078 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f62f344c5a648328ca2de8ca1069c19" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f"
I20250113 02:51:42.581768 883 tablet_service.cc:1514] Processing DeleteTablet for tablet 80d31d5ead804ddaa032969e401a8f7e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:22 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:42.582320 891 tablet_service.cc:1514] Processing DeleteTablet for tablet d8bb60fdd06946739b75b255303d9d24 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:26 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:42.582803 885 tablet_service.cc:1514] Processing DeleteTablet for tablet 30ca8a716bfc4ab789b216e17567785e with delete_type TABLET_DATA_DELETED (replaced by tablet 5f7c91d8c018455e847e5969f7b5e1cc) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:42.585335 1196 tablet_replica.cc:331] T 80d31d5ead804ddaa032969e401a8f7e P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:42.585552 1196 raft_consensus.cc:2238] T 80d31d5ead804ddaa032969e401a8f7e P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:42.585703 1196 raft_consensus.cc:2267] T 80d31d5ead804ddaa032969e401a8f7e P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:42.587101 1196 ts_tablet_manager.cc:1905] T 80d31d5ead804ddaa032969e401a8f7e P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250113 02:51:42.585511 998 leader_election.cc:343] T 3f62f344c5a648328ca2de8ca1069c19 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:42.589715 2501 raft_consensus.cc:491] T 938cc1ccd8dd449996836188ba9e64ff P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:42.590389 2501 raft_consensus.cc:513] T 938cc1ccd8dd449996836188ba9e64ff P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:42.591065 2501 leader_election.cc:290] T 938cc1ccd8dd449996836188ba9e64ff P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:42.580384 1195 ts_tablet_manager.cc:1918] T f8a6eebb6b6d4b63903e68750e181707 P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250113 02:51:42.591922 1195 log.cc:1198] T f8a6eebb6b6d4b63903e68750e181707 P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/f8a6eebb6b6d4b63903e68750e181707
I20250113 02:51:42.592356 1195 ts_tablet_manager.cc:1939] T f8a6eebb6b6d4b63903e68750e181707 P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:42.592283 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "938cc1ccd8dd449996836188ba9e64ff" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:42.592931 711 leader_election.cc:343] T 938cc1ccd8dd449996836188ba9e64ff P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:42.593154 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1109) took 5058 ms. Trace:
I20250113 02:51:42.593236 1195 rpcz_store.cc:276] 0113 02:51:37.534826 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:37.535958 (+ 1132us) service_pool.cc:225] Handling call
0113 02:51:42.408797 (+4872839us) ts_tablet_manager.cc:1145] Deleting tablet f8a6eebb6b6d4b63903e68750e181707
0113 02:51:42.408808 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:42.535234 (+126426us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:42.580271 (+ 45037us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:42.593136 (+ 12865us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4872668}
I20250113 02:51:42.593405 1058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "938cc1ccd8dd449996836188ba9e64ff" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:42.593595 1058 raft_consensus.cc:2463] T 938cc1ccd8dd449996836188ba9e64ff P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1186fa869eb14812832fa6562531f196 in term 0.
I20250113 02:51:42.593672 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet f8a6eebb6b6d4b63903e68750e181707 (table default.Table_4484fe40b4454c5ea4943a4d38f49c5e [id=a0e91da6e0ee449b92b7afbc6f68087d]) successfully deleted
I20250113 02:51:42.594094 709 leader_election.cc:304] T 938cc1ccd8dd449996836188ba9e64ff P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters: df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:42.594552 2501 raft_consensus.cc:2798] T 938cc1ccd8dd449996836188ba9e64ff P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 02:51:42.594628 2501 raft_consensus.cc:491] T 938cc1ccd8dd449996836188ba9e64ff P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 02:51:42.594697 2501 raft_consensus.cc:3054] T 938cc1ccd8dd449996836188ba9e64ff P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:42.594949 1196 ts_tablet_manager.cc:1918] T 80d31d5ead804ddaa032969e401a8f7e P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:42.595059 1196 log.cc:1198] T 80d31d5ead804ddaa032969e401a8f7e P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/80d31d5ead804ddaa032969e401a8f7e
I20250113 02:51:42.595378 1196 ts_tablet_manager.cc:1939] T 80d31d5ead804ddaa032969e401a8f7e P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:42.595834 1196 tablet_replica.cc:331] T d8bb60fdd06946739b75b255303d9d24 P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:42.596212 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet 80d31d5ead804ddaa032969e401a8f7e (table default.Table_2d201bcde7f04c76b66dd5c6a09ddbff [id=e0d56fb485af49c8bcf9152f95f89b2a]) successfully deleted
I20250113 02:51:42.599963 1196 raft_consensus.cc:2238] T d8bb60fdd06946739b75b255303d9d24 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:42.600183 1196 raft_consensus.cc:2267] T d8bb60fdd06946739b75b255303d9d24 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:42.600318 2501 raft_consensus.cc:513] T 938cc1ccd8dd449996836188ba9e64ff P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:42.601447 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "938cc1ccd8dd449996836188ba9e64ff" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f"
W20250113 02:51:42.601954 711 leader_election.cc:343] T 938cc1ccd8dd449996836188ba9e64ff P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:42.602414 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 125ebde7a9f14918ba328a75e88a7746
I20250113 02:51:42.603444 748 tablet_service.cc:1514] Processing DeleteTablet for tablet 125ebde7a9f14918ba328a75e88a7746 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:42 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:42.603662 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet 125ebde7a9f14918ba328a75e88a7746 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:42 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:42.604104 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 166f58d45b7b443ea54087e4ab48eaba
I20250113 02:51:42.604235 885 tablet_service.cc:1514] Processing DeleteTablet for tablet 125ebde7a9f14918ba328a75e88a7746 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:42 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:42.604777 1196 ts_tablet_manager.cc:1905] T d8bb60fdd06946739b75b255303d9d24 P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:42.605036 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet 166f58d45b7b443ea54087e4ab48eaba with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:42 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:42.605123 748 tablet_service.cc:1514] Processing DeleteTablet for tablet 166f58d45b7b443ea54087e4ab48eaba with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:42 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:42.605266 885 tablet_service.cc:1514] Processing DeleteTablet for tablet 166f58d45b7b443ea54087e4ab48eaba with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:42 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:42.606179 2501 leader_election.cc:290] T 938cc1ccd8dd449996836188ba9e64ff P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 election: Requested vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:42.606784 448 catalog_manager.cc:5618] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095) does not have the latest schema for tablet b815003a27f64809baaa2d43d6293cef (table default.Table_c2e8fd7d2cc14fa39c0abfece783c6c9 [id=42e19138c08f422a92ba9621e9609957]). Expected version 1 got 0
I20250113 02:51:42.607030 448 catalog_manager.cc:5618] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095) does not have the latest schema for tablet 019eb21c661547cfa0b619d119eb024e (table default.Table_022a7c61a6864393acd739f562893b7c [id=97bfbd46eecd4e57b7f9757c413f6795]). Expected version 1 got 0
I20250113 02:51:42.608530 748 tablet_service.cc:1514] Processing DeleteTablet for tablet 8cc78bf7d6324b7ba10a1b1045923f2b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:32 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:42.624970 1194 tablet_bootstrap.cc:492] T 9e736211eb3c4536b9fd1d90d2d83838 P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:42.625144 1194 ts_tablet_manager.cc:1397] T 9e736211eb3c4536b9fd1d90d2d83838 P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.079s user 0.001s sys 0.003s
I20250113 02:51:42.626106 1194 raft_consensus.cc:357] T 9e736211eb3c4536b9fd1d90d2d83838 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:42.626361 1194 raft_consensus.cc:383] T 9e736211eb3c4536b9fd1d90d2d83838 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:42.626413 1194 raft_consensus.cc:738] T 9e736211eb3c4536b9fd1d90d2d83838 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:42.626684 1194 consensus_queue.cc:260] T 9e736211eb3c4536b9fd1d90d2d83838 P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:42.630956 1194 ts_tablet_manager.cc:1428] T 9e736211eb3c4536b9fd1d90d2d83838 P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.006s user 0.001s sys 0.002s
I20250113 02:51:42.631318 1194 tablet_bootstrap.cc:492] T f8a6eebb6b6d4b63903e68750e181707 P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:42.632241 2368 raft_consensus.cc:491] T 6a4c6dc08df744e6856bdf99ab65c575 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:42.632426 2368 raft_consensus.cc:513] T 6a4c6dc08df744e6856bdf99ab65c575 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:42.633137 2368 leader_election.cc:290] T 6a4c6dc08df744e6856bdf99ab65c575 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:42.633809 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a4c6dc08df744e6856bdf99ab65c575" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:42.634406 998 leader_election.cc:343] T 6a4c6dc08df744e6856bdf99ab65c575 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:42.635135 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a4c6dc08df744e6856bdf99ab65c575" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
I20250113 02:51:42.635386 772 raft_consensus.cc:2463] T 6a4c6dc08df744e6856bdf99ab65c575 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 0.
I20250113 02:51:42.635895 995 leader_election.cc:304] T 6a4c6dc08df744e6856bdf99ab65c575 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters: df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:42.636415 2368 raft_consensus.cc:2798] T 6a4c6dc08df744e6856bdf99ab65c575 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 02:51:42.636502 2368 raft_consensus.cc:491] T 6a4c6dc08df744e6856bdf99ab65c575 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 02:51:42.636576 2368 raft_consensus.cc:3054] T 6a4c6dc08df744e6856bdf99ab65c575 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:42.639729 2368 raft_consensus.cc:513] T 6a4c6dc08df744e6856bdf99ab65c575 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:42.641064 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a4c6dc08df744e6856bdf99ab65c575" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f"
W20250113 02:51:42.641644 998 leader_election.cc:343] T 6a4c6dc08df744e6856bdf99ab65c575 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:42.641783 1058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "938cc1ccd8dd449996836188ba9e64ff" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca"
I20250113 02:51:42.641973 1058 raft_consensus.cc:3054] T 938cc1ccd8dd449996836188ba9e64ff P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:42.642380 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a4c6dc08df744e6856bdf99ab65c575" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196"
I20250113 02:51:42.642585 772 raft_consensus.cc:3054] T 6a4c6dc08df744e6856bdf99ab65c575 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:42.644603 1058 raft_consensus.cc:2463] T 938cc1ccd8dd449996836188ba9e64ff P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1186fa869eb14812832fa6562531f196 in term 1.
I20250113 02:51:42.645920 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1176) took 3431 ms. Trace:
I20250113 02:51:42.646157 455 catalog_manager.cc:5618] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587) does not have the latest schema for tablet c13e9a51c1e74b44b352bade78390ba5 (table default.Table_0468611ec68e47109f5357935e50874d [id=caf098e2b57849a997bd7c8eaf1f9326]). Expected version 1 got 0
I20250113 02:51:42.647739 772 raft_consensus.cc:2463] T 6a4c6dc08df744e6856bdf99ab65c575 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 1.
I20250113 02:51:42.648465 709 leader_election.cc:304] T 938cc1ccd8dd449996836188ba9e64ff P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters: df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:42.641845 2368 leader_election.cc:290] T 6a4c6dc08df744e6856bdf99ab65c575 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Requested vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:42.649094 2501 raft_consensus.cc:2798] T 938cc1ccd8dd449996836188ba9e64ff P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Leader election won for term 1
I20250113 02:51:42.649235 2501 raft_consensus.cc:695] T 938cc1ccd8dd449996836188ba9e64ff P 1186fa869eb14812832fa6562531f196 [term 1 LEADER]: Becoming Leader. State: Replica: 1186fa869eb14812832fa6562531f196, State: Running, Role: LEADER
I20250113 02:51:42.649456 2501 consensus_queue.cc:237] T 938cc1ccd8dd449996836188ba9e64ff P 1186fa869eb14812832fa6562531f196 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:42.646021 1198 rpcz_store.cc:276] 0113 02:51:39.214313 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:39.219216 (+ 4903us) service_pool.cc:225] Handling call
0113 02:51:42.600485 (+3381269us) ts_tablet_manager.cc:1145] Deleting tablet 960a7078181f4621a5ff4f3eb3050de3
0113 02:51:42.640165 (+ 39680us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:42.645895 (+ 5730us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3381073}
I20250113 02:51:42.650583 885 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e736211eb3c4536b9fd1d90d2d83838 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:24 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:42.651017 1198 tablet_replica.cc:331] T f1e32e09a8244957a99bdfea5812ff31 P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:42.651654 1198 raft_consensus.cc:2238] T f1e32e09a8244957a99bdfea5812ff31 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:42.653470 1198 raft_consensus.cc:2267] T f1e32e09a8244957a99bdfea5812ff31 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:42.654518 748 tablet_service.cc:1467] Processing CreateTablet for tablet a5565bcb7761428fbf136ed8ce500642 (DEFAULT_TABLE table=default.Table_302b76de6d984d2586c6a6e182f85156 [id=095952e818d24cb08c5e1ff3cc5c0a8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:42.654942 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a5565bcb7761428fbf136ed8ce500642. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:42.657282 1191 tablet_bootstrap.cc:492] T e01e7f8db83d461bbfbb2e901419ce0b P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:42.657399 1191 ts_tablet_manager.cc:1397] T e01e7f8db83d461bbfbb2e901419ce0b P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.298s user 0.049s sys 0.098s
I20250113 02:51:42.658133 1191 raft_consensus.cc:357] T e01e7f8db83d461bbfbb2e901419ce0b P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:42.658308 1191 raft_consensus.cc:383] T e01e7f8db83d461bbfbb2e901419ce0b P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:42.658357 1191 raft_consensus.cc:738] T e01e7f8db83d461bbfbb2e901419ce0b P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:42.658528 1191 consensus_queue.cc:260] T e01e7f8db83d461bbfbb2e901419ce0b P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:42.659003 1191 ts_tablet_manager.cc:1428] T e01e7f8db83d461bbfbb2e901419ce0b P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250113 02:51:42.659255 1191 tablet_bootstrap.cc:492] T b3ce6d851f564b849796f174860aacd5 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:42.660892 1191 tablet_bootstrap.cc:654] T b3ce6d851f564b849796f174860aacd5 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:42.661839 885 tablet_service.cc:1467] Processing CreateTablet for tablet a5565bcb7761428fbf136ed8ce500642 (DEFAULT_TABLE table=default.Table_302b76de6d984d2586c6a6e182f85156 [id=095952e818d24cb08c5e1ff3cc5c0a8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:42.662233 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a5565bcb7761428fbf136ed8ce500642. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:42.662386 1198 ts_tablet_manager.cc:1905] T f1e32e09a8244957a99bdfea5812ff31 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:42.662901 455 catalog_manager.cc:5526] T 938cc1ccd8dd449996836188ba9e64ff P 1186fa869eb14812832fa6562531f196 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1186fa869eb14812832fa6562531f196 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "1186fa869eb14812832fa6562531f196" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } health_report { overall_health: HEALTHY } } }
I20250113 02:51:42.663487 455 catalog_manager.cc:5618] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095) does not have the latest schema for tablet 938cc1ccd8dd449996836188ba9e64ff (table default.Table_e5b4d614e4b743d7982b025ca6d297e5 [id=5824acd778ed47d09f087bbf8e7822d6]). Expected version 1 got 0
I20250113 02:51:42.666334 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1117) took 5071 ms. Trace:
I20250113 02:51:42.652149 995 leader_election.cc:304] T 6a4c6dc08df744e6856bdf99ab65c575 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters: df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:42.666411 1195 rpcz_store.cc:276] 0113 02:51:37.594517 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:37.594588 (+ 71us) service_pool.cc:225] Handling call
0113 02:51:42.666174 (+5071586us) ts_tablet_manager.cc:1145] Deleting tablet abc49f968ea94bdbb0443fba6864a04a
0113 02:51:42.666187 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:42.666316 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5071321}
I20250113 02:51:42.666683 1195 tablet_replica.cc:331] T aa5c172a69bf4ca79999f4b58df08de8 P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:42.666867 1195 raft_consensus.cc:2238] T aa5c172a69bf4ca79999f4b58df08de8 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:42.666958 2368 raft_consensus.cc:2798] T 6a4c6dc08df744e6856bdf99ab65c575 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Leader election won for term 1
I20250113 02:51:42.667011 1195 raft_consensus.cc:2267] T aa5c172a69bf4ca79999f4b58df08de8 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:42.667107 2368 raft_consensus.cc:695] T 6a4c6dc08df744e6856bdf99ab65c575 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: Becoming Leader. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Running, Role: LEADER
I20250113 02:51:42.667515 1195 ts_tablet_manager.cc:1905] T aa5c172a69bf4ca79999f4b58df08de8 P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:42.667405 2368 consensus_queue.cc:237] T 6a4c6dc08df744e6856bdf99ab65c575 P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
W20250113 02:51:42.668007 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet abc49f968ea94bdbb0443fba6864a04a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet abc49f968ea94bdbb0443fba6864a04a already in progress: creating tablet
I20250113 02:51:42.672677 1191 tablet_bootstrap.cc:492] T b3ce6d851f564b849796f174860aacd5 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:42.672812 1191 ts_tablet_manager.cc:1397] T b3ce6d851f564b849796f174860aacd5 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.000s
W20250113 02:51:42.672978 440 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.61:35271, user_credentials={real_user=slave}} blocked reactor thread for 62529us
W20250113 02:51:42.673497 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 960a7078181f4621a5ff4f3eb3050de3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 960a7078181f4621a5ff4f3eb3050de3 already in progress: creating tablet
I20250113 02:51:42.673811 1024 tablet_service.cc:1467] Processing CreateTablet for tablet a5565bcb7761428fbf136ed8ce500642 (DEFAULT_TABLE table=default.Table_302b76de6d984d2586c6a6e182f85156 [id=095952e818d24cb08c5e1ff3cc5c0a8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:42.674152 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a5565bcb7761428fbf136ed8ce500642. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:42.680275 1191 raft_consensus.cc:357] T b3ce6d851f564b849796f174860aacd5 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:42.680480 1191 raft_consensus.cc:383] T b3ce6d851f564b849796f174860aacd5 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:42.680533 1191 raft_consensus.cc:738] T b3ce6d851f564b849796f174860aacd5 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:42.682430 1191 consensus_queue.cc:260] T b3ce6d851f564b849796f174860aacd5 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:42.688514 1191 ts_tablet_manager.cc:1428] T b3ce6d851f564b849796f174860aacd5 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20250113 02:51:42.688822 1191 tablet_bootstrap.cc:492] T 247af1a17df04b27898f86642336d51e P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:42.695358 448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_3cf8774e45444063a6da1fec1b7ed5e2" } new_table_name: "default.Table_74bd8fe399954ec4a53885440efbfe23" modify_external_catalogs: true
I20250113 02:51:42.697263 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_c569a6d1ad344d33920376cb1ba96eba"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:42.697436 454 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_2d9099633e344b5886574d99e90fe2b7" } new_table_name: "default.Table_20ecc91dbd6f4a03908c4a063388c8b7" modify_external_catalogs: true
W20250113 02:51:42.697822 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c569a6d1ad344d33920376cb1ba96eba in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:42.697757 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_21dd5e7f8388499ea952554e9d7df194"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:42.698173 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_21dd5e7f8388499ea952554e9d7df194 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:42.698288 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_ca9b419c256442dfb375e4043a0e0fa9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:42.698858 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ca9b419c256442dfb375e4043a0e0fa9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:42.699138 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_6fa6ea162f4c46b19013a23599df0f49"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:42.699630 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6fa6ea162f4c46b19013a23599df0f49 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:42.700011 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_0b86c10add464ae3a38d1ec5305117b4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:42.700515 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0b86c10add464ae3a38d1ec5305117b4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:42.710433 1191 tablet_bootstrap.cc:654] T 247af1a17df04b27898f86642336d51e P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:42.713924 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 4791a82b5e13429184d0c7192dd7aef6
I20250113 02:51:42.714761 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_d84fa5147d8a430ca92f49e98b2f7839" } new_table_name: "default.Table_9ca0c47692a840c39dbc1c73c7a6bc7c" modify_external_catalogs: true
I20250113 02:51:42.716687 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 495064a93dfb41ff999fc73b579baca0
I20250113 02:51:42.717337 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_fe0c2876b17f4e318de4048a7fba8c7a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:42.717793 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fe0c2876b17f4e318de4048a7fba8c7a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:42.719122 885 tablet_service.cc:1514] Processing DeleteTablet for tablet 4791a82b5e13429184d0c7192dd7aef6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:42 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:42.719645 885 tablet_service.cc:1514] Processing DeleteTablet for tablet 495064a93dfb41ff999fc73b579baca0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:42 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:42.722261 455 catalog_manager.cc:5618] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095) does not have the latest schema for tablet e01e7f8db83d461bbfbb2e901419ce0b (table default.Table_e27f8b6f8232415388a5e53ab15494b7 [id=efc4ebba97f74c9ba12ab1dcc6577da0]). Expected version 1 got 0
I20250113 02:51:42.723315 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_f62a86c4159541ab9c2abe3d3492f790"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:42.725476 1194 tablet_bootstrap.cc:654] T f8a6eebb6b6d4b63903e68750e181707 P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:42.726528 748 tablet_service.cc:1514] Processing DeleteTablet for tablet 4791a82b5e13429184d0c7192dd7aef6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:42 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:42.727375 748 tablet_service.cc:1514] Processing DeleteTablet for tablet 495064a93dfb41ff999fc73b579baca0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:42 UTC) from {username='slave'} at 127.0.0.1:55398
W20250113 02:51:42.728123 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3f62f344c5a648328ca2de8ca1069c19 consensus state has no leader
I20250113 02:51:42.732069 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet 495064a93dfb41ff999fc73b579baca0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:42 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:42.732300 1032 tablet_service.cc:1514] Processing DeleteTablet for tablet 4791a82b5e13429184d0c7192dd7aef6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:42 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:42.736162 2501 raft_consensus.cc:491] T f105f8b32a654e4ca154e8fc0cb96091 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:42.736333 2501 raft_consensus.cc:513] T f105f8b32a654e4ca154e8fc0cb96091 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:42.737013 2501 leader_election.cc:290] T f105f8b32a654e4ca154e8fc0cb96091 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:42.738696 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f105f8b32a654e4ca154e8fc0cb96091" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
W20250113 02:51:42.739339 709 leader_election.cc:343] T f105f8b32a654e4ca154e8fc0cb96091 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:42.748814 1191 tablet_bootstrap.cc:492] T 247af1a17df04b27898f86642336d51e P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:42.749013 1191 ts_tablet_manager.cc:1397] T 247af1a17df04b27898f86642336d51e P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.060s user 0.002s sys 0.006s
I20250113 02:51:42.749765 1191 raft_consensus.cc:357] T 247af1a17df04b27898f86642336d51e P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:42.749963 1191 raft_consensus.cc:383] T 247af1a17df04b27898f86642336d51e P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:42.750021 1191 raft_consensus.cc:738] T 247af1a17df04b27898f86642336d51e P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:42.752137 1195 ts_tablet_manager.cc:1918] T aa5c172a69bf4ca79999f4b58df08de8 P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250113 02:51:42.752233 1195 log.cc:1198] T aa5c172a69bf4ca79999f4b58df08de8 P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/aa5c172a69bf4ca79999f4b58df08de8
I20250113 02:51:42.752596 1195 ts_tablet_manager.cc:1939] T aa5c172a69bf4ca79999f4b58df08de8 P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:42.753261 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1116) took 5160 ms. Trace:
I20250113 02:51:42.753348 1195 rpcz_store.cc:276] 0113 02:51:37.593228 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:37.594889 (+ 1661us) service_pool.cc:225] Handling call
0113 02:51:42.666641 (+5071752us) ts_tablet_manager.cc:1145] Deleting tablet aa5c172a69bf4ca79999f4b58df08de8
0113 02:51:42.666649 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:42.710313 (+ 43664us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:42.752041 (+ 41728us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:42.753237 (+ 1196us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5071554}
I20250113 02:51:42.753738 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet aa5c172a69bf4ca79999f4b58df08de8 (table default.Table_3f1e7146fed34d53896f138e43a4d26e [id=5dceb76e1dbe422ab98d4da5bf6c45a0]) successfully deleted
I20250113 02:51:42.754407 1192 tablet_bootstrap.cc:492] T e570465a3d4046b1ac19a3a034fe91f1 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:42.754671 1192 ts_tablet_manager.cc:1397] T e570465a3d4046b1ac19a3a034fe91f1 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.172s user 0.030s sys 0.053s
I20250113 02:51:42.755386 1192 raft_consensus.cc:357] T e570465a3d4046b1ac19a3a034fe91f1 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:42.755587 1192 raft_consensus.cc:383] T e570465a3d4046b1ac19a3a034fe91f1 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:42.755612 1198 ts_tablet_manager.cc:1918] T f1e32e09a8244957a99bdfea5812ff31 P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:42.755652 1192 raft_consensus.cc:738] T e570465a3d4046b1ac19a3a034fe91f1 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:42.755694 1198 log.cc:1198] T f1e32e09a8244957a99bdfea5812ff31 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/f1e32e09a8244957a99bdfea5812ff31
I20250113 02:51:42.755992 1198 ts_tablet_manager.cc:1939] T f1e32e09a8244957a99bdfea5812ff31 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:42.755884 1192 consensus_queue.cc:260] T e570465a3d4046b1ac19a3a034fe91f1 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:42.756476 1192 ts_tablet_manager.cc:1428] T e570465a3d4046b1ac19a3a034fe91f1 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250113 02:51:42.756707 1192 tablet_bootstrap.cc:492] T fd0b8592997d4a61b8ca854ecbb67c55 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:42.758299 1192 tablet_bootstrap.cc:654] T fd0b8592997d4a61b8ca854ecbb67c55 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:42.763345 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_2142dbe5da9c4c2c824c7dc752e2f15b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:42.763813 1195 tablet_replica.cc:331] T 378d59d314a84ddb84cf2ab041c51cf1 P 1186fa869eb14812832fa6562531f196: stopping tablet replica
W20250113 02:51:42.763943 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2142dbe5da9c4c2c824c7dc752e2f15b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:42.765295 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_e6725383626f4287a24babc4e253e7b1" } new_table_name: "default.Table_b9056fab83844c22ba28394a7287a942" modify_external_catalogs: true
I20250113 02:51:42.765283 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_2a70bebef78f48f09c6c3982df867cdc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:42.765750 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2a70bebef78f48f09c6c3982df867cdc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:42.766582 1195 raft_consensus.cc:2238] T 378d59d314a84ddb84cf2ab041c51cf1 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:42.767194 1192 tablet_bootstrap.cc:492] T fd0b8592997d4a61b8ca854ecbb67c55 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:42.767321 1192 ts_tablet_manager.cc:1397] T fd0b8592997d4a61b8ca854ecbb67c55 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.004s
I20250113 02:51:42.767488 1195 raft_consensus.cc:2267] T 378d59d314a84ddb84cf2ab041c51cf1 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:42.768088 1195 ts_tablet_manager.cc:1905] T 378d59d314a84ddb84cf2ab041c51cf1 P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:42.767966 1192 raft_consensus.cc:357] T fd0b8592997d4a61b8ca854ecbb67c55 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:42.768172 1192 raft_consensus.cc:383] T fd0b8592997d4a61b8ca854ecbb67c55 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:42.768229 1192 raft_consensus.cc:738] T fd0b8592997d4a61b8ca854ecbb67c55 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:42.768467 1192 consensus_queue.cc:260] T fd0b8592997d4a61b8ca854ecbb67c55 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:42.769063 1192 ts_tablet_manager.cc:1428] T fd0b8592997d4a61b8ca854ecbb67c55 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250113 02:51:42.769781 1192 tablet_bootstrap.cc:492] T 1968c7e9357c4c6e901d334ace0467fb P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:42.771042 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1182) took 3462 ms. Trace:
I20250113 02:51:42.771114 1198 rpcz_store.cc:276] 0113 02:51:39.308849 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:39.309309 (+ 460us) service_pool.cc:225] Handling call
0113 02:51:42.650958 (+3341649us) ts_tablet_manager.cc:1145] Deleting tablet f1e32e09a8244957a99bdfea5812ff31
0113 02:51:42.650969 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:42.693157 (+ 42188us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:42.755521 (+ 62364us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:42.771023 (+ 15502us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3341396}
I20250113 02:51:42.771494 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1199) took 3203 ms. Trace:
I20250113 02:51:42.771533 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet f1e32e09a8244957a99bdfea5812ff31 (table default.Table_12a092a4543e4b04a466945f4b694c53 [id=c84b256e3b5744edbbf4de12eeaff0a3]) successfully deleted
I20250113 02:51:42.771564 1198 rpcz_store.cc:276] 0113 02:51:39.568418 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:39.568935 (+ 517us) service_pool.cc:225] Handling call
0113 02:51:42.771375 (+3202440us) ts_tablet_manager.cc:1145] Deleting tablet e5ed7016af184b86be4c234f42e2a6ef
0113 02:51:42.771384 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:42.771479 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3202269}
I20250113 02:51:42.771903 1192 tablet_bootstrap.cc:654] T 1968c7e9357c4c6e901d334ace0467fb P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
W20250113 02:51:42.771932 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet e5ed7016af184b86be4c234f42e2a6ef because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e5ed7016af184b86be4c234f42e2a6ef already in progress: creating tablet
I20250113 02:51:42.737653 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f105f8b32a654e4ca154e8fc0cb96091" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:42.774602 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1197) took 3206 ms. Trace:
I20250113 02:51:42.774701 1198 rpcz_store.cc:276] 0113 02:51:39.568246 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:39.572176 (+ 3930us) service_pool.cc:225] Handling call
0113 02:51:42.774489 (+3202313us) ts_tablet_manager.cc:1145] Deleting tablet 16f547e3cec54baf86b96ad40e3afb6a
0113 02:51:42.774499 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:42.774585 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3202138}
W20250113 02:51:42.775389 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 16f547e3cec54baf86b96ad40e3afb6a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 16f547e3cec54baf86b96ad40e3afb6a already in progress: creating tablet
I20250113 02:51:42.778685 1196 ts_tablet_manager.cc:1918] T d8bb60fdd06946739b75b255303d9d24 P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:42.778800 1196 log.cc:1198] T d8bb60fdd06946739b75b255303d9d24 P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/d8bb60fdd06946739b75b255303d9d24
I20250113 02:51:42.779079 1196 ts_tablet_manager.cc:1939] T d8bb60fdd06946739b75b255303d9d24 P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:42.779606 1196 tablet_replica.cc:331] T 30ca8a716bfc4ab789b216e17567785e P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:42.779728 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet d8bb60fdd06946739b75b255303d9d24 (table default.Table_ba98ed25e93a45a288784d4e5bdee029 [id=7c84304f6d1e4b4ea99f0bc38215d74c]) successfully deleted
I20250113 02:51:42.779762 1196 raft_consensus.cc:2238] T 30ca8a716bfc4ab789b216e17567785e P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:42.779882 1196 raft_consensus.cc:2267] T 30ca8a716bfc4ab789b216e17567785e P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:42.781543 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1198) took 3213 ms. Trace:
I20250113 02:51:42.781626 1198 rpcz_store.cc:276] 0113 02:51:39.568331 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:39.568390 (+ 59us) service_pool.cc:225] Handling call
0113 02:51:42.780090 (+3211700us) ts_tablet_manager.cc:1145] Deleting tablet 128f74fdc98f4135b39dbf23f1d987b0
0113 02:51:42.781424 (+ 1334us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:42.781525 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3193621}
W20250113 02:51:42.781883 448 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1c82cf9373d24fedad88b5df7d5a8b64 consensus state has no leader
W20250113 02:51:42.782685 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 128f74fdc98f4135b39dbf23f1d987b0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 128f74fdc98f4135b39dbf23f1d987b0 already in progress: creating tablet
W20250113 02:51:42.782979 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f62a86c4159541ab9c2abe3d3492f790 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:42.786327 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1226) took 2812 ms. Trace:
I20250113 02:51:42.786409 1198 rpcz_store.cc:276] 0113 02:51:39.973863 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:39.973910 (+ 47us) service_pool.cc:225] Handling call
0113 02:51:42.781942 (+2808032us) ts_tablet_manager.cc:1145] Deleting tablet 045167a6122a4560b298aae2358a485c
0113 02:51:42.786199 (+ 4257us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:42.786309 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2807851}
W20250113 02:51:42.786761 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 045167a6122a4560b298aae2358a485c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 045167a6122a4560b298aae2358a485c already in progress: creating tablet
I20250113 02:51:42.787402 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1225) took 2813 ms. Trace:
I20250113 02:51:42.787472 1198 rpcz_store.cc:276] 0113 02:51:39.973735 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:40.016035 (+ 42300us) service_pool.cc:225] Handling call
0113 02:51:42.787300 (+2771265us) ts_tablet_manager.cc:1145] Deleting tablet 61c79029dfb74f01aaac0f7f828880d0
0113 02:51:42.787309 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:42.787387 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2758992}
W20250113 02:51:42.787801 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 61c79029dfb74f01aaac0f7f828880d0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 61c79029dfb74f01aaac0f7f828880d0 already in progress: creating tablet
I20250113 02:51:42.789170 1194 tablet_bootstrap.cc:492] T f8a6eebb6b6d4b63903e68750e181707 P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:42.789417 1194 ts_tablet_manager.cc:1397] T f8a6eebb6b6d4b63903e68750e181707 P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.158s user 0.002s sys 0.002s
I20250113 02:51:42.789824 1196 ts_tablet_manager.cc:1905] T 30ca8a716bfc4ab789b216e17567785e P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:42.790939 1194 raft_consensus.cc:357] T f8a6eebb6b6d4b63903e68750e181707 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:42.791188 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1227) took 2699 ms. Trace:
I20250113 02:51:42.791308 1198 rpcz_store.cc:276] 0113 02:51:40.091061 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:40.094427 (+ 3366us) service_pool.cc:225] Handling call
0113 02:51:42.790378 (+2695951us) ts_tablet_manager.cc:1145] Deleting tablet 432bcbc7ba6a44ba98cfe588f8292e32
0113 02:51:42.790391 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:42.790492 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2666482}
I20250113 02:51:42.791545 1192 tablet_bootstrap.cc:492] T 1968c7e9357c4c6e901d334ace0467fb P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:42.791664 1192 ts_tablet_manager.cc:1397] T 1968c7e9357c4c6e901d334ace0467fb P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.022s user 0.007s sys 0.001s
W20250113 02:51:42.791781 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 432bcbc7ba6a44ba98cfe588f8292e32 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 432bcbc7ba6a44ba98cfe588f8292e32 already in progress: creating tablet
I20250113 02:51:42.792207 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1228) took 2701 ms. Trace:
I20250113 02:51:42.792346 1194 raft_consensus.cc:383] T f8a6eebb6b6d4b63903e68750e181707 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:42.792455 1194 raft_consensus.cc:738] T f8a6eebb6b6d4b63903e68750e181707 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:42.792387 1192 raft_consensus.cc:357] T 1968c7e9357c4c6e901d334ace0467fb P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:42.792611 1192 raft_consensus.cc:383] T 1968c7e9357c4c6e901d334ace0467fb P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:42.792565 1198 rpcz_store.cc:276] 0113 02:51:40.091154 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:40.094620 (+ 3466us) service_pool.cc:225] Handling call
0113 02:51:42.792115 (+2697495us) ts_tablet_manager.cc:1145] Deleting tablet 797d91d4aae240bd82b00021717772ae
0113 02:51:42.792124 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:42.792192 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2668088}
I20250113 02:51:42.793105 1194 consensus_queue.cc:260] T f8a6eebb6b6d4b63903e68750e181707 P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
W20250113 02:51:42.793706 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 797d91d4aae240bd82b00021717772ae because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 797d91d4aae240bd82b00021717772ae already in progress: creating tablet
I20250113 02:51:42.793920 2368 raft_consensus.cc:491] T c13e9a51c1e74b44b352bade78390ba5 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:42.794020 2368 raft_consensus.cc:513] T c13e9a51c1e74b44b352bade78390ba5 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:42.794554 2368 leader_election.cc:290] T c13e9a51c1e74b44b352bade78390ba5 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
I20250113 02:51:42.792667 1192 raft_consensus.cc:738] T 1968c7e9357c4c6e901d334ace0467fb P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
W20250113 02:51:42.795653 454 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 71122b6a1c0145b4a674815f3cbf6e92 consensus state has no leader
I20250113 02:51:42.795769 1192 consensus_queue.cc:260] T 1968c7e9357c4c6e901d334ace0467fb P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:42.795890 454 master_service.cc:946] ReplaceTablet: received request to replace tablet e3a1e65ec9874cc7b83485683aafff77 from {username='slave'} at 127.0.0.1:49258
W20250113 02:51:42.772902 711 leader_election.cc:343] T f105f8b32a654e4ca154e8fc0cb96091 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:42.796942 711 leader_election.cc:304] T f105f8b32a654e4ca154e8fc0cb96091 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:42.797441 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1229) took 2706 ms. Trace:
I20250113 02:51:42.797513 1198 rpcz_store.cc:276] 0113 02:51:40.091219 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:40.094728 (+ 3509us) service_pool.cc:225] Handling call
0113 02:51:42.797338 (+2702610us) ts_tablet_manager.cc:1145] Deleting tablet 67fe2f52d98048dab9f37dd820b7f712
0113 02:51:42.797347 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:42.797426 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2673218}
I20250113 02:51:42.797646 2501 raft_consensus.cc:2743] T f105f8b32a654e4ca154e8fc0cb96091 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250113 02:51:42.797891 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 67fe2f52d98048dab9f37dd820b7f712 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 67fe2f52d98048dab9f37dd820b7f712 already in progress: creating tablet
I20250113 02:51:42.798342 448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_5bc5d0b055904e50b74e0981675f4abb" } new_table_name: "default.Table_336aa2fec26e43f5b0579d8756140276" modify_external_catalogs: true
I20250113 02:51:42.794904 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c13e9a51c1e74b44b352bade78390ba5" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:42.799192 998 leader_election.cc:343] T c13e9a51c1e74b44b352bade78390ba5 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:42.799700 1194 ts_tablet_manager.cc:1428] T f8a6eebb6b6d4b63903e68750e181707 P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20250113 02:51:42.799886 1198 tablet_replica.cc:331] T c7842d6f2c074e3b9d1a3378db9856be P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:42.800112 1198 raft_consensus.cc:2238] T c7842d6f2c074e3b9d1a3378db9856be P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:42.801342 449 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_54127c30dcd54c1dbe1f089100b2a4b6" } modify_external_catalogs: true
I20250113 02:51:42.801491 449 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_54127c30dcd54c1dbe1f089100b2a4b6" } modify_external_catalogs: true
I20250113 02:51:42.802695 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_9eb9691792fb4e5da6a1299fbc96e962"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:42.803205 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9eb9691792fb4e5da6a1299fbc96e962 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:42.803767 1194 tablet_bootstrap.cc:492] T dd566a49d3ff4d3889b80d8d2ccbee0e P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:42.750212 1191 consensus_queue.cc:260] T 247af1a17df04b27898f86642336d51e P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:42.805370 1194 tablet_bootstrap.cc:654] T dd566a49d3ff4d3889b80d8d2ccbee0e P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:42.805545 1198 raft_consensus.cc:2267] T c7842d6f2c074e3b9d1a3378db9856be P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:42.805888 1191 ts_tablet_manager.cc:1428] T 247af1a17df04b27898f86642336d51e P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.057s user 0.001s sys 0.001s
I20250113 02:51:42.806142 1198 ts_tablet_manager.cc:1905] T c7842d6f2c074e3b9d1a3378db9856be P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:42.806250 1191 tablet_bootstrap.cc:492] T 9b2d0aa6f0e9456d95f4e48c360ab6e8 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:42.795221 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c13e9a51c1e74b44b352bade78390ba5" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
I20250113 02:51:42.807140 772 raft_consensus.cc:2463] T c13e9a51c1e74b44b352bade78390ba5 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 0.
I20250113 02:51:42.810528 1191 tablet_bootstrap.cc:654] T 9b2d0aa6f0e9456d95f4e48c360ab6e8 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:42.811582 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_b95a9b1d12284c9aac701a8ef9c93482" } modify_external_catalogs: true
I20250113 02:51:42.811740 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_b95a9b1d12284c9aac701a8ef9c93482" } modify_external_catalogs: true
I20250113 02:51:42.815009 995 leader_election.cc:304] T c13e9a51c1e74b44b352bade78390ba5 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters: df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:42.818861 1198 ts_tablet_manager.cc:1918] T c7842d6f2c074e3b9d1a3378db9856be P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:42.818984 1198 log.cc:1198] T c7842d6f2c074e3b9d1a3378db9856be P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/c7842d6f2c074e3b9d1a3378db9856be
I20250113 02:51:42.819193 2547 raft_consensus.cc:2798] T c13e9a51c1e74b44b352bade78390ba5 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 02:51:42.819283 2547 raft_consensus.cc:491] T c13e9a51c1e74b44b352bade78390ba5 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 02:51:42.819370 2547 raft_consensus.cc:3054] T c13e9a51c1e74b44b352bade78390ba5 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:42.822150 2476 raft_consensus.cc:491] T f8a6eebb6b6d4b63903e68750e181707 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:42.822145 2547 raft_consensus.cc:513] T c13e9a51c1e74b44b352bade78390ba5 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:42.822266 2476 raft_consensus.cc:513] T f8a6eebb6b6d4b63903e68750e181707 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:42.822773 2476 leader_election.cc:290] T f8a6eebb6b6d4b63903e68750e181707 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:42.823020 2547 leader_election.cc:290] T c13e9a51c1e74b44b352bade78390ba5 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Requested vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
I20250113 02:51:42.823491 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c13e9a51c1e74b44b352bade78390ba5" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f"
W20250113 02:51:42.823962 998 leader_election.cc:343] T c13e9a51c1e74b44b352bade78390ba5 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:42.824694 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c13e9a51c1e74b44b352bade78390ba5" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196"
I20250113 02:51:42.824884 769 raft_consensus.cc:3054] T c13e9a51c1e74b44b352bade78390ba5 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Advancing to term 1
W20250113 02:51:42.824913 851 leader_election.cc:343] T f8a6eebb6b6d4b63903e68750e181707 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Not found: Tablet not found: f8a6eebb6b6d4b63903e68750e181707
I20250113 02:51:42.819303 1198 ts_tablet_manager.cc:1939] T c7842d6f2c074e3b9d1a3378db9856be P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:42.826161 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1230) took 2734 ms. Trace:
I20250113 02:51:42.826236 1198 rpcz_store.cc:276] 0113 02:51:40.091285 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:40.094828 (+ 3543us) service_pool.cc:225] Handling call
0113 02:51:42.799842 (+2705014us) ts_tablet_manager.cc:1145] Deleting tablet c7842d6f2c074e3b9d1a3378db9856be
0113 02:51:42.799851 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:42.810524 (+ 10673us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:42.818743 (+ 8219us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:42.826142 (+ 7399us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2675675}
I20250113 02:51:42.826623 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet c7842d6f2c074e3b9d1a3378db9856be (table default.Table_59401ddb940e4e789a5bd54f96548ba5 [id=0ac370ed485e4d64a3e1aa1e8d67ea23]) successfully deleted
I20250113 02:51:42.827462 769 raft_consensus.cc:2463] T c13e9a51c1e74b44b352bade78390ba5 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 1.
I20250113 02:51:42.827518 1194 tablet_bootstrap.cc:492] T dd566a49d3ff4d3889b80d8d2ccbee0e P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:42.827632 1194 ts_tablet_manager.cc:1397] T dd566a49d3ff4d3889b80d8d2ccbee0e P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.024s user 0.004s sys 0.000s
I20250113 02:51:42.827454 2576 sys_catalog.cc:455] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 4 leader_uuid: "e13f5bcdcee34225a46c6d2651e5ede9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e13f5bcdcee34225a46c6d2651e5ede9" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44217 } } peers { permanent_uuid: "6bbc3541604a455584bb32dfc2cc4efd" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 35271 } } peers { permanent_uuid: "45e9d43ecac0401fb2c18fe5eaf089b5" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42067 } } }
I20250113 02:51:42.827706 2576 sys_catalog.cc:458] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9 [sys.catalog]: This master's current role is: LEADER
I20250113 02:51:42.828361 995 leader_election.cc:304] T c13e9a51c1e74b44b352bade78390ba5 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters: df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:42.828491 1194 raft_consensus.cc:357] T dd566a49d3ff4d3889b80d8d2ccbee0e P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:42.828657 1194 raft_consensus.cc:383] T dd566a49d3ff4d3889b80d8d2ccbee0e P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:42.828713 1194 raft_consensus.cc:738] T dd566a49d3ff4d3889b80d8d2ccbee0e P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:42.828958 1194 consensus_queue.cc:260] T dd566a49d3ff4d3889b80d8d2ccbee0e P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:42.830505 455 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_921e31e7ce184317873cbfa17f6f32ac" } modify_external_catalogs: true
W20250113 02:51:42.830641 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 01371679919c4daaad9451a7279898e5 consensus state has no leader
I20250113 02:51:42.830665 455 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_921e31e7ce184317873cbfa17f6f32ac" } modify_external_catalogs: true
I20250113 02:51:42.831099 2547 raft_consensus.cc:2798] T c13e9a51c1e74b44b352bade78390ba5 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Leader election won for term 1
I20250113 02:51:42.831270 2547 raft_consensus.cc:695] T c13e9a51c1e74b44b352bade78390ba5 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: Becoming Leader. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Running, Role: LEADER
I20250113 02:51:42.831557 2547 consensus_queue.cc:237] T c13e9a51c1e74b44b352bade78390ba5 P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:42.832883 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet e3a1e65ec9874cc7b83485683aafff77
I20250113 02:51:42.833295 454 catalog_manager.cc:6596] ReplaceTablet: tablet e3a1e65ec9874cc7b83485683aafff77 deleted and replaced by tablet 6d859229bd084f35975a615777be80a4
I20250113 02:51:42.834053 748 tablet_service.cc:1514] Processing DeleteTablet for tablet e3a1e65ec9874cc7b83485683aafff77 with delete_type TABLET_DATA_DELETED (replaced by tablet 6d859229bd084f35975a615777be80a4) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:42.834275 891 tablet_service.cc:1514] Processing DeleteTablet for tablet e3a1e65ec9874cc7b83485683aafff77 with delete_type TABLET_DATA_DELETED (replaced by tablet 6d859229bd084f35975a615777be80a4) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:42.834481 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet e3a1e65ec9874cc7b83485683aafff77 with delete_type TABLET_DATA_DELETED (replaced by tablet 6d859229bd084f35975a615777be80a4) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:42.834787 885 tablet_service.cc:1514] Processing DeleteTablet for tablet f8a6eebb6b6d4b63903e68750e181707 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:31 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:42.834929 1194 ts_tablet_manager.cc:1428] T dd566a49d3ff4d3889b80d8d2ccbee0e P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
W20250113 02:51:42.835279 852 leader_election.cc:343] T f8a6eebb6b6d4b63903e68750e181707 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Not found: Tablet not found: f8a6eebb6b6d4b63903e68750e181707
I20250113 02:51:42.835422 852 leader_election.cc:304] T f8a6eebb6b6d4b63903e68750e181707 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df69f48656b44bb890f76587ee69cc0f; no voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca
I20250113 02:51:42.836042 1194 tablet_bootstrap.cc:492] T 44d2c7d5334c476fac60fdd57e2c02b6 P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:42.836192 2476 raft_consensus.cc:2743] T f8a6eebb6b6d4b63903e68750e181707 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:42.837603 1194 tablet_bootstrap.cc:654] T 44d2c7d5334c476fac60fdd57e2c02b6 P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:42.838647 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1231) took 2680 ms. Trace:
I20250113 02:51:42.838734 1198 rpcz_store.cc:276] 0113 02:51:40.157859 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:40.176059 (+ 18200us) service_pool.cc:225] Handling call
0113 02:51:42.838519 (+2662460us) ts_tablet_manager.cc:1145] Deleting tablet efe5f5ffd2c8423686b4007ded447466
0113 02:51:42.838531 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:42.838628 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2662250}
I20250113 02:51:42.839480 885 tablet_service.cc:1514] Processing DeleteTablet for tablet dd566a49d3ff4d3889b80d8d2ccbee0e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:35 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:42.840286 1191 tablet_bootstrap.cc:492] T 9b2d0aa6f0e9456d95f4e48c360ab6e8 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:42.840413 1191 ts_tablet_manager.cc:1397] T 9b2d0aa6f0e9456d95f4e48c360ab6e8 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.034s user 0.002s sys 0.001s
I20250113 02:51:42.841096 1191 raft_consensus.cc:357] T 9b2d0aa6f0e9456d95f4e48c360ab6e8 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:42.841277 1191 raft_consensus.cc:383] T 9b2d0aa6f0e9456d95f4e48c360ab6e8 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:42.841322 1191 raft_consensus.cc:738] T 9b2d0aa6f0e9456d95f4e48c360ab6e8 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:42.841596 1198 tablet_replica.cc:331] T c518f791b4d74c54ab0c0364d29b4a64 P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:42.841508 1191 consensus_queue.cc:260] T 9b2d0aa6f0e9456d95f4e48c360ab6e8 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:42.841786 1198 raft_consensus.cc:2238] T c518f791b4d74c54ab0c0364d29b4a64 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:42.841967 1191 ts_tablet_manager.cc:1428] T 9b2d0aa6f0e9456d95f4e48c360ab6e8 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250113 02:51:42.842059 1198 raft_consensus.cc:2267] T c518f791b4d74c54ab0c0364d29b4a64 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:42.842247 1191 tablet_bootstrap.cc:492] T 59fadc693f554854bc2874f8db7c64c8 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:42.842574 1198 ts_tablet_manager.cc:1905] T c518f791b4d74c54ab0c0364d29b4a64 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:42.843837 1191 tablet_bootstrap.cc:654] T 59fadc693f554854bc2874f8db7c64c8 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:42.845510 2368 raft_consensus.cc:491] T aa1ebdb92a5f4c92a5195b614689c100 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:42.845695 2368 raft_consensus.cc:513] T aa1ebdb92a5f4c92a5195b614689c100 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:42.846863 2368 leader_election.cc:290] T aa1ebdb92a5f4c92a5195b614689c100 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:42.846976 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa1ebdb92a5f4c92a5195b614689c100" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:42.847437 998 leader_election.cc:343] T aa1ebdb92a5f4c92a5195b614689c100 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:42.847517 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa1ebdb92a5f4c92a5195b614689c100" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
I20250113 02:51:42.847692 769 raft_consensus.cc:2463] T aa1ebdb92a5f4c92a5195b614689c100 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 0.
I20250113 02:51:42.848300 2473 raft_consensus.cc:491] T b38cd34d67834a3f9bf5b5636bb79217 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:42.848407 2473 raft_consensus.cc:513] T b38cd34d67834a3f9bf5b5636bb79217 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:42.848958 2473 leader_election.cc:290] T b38cd34d67834a3f9bf5b5636bb79217 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:42.849486 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b38cd34d67834a3f9bf5b5636bb79217" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:42.849646 1059 raft_consensus.cc:2463] T b38cd34d67834a3f9bf5b5636bb79217 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1186fa869eb14812832fa6562531f196 in term 0.
I20250113 02:51:42.850674 709 leader_election.cc:304] T b38cd34d67834a3f9bf5b5636bb79217 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters:
I20250113 02:51:42.851181 2473 raft_consensus.cc:2798] T b38cd34d67834a3f9bf5b5636bb79217 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 02:51:42.851254 2473 raft_consensus.cc:491] T b38cd34d67834a3f9bf5b5636bb79217 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 02:51:42.851352 2473 raft_consensus.cc:3054] T b38cd34d67834a3f9bf5b5636bb79217 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:42.851414 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b38cd34d67834a3f9bf5b5636bb79217" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:42.852193 711 leader_election.cc:343] T b38cd34d67834a3f9bf5b5636bb79217 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:42.853786 2473 raft_consensus.cc:513] T b38cd34d67834a3f9bf5b5636bb79217 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:42.854700 2473 leader_election.cc:290] T b38cd34d67834a3f9bf5b5636bb79217 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 election: Requested vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:42.854974 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b38cd34d67834a3f9bf5b5636bb79217" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca"
I20250113 02:51:42.855186 1059 raft_consensus.cc:3054] T b38cd34d67834a3f9bf5b5636bb79217 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:42.857887 1059 raft_consensus.cc:2463] T b38cd34d67834a3f9bf5b5636bb79217 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1186fa869eb14812832fa6562531f196 in term 1.
I20250113 02:51:42.858250 709 leader_election.cc:304] T b38cd34d67834a3f9bf5b5636bb79217 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters:
I20250113 02:51:42.858569 2501 raft_consensus.cc:2798] T b38cd34d67834a3f9bf5b5636bb79217 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Leader election won for term 1
I20250113 02:51:42.858650 2501 raft_consensus.cc:695] T b38cd34d67834a3f9bf5b5636bb79217 P 1186fa869eb14812832fa6562531f196 [term 1 LEADER]: Becoming Leader. State: Replica: 1186fa869eb14812832fa6562531f196, State: Running, Role: LEADER
I20250113 02:51:42.858851 2501 consensus_queue.cc:237] T b38cd34d67834a3f9bf5b5636bb79217 P 1186fa869eb14812832fa6562531f196 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:42.860653 1198 ts_tablet_manager.cc:1918] T c518f791b4d74c54ab0c0364d29b4a64 P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:42.860759 1198 log.cc:1198] T c518f791b4d74c54ab0c0364d29b4a64 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/c518f791b4d74c54ab0c0364d29b4a64
I20250113 02:51:42.861053 1198 ts_tablet_manager.cc:1939] T c518f791b4d74c54ab0c0364d29b4a64 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:42.861950 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1232) took 2638 ms. Trace:
I20250113 02:51:42.862061 1198 rpcz_store.cc:276] 0113 02:51:40.223916 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:40.235837 (+ 11921us) service_pool.cc:225] Handling call
0113 02:51:42.841544 (+2605707us) ts_tablet_manager.cc:1145] Deleting tablet c518f791b4d74c54ab0c0364d29b4a64
0113 02:51:42.841555 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:42.852781 (+ 11226us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:42.860551 (+ 7770us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:42.861926 (+ 1375us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2605460}
I20250113 02:51:42.862146 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b38cd34d67834a3f9bf5b5636bb79217" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f"
I20250113 02:51:42.862548 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet c518f791b4d74c54ab0c0364d29b4a64 (table default.Table_157d1350cce74502a990462ac9bd0e5f [id=f96a39bfcdf442f7965a7b16d9aa432f]) successfully deleted
W20250113 02:51:42.862876 448 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ffb27f71b1464e18977f46d954194bef consensus state has no leader
W20250113 02:51:42.862900 711 leader_election.cc:343] T b38cd34d67834a3f9bf5b5636bb79217 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:42.862993 995 leader_election.cc:304] T aa1ebdb92a5f4c92a5195b614689c100 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters: df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:42.863268 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet a446180f980640268170b4ab652d32cc
I20250113 02:51:42.864049 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet a446180f980640268170b4ab652d32cc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:42 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:42.864244 2474 raft_consensus.cc:2798] T aa1ebdb92a5f4c92a5195b614689c100 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 02:51:42.864365 2474 raft_consensus.cc:491] T aa1ebdb92a5f4c92a5195b614689c100 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 02:51:42.864445 2474 raft_consensus.cc:3054] T aa1ebdb92a5f4c92a5195b614689c100 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:42.865075 885 tablet_service.cc:1514] Processing DeleteTablet for tablet a446180f980640268170b4ab652d32cc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:42 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:42.867071 2474 raft_consensus.cc:513] T aa1ebdb92a5f4c92a5195b614689c100 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:42.867506 1195 ts_tablet_manager.cc:1918] T 378d59d314a84ddb84cf2ab041c51cf1 P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250113 02:51:42.867594 1195 log.cc:1198] T 378d59d314a84ddb84cf2ab041c51cf1 P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/378d59d314a84ddb84cf2ab041c51cf1
I20250113 02:51:42.867913 1195 ts_tablet_manager.cc:1939] T 378d59d314a84ddb84cf2ab041c51cf1 P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:42.867956 2474 leader_election.cc:290] T aa1ebdb92a5f4c92a5195b614689c100 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Requested vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:42.872130 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa1ebdb92a5f4c92a5195b614689c100" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f"
W20250113 02:51:42.872777 998 leader_election.cc:343] T aa1ebdb92a5f4c92a5195b614689c100 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:42.872846 1198 tablet_replica.cc:331] T bbc7832c50114d6f890fec45a3d994e9 P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:42.873227 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1118) took 5244 ms. Trace:
I20250113 02:51:42.873339 1195 rpcz_store.cc:276] 0113 02:51:37.628334 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:37.628401 (+ 67us) service_pool.cc:225] Handling call
0113 02:51:42.763764 (+5135363us) ts_tablet_manager.cc:1145] Deleting tablet 378d59d314a84ddb84cf2ab041c51cf1
0113 02:51:42.763774 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:42.799919 (+ 36145us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:42.867413 (+ 67494us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:42.873203 (+ 5790us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5135097}
I20250113 02:51:42.873479 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 0b119704bc774ca9b4155c9619d4765b
I20250113 02:51:42.873596 1196 ts_tablet_manager.cc:1918] T 30ca8a716bfc4ab789b216e17567785e P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:42.873715 1196 log.cc:1198] T 30ca8a716bfc4ab789b216e17567785e P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/30ca8a716bfc4ab789b216e17567785e
I20250113 02:51:42.874503 885 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b119704bc774ca9b4155c9619d4765b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:42 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:42.875866 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa1ebdb92a5f4c92a5195b614689c100" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196"
I20250113 02:51:42.876112 769 raft_consensus.cc:3054] T aa1ebdb92a5f4c92a5195b614689c100 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:42.876590 1196 ts_tablet_manager.cc:1939] T 30ca8a716bfc4ab789b216e17567785e P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:42.877631 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet 30ca8a716bfc4ab789b216e17567785e (table default.Table_aac49fd87c1f4c3f9fc59bd8afe6e48f [id=9eaf97b701fd4e79847dce3a9d259beb]) successfully deleted
I20250113 02:51:42.878823 748 tablet_service.cc:1514] Processing DeleteTablet for tablet a446180f980640268170b4ab652d32cc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:42 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:42.878657 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet 378d59d314a84ddb84cf2ab041c51cf1 (table default.Table_49dafeaa3df04cb789a3e54f469973e8 [id=0c01441cec7043b687d6744e72a0e45d]) successfully deleted
I20250113 02:51:42.879025 739 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b119704bc774ca9b4155c9619d4765b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:42 UTC) from {username='slave'} at 127.0.0.1:55398
W20250113 02:51:42.879498 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 125ebde7a9f14918ba328a75e88a7746 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 125ebde7a9f14918ba328a75e88a7746 already in progress: creating tablet
I20250113 02:51:42.882266 2363 consensus_queue.cc:1035] T 82505f1389184d0ca12ec0e2d6084d22 P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 02:51:42.883169 2363 consensus_queue.cc:1035] T b33025a462e74f2aa26cc3779f36615b P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250113 02:51:42.884469 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 166f58d45b7b443ea54087e4ab48eaba because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 166f58d45b7b443ea54087e4ab48eaba already in progress: creating tablet
I20250113 02:51:42.886941 1196 tablet_replica.cc:331] T 9e736211eb3c4536b9fd1d90d2d83838 P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:42.887152 1196 raft_consensus.cc:2238] T 9e736211eb3c4536b9fd1d90d2d83838 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:42.887275 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskXJrxug/build/asan/bin/kudu
/tmp/dist-test-taskXJrxug/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.60:42067
--webserver_interface=127.0.104.60
--webserver_port=38411
--builtin_ntp_servers=127.0.104.20:45573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:44217,127.0.104.61:35271,127.0.104.60:42067
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250113 02:51:42.887317 1196 raft_consensus.cc:2267] T 9e736211eb3c4536b9fd1d90d2d83838 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:42.892143 1198 raft_consensus.cc:2238] T bbc7832c50114d6f890fec45a3d994e9 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:42.892321 1198 raft_consensus.cc:2267] T bbc7832c50114d6f890fec45a3d994e9 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:42.892868 1198 ts_tablet_manager.cc:1905] T bbc7832c50114d6f890fec45a3d994e9 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:42.896783 1192 ts_tablet_manager.cc:1428] T 1968c7e9357c4c6e901d334ace0467fb P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.105s user 0.002s sys 0.000s
I20250113 02:51:42.879418 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1119) took 5246 ms. Trace:
I20250113 02:51:42.880242 769 raft_consensus.cc:2463] T aa1ebdb92a5f4c92a5195b614689c100 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 1.
I20250113 02:51:42.897075 1195 rpcz_store.cc:276] 0113 02:51:37.633347 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:37.635367 (+ 2020us) service_pool.cc:225] Handling call
0113 02:51:42.879291 (+5243924us) ts_tablet_manager.cc:1145] Deleting tablet dff74d7c093b45eba7a47d8eceb0c1d1
0113 02:51:42.879302 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:42.879401 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5243661}
I20250113 02:51:42.897306 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b119704bc774ca9b4155c9619d4765b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:42 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:42.897501 1195 tablet_replica.cc:331] T 438604e8859f422f9caccb61c5b5b125 P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:42.897680 1195 raft_consensus.cc:2238] T 438604e8859f422f9caccb61c5b5b125 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Raft consensus shutting down.
W20250113 02:51:42.897642 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet dff74d7c093b45eba7a47d8eceb0c1d1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dff74d7c093b45eba7a47d8eceb0c1d1 already in progress: creating tablet
I20250113 02:51:42.897812 1195 raft_consensus.cc:2267] T 438604e8859f422f9caccb61c5b5b125 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:42.898332 1195 ts_tablet_manager.cc:1905] T 438604e8859f422f9caccb61c5b5b125 P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:42.899112 995 leader_election.cc:304] T aa1ebdb92a5f4c92a5195b614689c100 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters: df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:42.901433 2596 raft_consensus.cc:2798] T aa1ebdb92a5f4c92a5195b614689c100 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Leader election won for term 1
I20250113 02:51:42.901696 2596 raft_consensus.cc:695] T aa1ebdb92a5f4c92a5195b614689c100 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: Becoming Leader. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Running, Role: LEADER
I20250113 02:51:42.901897 1192 tablet_bootstrap.cc:492] T 67baaab1e36e415381579d7385d4ab5a P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:42.902019 2596 consensus_queue.cc:237] T aa1ebdb92a5f4c92a5195b614689c100 P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:42.902979 2588 raft_consensus.cc:491] T 283fc7f1c74e480c8a63c74d3ae9eb7d P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:42.903141 2588 raft_consensus.cc:513] T 283fc7f1c74e480c8a63c74d3ae9eb7d P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:42.905377 1192 tablet_bootstrap.cc:654] T 67baaab1e36e415381579d7385d4ab5a P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:42.906105 2501 raft_consensus.cc:491] T e01e7f8db83d461bbfbb2e901419ce0b P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:42.906296 2501 raft_consensus.cc:513] T e01e7f8db83d461bbfbb2e901419ce0b P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:42.907097 2501 leader_election.cc:290] T e01e7f8db83d461bbfbb2e901419ce0b P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:42.907601 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e01e7f8db83d461bbfbb2e901419ce0b" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:42.908138 711 leader_election.cc:343] T e01e7f8db83d461bbfbb2e901419ce0b P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:42.909365 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "283fc7f1c74e480c8a63c74d3ae9eb7d" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:42.910151 711 leader_election.cc:343] T 283fc7f1c74e480c8a63c74d3ae9eb7d P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:42.913053 1191 tablet_bootstrap.cc:492] T 59fadc693f554854bc2874f8db7c64c8 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:42.913343 1191 ts_tablet_manager.cc:1397] T 59fadc693f554854bc2874f8db7c64c8 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.071s user 0.007s sys 0.001s
I20250113 02:51:42.914171 1191 raft_consensus.cc:357] T 59fadc693f554854bc2874f8db7c64c8 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:42.914367 1191 raft_consensus.cc:383] T 59fadc693f554854bc2874f8db7c64c8 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:42.914430 1191 raft_consensus.cc:738] T 59fadc693f554854bc2874f8db7c64c8 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:42.914727 2588 leader_election.cc:290] T 283fc7f1c74e480c8a63c74d3ae9eb7d P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:42.914793 1191 consensus_queue.cc:260] T 59fadc693f554854bc2874f8db7c64c8 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:42.915551 1191 ts_tablet_manager.cc:1428] T 59fadc693f554854bc2874f8db7c64c8 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250113 02:51:42.916946 1191 tablet_bootstrap.cc:492] T 2470b010b571444eba3f132323038f33 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:42.919170 1191 tablet_bootstrap.cc:654] T 2470b010b571444eba3f132323038f33 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:42.921532 1196 ts_tablet_manager.cc:1905] T 9e736211eb3c4536b9fd1d90d2d83838 P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:42.908305 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e01e7f8db83d461bbfbb2e901419ce0b" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:42.914535 1058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "283fc7f1c74e480c8a63c74d3ae9eb7d" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
W20250113 02:51:42.924541 709 leader_election.cc:343] T 283fc7f1c74e480c8a63c74d3ae9eb7d P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:42.924669 709 leader_election.cc:304] T 283fc7f1c74e480c8a63c74d3ae9eb7d P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
W20250113 02:51:42.925100 709 leader_election.cc:343] T e01e7f8db83d461bbfbb2e901419ce0b P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:42.925205 709 leader_election.cc:304] T e01e7f8db83d461bbfbb2e901419ce0b P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:42.916651 1198 ts_tablet_manager.cc:1918] T bbc7832c50114d6f890fec45a3d994e9 P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:42.925866 1198 log.cc:1198] T bbc7832c50114d6f890fec45a3d994e9 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/bbc7832c50114d6f890fec45a3d994e9
I20250113 02:51:42.925874 2501 raft_consensus.cc:2743] T 283fc7f1c74e480c8a63c74d3ae9eb7d P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:42.926178 2501 raft_consensus.cc:2743] T e01e7f8db83d461bbfbb2e901419ce0b P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:42.926220 1198 ts_tablet_manager.cc:1939] T bbc7832c50114d6f890fec45a3d994e9 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:42.926985 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1235) took 2680 ms. Trace:
I20250113 02:51:42.927093 1198 rpcz_store.cc:276] 0113 02:51:40.246295 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:40.246342 (+ 47us) service_pool.cc:225] Handling call
0113 02:51:42.872796 (+2626454us) ts_tablet_manager.cc:1145] Deleting tablet bbc7832c50114d6f890fec45a3d994e9
0113 02:51:42.872808 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:42.895690 (+ 22882us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:42.916532 (+ 20842us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:42.926968 (+ 10436us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2626261}
I20250113 02:51:42.927421 1198 tablet_replica.cc:331] T 788ab82626424b6e833e31761a15b9b2 P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:42.927577 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet bbc7832c50114d6f890fec45a3d994e9 (table default.Table_3c637eb181574498b44ecb4fc41f0117 [id=10c0207c68d8463e91f09972031ab8d6]) successfully deleted
I20250113 02:51:42.927634 1198 raft_consensus.cc:2238] T 788ab82626424b6e833e31761a15b9b2 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:42.927781 1198 raft_consensus.cc:2267] T 788ab82626424b6e833e31761a15b9b2 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:42.928386 1198 ts_tablet_manager.cc:1905] T 788ab82626424b6e833e31761a15b9b2 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:42.933648 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_6ca7cb0180f844548bfb870bb68aaeb9" } modify_external_catalogs: true
I20250113 02:51:42.933929 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_6ca7cb0180f844548bfb870bb68aaeb9" } modify_external_catalogs: true
I20250113 02:51:42.934350 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_9887bd371c8046b49c4ae84f97f16802" } new_table_name: "default.Table_27220e70104c4e69aab7262aef046bdb" modify_external_catalogs: true
I20250113 02:51:42.939177 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_5be43673265c48b1905509b7852f0c11"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:42.939349 448 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_37d63a1b8622483a855edf2ce71c8d71" } modify_external_catalogs: true
I20250113 02:51:42.939469 448 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_37d63a1b8622483a855edf2ce71c8d71" } modify_external_catalogs: true
W20250113 02:51:42.939734 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5be43673265c48b1905509b7852f0c11 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:42.939188 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_55663eac113f43f4ad701ef384b143cb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:42.941207 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_55663eac113f43f4ad701ef384b143cb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250113 02:51:42.945022 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6d859229bd084f35975a615777be80a4 has no consensus state
W20250113 02:51:42.947239 998 consensus_peers.cc:487] T 82505f1389184d0ca12ec0e2d6084d22 P 4216a7f853d642b7bc3e5cff458f3aca -> Peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Couldn't send request to peer df69f48656b44bb890f76587ee69cc0f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250113 02:51:42.948980 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 8337f3c82d654ecdac2cbc1ffa88ef05
W20250113 02:51:42.945667 421 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.104.2:38467, user_credentials={real_user=slave}} blocked reactor thread for 63699.2us
I20250113 02:51:42.951574 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet 8337f3c82d654ecdac2cbc1ffa88ef05 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:42 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:42.952281 885 tablet_service.cc:1514] Processing DeleteTablet for tablet 8337f3c82d654ecdac2cbc1ffa88ef05 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:42 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:42.964289 1191 tablet_bootstrap.cc:492] T 2470b010b571444eba3f132323038f33 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:42.964454 1191 ts_tablet_manager.cc:1397] T 2470b010b571444eba3f132323038f33 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.048s user 0.002s sys 0.002s
I20250113 02:51:42.965261 1191 raft_consensus.cc:357] T 2470b010b571444eba3f132323038f33 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:42.965476 1191 raft_consensus.cc:383] T 2470b010b571444eba3f132323038f33 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:42.965531 1191 raft_consensus.cc:738] T 2470b010b571444eba3f132323038f33 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:42.965757 1191 consensus_queue.cc:260] T 2470b010b571444eba3f132323038f33 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:42.966322 1191 ts_tablet_manager.cc:1428] T 2470b010b571444eba3f132323038f33 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250113 02:51:42.966599 1191 tablet_bootstrap.cc:492] T 65d9bbd8c15f40a4a50c1695fb5e648e P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:42.973290 1195 ts_tablet_manager.cc:1918] T 438604e8859f422f9caccb61c5b5b125 P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:42.973484 1195 log.cc:1198] T 438604e8859f422f9caccb61c5b5b125 P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/438604e8859f422f9caccb61c5b5b125
I20250113 02:51:42.973850 1195 ts_tablet_manager.cc:1939] T 438604e8859f422f9caccb61c5b5b125 P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:42.974622 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1120) took 5333 ms. Trace:
I20250113 02:51:42.974757 1195 rpcz_store.cc:276] 0113 02:51:37.641022 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:37.642343 (+ 1321us) service_pool.cc:225] Handling call
0113 02:51:42.897453 (+5255110us) ts_tablet_manager.cc:1145] Deleting tablet 438604e8859f422f9caccb61c5b5b125
0113 02:51:42.897463 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:42.901732 (+ 4269us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:42.973187 (+ 71455us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:42.974602 (+ 1415us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5254232}
I20250113 02:51:42.975217 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet 438604e8859f422f9caccb61c5b5b125 (table default.Table_2cb862a84f3f44c5bcfb646461a5af1c [id=c6e65e4bab5040cca1f0f55721639d2e]) successfully deleted
I20250113 02:51:42.975397 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1125) took 5217 ms. Trace:
I20250113 02:51:42.975621 739 tablet_service.cc:1514] Processing DeleteTablet for tablet 8337f3c82d654ecdac2cbc1ffa88ef05 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:42 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:42.975476 1195 rpcz_store.cc:276] 0113 02:51:37.757492 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:37.761429 (+ 3937us) service_pool.cc:225] Handling call
0113 02:51:42.975246 (+5213817us) ts_tablet_manager.cc:1145] Deleting tablet 97159fccfa154465a33ba90a27576f0a
0113 02:51:42.975268 (+ 22us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:42.975381 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5213623}
I20250113 02:51:42.976490 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1127) took 5129 ms. Trace:
W20250113 02:51:42.976485 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 97159fccfa154465a33ba90a27576f0a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 97159fccfa154465a33ba90a27576f0a already in progress: creating tablet
I20250113 02:51:42.976568 1195 rpcz_store.cc:276] 0113 02:51:37.846686 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:37.852647 (+ 5961us) service_pool.cc:225] Handling call
0113 02:51:42.976371 (+5123724us) ts_tablet_manager.cc:1145] Deleting tablet 524b4bd640a04123b4abbd716e7d40f2
0113 02:51:42.976381 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:42.976469 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5123466}
W20250113 02:51:42.976874 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 524b4bd640a04123b4abbd716e7d40f2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 524b4bd640a04123b4abbd716e7d40f2 already in progress: creating tablet
I20250113 02:51:42.976938 1195 tablet_replica.cc:331] T 1d2fdc75d00f4e749d322ac6cd5fc410 P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:42.977130 1195 raft_consensus.cc:2238] T 1d2fdc75d00f4e749d322ac6cd5fc410 P 1186fa869eb14812832fa6562531f196 [term 1 LEADER]: Raft consensus shutting down.
I20250113 02:51:42.977348 1195 pending_rounds.cc:62] T 1d2fdc75d00f4e749d322ac6cd5fc410 P 1186fa869eb14812832fa6562531f196: Trying to abort 1 pending ops.
I20250113 02:51:42.977396 1195 pending_rounds.cc:69] T 1d2fdc75d00f4e749d322ac6cd5fc410 P 1186fa869eb14812832fa6562531f196: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113673532003876864 op_type: NO_OP noop_request { }
I20250113 02:51:42.977504 1195 raft_consensus.cc:2883] T 1d2fdc75d00f4e749d322ac6cd5fc410 P 1186fa869eb14812832fa6562531f196 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250113 02:51:42.977584 1195 raft_consensus.cc:2267] T 1d2fdc75d00f4e749d322ac6cd5fc410 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:42.978384 1192 tablet_bootstrap.cc:492] T 67baaab1e36e415381579d7385d4ab5a P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:42.978904 1191 tablet_bootstrap.cc:654] T 65d9bbd8c15f40a4a50c1695fb5e648e P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:42.984643 1192 ts_tablet_manager.cc:1397] T 67baaab1e36e415381579d7385d4ab5a P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.083s user 0.004s sys 0.000s
I20250113 02:51:42.985167 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_906514aed3bf4e5e87a1748d6fffc261"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:42.985574 1192 raft_consensus.cc:357] T 67baaab1e36e415381579d7385d4ab5a P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
W20250113 02:51:42.985741 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_906514aed3bf4e5e87a1748d6fffc261 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:42.985867 1192 raft_consensus.cc:383] T 67baaab1e36e415381579d7385d4ab5a P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:42.985978 1192 raft_consensus.cc:738] T 67baaab1e36e415381579d7385d4ab5a P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:42.986270 1192 consensus_queue.cc:260] T 67baaab1e36e415381579d7385d4ab5a P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:42.986912 1192 ts_tablet_manager.cc:1428] T 67baaab1e36e415381579d7385d4ab5a P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250113 02:51:42.993701 454 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_5576eb7eae61401bbb7065ffbd571846" } new_table_name: "default.Table_69c5fc645ba3452c9ecd494f0d5b1dd2" modify_external_catalogs: true
I20250113 02:51:42.996374 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_9054f47a8da54f029277b3cbaf88e62c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:42.996634 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_f02ddc5b50b94468a206c5bb80525f28"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:42.996870 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9054f47a8da54f029277b3cbaf88e62c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250113 02:51:42.996956 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f02ddc5b50b94468a206c5bb80525f28 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:42.997177 1194 tablet_bootstrap.cc:492] T 44d2c7d5334c476fac60fdd57e2c02b6 P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:42.997309 1194 ts_tablet_manager.cc:1397] T 44d2c7d5334c476fac60fdd57e2c02b6 P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.161s user 0.015s sys 0.010s
I20250113 02:51:42.996394 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_5979f36091aa4a0c89338e504b22b891"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:42.997941 1194 raft_consensus.cc:357] T 44d2c7d5334c476fac60fdd57e2c02b6 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:42.998104 1194 raft_consensus.cc:383] T 44d2c7d5334c476fac60fdd57e2c02b6 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:42.998152 1194 raft_consensus.cc:738] T 44d2c7d5334c476fac60fdd57e2c02b6 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:42.971906 1198 ts_tablet_manager.cc:1918] T 788ab82626424b6e833e31761a15b9b2 P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:42.998374 1198 log.cc:1198] T 788ab82626424b6e833e31761a15b9b2 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/788ab82626424b6e833e31761a15b9b2
W20250113 02:51:42.998385 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5979f36091aa4a0c89338e504b22b891 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:42.998363 1194 consensus_queue.cc:260] T 44d2c7d5334c476fac60fdd57e2c02b6 P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:42.999038 1194 ts_tablet_manager.cc:1428] T 44d2c7d5334c476fac60fdd57e2c02b6 P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250113 02:51:42.999280 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_05b6401543ad445d8ba261889e75c3f8" } new_table_name: "default.Table_a44fcb8c7ef04ef68933abf5950c4d3a" modify_external_catalogs: true
I20250113 02:51:43.000249 1194 tablet_bootstrap.cc:492] T d9df314ffaf042ec80180868d7e7bdf7 P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:43.002115 1194 tablet_bootstrap.cc:654] T d9df314ffaf042ec80180868d7e7bdf7 P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:43.003741 446 catalog_manager.cc:5618] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467) does not have the latest schema for tablet 44d2c7d5334c476fac60fdd57e2c02b6 (table default.Table_6daec99b05744d0caed86e51e86f24c2 [id=2931f217041c46cc8ae52bd6ce8cfd17]). Expected version 1 got 0
I20250113 02:51:42.998695 1198 ts_tablet_manager.cc:1939] T 788ab82626424b6e833e31761a15b9b2 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:43.006721 1196 ts_tablet_manager.cc:1918] T 9e736211eb3c4536b9fd1d90d2d83838 P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:43.006814 1196 log.cc:1198] T 9e736211eb3c4536b9fd1d90d2d83838 P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/9e736211eb3c4536b9fd1d90d2d83838
I20250113 02:51:43.006844 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1234) took 2760 ms. Trace:
I20250113 02:51:43.007081 1196 ts_tablet_manager.cc:1939] T 9e736211eb3c4536b9fd1d90d2d83838 P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:43.006925 1198 rpcz_store.cc:276] 0113 02:51:40.246185 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:40.248963 (+ 2778us) service_pool.cc:225] Handling call
0113 02:51:42.927375 (+2678412us) ts_tablet_manager.cc:1145] Deleting tablet 788ab82626424b6e833e31761a15b9b2
0113 02:51:42.927386 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:42.968365 (+ 40979us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:42.971784 (+ 3419us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:43.006825 (+ 35041us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2678224}
I20250113 02:51:43.007359 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet 788ab82626424b6e833e31761a15b9b2 (table default.Table_9494f6e5f96141dbaec6769ed51d39b2 [id=0f07147c73f24decb09399a1dd188e1f]) successfully deleted
I20250113 02:51:43.003223 1192 tablet_bootstrap.cc:492] T 21bcdd741104474bad7023426856d18f P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:43.008913 1192 tablet_bootstrap.cc:654] T 21bcdd741104474bad7023426856d18f P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:43.009377 446 catalog_manager.cc:5618] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467) does not have the latest schema for tablet 44d2c7d5334c476fac60fdd57e2c02b6 (table default.Table_6daec99b05744d0caed86e51e86f24c2 [id=2931f217041c46cc8ae52bd6ce8cfd17]). Expected version 1 got 0
I20250113 02:51:43.010564 1195 ts_tablet_manager.cc:1905] T 1d2fdc75d00f4e749d322ac6cd5fc410 P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:43.011301 446 master_service.cc:946] ReplaceTablet: received request to replace tablet f86318637d5b4e378ce47e89a5d52b07 from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:43.018355 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1237) took 2738 ms. Trace:
I20250113 02:51:43.018451 1198 rpcz_store.cc:276] 0113 02:51:40.280062 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:40.280105 (+ 43us) service_pool.cc:225] Handling call
0113 02:51:43.018221 (+2738116us) ts_tablet_manager.cc:1145] Deleting tablet 368e5f00861b40ffb76e79376245b885
0113 02:51:43.018231 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:43.018335 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2733684}
I20250113 02:51:43.018798 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1238) took 2710 ms. Trace:
I20250113 02:51:43.018872 1198 rpcz_store.cc:276] 0113 02:51:40.307935 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:40.308019 (+ 84us) service_pool.cc:225] Handling call
0113 02:51:43.018701 (+2710682us) ts_tablet_manager.cc:1145] Deleting tablet 59fadc693f554854bc2874f8db7c64c8
0113 02:51:43.018710 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:43.018784 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2710497}
W20250113 02:51:43.018958 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 368e5f00861b40ffb76e79376245b885 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 368e5f00861b40ffb76e79376245b885 already in progress: creating tablet
I20250113 02:51:43.019166 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1239) took 2710 ms. Trace:
I20250113 02:51:43.019237 1198 rpcz_store.cc:276] 0113 02:51:40.308273 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:40.310890 (+ 2617us) service_pool.cc:225] Handling call
0113 02:51:43.019080 (+2708190us) ts_tablet_manager.cc:1145] Deleting tablet 348c21d783ef4103a37409ef8634eb35
0113 02:51:43.019088 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:43.019153 (+ 65us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2707961}
I20250113 02:51:43.019520 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1240) took 2703 ms. Trace:
W20250113 02:51:43.019587 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 59fadc693f554854bc2874f8db7c64c8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 59fadc693f554854bc2874f8db7c64c8 already in progress: creating tablet
I20250113 02:51:43.019596 1198 rpcz_store.cc:276] 0113 02:51:40.315954 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:40.316027 (+ 73us) service_pool.cc:225] Handling call
0113 02:51:43.019431 (+2703404us) ts_tablet_manager.cc:1145] Deleting tablet 348c21d783ef4103a37409ef8634eb35
0113 02:51:43.019438 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:43.019509 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2703250}
W20250113 02:51:43.019764 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 348c21d783ef4103a37409ef8634eb35 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 348c21d783ef4103a37409ef8634eb35 already in progress: creating tablet
W20250113 02:51:43.020197 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 348c21d783ef4103a37409ef8634eb35 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 348c21d783ef4103a37409ef8634eb35 already in progress: creating tablet
I20250113 02:51:43.024215 1191 tablet_bootstrap.cc:492] T 65d9bbd8c15f40a4a50c1695fb5e648e P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:43.024406 1191 ts_tablet_manager.cc:1397] T 65d9bbd8c15f40a4a50c1695fb5e648e P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.058s user 0.005s sys 0.003s
I20250113 02:51:43.024713 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet 9e736211eb3c4536b9fd1d90d2d83838 (table default.Table_f8af5d3e0dc64d3fae054f82a7b33ccf [id=4b739228564344b8b394a4718750dbb0]) successfully deleted
I20250113 02:51:43.025159 1191 raft_consensus.cc:357] T 65d9bbd8c15f40a4a50c1695fb5e648e P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:43.025354 1191 raft_consensus.cc:383] T 65d9bbd8c15f40a4a50c1695fb5e648e P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:43.025413 1191 raft_consensus.cc:738] T 65d9bbd8c15f40a4a50c1695fb5e648e P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:43.025635 1191 consensus_queue.cc:260] T 65d9bbd8c15f40a4a50c1695fb5e648e P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:43.027735 1191 ts_tablet_manager.cc:1428] T 65d9bbd8c15f40a4a50c1695fb5e648e P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250113 02:51:43.028074 1191 tablet_bootstrap.cc:492] T 542df00229f645788f1b821c0ce00c46 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:43.029469 1191 tablet_bootstrap.cc:654] T 542df00229f645788f1b821c0ce00c46 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:43.034540 2588 raft_consensus.cc:491] T 8cc78bf7d6324b7ba10a1b1045923f2b P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:43.034669 2588 raft_consensus.cc:513] T 8cc78bf7d6324b7ba10a1b1045923f2b P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:43.035245 2588 leader_election.cc:290] T 8cc78bf7d6324b7ba10a1b1045923f2b P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:43.035610 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cc78bf7d6324b7ba10a1b1045923f2b" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
W20250113 02:51:43.036168 709 leader_election.cc:343] T 8cc78bf7d6324b7ba10a1b1045923f2b P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:43.036361 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cc78bf7d6324b7ba10a1b1045923f2b" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:43.036842 711 leader_election.cc:343] T 8cc78bf7d6324b7ba10a1b1045923f2b P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:43.036947 711 leader_election.cc:304] T 8cc78bf7d6324b7ba10a1b1045923f2b P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:43.038543 2588 raft_consensus.cc:2743] T 8cc78bf7d6324b7ba10a1b1045923f2b P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:43.040807 453 catalog_manager.cc:5618] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095) does not have the latest schema for tablet 247af1a17df04b27898f86642336d51e (table default.Table_21fe260f30524201a0367153afecc2d8 [id=43940b76df2c4fb5bc58e5e2975cac8b]). Expected version 1 got 0
I20250113 02:51:43.041774 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_597caad30c254169ad25df814c309739"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:43.042289 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_597caad30c254169ad25df814c309739 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250113 02:51:43.042564 454 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0f091eb311d94956bb398b6c5efce284 consensus state has no leader
I20250113 02:51:43.043004 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_3263b752318740a49b84af744eac2002"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:43.043480 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3263b752318740a49b84af744eac2002 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:43.044119 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1242) took 2686 ms. Trace:
I20250113 02:51:43.044209 1198 rpcz_store.cc:276] 0113 02:51:40.357720 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:40.357778 (+ 58us) service_pool.cc:225] Handling call
0113 02:51:43.042425 (+2684647us) ts_tablet_manager.cc:1145] Deleting tablet c368b5b0c3a940c4b4efb64c946375c3
0113 02:51:43.042437 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:43.044097 (+ 1660us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2682532}
I20250113 02:51:43.044413 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_4ab048b5ba254ac6af616103105d618e" } new_table_name: "default.Table_cda7b743f02745f3af60f2687a65bb81" modify_external_catalogs: true
I20250113 02:51:43.044703 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1243) took 2664 ms. Trace:
I20250113 02:51:43.044773 1198 rpcz_store.cc:276] 0113 02:51:40.380162 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:40.380217 (+ 55us) service_pool.cc:225] Handling call
0113 02:51:43.044597 (+2664380us) ts_tablet_manager.cc:1145] Deleting tablet e4998b86d32040ec959ff6fd94b1e9eb
0113 02:51:43.044606 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:43.044687 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2664187}
I20250113 02:51:43.045130 1198 tablet_replica.cc:331] T eceed0f52c9f46bbaa78ca773a7afde5 P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:43.045315 1198 raft_consensus.cc:2238] T eceed0f52c9f46bbaa78ca773a7afde5 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: Raft consensus shutting down.
I20250113 02:51:43.045622 1198 raft_consensus.cc:2267] T eceed0f52c9f46bbaa78ca773a7afde5 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:43.046132 1198 ts_tablet_manager.cc:1905] T eceed0f52c9f46bbaa78ca773a7afde5 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:43.046980 450 catalog_manager.cc:5526] T 3f62f344c5a648328ca2de8ca1069c19 P 4216a7f853d642b7bc3e5cff458f3aca reported cstate change: term changed from 0 to 1, leader changed from <none> to 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3). New cstate: current_term: 1 leader_uuid: "4216a7f853d642b7bc3e5cff458f3aca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } health_report { overall_health: HEALTHY } } }
I20250113 02:51:43.047472 450 catalog_manager.cc:5618] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587) does not have the latest schema for tablet 3f62f344c5a648328ca2de8ca1069c19 (table default.Table_9ca0c47692a840c39dbc1c73c7a6bc7c [id=7783df86a6ee404f87055036508adb20]). Expected version 1 got 0
I20250113 02:51:43.047717 450 catalog_manager.cc:5618] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587) does not have the latest schema for tablet b815003a27f64809baaa2d43d6293cef (table default.Table_c2e8fd7d2cc14fa39c0abfece783c6c9 [id=42e19138c08f422a92ba9621e9609957]). Expected version 1 got 0
I20250113 02:51:43.048003 450 catalog_manager.cc:5618] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587) does not have the latest schema for tablet 938cc1ccd8dd449996836188ba9e64ff (table default.Table_e5b4d614e4b743d7982b025ca6d297e5 [id=5824acd778ed47d09f087bbf8e7822d6]). Expected version 1 got 0
I20250113 02:51:43.048806 1194 tablet_bootstrap.cc:492] T d9df314ffaf042ec80180868d7e7bdf7 P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:43.048977 1194 ts_tablet_manager.cc:1397] T d9df314ffaf042ec80180868d7e7bdf7 P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.049s user 0.001s sys 0.002s
I20250113 02:51:43.049678 1194 raft_consensus.cc:357] T d9df314ffaf042ec80180868d7e7bdf7 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:43.049863 1194 raft_consensus.cc:383] T d9df314ffaf042ec80180868d7e7bdf7 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:43.049914 1194 raft_consensus.cc:738] T d9df314ffaf042ec80180868d7e7bdf7 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:43.050104 1194 consensus_queue.cc:260] T d9df314ffaf042ec80180868d7e7bdf7 P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
W20250113 02:51:43.050606 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 4791a82b5e13429184d0c7192dd7aef6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4791a82b5e13429184d0c7192dd7aef6 already in progress: creating tablet
I20250113 02:51:43.052229 1196 tablet_replica.cc:331] T f8a6eebb6b6d4b63903e68750e181707 P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:43.052556 885 tablet_service.cc:1467] Processing CreateTablet for tablet 8079e324ccb446609677074c8af773b7 (DEFAULT_TABLE table=default.Table_4ab048b5ba254ac6af616103105d618e [id=6a08527d21954bbcb446e8600758d626]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:43.052947 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8079e324ccb446609677074c8af773b7. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:43.053166 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet c368b5b0c3a940c4b4efb64c946375c3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c368b5b0c3a940c4b4efb64c946375c3 already in progress: creating tablet
W20250113 02:51:43.053393 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet e4998b86d32040ec959ff6fd94b1e9eb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e4998b86d32040ec959ff6fd94b1e9eb already in progress: creating tablet
W20250113 02:51:43.053910 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 495064a93dfb41ff999fc73b579baca0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 495064a93dfb41ff999fc73b579baca0 already in progress: creating tablet
W20250113 02:51:43.054111 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet e3a1e65ec9874cc7b83485683aafff77 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e3a1e65ec9874cc7b83485683aafff77 already in progress: creating tablet
I20250113 02:51:43.054816 1194 ts_tablet_manager.cc:1428] T d9df314ffaf042ec80180868d7e7bdf7 P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20250113 02:51:43.055059 1196 raft_consensus.cc:2238] T f8a6eebb6b6d4b63903e68750e181707 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:43.055212 1196 raft_consensus.cc:2267] T f8a6eebb6b6d4b63903e68750e181707 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:43.058818 2588 raft_consensus.cc:491] T 2470b010b571444eba3f132323038f33 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:43.058955 2588 raft_consensus.cc:513] T 2470b010b571444eba3f132323038f33 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:43.059532 2588 leader_election.cc:290] T 2470b010b571444eba3f132323038f33 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:43.060103 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2470b010b571444eba3f132323038f33" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:43.060874 711 leader_election.cc:343] T 2470b010b571444eba3f132323038f33 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:43.061476 1194 tablet_bootstrap.cc:492] T f1e32e09a8244957a99bdfea5812ff31 P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:43.061990 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2470b010b571444eba3f132323038f33" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
W20250113 02:51:43.062564 709 leader_election.cc:343] T 2470b010b571444eba3f132323038f33 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:43.062672 709 leader_election.cc:304] T 2470b010b571444eba3f132323038f33 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:43.063004 2588 raft_consensus.cc:2743] T 2470b010b571444eba3f132323038f33 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250113 02:51:43.063889 998 consensus_peers.cc:487] T 42dbd37e20fa47318bc6497423dbe565 P 4216a7f853d642b7bc3e5cff458f3aca -> Peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Couldn't send request to peer df69f48656b44bb890f76587ee69cc0f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250113 02:51:43.064105 1196 ts_tablet_manager.cc:1905] T f8a6eebb6b6d4b63903e68750e181707 P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:43.068405 1194 tablet_bootstrap.cc:654] T f1e32e09a8244957a99bdfea5812ff31 P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:43.068774 891 tablet_service.cc:1514] Processing DeleteTablet for tablet d9df314ffaf042ec80180868d7e7bdf7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:24 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:43.072293 1196 ts_tablet_manager.cc:1918] T f8a6eebb6b6d4b63903e68750e181707 P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:43.072394 1196 log.cc:1198] T f8a6eebb6b6d4b63903e68750e181707 P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/f8a6eebb6b6d4b63903e68750e181707
I20250113 02:51:43.072644 1196 ts_tablet_manager.cc:1939] T f8a6eebb6b6d4b63903e68750e181707 P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:43.073601 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_534ccaa08cbf4ee6adaacca75de576bf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:43.074203 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_534ccaa08cbf4ee6adaacca75de576bf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:43.074781 1191 tablet_bootstrap.cc:492] T 542df00229f645788f1b821c0ce00c46 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
W20250113 02:51:43.075484 451 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5fead010e60544d394e0114ee4afc00c consensus state has no leader
I20250113 02:51:43.076048 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_f1d37dd0db604e66abe5918b55426fa2" } new_table_name: "default.Table_1c44e4571363423c9348ca3bb8c0e19a" modify_external_catalogs: true
I20250113 02:51:43.076407 1191 ts_tablet_manager.cc:1397] T 542df00229f645788f1b821c0ce00c46 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.048s user 0.004s sys 0.000s
I20250113 02:51:43.078341 1191 raft_consensus.cc:357] T 542df00229f645788f1b821c0ce00c46 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:43.078603 1191 raft_consensus.cc:383] T 542df00229f645788f1b821c0ce00c46 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:43.078711 1191 raft_consensus.cc:738] T 542df00229f645788f1b821c0ce00c46 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:43.078959 1191 consensus_queue.cc:260] T 542df00229f645788f1b821c0ce00c46 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:43.079535 1191 ts_tablet_manager.cc:1428] T 542df00229f645788f1b821c0ce00c46 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250113 02:51:43.079854 1191 tablet_bootstrap.cc:492] T f2c6c8c17d6e499d84ad7c0c23889cc4 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:43.081267 1191 tablet_bootstrap.cc:654] T f2c6c8c17d6e499d84ad7c0c23889cc4 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:43.074888 2547 consensus_queue.cc:1035] T 3d85539cf0074a03b167d3279e940079 P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.006s
I20250113 02:51:43.066221 1198 ts_tablet_manager.cc:1918] T eceed0f52c9f46bbaa78ca773a7afde5 P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250113 02:51:43.082213 1198 log.cc:1198] T eceed0f52c9f46bbaa78ca773a7afde5 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/eceed0f52c9f46bbaa78ca773a7afde5
I20250113 02:51:43.083670 1198 ts_tablet_manager.cc:1939] T eceed0f52c9f46bbaa78ca773a7afde5 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:43.085404 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1275) took 2461 ms. Trace:
I20250113 02:51:43.085453 2547 consensus_queue.cc:1035] T 64aa8390b1b1475c8743b6491d40674c P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 02:51:43.085542 1198 rpcz_store.cc:276] 0113 02:51:40.624349 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:40.625078 (+ 729us) service_pool.cc:225] Handling call
0113 02:51:43.045086 (+2420008us) ts_tablet_manager.cc:1145] Deleting tablet eceed0f52c9f46bbaa78ca773a7afde5
0113 02:51:43.045095 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:43.053747 (+ 8652us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:43.066119 (+ 12372us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:43.085385 (+ 19266us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2419279}
I20250113 02:51:43.086146 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet eceed0f52c9f46bbaa78ca773a7afde5 (table default.Table_6a0195b054a043269e133f6ed8dd5979 [id=4b2bd999368c4b628c70cb3ee8c2b1f3]) successfully deleted
I20250113 02:51:43.086625 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1290) took 2431 ms. Trace:
I20250113 02:51:43.087775 1198 rpcz_store.cc:276] 0113 02:51:40.655142 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:40.725037 (+ 69895us) service_pool.cc:225] Handling call
0113 02:51:43.086485 (+2361448us) ts_tablet_manager.cc:1145] Deleting tablet 42a8e01bb71f49b09c83a8f5c905f38d
0113 02:51:43.086495 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:43.086607 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2361239}
W20250113 02:51:43.088579 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 42a8e01bb71f49b09c83a8f5c905f38d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 42a8e01bb71f49b09c83a8f5c905f38d already in progress: creating tablet
I20250113 02:51:43.089373 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 1c3c3b7cf89e4f22b8a5dd7ba4f8f5ec
I20250113 02:51:43.089867 453 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_e32cce9feadb4573a31e4b3ddf77e835" } modify_external_catalogs: true
I20250113 02:51:43.090026 453 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_e32cce9feadb4573a31e4b3ddf77e835" } modify_external_catalogs: true
I20250113 02:51:43.090258 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_7cdec29878244c9a9b06a312482a2ed6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:43.090766 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet d1f30873a02f4b61ac6940e44c332ad3
W20250113 02:51:43.090927 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7cdec29878244c9a9b06a312482a2ed6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:43.091372 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_c8b93363989c4e109540af6bef426f73" } modify_external_catalogs: true
I20250113 02:51:43.091504 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_c8b93363989c4e109540af6bef426f73" } modify_external_catalogs: true
I20250113 02:51:43.091841 739 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c3c3b7cf89e4f22b8a5dd7ba4f8f5ec with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:42 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:43.092712 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet f86318637d5b4e378ce47e89a5d52b07
I20250113 02:51:43.093137 446 catalog_manager.cc:6596] ReplaceTablet: tablet f86318637d5b4e378ce47e89a5d52b07 deleted and replaced by tablet 7c7f54f1b0754423bcac4686d5f3a420
I20250113 02:51:43.093291 739 tablet_service.cc:1514] Processing DeleteTablet for tablet f86318637d5b4e378ce47e89a5d52b07 with delete_type TABLET_DATA_DELETED (replaced by tablet 7c7f54f1b0754423bcac4686d5f3a420) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:43.093262 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c3c3b7cf89e4f22b8a5dd7ba4f8f5ec with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:42 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:43.093264 1032 tablet_service.cc:1514] Processing DeleteTablet for tablet d1f30873a02f4b61ac6940e44c332ad3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:42 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:43.094028 1032 tablet_service.cc:1514] Processing DeleteTablet for tablet f86318637d5b4e378ce47e89a5d52b07 with delete_type TABLET_DATA_DELETED (replaced by tablet 7c7f54f1b0754423bcac4686d5f3a420) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:43.095053 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet aa1ebdb92a5f4c92a5195b614689c100 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:29 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:43.095271 1032 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a4c6dc08df744e6856bdf99ab65c575 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:26 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:43.096097 748 tablet_service.cc:1514] Processing DeleteTablet for tablet d1f30873a02f4b61ac6940e44c332ad3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:42 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:43.096922 448 catalog_manager.cc:5526] T b38cd34d67834a3f9bf5b5636bb79217 P 1186fa869eb14812832fa6562531f196 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1186fa869eb14812832fa6562531f196 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "1186fa869eb14812832fa6562531f196" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } health_report { overall_health: UNKNOWN } } }
I20250113 02:51:43.097630 1196 tablet_replica.cc:331] T dd566a49d3ff4d3889b80d8d2ccbee0e P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:43.097802 1196 raft_consensus.cc:2238] T dd566a49d3ff4d3889b80d8d2ccbee0e P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:43.097941 1196 raft_consensus.cc:2267] T dd566a49d3ff4d3889b80d8d2ccbee0e P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:43.098544 885 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c3c3b7cf89e4f22b8a5dd7ba4f8f5ec with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:42 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:43.099237 891 tablet_service.cc:1514] Processing DeleteTablet for tablet d1f30873a02f4b61ac6940e44c332ad3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:42 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:43.099539 883 tablet_service.cc:1514] Processing DeleteTablet for tablet f86318637d5b4e378ce47e89a5d52b07 with delete_type TABLET_DATA_DELETED (replaced by tablet 7c7f54f1b0754423bcac4686d5f3a420) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:43.099948 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet f8a6eebb6b6d4b63903e68750e181707 (table default.Table_4484fe40b4454c5ea4943a4d38f49c5e [id=a0e91da6e0ee449b92b7afbc6f68087d]) successfully deleted
I20250113 02:51:43.100138 1192 tablet_bootstrap.cc:492] T 21bcdd741104474bad7023426856d18f P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:43.100265 1192 ts_tablet_manager.cc:1397] T 21bcdd741104474bad7023426856d18f P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.097s user 0.003s sys 0.000s
I20250113 02:51:43.101123 1192 raft_consensus.cc:357] T 21bcdd741104474bad7023426856d18f P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:43.101359 1192 raft_consensus.cc:383] T 21bcdd741104474bad7023426856d18f P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:43.101418 1192 raft_consensus.cc:738] T 21bcdd741104474bad7023426856d18f P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:43.101647 1192 consensus_queue.cc:260] T 21bcdd741104474bad7023426856d18f P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:43.102264 1192 ts_tablet_manager.cc:1428] T 21bcdd741104474bad7023426856d18f P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250113 02:51:43.102532 1192 tablet_bootstrap.cc:492] T 283fc7f1c74e480c8a63c74d3ae9eb7d P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:43.104733 2588 raft_consensus.cc:491] T 9f8eb43a39774d2c8e9cb4e88a405b0f P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:43.104863 2588 raft_consensus.cc:513] T 9f8eb43a39774d2c8e9cb4e88a405b0f P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:43.105437 2588 leader_election.cc:290] T 9f8eb43a39774d2c8e9cb4e88a405b0f P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:43.105927 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f8eb43a39774d2c8e9cb4e88a405b0f" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:43.106446 711 leader_election.cc:343] T 9f8eb43a39774d2c8e9cb4e88a405b0f P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:43.109127 1191 tablet_bootstrap.cc:492] T f2c6c8c17d6e499d84ad7c0c23889cc4 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:43.109258 1191 ts_tablet_manager.cc:1397] T f2c6c8c17d6e499d84ad7c0c23889cc4 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.029s user 0.002s sys 0.000s
I20250113 02:51:43.109856 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f8eb43a39774d2c8e9cb4e88a405b0f" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:43.109870 1191 raft_consensus.cc:357] T f2c6c8c17d6e499d84ad7c0c23889cc4 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:43.110052 1191 raft_consensus.cc:383] T f2c6c8c17d6e499d84ad7c0c23889cc4 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:43.110100 1191 raft_consensus.cc:738] T f2c6c8c17d6e499d84ad7c0c23889cc4 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:43.110324 1191 consensus_queue.cc:260] T f2c6c8c17d6e499d84ad7c0c23889cc4 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
W20250113 02:51:43.110476 709 leader_election.cc:343] T 9f8eb43a39774d2c8e9cb4e88a405b0f P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:43.110589 709 leader_election.cc:304] T 9f8eb43a39774d2c8e9cb4e88a405b0f P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:43.110823 1191 ts_tablet_manager.cc:1428] T f2c6c8c17d6e499d84ad7c0c23889cc4 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20250113 02:51:43.111097 1191 tablet_bootstrap.cc:492] T b217cb8c09fb41cab4f53e605c37efe8 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:43.111326 2501 raft_consensus.cc:2743] T 9f8eb43a39774d2c8e9cb4e88a405b0f P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:43.112553 1191 tablet_bootstrap.cc:654] T b217cb8c09fb41cab4f53e605c37efe8 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:43.112920 1192 tablet_bootstrap.cc:654] T 283fc7f1c74e480c8a63c74d3ae9eb7d P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:43.114370 1196 ts_tablet_manager.cc:1905] T dd566a49d3ff4d3889b80d8d2ccbee0e P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:43.118085 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1291) took 2462 ms. Trace:
I20250113 02:51:43.118455 1198 rpcz_store.cc:276] 0113 02:51:40.655204 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:40.725272 (+ 70068us) service_pool.cc:225] Handling call
0113 02:51:43.117935 (+2392663us) ts_tablet_manager.cc:1145] Deleting tablet 94307d22b7f842ca89844768e4a2a2ab
0113 02:51:43.117948 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:43.118067 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2369041}
W20250113 02:51:43.119110 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 94307d22b7f842ca89844768e4a2a2ab because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 94307d22b7f842ca89844768e4a2a2ab already in progress: creating tablet
I20250113 02:51:43.119625 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1292) took 2454 ms. Trace:
I20250113 02:51:43.121260 1194 tablet_bootstrap.cc:492] T f1e32e09a8244957a99bdfea5812ff31 P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:43.121407 1194 ts_tablet_manager.cc:1397] T f1e32e09a8244957a99bdfea5812ff31 P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.060s user 0.001s sys 0.009s
I20250113 02:51:43.122148 1194 raft_consensus.cc:357] T f1e32e09a8244957a99bdfea5812ff31 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:43.122354 1194 raft_consensus.cc:383] T f1e32e09a8244957a99bdfea5812ff31 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:43.122403 1194 raft_consensus.cc:738] T f1e32e09a8244957a99bdfea5812ff31 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:43.122608 1194 consensus_queue.cc:260] T f1e32e09a8244957a99bdfea5812ff31 P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:43.122675 1198 rpcz_store.cc:276] 0113 02:51:40.664994 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:40.725418 (+ 60424us) service_pool.cc:225] Handling call
0113 02:51:43.119526 (+2394108us) ts_tablet_manager.cc:1145] Deleting tablet a0c14501ad0449aca28909bc14428dad
0113 02:51:43.119536 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:43.119611 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2394004}
W20250113 02:51:43.124750 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet a0c14501ad0449aca28909bc14428dad because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a0c14501ad0449aca28909bc14428dad already in progress: creating tablet
I20250113 02:51:43.125269 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1294) took 2451 ms. Trace:
I20250113 02:51:43.126760 1198 rpcz_store.cc:276] 0113 02:51:40.674052 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:40.725879 (+ 51827us) service_pool.cc:225] Handling call
0113 02:51:43.125153 (+2399274us) ts_tablet_manager.cc:1145] Deleting tablet 6001369444894b48b51a48fb60ddc52a
0113 02:51:43.125165 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:43.125252 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2399131}
I20250113 02:51:43.127100 883 tablet_service.cc:1514] Processing DeleteTablet for tablet f1e32e09a8244957a99bdfea5812ff31 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:25 UTC) from {username='slave'} at 127.0.0.1:37366
W20250113 02:51:43.127497 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 6001369444894b48b51a48fb60ddc52a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6001369444894b48b51a48fb60ddc52a already in progress: creating tablet
I20250113 02:51:43.127791 1194 ts_tablet_manager.cc:1428] T f1e32e09a8244957a99bdfea5812ff31 P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20250113 02:51:43.129176 1198 tablet_replica.cc:331] T b33025a462e74f2aa26cc3779f36615b P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:43.129611 1198 raft_consensus.cc:2238] T b33025a462e74f2aa26cc3779f36615b P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: Raft consensus shutting down.
I20250113 02:51:43.130327 1198 raft_consensus.cc:2267] T b33025a462e74f2aa26cc3779f36615b P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Raft consensus is shut down!
W20250113 02:51:43.132622 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8079e324ccb446609677074c8af773b7 consensus state has no leader
W20250113 02:51:43.133249 998 consensus_peers.cc:487] T 64aa8390b1b1475c8743b6491d40674c P 4216a7f853d642b7bc3e5cff458f3aca -> Peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Couldn't send request to peer df69f48656b44bb890f76587ee69cc0f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250113 02:51:43.133872 1032 tablet_service.cc:1467] Processing CreateTablet for tablet 8079e324ccb446609677074c8af773b7 (DEFAULT_TABLE table=default.Table_4ab048b5ba254ac6af616103105d618e [id=6a08527d21954bbcb446e8600758d626]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:43.134268 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8079e324ccb446609677074c8af773b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:43.134967 748 tablet_service.cc:1467] Processing CreateTablet for tablet 8079e324ccb446609677074c8af773b7 (DEFAULT_TABLE table=default.Table_cda7b743f02745f3af60f2687a65bb81 [id=6a08527d21954bbcb446e8600758d626]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:43.135066 739 tablet_service.cc:1467] Processing CreateTablet for tablet fbaaa0a7124b4b3d8185220fcfeb9e7f (DEFAULT_TABLE table=default.Table_3a930068998b48628b755bc9782bc464 [id=f7ebe5538097479baa99cb88ba484f75]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:43.135383 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fbaaa0a7124b4b3d8185220fcfeb9e7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:43.135969 883 tablet_service.cc:1467] Processing CreateTablet for tablet fbaaa0a7124b4b3d8185220fcfeb9e7f (DEFAULT_TABLE table=default.Table_3a930068998b48628b755bc9782bc464 [id=f7ebe5538097479baa99cb88ba484f75]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:43.136351 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fbaaa0a7124b4b3d8185220fcfeb9e7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:43.140267 891 tablet_service.cc:1467] Processing CreateTablet for tablet 245ea9f0ae2244e1a1044be53de738e1 (DEFAULT_TABLE table=default.Table_e5d873c74354424aad0f76593065cacd [id=d44298f6cd75487e99a58556f46a796d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:43.140657 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 245ea9f0ae2244e1a1044be53de738e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:43.142436 1195 ts_tablet_manager.cc:1918] T 1d2fdc75d00f4e749d322ac6cd5fc410 P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250113 02:51:43.142541 1195 log.cc:1198] T 1d2fdc75d00f4e749d322ac6cd5fc410 P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/1d2fdc75d00f4e749d322ac6cd5fc410
I20250113 02:51:43.142858 1195 ts_tablet_manager.cc:1939] T 1d2fdc75d00f4e749d322ac6cd5fc410 P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:43.143174 1032 tablet_service.cc:1467] Processing CreateTablet for tablet fbaaa0a7124b4b3d8185220fcfeb9e7f (DEFAULT_TABLE table=default.Table_3a930068998b48628b755bc9782bc464 [id=f7ebe5538097479baa99cb88ba484f75]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:43.143551 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fbaaa0a7124b4b3d8185220fcfeb9e7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:43.144048 739 tablet_service.cc:1467] Processing CreateTablet for tablet 245ea9f0ae2244e1a1044be53de738e1 (DEFAULT_TABLE table=default.Table_e5d873c74354424aad0f76593065cacd [id=d44298f6cd75487e99a58556f46a796d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:43.144440 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 245ea9f0ae2244e1a1044be53de738e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:43.144522 1024 tablet_service.cc:1467] Processing CreateTablet for tablet 245ea9f0ae2244e1a1044be53de738e1 (DEFAULT_TABLE table=default.Table_e5d873c74354424aad0f76593065cacd [id=d44298f6cd75487e99a58556f46a796d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:43.144721 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8079e324ccb446609677074c8af773b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:43.144870 885 tablet_service.cc:1467] Processing CreateTablet for tablet 9cedef8390f94111a6b3d8d229c75b27 (DEFAULT_TABLE table=default.Table_30951971e07445449dd042657deb91ec [id=38f42d9ea75340fdb8b32eff730a63e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:43.145061 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 245ea9f0ae2244e1a1044be53de738e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:43.145206 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9cedef8390f94111a6b3d8d229c75b27. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:43.146700 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_dd419a5795284345929ba3de5c1a0087" } new_table_name: "default.Table_77c140d8e2c24ad8a08b0c2ed713ba02" modify_external_catalogs: true
I20250113 02:51:43.153604 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_9c2b9596afed4532858a877b5e954b4d" } modify_external_catalogs: true
I20250113 02:51:43.153780 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_9c2b9596afed4532858a877b5e954b4d" } modify_external_catalogs: true
I20250113 02:51:43.155310 1194 tablet_bootstrap.cc:492] T 655b63ff61ff4f4aa08e0449df39352a P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:43.156919 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_82bda2c55fed4358ba3769c07352f440"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:43.157385 1194 tablet_bootstrap.cc:654] T 655b63ff61ff4f4aa08e0449df39352a P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
W20250113 02:51:43.157440 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_82bda2c55fed4358ba3769c07352f440 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250113 02:51:43.158176 998 consensus_peers.cc:487] T 3d85539cf0074a03b167d3279e940079 P 4216a7f853d642b7bc3e5cff458f3aca -> Peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Couldn't send request to peer df69f48656b44bb890f76587ee69cc0f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250113 02:51:43.158921 421 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.104.2:38467, user_credentials={real_user=slave}} blocked reactor thread for 48834us
I20250113 02:51:43.164058 1198 ts_tablet_manager.cc:1905] T b33025a462e74f2aa26cc3779f36615b P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250113 02:51:43.174839 451 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 65139dcfe89249ec95f5425b294f04d8 consensus state has no leader
I20250113 02:51:43.175297 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_4600561619ef45df810b4bb624864f7d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:43.175813 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4600561619ef45df810b4bb624864f7d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:43.178057 1192 tablet_bootstrap.cc:492] T 283fc7f1c74e480c8a63c74d3ae9eb7d P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:43.178659 1192 ts_tablet_manager.cc:1397] T 283fc7f1c74e480c8a63c74d3ae9eb7d P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.076s user 0.007s sys 0.000s
I20250113 02:51:43.179375 1192 raft_consensus.cc:357] T 283fc7f1c74e480c8a63c74d3ae9eb7d P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:43.179625 1192 raft_consensus.cc:383] T 283fc7f1c74e480c8a63c74d3ae9eb7d P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:43.179713 1192 raft_consensus.cc:738] T 283fc7f1c74e480c8a63c74d3ae9eb7d P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:43.179944 1192 consensus_queue.cc:260] T 283fc7f1c74e480c8a63c74d3ae9eb7d P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:43.180553 1192 ts_tablet_manager.cc:1428] T 283fc7f1c74e480c8a63c74d3ae9eb7d P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250113 02:51:43.180881 1192 tablet_bootstrap.cc:492] T 9f8eb43a39774d2c8e9cb4e88a405b0f P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:43.182258 1192 tablet_bootstrap.cc:654] T 9f8eb43a39774d2c8e9cb4e88a405b0f P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:43.182507 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 7da6f8a4b89647968609f73f9ad5ba14
I20250113 02:51:43.182804 1196 ts_tablet_manager.cc:1918] T dd566a49d3ff4d3889b80d8d2ccbee0e P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:43.182926 1196 log.cc:1198] T dd566a49d3ff4d3889b80d8d2ccbee0e P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/dd566a49d3ff4d3889b80d8d2ccbee0e
I20250113 02:51:43.183234 1196 ts_tablet_manager.cc:1939] T dd566a49d3ff4d3889b80d8d2ccbee0e P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
W20250113 02:51:43.183512 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ccc24d92091a4643a66959d1f8ddb865 consensus state has no leader
I20250113 02:51:43.183607 748 tablet_service.cc:1514] Processing DeleteTablet for tablet 7da6f8a4b89647968609f73f9ad5ba14 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:43 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:43.183835 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet ef078624960843eead2edd1e39b10118
I20250113 02:51:43.184641 748 tablet_service.cc:1514] Processing DeleteTablet for tablet ef078624960843eead2edd1e39b10118 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:43 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:43.185109 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet dd566a49d3ff4d3889b80d8d2ccbee0e (table default.Table_34e676c9d2fb4f7ba8a20c74bc14eccf [id=0e2055968e5a492492305d4513f0edba]) successfully deleted
I20250113 02:51:43.185740 891 tablet_service.cc:1514] Processing DeleteTablet for tablet ef078624960843eead2edd1e39b10118 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:43 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:43.186318 883 tablet_service.cc:1514] Processing DeleteTablet for tablet 7da6f8a4b89647968609f73f9ad5ba14 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:43 UTC) from {username='slave'} at 127.0.0.1:37366
W20250113 02:51:43.186421 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet a446180f980640268170b4ab652d32cc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a446180f980640268170b4ab652d32cc already in progress: creating tablet
I20250113 02:51:43.186776 1196 tablet_replica.cc:331] T 0b119704bc774ca9b4155c9619d4765b P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:43.187040 1196 raft_consensus.cc:2238] T 0b119704bc774ca9b4155c9619d4765b P df69f48656b44bb890f76587ee69cc0f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:43.188160 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet ef078624960843eead2edd1e39b10118 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:43 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:43.189374 1032 tablet_service.cc:1514] Processing DeleteTablet for tablet 7da6f8a4b89647968609f73f9ad5ba14 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:43 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:43.189709 1196 raft_consensus.cc:2267] T 0b119704bc774ca9b4155c9619d4765b P df69f48656b44bb890f76587ee69cc0f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:43.191195 2596 raft_consensus.cc:491] T fd0b8592997d4a61b8ca854ecbb67c55 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:43.191334 2596 raft_consensus.cc:513] T fd0b8592997d4a61b8ca854ecbb67c55 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:43.191941 2596 leader_election.cc:290] T fd0b8592997d4a61b8ca854ecbb67c55 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:43.192183 1196 ts_tablet_manager.cc:1905] T 0b119704bc774ca9b4155c9619d4765b P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:43.193140 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd0b8592997d4a61b8ca854ecbb67c55" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
I20250113 02:51:43.193339 769 raft_consensus.cc:2463] T fd0b8592997d4a61b8ca854ecbb67c55 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 0.
I20250113 02:51:43.193717 995 leader_election.cc:304] T fd0b8592997d4a61b8ca854ecbb67c55 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters:
I20250113 02:51:43.194162 2547 raft_consensus.cc:2798] T fd0b8592997d4a61b8ca854ecbb67c55 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 02:51:43.194226 2547 raft_consensus.cc:491] T fd0b8592997d4a61b8ca854ecbb67c55 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 02:51:43.194290 2547 raft_consensus.cc:3054] T fd0b8592997d4a61b8ca854ecbb67c55 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:43.194923 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_07901047d0824d36a72c975873c14d3a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:43.195417 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_07901047d0824d36a72c975873c14d3a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:43.196559 453 master_service.cc:946] ReplaceTablet: received request to replace tablet b16ff51101e0452ebc05374f0d1f4917 from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:43.196648 2547 raft_consensus.cc:513] T fd0b8592997d4a61b8ca854ecbb67c55 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:43.197225 2547 leader_election.cc:290] T fd0b8592997d4a61b8ca854ecbb67c55 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Requested vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:43.198297 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd0b8592997d4a61b8ca854ecbb67c55" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:43.198542 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd0b8592997d4a61b8ca854ecbb67c55" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f"
I20250113 02:51:43.199023 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd0b8592997d4a61b8ca854ecbb67c55" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196"
I20250113 02:51:43.199198 769 raft_consensus.cc:3054] T fd0b8592997d4a61b8ca854ecbb67c55 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Advancing to term 1
W20250113 02:51:43.200107 998 leader_election.cc:343] T fd0b8592997d4a61b8ca854ecbb67c55 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
W20250113 02:51:43.200407 998 leader_election.cc:343] T fd0b8592997d4a61b8ca854ecbb67c55 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:43.203087 1198 ts_tablet_manager.cc:1918] T b33025a462e74f2aa26cc3779f36615b P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250113 02:51:43.203868 454 catalog_manager.cc:5526] T c13e9a51c1e74b44b352bade78390ba5 P 4216a7f853d642b7bc3e5cff458f3aca reported cstate change: term changed from 0 to 1, leader changed from <none> to 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3). New cstate: current_term: 1 leader_uuid: "4216a7f853d642b7bc3e5cff458f3aca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } health_report { overall_health: UNKNOWN } } }
I20250113 02:51:43.204335 454 catalog_manager.cc:5618] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587) does not have the latest schema for tablet c13e9a51c1e74b44b352bade78390ba5 (table default.Table_0468611ec68e47109f5357935e50874d [id=caf098e2b57849a997bd7c8eaf1f9326]). Expected version 1 got 0
I20250113 02:51:43.204373 769 raft_consensus.cc:2463] T fd0b8592997d4a61b8ca854ecbb67c55 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 1.
I20250113 02:51:43.204509 1198 log.cc:1198] T b33025a462e74f2aa26cc3779f36615b P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/b33025a462e74f2aa26cc3779f36615b
I20250113 02:51:43.204878 1198 ts_tablet_manager.cc:1939] T b33025a462e74f2aa26cc3779f36615b P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:43.205017 995 leader_election.cc:304] T fd0b8592997d4a61b8ca854ecbb67c55 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters: df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:43.205394 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 7527e6e0b65e4a43b7f48899a34cdf7f
I20250113 02:51:43.205457 2596 raft_consensus.cc:2798] T fd0b8592997d4a61b8ca854ecbb67c55 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Leader election won for term 1
I20250113 02:51:43.205561 2596 raft_consensus.cc:695] T fd0b8592997d4a61b8ca854ecbb67c55 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: Becoming Leader. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Running, Role: LEADER
I20250113 02:51:43.205639 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1295) took 2506 ms. Trace:
I20250113 02:51:43.205771 2596 consensus_queue.cc:237] T fd0b8592997d4a61b8ca854ecbb67c55 P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:43.206574 883 tablet_service.cc:1514] Processing DeleteTablet for tablet 7527e6e0b65e4a43b7f48899a34cdf7f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:43 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:43.207288 748 tablet_service.cc:1514] Processing DeleteTablet for tablet 7527e6e0b65e4a43b7f48899a34cdf7f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:43 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:43.207429 736 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b2d0aa6f0e9456d95f4e48c360ab6e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:33 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:43.207538 748 tablet_service.cc:1514] Processing DeleteTablet for tablet 2470b010b571444eba3f132323038f33 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:24 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:43.207789 748 tablet_service.cc:1514] Processing DeleteTablet for tablet 65d9bbd8c15f40a4a50c1695fb5e648e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:24 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:43.207959 2476 raft_consensus.cc:491] T d9df314ffaf042ec80180868d7e7bdf7 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:43.208142 2476 raft_consensus.cc:513] T d9df314ffaf042ec80180868d7e7bdf7 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:43.208277 1032 tablet_service.cc:1514] Processing DeleteTablet for tablet 7527e6e0b65e4a43b7f48899a34cdf7f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:43 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:43.208770 2476 leader_election.cc:290] T d9df314ffaf042ec80180868d7e7bdf7 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
W20250113 02:51:43.209962 851 leader_election.cc:343] T d9df314ffaf042ec80180868d7e7bdf7 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Not found: Tablet not found: d9df314ffaf042ec80180868d7e7bdf7
W20250113 02:51:43.210938 852 leader_election.cc:343] T d9df314ffaf042ec80180868d7e7bdf7 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Not found: Tablet not found: d9df314ffaf042ec80180868d7e7bdf7
I20250113 02:51:43.211072 852 leader_election.cc:304] T d9df314ffaf042ec80180868d7e7bdf7 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df69f48656b44bb890f76587ee69cc0f; no voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca
I20250113 02:51:43.211571 2476 raft_consensus.cc:2743] T d9df314ffaf042ec80180868d7e7bdf7 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:43.205698 1198 rpcz_store.cc:276] 0113 02:51:40.699607 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:40.726032 (+ 26425us) service_pool.cc:225] Handling call
0113 02:51:43.129111 (+2403079us) ts_tablet_manager.cc:1145] Deleting tablet b33025a462e74f2aa26cc3779f36615b
0113 02:51:43.129125 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:43.185657 (+ 56532us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:43.202977 (+ 17320us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:43.205622 (+ 2645us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2401719}
I20250113 02:51:43.212383 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1296) took 2426 ms. Trace:
I20250113 02:51:43.212757 1198 rpcz_store.cc:276] 0113 02:51:40.785578 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:40.788196 (+ 2618us) service_pool.cc:225] Handling call
0113 02:51:43.212273 (+2424077us) ts_tablet_manager.cc:1145] Deleting tablet a894199f21aa4dba8a03583ea14a9092
0113 02:51:43.212285 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:43.212369 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2423875}
I20250113 02:51:43.213431 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1300) took 2390 ms. Trace:
I20250113 02:51:43.213789 1198 rpcz_store.cc:276] 0113 02:51:40.823001 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:40.823489 (+ 488us) service_pool.cc:225] Handling call
0113 02:51:43.213343 (+2389854us) ts_tablet_manager.cc:1145] Deleting tablet 8f2b4df7c6a74896ade95a490ea9b54c
0113 02:51:43.213352 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:43.213419 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2384207}
I20250113 02:51:43.214417 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1302) took 2383 ms. Trace:
I20250113 02:51:43.216267 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet b33025a462e74f2aa26cc3779f36615b (table default.Table_23c1655ab4cf4a30b67185ccf4fbb76d [id=a8e428598ca94e15b04f1df8112a4abc]) successfully deleted
W20250113 02:51:43.216503 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet a894199f21aa4dba8a03583ea14a9092 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a894199f21aa4dba8a03583ea14a9092 already in progress: creating tablet
W20250113 02:51:43.216667 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 8f2b4df7c6a74896ade95a490ea9b54c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8f2b4df7c6a74896ade95a490ea9b54c already in progress: creating tablet
I20250113 02:51:43.214656 1032 tablet_service.cc:1514] Processing DeleteTablet for tablet 67baaab1e36e415381579d7385d4ab5a with delete_type TABLET_DATA_DELETED (replaced by tablet c368b5b0c3a940c4b4efb64c946375c3) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:43.214941 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet e570465a3d4046b1ac19a3a034fe91f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:27 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:43.215406 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet 1968c7e9357c4c6e901d334ace0467fb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:26 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:43.215870 1039 tablet_service.cc:1514] Processing DeleteTablet for tablet fd0b8592997d4a61b8ca854ecbb67c55 with delete_type TABLET_DATA_DELETED (replaced by tablet e312e70334e04ecd843159ac72f4a2a6) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:43.219628 1198 rpcz_store.cc:276] 0113 02:51:40.831365 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:40.831685 (+ 320us) service_pool.cc:225] Handling call
0113 02:51:43.214334 (+2382649us) ts_tablet_manager.cc:1145] Deleting tablet 3c623bd0fbbf4922aedda031a88b79eb
0113 02:51:43.214342 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:43.214405 (+ 63us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2382500}
I20250113 02:51:43.220504 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1317) took 2261 ms. Trace:
I20250113 02:51:43.220955 1198 rpcz_store.cc:276] 0113 02:51:40.959430 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:40.959726 (+ 296us) service_pool.cc:225] Handling call
0113 02:51:43.220391 (+2260665us) ts_tablet_manager.cc:1145] Deleting tablet aef2ffd67def48eea501a98fbc9dc43b
0113 02:51:43.220401 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:43.220488 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2022367}
W20250113 02:51:43.221503 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 3c623bd0fbbf4922aedda031a88b79eb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3c623bd0fbbf4922aedda031a88b79eb already in progress: creating tablet
I20250113 02:51:43.221280 2596 consensus_queue.cc:1035] T 42dbd37e20fa47318bc6497423dbe565 P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 02:51:43.221845 1198 tablet_replica.cc:331] T 438d1d2300e044f39a15586f335935ba P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:43.221938 2596 raft_consensus.cc:491] T 1968c7e9357c4c6e901d334ace0467fb P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250113 02:51:43.221971 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet aef2ffd67def48eea501a98fbc9dc43b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet aef2ffd67def48eea501a98fbc9dc43b already in progress: creating tablet
I20250113 02:51:43.222044 1198 raft_consensus.cc:2238] T 438d1d2300e044f39a15586f335935ba P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: Raft consensus shutting down.
I20250113 02:51:43.222030 2596 raft_consensus.cc:513] T 1968c7e9357c4c6e901d334ace0467fb P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:43.222304 1198 raft_consensus.cc:2267] T 438d1d2300e044f39a15586f335935ba P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:43.222576 2596 leader_election.cc:290] T 1968c7e9357c4c6e901d334ace0467fb P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
I20250113 02:51:43.222846 1198 ts_tablet_manager.cc:1905] T 438d1d2300e044f39a15586f335935ba P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:43.223169 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1968c7e9357c4c6e901d334ace0467fb" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:43.223852 998 leader_election.cc:343] T 1968c7e9357c4c6e901d334ace0467fb P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:43.226843 1196 ts_tablet_manager.cc:1918] T 0b119704bc774ca9b4155c9619d4765b P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250113 02:51:43.226979 1196 log.cc:1198] T 0b119704bc774ca9b4155c9619d4765b P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/0b119704bc774ca9b4155c9619d4765b
I20250113 02:51:43.226933 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1968c7e9357c4c6e901d334ace0467fb" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
I20250113 02:51:43.227157 773 raft_consensus.cc:2463] T 1968c7e9357c4c6e901d334ace0467fb P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 0.
I20250113 02:51:43.227371 1196 ts_tablet_manager.cc:1939] T 0b119704bc774ca9b4155c9619d4765b P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:43.227946 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet 0b119704bc774ca9b4155c9619d4765b (table default.Table_b95a9b1d12284c9aac701a8ef9c93482 [id=843aabcff7074acd98e5801b21282faa]) successfully deleted
I20250113 02:51:43.228593 995 leader_election.cc:304] T 1968c7e9357c4c6e901d334ace0467fb P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters: df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:43.229818 2547 raft_consensus.cc:2798] T 1968c7e9357c4c6e901d334ace0467fb P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 02:51:43.229925 2547 raft_consensus.cc:491] T 1968c7e9357c4c6e901d334ace0467fb P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 02:51:43.230005 2547 raft_consensus.cc:3054] T 1968c7e9357c4c6e901d334ace0467fb P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:43.231617 1196 tablet_replica.cc:331] T 8337f3c82d654ecdac2cbc1ffa88ef05 P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:43.232141 1196 raft_consensus.cc:2238] T 8337f3c82d654ecdac2cbc1ffa88ef05 P df69f48656b44bb890f76587ee69cc0f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:43.232471 1032 tablet_service.cc:1514] Processing DeleteTablet for tablet 21bcdd741104474bad7023426856d18f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:33 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:43.232709 1032 tablet_service.cc:1514] Processing DeleteTablet for tablet 283fc7f1c74e480c8a63c74d3ae9eb7d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:24 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:43.233013 2547 raft_consensus.cc:513] T 1968c7e9357c4c6e901d334ace0467fb P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:43.233702 1196 raft_consensus.cc:2267] T 8337f3c82d654ecdac2cbc1ffa88ef05 P df69f48656b44bb890f76587ee69cc0f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:43.234179 2547 leader_election.cc:290] T 1968c7e9357c4c6e901d334ace0467fb P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Requested vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
I20250113 02:51:43.234169 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1968c7e9357c4c6e901d334ace0467fb" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f"
I20250113 02:51:43.234654 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1968c7e9357c4c6e901d334ace0467fb" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196"
I20250113 02:51:43.234879 773 raft_consensus.cc:3054] T 1968c7e9357c4c6e901d334ace0467fb P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:43.235414 1196 ts_tablet_manager.cc:1905] T 8337f3c82d654ecdac2cbc1ffa88ef05 P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:43.237444 773 raft_consensus.cc:2463] T 1968c7e9357c4c6e901d334ace0467fb P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 1.
W20250113 02:51:43.238009 998 leader_election.cc:343] T 1968c7e9357c4c6e901d334ace0467fb P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:43.238525 995 leader_election.cc:304] T 1968c7e9357c4c6e901d334ace0467fb P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters: df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:43.239413 2596 raft_consensus.cc:2798] T 1968c7e9357c4c6e901d334ace0467fb P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Leader election won for term 1
I20250113 02:51:43.239537 2596 raft_consensus.cc:695] T 1968c7e9357c4c6e901d334ace0467fb P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: Becoming Leader. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Running, Role: LEADER
I20250113 02:51:43.239778 2596 consensus_queue.cc:237] T 1968c7e9357c4c6e901d334ace0467fb P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:43.240954 2588 raft_consensus.cc:491] T b3ce6d851f564b849796f174860aacd5 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:43.241084 2588 raft_consensus.cc:513] T b3ce6d851f564b849796f174860aacd5 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:43.241675 2588 leader_election.cc:290] T b3ce6d851f564b849796f174860aacd5 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:43.242322 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3ce6d851f564b849796f174860aacd5" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:43.243098 711 leader_election.cc:343] T b3ce6d851f564b849796f174860aacd5 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:43.244042 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3ce6d851f564b849796f174860aacd5" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
W20250113 02:51:43.244593 709 leader_election.cc:343] T b3ce6d851f564b849796f174860aacd5 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:43.244700 709 leader_election.cc:304] T b3ce6d851f564b849796f174860aacd5 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:43.245126 2588 raft_consensus.cc:2743] T b3ce6d851f564b849796f174860aacd5 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:43.245922 1198 ts_tablet_manager.cc:1918] T 438d1d2300e044f39a15586f335935ba P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250113 02:51:43.251026 1198 log.cc:1198] T 438d1d2300e044f39a15586f335935ba P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/438d1d2300e044f39a15586f335935ba
I20250113 02:51:43.251475 1198 ts_tablet_manager.cc:1939] T 438d1d2300e044f39a15586f335935ba P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:43.252653 1192 tablet_bootstrap.cc:492] T 9f8eb43a39774d2c8e9cb4e88a405b0f P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:43.252803 1192 ts_tablet_manager.cc:1397] T 9f8eb43a39774d2c8e9cb4e88a405b0f P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.072s user 0.004s sys 0.010s
I20250113 02:51:43.253122 448 catalog_manager.cc:5618] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095) does not have the latest schema for tablet f2c6c8c17d6e499d84ad7c0c23889cc4 (table default.Table_92ea541283614df981d96c5b74362f1d [id=9e58548d1a184cc0b362f148c44005ed]). Expected version 2 got 0
I20250113 02:51:43.253532 2588 raft_consensus.cc:491] T 542df00229f645788f1b821c0ce00c46 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:43.253679 2588 raft_consensus.cc:513] T 542df00229f645788f1b821c0ce00c46 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:43.254317 2588 leader_election.cc:290] T 542df00229f645788f1b821c0ce00c46 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:43.254317 1192 raft_consensus.cc:357] T 9f8eb43a39774d2c8e9cb4e88a405b0f P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:43.254518 1192 raft_consensus.cc:383] T 9f8eb43a39774d2c8e9cb4e88a405b0f P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:43.254429 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_f03ffbab220d4fb89b7c7baf4b03aa96"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:43.254581 1192 raft_consensus.cc:738] T 9f8eb43a39774d2c8e9cb4e88a405b0f P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:43.254807 1192 consensus_queue.cc:260] T 9f8eb43a39774d2c8e9cb4e88a405b0f P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
W20250113 02:51:43.254990 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f03ffbab220d4fb89b7c7baf4b03aa96 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:43.255703 1192 ts_tablet_manager.cc:1428] T 9f8eb43a39774d2c8e9cb4e88a405b0f P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20250113 02:51:43.255961 1192 tablet_bootstrap.cc:492] T fe8f075d4c9a4f9abe372bfdb81a829c P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:43.256551 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "542df00229f645788f1b821c0ce00c46" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:43.257426 711 leader_election.cc:343] T 542df00229f645788f1b821c0ce00c46 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:43.258036 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "542df00229f645788f1b821c0ce00c46" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
W20250113 02:51:43.258525 709 leader_election.cc:343] T 542df00229f645788f1b821c0ce00c46 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:43.258630 709 leader_election.cc:304] T 542df00229f645788f1b821c0ce00c46 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:43.259060 2588 raft_consensus.cc:2743] T 542df00229f645788f1b821c0ce00c46 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:43.259747 1192 tablet_bootstrap.cc:654] T fe8f075d4c9a4f9abe372bfdb81a829c P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:43.260685 454 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_f68a86b5448042cd898cfd456b3924a7" } new_table_name: "default.Table_5b1a111b11c84701a3c53a60069f114c" modify_external_catalogs: true
I20250113 02:51:43.263137 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1323) took 2224 ms. Trace:
I20250113 02:51:43.263301 1198 rpcz_store.cc:276] 0113 02:51:41.038558 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:41.038607 (+ 49us) service_pool.cc:225] Handling call
0113 02:51:43.221803 (+2183196us) ts_tablet_manager.cc:1145] Deleting tablet 438d1d2300e044f39a15586f335935ba
0113 02:51:43.221812 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:43.238441 (+ 16629us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:43.245819 (+ 7378us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:43.263118 (+ 17299us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2009022}
I20250113 02:51:43.263799 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1324) took 2164 ms. Trace:
I20250113 02:51:43.265041 1196 ts_tablet_manager.cc:1918] T 8337f3c82d654ecdac2cbc1ffa88ef05 P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250113 02:51:43.265174 1198 rpcz_store.cc:276] 0113 02:51:41.099091 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:41.099144 (+ 53us) service_pool.cc:225] Handling call
0113 02:51:43.263682 (+2164538us) ts_tablet_manager.cc:1145] Deleting tablet 01a455eda0ee488fb3d45a6a09c43833
0113 02:51:43.263692 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:43.263786 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2050855}
I20250113 02:51:43.265429 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet 438d1d2300e044f39a15586f335935ba (table default.Table_d7d1f5a34e5f4c05b474b0b61f912cd9 [id=2c7925ec0d8d4cd7b8761268d3962f15]) successfully deleted
I20250113 02:51:43.265609 1196 log.cc:1198] T 8337f3c82d654ecdac2cbc1ffa88ef05 P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/8337f3c82d654ecdac2cbc1ffa88ef05
W20250113 02:51:43.265841 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 01a455eda0ee488fb3d45a6a09c43833 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 01a455eda0ee488fb3d45a6a09c43833 already in progress: creating tablet
I20250113 02:51:43.266057 1198 tablet_replica.cc:331] T 82505f1389184d0ca12ec0e2d6084d22 P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:43.266253 1198 raft_consensus.cc:2238] T 82505f1389184d0ca12ec0e2d6084d22 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: Raft consensus shutting down.
I20250113 02:51:43.266738 1198 raft_consensus.cc:2267] T 82505f1389184d0ca12ec0e2d6084d22 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:43.269729 1198 ts_tablet_manager.cc:1905] T 82505f1389184d0ca12ec0e2d6084d22 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:43.269685 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_db479ab5faf8447d825ff655bfed0245"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:43.270351 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_db479ab5faf8447d825ff655bfed0245 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250113 02:51:43.271697 454 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5af53982fd3f4c0aa45cbde8a5ca2241 consensus state has no leader
I20250113 02:51:43.272192 1196 ts_tablet_manager.cc:1939] T 8337f3c82d654ecdac2cbc1ffa88ef05 P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:43.272743 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet 8337f3c82d654ecdac2cbc1ffa88ef05 (table default.Table_37d63a1b8622483a855edf2ce71c8d71 [id=885fa0a4645f4211bd874149829284c4]) successfully deleted
I20250113 02:51:43.273326 1196 tablet_replica.cc:331] T d9df314ffaf042ec80180868d7e7bdf7 P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:43.273562 1196 raft_consensus.cc:2238] T d9df314ffaf042ec80180868d7e7bdf7 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:43.273764 1196 raft_consensus.cc:2267] T d9df314ffaf042ec80180868d7e7bdf7 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:43.277408 1196 ts_tablet_manager.cc:1905] T d9df314ffaf042ec80180868d7e7bdf7 P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:43.278255 1192 tablet_bootstrap.cc:492] T fe8f075d4c9a4f9abe372bfdb81a829c P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:43.278429 1192 ts_tablet_manager.cc:1397] T fe8f075d4c9a4f9abe372bfdb81a829c P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.023s user 0.001s sys 0.002s
I20250113 02:51:43.279587 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_5627e75046d44ac98b829afa751f42d0" } new_table_name: "default.Table_ebc5fa342bb4429abe48397fab36abcd" modify_external_catalogs: true
I20250113 02:51:43.280884 1192 raft_consensus.cc:357] T fe8f075d4c9a4f9abe372bfdb81a829c P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:43.281145 1192 raft_consensus.cc:383] T fe8f075d4c9a4f9abe372bfdb81a829c P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:43.281236 1192 raft_consensus.cc:738] T fe8f075d4c9a4f9abe372bfdb81a829c P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:43.281471 1192 consensus_queue.cc:260] T fe8f075d4c9a4f9abe372bfdb81a829c P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:43.282002 1192 ts_tablet_manager.cc:1428] T fe8f075d4c9a4f9abe372bfdb81a829c P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250113 02:51:43.282146 2596 consensus_queue.cc:1035] T acd83024592e431898e9675c17f50d4f P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 02:51:43.282862 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_d9aa4ecc3d6143ce8ba0e6b1f5f87df0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:43.283537 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d9aa4ecc3d6143ce8ba0e6b1f5f87df0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:43.287556 451 master_service.cc:946] ReplaceTablet: received request to replace tablet d565b77f90784d87b64996772c938085 from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:43.288396 1192 tablet_bootstrap.cc:492] T fb2458e2044d49029ed2d6851db9de25 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:43.290613 1192 tablet_bootstrap.cc:654] T fb2458e2044d49029ed2d6851db9de25 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:43.296228 1032 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f8eb43a39774d2c8e9cb4e88a405b0f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:23 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:43.297040 1198 ts_tablet_manager.cc:1918] T 82505f1389184d0ca12ec0e2d6084d22 P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250113 02:51:43.297400 1198 log.cc:1198] T 82505f1389184d0ca12ec0e2d6084d22 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/82505f1389184d0ca12ec0e2d6084d22
W20250113 02:51:43.297866 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 37b7e05f7bbd4a4180cc40e76868469b consensus state has no leader
I20250113 02:51:43.298946 1192 tablet_bootstrap.cc:492] T fb2458e2044d49029ed2d6851db9de25 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:43.299149 1192 ts_tablet_manager.cc:1397] T fb2458e2044d49029ed2d6851db9de25 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.000s
I20250113 02:51:43.299942 449 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_cdfc4b389e7c4bd984d3dd2c3b33b1c8" } modify_external_catalogs: true
I20250113 02:51:43.300132 449 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_cdfc4b389e7c4bd984d3dd2c3b33b1c8" } modify_external_catalogs: true
I20250113 02:51:43.300134 1192 raft_consensus.cc:357] T fb2458e2044d49029ed2d6851db9de25 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:43.300299 1192 raft_consensus.cc:383] T fb2458e2044d49029ed2d6851db9de25 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:43.300339 1192 raft_consensus.cc:738] T fb2458e2044d49029ed2d6851db9de25 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:43.300531 1192 consensus_queue.cc:260] T fb2458e2044d49029ed2d6851db9de25 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:43.302260 1192 ts_tablet_manager.cc:1428] T fb2458e2044d49029ed2d6851db9de25 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250113 02:51:43.302946 1192 tablet_bootstrap.cc:492] T 4c759831773849f2bc52e484ffa0cf49 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:43.303077 1198 ts_tablet_manager.cc:1939] T 82505f1389184d0ca12ec0e2d6084d22 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:43.305989 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1326) took 2076 ms. Trace:
I20250113 02:51:43.306409 1032 tablet_service.cc:1514] Processing DeleteTablet for tablet fe8f075d4c9a4f9abe372bfdb81a829c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:36 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:43.306250 1198 rpcz_store.cc:276] 0113 02:51:41.229384 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:41.234653 (+ 5269us) service_pool.cc:225] Handling call
0113 02:51:43.265512 (+2030859us) ts_tablet_manager.cc:1145] Deleting tablet 82505f1389184d0ca12ec0e2d6084d22
0113 02:51:43.265994 (+ 482us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:43.274962 (+ 8968us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:43.296868 (+ 21906us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:43.305887 (+ 9019us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2816,"tablet-delete.queue_time_us":2030466}
I20250113 02:51:43.307823 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_b69c092ae6994928ad1eb5c10421a027"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:43.308135 1198 tablet_replica.cc:331] T 42dbd37e20fa47318bc6497423dbe565 P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:43.308398 1198 raft_consensus.cc:2238] T 42dbd37e20fa47318bc6497423dbe565 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: Raft consensus shutting down.
W20250113 02:51:43.308663 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b69c092ae6994928ad1eb5c10421a027 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:43.309532 1198 raft_consensus.cc:2267] T 42dbd37e20fa47318bc6497423dbe565 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:43.309615 1192 tablet_bootstrap.cc:654] T 4c759831773849f2bc52e484ffa0cf49 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:43.311697 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_37bcc8921e964a6391e0c8adde667af8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:43.312162 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_37bcc8921e964a6391e0c8adde667af8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:43.312471 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_f03ffbab220d4fb89b7c7baf4b03aa96" } modify_external_catalogs: true
I20250113 02:51:43.312747 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_f03ffbab220d4fb89b7c7baf4b03aa96" } modify_external_catalogs: true
I20250113 02:51:43.314512 1198 ts_tablet_manager.cc:1905] T 42dbd37e20fa47318bc6497423dbe565 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:43.316473 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet 82505f1389184d0ca12ec0e2d6084d22 (table default.Table_855f1069eae9472998eb347f20b003b8 [id=210a731382514eb098fb9442e6c290ba]) successfully deleted
I20250113 02:51:43.320911 748 tablet_service.cc:1467] Processing CreateTablet for tablet 9cedef8390f94111a6b3d8d229c75b27 (DEFAULT_TABLE table=default.Table_30951971e07445449dd042657deb91ec [id=38f42d9ea75340fdb8b32eff730a63e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:43.321326 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9cedef8390f94111a6b3d8d229c75b27. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:43.322093 1194 tablet_bootstrap.cc:492] T 655b63ff61ff4f4aa08e0449df39352a P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:43.322294 1194 ts_tablet_manager.cc:1397] T 655b63ff61ff4f4aa08e0449df39352a P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.167s user 0.020s sys 0.004s
I20250113 02:51:43.323362 1194 raft_consensus.cc:357] T 655b63ff61ff4f4aa08e0449df39352a P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:43.323643 1194 raft_consensus.cc:383] T 655b63ff61ff4f4aa08e0449df39352a P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:43.323748 1194 raft_consensus.cc:738] T 655b63ff61ff4f4aa08e0449df39352a P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:43.324165 1194 consensus_queue.cc:260] T 655b63ff61ff4f4aa08e0449df39352a P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:43.326526 1032 tablet_service.cc:1514] Processing DeleteTablet for tablet fb2458e2044d49029ed2d6851db9de25 with delete_type TABLET_DATA_DELETED (replaced by tablet 32e3dc0b7c044415a6343368ffa08552) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:43.327647 1191 tablet_bootstrap.cc:492] T b217cb8c09fb41cab4f53e605c37efe8 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:43.328784 1191 ts_tablet_manager.cc:1397] T b217cb8c09fb41cab4f53e605c37efe8 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.218s user 0.003s sys 0.000s
I20250113 02:51:43.329844 1191 raft_consensus.cc:357] T b217cb8c09fb41cab4f53e605c37efe8 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:43.330122 1191 raft_consensus.cc:383] T b217cb8c09fb41cab4f53e605c37efe8 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:43.330230 1191 raft_consensus.cc:738] T b217cb8c09fb41cab4f53e605c37efe8 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:43.330569 1191 consensus_queue.cc:260] T b217cb8c09fb41cab4f53e605c37efe8 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:43.331816 883 tablet_service.cc:1514] Processing DeleteTablet for tablet 655b63ff61ff4f4aa08e0449df39352a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:24 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:43.332401 1191 ts_tablet_manager.cc:1428] T b217cb8c09fb41cab4f53e605c37efe8 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250113 02:51:43.333782 1191 tablet_bootstrap.cc:492] T 569d666c128a45b7a526c13be5090ae3 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:43.334329 1192 tablet_bootstrap.cc:492] T 4c759831773849f2bc52e484ffa0cf49 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:43.334245 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_1ebb0d3cf33347f79b69a2951a6718cc" } new_table_name: "default.Table_f2056f5889be4891a565f2a9ddaa0a9c" modify_external_catalogs: true
I20250113 02:51:43.334398 2588 raft_consensus.cc:491] T 9b2d0aa6f0e9456d95f4e48c360ab6e8 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:43.334511 2588 raft_consensus.cc:513] T 9b2d0aa6f0e9456d95f4e48c360ab6e8 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:43.334571 1192 ts_tablet_manager.cc:1397] T 4c759831773849f2bc52e484ffa0cf49 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.032s user 0.007s sys 0.000s
W20250113 02:51:43.334791 439 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 8079e324ccb446609677074c8af773b7 (table default.Table_cda7b743f02745f3af60f2687a65bb81 [id=6a08527d21954bbcb446e8600758d626], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 8079e324ccb446609677074c8af773b7 consensus state has no leader
I20250113 02:51:43.335057 2588 leader_election.cc:290] T 9b2d0aa6f0e9456d95f4e48c360ab6e8 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:43.335819 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b2d0aa6f0e9456d95f4e48c360ab6e8" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:43.336073 1192 raft_consensus.cc:357] T 4c759831773849f2bc52e484ffa0cf49 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:43.336272 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b2d0aa6f0e9456d95f4e48c360ab6e8" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:43.336352 1192 raft_consensus.cc:383] T 4c759831773849f2bc52e484ffa0cf49 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250113 02:51:43.336357 709 leader_election.cc:343] T 9b2d0aa6f0e9456d95f4e48c360ab6e8 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:43.336458 1192 raft_consensus.cc:738] T 4c759831773849f2bc52e484ffa0cf49 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
W20250113 02:51:43.336808 711 leader_election.cc:343] T 9b2d0aa6f0e9456d95f4e48c360ab6e8 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:43.336911 711 leader_election.cc:304] T 9b2d0aa6f0e9456d95f4e48c360ab6e8 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:43.336820 1192 consensus_queue.cc:260] T 4c759831773849f2bc52e484ffa0cf49 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:43.337956 1191 tablet_bootstrap.cc:654] T 569d666c128a45b7a526c13be5090ae3 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:43.338944 1194 ts_tablet_manager.cc:1428] T 655b63ff61ff4f4aa08e0449df39352a P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.017s user 0.002s sys 0.000s
I20250113 02:51:43.340498 2588 raft_consensus.cc:2743] T 9b2d0aa6f0e9456d95f4e48c360ab6e8 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:43.341017 1194 tablet_bootstrap.cc:492] T b0113b16110e4787ac1c4676b2c8bd1e P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:43.343367 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1129) took 5473 ms. Trace:
I20250113 02:51:43.343458 1195 rpcz_store.cc:276] 0113 02:51:37.869693 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:37.869754 (+ 61us) service_pool.cc:225] Handling call
0113 02:51:42.976893 (+5107139us) ts_tablet_manager.cc:1145] Deleting tablet 1d2fdc75d00f4e749d322ac6cd5fc410
0113 02:51:42.976903 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:43.013548 (+ 36645us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:43.142351 (+128803us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:43.343348 (+200997us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":31987,"tablet-delete.queue_time_us":5106907}
I20250113 02:51:43.343878 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1131) took 5468 ms. Trace:
I20250113 02:51:43.343986 1195 rpcz_store.cc:276] 0113 02:51:37.875538 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:37.875581 (+ 43us) service_pool.cc:225] Handling call
0113 02:51:43.343738 (+5468157us) ts_tablet_manager.cc:1145] Deleting tablet b1cd18ee2e2145ba9273f500d468343d
0113 02:51:43.343750 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:43.343861 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5467999}
I20250113 02:51:43.344336 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1130) took 5468 ms. Trace:
I20250113 02:51:43.344404 1195 rpcz_store.cc:276] 0113 02:51:37.875398 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:37.875451 (+ 53us) service_pool.cc:225] Handling call
0113 02:51:43.344195 (+5468744us) ts_tablet_manager.cc:1145] Deleting tablet a8aedbc8d7bb4183bb1ab9902f07bea7
0113 02:51:43.344205 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:43.344322 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5468363}
I20250113 02:51:43.344430 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet 1d2fdc75d00f4e749d322ac6cd5fc410 (table default.Table_3fa4f9f57be84226a01935b139545f6e [id=0ab8e0c80dda42fcb07f2fd6fbf01609]) successfully deleted
I20250113 02:51:43.344686 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1132) took 5467 ms. Trace:
W20250113 02:51:43.344750 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet b1cd18ee2e2145ba9273f500d468343d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b1cd18ee2e2145ba9273f500d468343d already in progress: creating tablet
I20250113 02:51:43.344753 1195 rpcz_store.cc:276] 0113 02:51:37.877093 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:37.877143 (+ 50us) service_pool.cc:225] Handling call
0113 02:51:43.344592 (+5467449us) ts_tablet_manager.cc:1145] Deleting tablet de806919192f4691bf9ee088ec77963c
0113 02:51:43.344600 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:43.344673 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5467274}
I20250113 02:51:43.344981 1195 tablet_replica.cc:331] T 64aa8390b1b1475c8743b6491d40674c P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:43.345183 1195 raft_consensus.cc:2238] T 64aa8390b1b1475c8743b6491d40674c P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:43.345361 1195 raft_consensus.cc:2267] T 64aa8390b1b1475c8743b6491d40674c P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:43.347987 1032 tablet_service.cc:1467] Processing CreateTablet for tablet 9cedef8390f94111a6b3d8d229c75b27 (DEFAULT_TABLE table=default.Table_30951971e07445449dd042657deb91ec [id=38f42d9ea75340fdb8b32eff730a63e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:43.348379 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9cedef8390f94111a6b3d8d229c75b27. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:43.348971 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet a8aedbc8d7bb4183bb1ab9902f07bea7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a8aedbc8d7bb4183bb1ab9902f07bea7 already in progress: creating tablet
W20250113 02:51:43.349152 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet de806919192f4691bf9ee088ec77963c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet de806919192f4691bf9ee088ec77963c already in progress: creating tablet
I20250113 02:51:43.352072 748 tablet_service.cc:1514] Processing DeleteTablet for tablet 542df00229f645788f1b821c0ce00c46 with delete_type TABLET_DATA_DELETED (replaced by tablet 2460e7cddcd84788a54dab342e7bb27b) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:43.354223 748 tablet_service.cc:1467] Processing CreateTablet for tablet eaf46d7cc6b04126b349fd7c2d3fdb7a (DEFAULT_TABLE table=default.Table_51c45c1994534c5593ec0872e3f10a9d [id=2ab212716782456ea7bdc0f0dd4af4e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:43.354601 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eaf46d7cc6b04126b349fd7c2d3fdb7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:43.355552 736 tablet_service.cc:1467] Processing CreateTablet for tablet 6e6c2c9a4c5e4d308b4db7ccf7dd4fd7 (DEFAULT_TABLE table=default.Table_cba6acff6d514b32be48c6da57f28ab4 [id=b9352633c7f94d47b55856e001ca1f1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:43.355926 736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e6c2c9a4c5e4d308b4db7ccf7dd4fd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:43.357321 883 tablet_service.cc:1467] Processing CreateTablet for tablet 6e6c2c9a4c5e4d308b4db7ccf7dd4fd7 (DEFAULT_TABLE table=default.Table_cba6acff6d514b32be48c6da57f28ab4 [id=b9352633c7f94d47b55856e001ca1f1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:43.357848 891 tablet_service.cc:1467] Processing CreateTablet for tablet 8f46d38f2c634b7d87df37e59ef4f5ef (DEFAULT_TABLE table=default.Table_d65b7b94ce864ac6a57e6b151433a14d [id=2df41304dc934077a63eda2694048d0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:43.360553 751 tablet_service.cc:1467] Processing CreateTablet for tablet 8f46d38f2c634b7d87df37e59ef4f5ef (DEFAULT_TABLE table=default.Table_d65b7b94ce864ac6a57e6b151433a14d [id=2df41304dc934077a63eda2694048d0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:43.360862 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f46d38f2c634b7d87df37e59ef4f5ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:43.363382 1192 ts_tablet_manager.cc:1428] T 4c759831773849f2bc52e484ffa0cf49 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.028s user 0.002s sys 0.000s
I20250113 02:51:43.367449 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f46d38f2c634b7d87df37e59ef4f5ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:43.371491 913 raft_consensus.cc:3054] T 44d2c7d5334c476fac60fdd57e2c02b6 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:43.372190 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e6c2c9a4c5e4d308b4db7ccf7dd4fd7. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:43.375623 998 consensus_peers.cc:487] T acd83024592e431898e9675c17f50d4f P 4216a7f853d642b7bc3e5cff458f3aca -> Peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Couldn't send request to peer df69f48656b44bb890f76587ee69cc0f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250113 02:51:43.376792 1196 ts_tablet_manager.cc:1918] T d9df314ffaf042ec80180868d7e7bdf7 P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:43.376978 1196 log.cc:1198] T d9df314ffaf042ec80180868d7e7bdf7 P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/d9df314ffaf042ec80180868d7e7bdf7
I20250113 02:51:43.377440 1196 ts_tablet_manager.cc:1939] T d9df314ffaf042ec80180868d7e7bdf7 P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:43.377948 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet d9df314ffaf042ec80180868d7e7bdf7 (table default.Table_2155bf71d1a04a93a28b13c97f636fab [id=00107bce6ce24feba58176198a222970]) successfully deleted
I20250113 02:51:43.378412 1192 tablet_bootstrap.cc:492] T ca75e7cae4f6423191a17bb5a8f150a9 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:43.380406 751 tablet_service.cc:1514] Processing DeleteTablet for tablet b217cb8c09fb41cab4f53e605c37efe8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:26 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:43.381322 448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_f8a6812c7fb94ea69cd9bf775b914ea6" } new_table_name: "default.Table_0ae4f97208974a97b60a2691c78ef43e" modify_external_catalogs: true
I20250113 02:51:43.387375 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_4c6fa7b378f94acea4082e8a5ad20579"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:43.387955 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_fb3b54ceea934445932e1192983f3e27"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:43.387461 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_77c140d8e2c24ad8a08b0c2ed713ba02" } new_table_name: "default.Table_23360f89a14b4e009d2bc211ec4cd735" modify_external_catalogs: true
W20250113 02:51:43.388394 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fb3b54ceea934445932e1192983f3e27 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:43.387714 455 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_74bd8fe399954ec4a53885440efbfe23" } modify_external_catalogs: true
I20250113 02:51:43.389487 455 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_74bd8fe399954ec4a53885440efbfe23" } modify_external_catalogs: true
W20250113 02:51:43.388394 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4c6fa7b378f94acea4082e8a5ad20579 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:43.392633 1039 tablet_service.cc:1467] Processing CreateTablet for tablet 6e6c2c9a4c5e4d308b4db7ccf7dd4fd7 (DEFAULT_TABLE table=default.Table_cba6acff6d514b32be48c6da57f28ab4 [id=b9352633c7f94d47b55856e001ca1f1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:43.393028 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e6c2c9a4c5e4d308b4db7ccf7dd4fd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:43.394093 1194 tablet_bootstrap.cc:654] T b0113b16110e4787ac1c4676b2c8bd1e P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:43.394948 885 tablet_service.cc:1467] Processing CreateTablet for tablet eaf46d7cc6b04126b349fd7c2d3fdb7a (DEFAULT_TABLE table=default.Table_51c45c1994534c5593ec0872e3f10a9d [id=2ab212716782456ea7bdc0f0dd4af4e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:43.395293 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eaf46d7cc6b04126b349fd7c2d3fdb7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:43.396201 1035 tablet_service.cc:1467] Processing CreateTablet for tablet 8f46d38f2c634b7d87df37e59ef4f5ef (DEFAULT_TABLE table=default.Table_d65b7b94ce864ac6a57e6b151433a14d [id=2df41304dc934077a63eda2694048d0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:43.396510 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f46d38f2c634b7d87df37e59ef4f5ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:43.401796 773 raft_consensus.cc:1270] T 3f62f344c5a648328ca2de8ca1069c19 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Refusing update from remote peer 4216a7f853d642b7bc3e5cff458f3aca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20250113 02:51:43.402521 998 consensus_peers.cc:487] T 0b119704bc774ca9b4155c9619d4765b P 4216a7f853d642b7bc3e5cff458f3aca -> Peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Couldn't send request to peer df69f48656b44bb890f76587ee69cc0f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0b119704bc774ca9b4155c9619d4765b. This is attempt 1: this message will repeat every 5th retry.
W20250113 02:51:43.402714 998 consensus_peers.cc:487] T 3f62f344c5a648328ca2de8ca1069c19 P 4216a7f853d642b7bc3e5cff458f3aca -> Peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Couldn't send request to peer df69f48656b44bb890f76587ee69cc0f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250113 02:51:43.403738 2363 consensus_queue.cc:1035] T 3f62f344c5a648328ca2de8ca1069c19 P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 02:51:43.406868 1032 tablet_service.cc:1467] Processing CreateTablet for tablet eaf46d7cc6b04126b349fd7c2d3fdb7a (DEFAULT_TABLE table=default.Table_51c45c1994534c5593ec0872e3f10a9d [id=2ab212716782456ea7bdc0f0dd4af4e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:43.407367 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eaf46d7cc6b04126b349fd7c2d3fdb7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:43.412292 1196 tablet_replica.cc:331] T f1e32e09a8244957a99bdfea5812ff31 P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:43.413817 1192 tablet_bootstrap.cc:654] T ca75e7cae4f6423191a17bb5a8f150a9 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:43.414793 1196 raft_consensus.cc:2238] T f1e32e09a8244957a99bdfea5812ff31 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
W20250113 02:51:43.415040 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 1c3c3b7cf89e4f22b8a5dd7ba4f8f5ec because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1c3c3b7cf89e4f22b8a5dd7ba4f8f5ec already in progress: creating tablet
W20250113 02:51:43.415275 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet d1f30873a02f4b61ac6940e44c332ad3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d1f30873a02f4b61ac6940e44c332ad3 already in progress: creating tablet
W20250113 02:51:43.415486 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet f86318637d5b4e378ce47e89a5d52b07 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f86318637d5b4e378ce47e89a5d52b07 already in progress: creating tablet
I20250113 02:51:43.418423 1196 raft_consensus.cc:2267] T f1e32e09a8244957a99bdfea5812ff31 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:43.419432 1196 ts_tablet_manager.cc:1905] T f1e32e09a8244957a99bdfea5812ff31 P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:43.384168 1198 ts_tablet_manager.cc:1918] T 42dbd37e20fa47318bc6497423dbe565 P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250113 02:51:43.421675 1198 log.cc:1198] T 42dbd37e20fa47318bc6497423dbe565 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/42dbd37e20fa47318bc6497423dbe565
I20250113 02:51:43.422150 1198 ts_tablet_manager.cc:1939] T 42dbd37e20fa47318bc6497423dbe565 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:43.422171 454 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_babc6b8af3474518b339d149d6a2ccb9" } modify_external_catalogs: true
I20250113 02:51:43.422382 454 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_babc6b8af3474518b339d149d6a2ccb9" } modify_external_catalogs: true
I20250113 02:51:43.422912 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1327) took 2179 ms. Trace:
I20250113 02:51:43.422979 1198 rpcz_store.cc:276] 0113 02:51:41.243560 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:41.247832 (+ 4272us) service_pool.cc:225] Handling call
0113 02:51:43.307673 (+2059841us) ts_tablet_manager.cc:1145] Deleting tablet 42dbd37e20fa47318bc6497423dbe565
0113 02:51:43.308077 (+ 404us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:43.325168 (+ 17091us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:43.384082 (+ 58914us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:43.422895 (+ 38813us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2053970}
I20250113 02:51:43.425344 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 77036be46b574f73a02ca656bf5cc0a3
I20250113 02:51:43.426225 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_27443429037f40faa2bad5d4eff2a571"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:43.426867 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_27443429037f40faa2bad5d4eff2a571 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:43.427485 1198 tablet_replica.cc:331] T ba8980ad3a364c888a825affbcbc9b1f P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:43.427664 1198 raft_consensus.cc:2238] T ba8980ad3a364c888a825affbcbc9b1f P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:43.427788 1198 raft_consensus.cc:2267] T ba8980ad3a364c888a825affbcbc9b1f P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:43.428303 1198 ts_tablet_manager.cc:1905] T ba8980ad3a364c888a825affbcbc9b1f P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:43.429987 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_2452909212ee4e0e9d228074c38e6949"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:43.430553 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2452909212ee4e0e9d228074c38e6949 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:43.433418 883 tablet_service.cc:1514] Processing DeleteTablet for tablet 77036be46b574f73a02ca656bf5cc0a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:43 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:43.433517 452 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet d464e98e11ba4aaca358381cae65580c
I20250113 02:51:43.433863 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_e85020a361f343dab7ddc0e6b11037ce"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:43.434927 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e85020a361f343dab7ddc0e6b11037ce in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:43.438782 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet 42dbd37e20fa47318bc6497423dbe565 (table default.Table_37b52d3cfe0f4b0fb1ef1468117512dd [id=45aa63632c914a4e99eb69759b56c842]) successfully deleted
I20250113 02:51:43.439395 1195 ts_tablet_manager.cc:1905] T 64aa8390b1b1475c8743b6491d40674c P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:43.441826 1032 tablet_service.cc:1514] Processing DeleteTablet for tablet 77036be46b574f73a02ca656bf5cc0a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:43 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:43.442510 2588 raft_consensus.cc:491] T 247af1a17df04b27898f86642336d51e P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:43.442628 2588 raft_consensus.cc:513] T 247af1a17df04b27898f86642336d51e P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:43.443194 2588 leader_election.cc:290] T 247af1a17df04b27898f86642336d51e P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:43.443848 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "247af1a17df04b27898f86642336d51e" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:43.444423 711 leader_election.cc:343] T 247af1a17df04b27898f86642336d51e P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:43.444855 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "247af1a17df04b27898f86642336d51e" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
W20250113 02:51:43.445394 709 leader_election.cc:343] T 247af1a17df04b27898f86642336d51e P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:43.445505 709 leader_election.cc:304] T 247af1a17df04b27898f86642336d51e P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:43.445914 2588 raft_consensus.cc:2743] T 247af1a17df04b27898f86642336d51e P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:43.449667 448 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_71699c9b0b1a46cda2db76dee82b90b4" } modify_external_catalogs: true
I20250113 02:51:43.449831 448 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_71699c9b0b1a46cda2db76dee82b90b4" } modify_external_catalogs: true
I20250113 02:51:43.450390 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 559c2f8ea91d4e81bf303b3a0c61c143
I20250113 02:51:43.451274 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 1c82cf9373d24fedad88b5df7d5a8b64
W20250113 02:51:43.451726 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 448d3daf897247f8bd53ffbcdc16a22c consensus state has no leader
W20250113 02:51:43.452047 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ccc24d92091a4643a66959d1f8ddb865 consensus state has no leader
I20250113 02:51:43.452437 2474 raft_consensus.cc:491] T e570465a3d4046b1ac19a3a034fe91f1 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:43.452570 2474 raft_consensus.cc:513] T e570465a3d4046b1ac19a3a034fe91f1 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:43.453207 1192 tablet_bootstrap.cc:492] T ca75e7cae4f6423191a17bb5a8f150a9 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:43.453343 1192 ts_tablet_manager.cc:1397] T ca75e7cae4f6423191a17bb5a8f150a9 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.087s user 0.003s sys 0.000s
I20250113 02:51:43.453512 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e570465a3d4046b1ac19a3a034fe91f1" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
I20250113 02:51:43.453578 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet 559c2f8ea91d4e81bf303b3a0c61c143 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:43 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:43.453686 769 raft_consensus.cc:2463] T e570465a3d4046b1ac19a3a034fe91f1 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 0.
I20250113 02:51:43.453766 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c759831773849f2bc52e484ffa0cf49 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:30 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:43.454293 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c82cf9373d24fedad88b5df7d5a8b64 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:43 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:43.455104 995 leader_election.cc:304] T e570465a3d4046b1ac19a3a034fe91f1 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters:
I20250113 02:51:43.455358 2596 raft_consensus.cc:2798] T e570465a3d4046b1ac19a3a034fe91f1 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 02:51:43.455430 2596 raft_consensus.cc:491] T e570465a3d4046b1ac19a3a034fe91f1 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 02:51:43.455494 2596 raft_consensus.cc:3054] T e570465a3d4046b1ac19a3a034fe91f1 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:43.457295 883 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c82cf9373d24fedad88b5df7d5a8b64 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:43 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:43.458029 2596 raft_consensus.cc:513] T e570465a3d4046b1ac19a3a034fe91f1 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:43.458083 1192 raft_consensus.cc:357] T ca75e7cae4f6423191a17bb5a8f150a9 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:43.458211 1192 raft_consensus.cc:383] T ca75e7cae4f6423191a17bb5a8f150a9 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:43.458253 1192 raft_consensus.cc:738] T ca75e7cae4f6423191a17bb5a8f150a9 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:43.458439 1192 consensus_queue.cc:260] T ca75e7cae4f6423191a17bb5a8f150a9 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:43.458577 2596 leader_election.cc:290] T e570465a3d4046b1ac19a3a034fe91f1 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Requested vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:43.458938 1192 ts_tablet_manager.cc:1428] T ca75e7cae4f6423191a17bb5a8f150a9 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20250113 02:51:43.460003 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e570465a3d4046b1ac19a3a034fe91f1" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196"
I20250113 02:51:43.460165 769 raft_consensus.cc:3054] T e570465a3d4046b1ac19a3a034fe91f1 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:43.462563 883 tablet_service.cc:1467] Processing CreateTablet for tablet 9572215fd2eb45faa99cc236fcc0d5ea (DEFAULT_TABLE table=default.Table_bbd8e889c0eb420fbda882e52e6beb0b [id=0277bf104f954c84b1dfb439d37da12b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:43.463024 769 raft_consensus.cc:2463] T e570465a3d4046b1ac19a3a034fe91f1 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 1.
I20250113 02:51:43.464833 995 leader_election.cc:304] T e570465a3d4046b1ac19a3a034fe91f1 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters:
I20250113 02:51:43.465085 2547 raft_consensus.cc:2798] T e570465a3d4046b1ac19a3a034fe91f1 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Leader election won for term 1
I20250113 02:51:43.465188 2547 raft_consensus.cc:695] T e570465a3d4046b1ac19a3a034fe91f1 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: Becoming Leader. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Running, Role: LEADER
I20250113 02:51:43.465279 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_33d141799a774a28887f4c91cbac6ad6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:43.465406 2547 consensus_queue.cc:237] T e570465a3d4046b1ac19a3a034fe91f1 P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
W20250113 02:51:43.465792 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_33d141799a774a28887f4c91cbac6ad6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:43.466421 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet e312e70334e04ecd843159ac72f4a2a6
I20250113 02:51:43.466862 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e570465a3d4046b1ac19a3a034fe91f1" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f"
I20250113 02:51:43.467756 882 tablet_service.cc:1514] Processing DeleteTablet for tablet e312e70334e04ecd843159ac72f4a2a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:43 UTC) from {username='slave'} at 127.0.0.1:37366
W20250113 02:51:43.468084 998 leader_election.cc:343] T e570465a3d4046b1ac19a3a034fe91f1 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:43.468466 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet e312e70334e04ecd843159ac72f4a2a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:43 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:43.469018 1032 tablet_service.cc:1467] Processing CreateTablet for tablet 9572215fd2eb45faa99cc236fcc0d5ea (DEFAULT_TABLE table=default.Table_bbd8e889c0eb420fbda882e52e6beb0b [id=0277bf104f954c84b1dfb439d37da12b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:43.469362 2368 raft_consensus.cc:491] T 283fc7f1c74e480c8a63c74d3ae9eb7d P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:43.469941 2368 raft_consensus.cc:513] T 283fc7f1c74e480c8a63c74d3ae9eb7d P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:43.470392 891 tablet_service.cc:1514] Processing DeleteTablet for tablet 559c2f8ea91d4e81bf303b3a0c61c143 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:43 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:43.470482 2368 leader_election.cc:290] T 283fc7f1c74e480c8a63c74d3ae9eb7d P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:43.471159 1194 tablet_bootstrap.cc:492] T b0113b16110e4787ac1c4676b2c8bd1e P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:43.471247 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "283fc7f1c74e480c8a63c74d3ae9eb7d" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:43.471757 998 leader_election.cc:343] T 283fc7f1c74e480c8a63c74d3ae9eb7d P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:43.469605 1192 tablet_bootstrap.cc:492] T f1237f7b04af4ebbb39b03aa3c78a499 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:43.471558 1194 ts_tablet_manager.cc:1397] T b0113b16110e4787ac1c4676b2c8bd1e P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.131s user 0.002s sys 0.001s
I20250113 02:51:43.473258 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9572215fd2eb45faa99cc236fcc0d5ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:43.473162 448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_7cdc614381e047e6a44f1c316214d076" } new_table_name: "default.Table_9dbbf836e760408cbd6abef0d6887f8b" modify_external_catalogs: true
I20250113 02:51:43.473284 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "283fc7f1c74e480c8a63c74d3ae9eb7d" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
I20250113 02:51:43.473474 769 raft_consensus.cc:2463] T 283fc7f1c74e480c8a63c74d3ae9eb7d P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 0.
I20250113 02:51:43.473901 995 leader_election.cc:304] T 283fc7f1c74e480c8a63c74d3ae9eb7d P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters: df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:43.474309 2596 raft_consensus.cc:2798] T 283fc7f1c74e480c8a63c74d3ae9eb7d P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 02:51:43.474385 2596 raft_consensus.cc:491] T 283fc7f1c74e480c8a63c74d3ae9eb7d P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 02:51:43.474442 2596 raft_consensus.cc:3054] T 283fc7f1c74e480c8a63c74d3ae9eb7d P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:43.476843 2596 raft_consensus.cc:513] T 283fc7f1c74e480c8a63c74d3ae9eb7d P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:43.477514 2596 leader_election.cc:290] T 283fc7f1c74e480c8a63c74d3ae9eb7d P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Requested vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:43.477919 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "283fc7f1c74e480c8a63c74d3ae9eb7d" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f"
I20250113 02:51:43.478302 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "283fc7f1c74e480c8a63c74d3ae9eb7d" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196"
W20250113 02:51:43.478467 998 leader_election.cc:343] T 283fc7f1c74e480c8a63c74d3ae9eb7d P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:43.478520 769 raft_consensus.cc:3054] T 283fc7f1c74e480c8a63c74d3ae9eb7d P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:43.479703 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9572215fd2eb45faa99cc236fcc0d5ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:43.479738 2596 raft_consensus.cc:491] T 21bcdd741104474bad7023426856d18f P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:43.479840 2596 raft_consensus.cc:513] T 21bcdd741104474bad7023426856d18f P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:43.480648 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21bcdd741104474bad7023426856d18f" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
I20250113 02:51:43.480671 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21bcdd741104474bad7023426856d18f" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:43.480818 772 raft_consensus.cc:2463] T 21bcdd741104474bad7023426856d18f P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 0.
W20250113 02:51:43.481159 998 leader_election.cc:343] T 21bcdd741104474bad7023426856d18f P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:43.481309 2596 leader_election.cc:290] T 21bcdd741104474bad7023426856d18f P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:43.481283 995 leader_election.cc:304] T 21bcdd741104474bad7023426856d18f P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters: df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:43.482177 1192 tablet_bootstrap.cc:654] T f1237f7b04af4ebbb39b03aa3c78a499 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:43.483232 769 raft_consensus.cc:2463] T 283fc7f1c74e480c8a63c74d3ae9eb7d P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 1.
I20250113 02:51:43.483572 2474 leader_election.cc:290] T e570465a3d4046b1ac19a3a034fe91f1 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:43.484105 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e570465a3d4046b1ac19a3a034fe91f1" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:43.484335 2474 raft_consensus.cc:2798] T 21bcdd741104474bad7023426856d18f P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 02:51:43.484542 2474 raft_consensus.cc:491] T 21bcdd741104474bad7023426856d18f P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 02:51:43.484758 2474 raft_consensus.cc:3054] T 21bcdd741104474bad7023426856d18f P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:43.485350 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_3d4ca369c9434e42b3455bd2f325eff9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:43.485942 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3d4ca369c9434e42b3455bd2f325eff9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250113 02:51:43.484637 998 leader_election.cc:343] T e570465a3d4046b1ac19a3a034fe91f1 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:43.488965 2474 raft_consensus.cc:513] T 21bcdd741104474bad7023426856d18f P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
W20250113 02:51:43.486132 711 consensus_peers.cc:487] T 938cc1ccd8dd449996836188ba9e64ff P 1186fa869eb14812832fa6562531f196 -> Peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Couldn't send request to peer df69f48656b44bb890f76587ee69cc0f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250113 02:51:43.489928 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21bcdd741104474bad7023426856d18f" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196"
I20250113 02:51:43.490762 772 raft_consensus.cc:3054] T 21bcdd741104474bad7023426856d18f P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:43.493381 772 raft_consensus.cc:2463] T 21bcdd741104474bad7023426856d18f P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 1.
W20250113 02:51:43.493450 448 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f59dc0fa43dd4eb5932a1ad8a5a08349 consensus state has no leader
I20250113 02:51:43.493813 995 leader_election.cc:304] T 21bcdd741104474bad7023426856d18f P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters:
I20250113 02:51:43.494109 2368 raft_consensus.cc:2798] T 21bcdd741104474bad7023426856d18f P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Leader election won for term 1
I20250113 02:51:43.494215 2368 raft_consensus.cc:695] T 21bcdd741104474bad7023426856d18f P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: Becoming Leader. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Running, Role: LEADER
I20250113 02:51:43.494444 2368 consensus_queue.cc:237] T 21bcdd741104474bad7023426856d18f P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:43.494886 1035 tablet_service.cc:1467] Processing CreateTablet for tablet 4310db82a66c4fe0963320dd6a84bbb2 (DEFAULT_TABLE table=default.Table_40a3924489b2446697bd846b8ab9d284 [id=c74125acebc74d9c9809bc30c7f49cb5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:43.495244 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4310db82a66c4fe0963320dd6a84bbb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:43.500541 891 tablet_service.cc:1467] Processing CreateTablet for tablet 7de409bf5dda4de9a488f7b4ce0deba7 (DEFAULT_TABLE table=default.Table_48b46990944b44dfae31af6b82c145e3 [id=9f434a2091d1418c8daeae5430b33653]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:43.501004 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7de409bf5dda4de9a488f7b4ce0deba7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:43.502061 995 leader_election.cc:304] T 283fc7f1c74e480c8a63c74d3ae9eb7d P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters: df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:43.502430 1194 raft_consensus.cc:357] T b0113b16110e4787ac1c4676b2c8bd1e P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:43.505328 448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_a2a4fd46d33f4ea6ae4f3a7883f4ff1c" } new_table_name: "default.Table_2d5dc09e2aad46f3abdd90ea9e4a6703" modify_external_catalogs: true
I20250113 02:51:43.506982 739 tablet_service.cc:1514] Processing DeleteTablet for tablet 559c2f8ea91d4e81bf303b3a0c61c143 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:43 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:43.507172 736 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c82cf9373d24fedad88b5df7d5a8b64 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:43 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:43.509176 2596 raft_consensus.cc:2798] T 283fc7f1c74e480c8a63c74d3ae9eb7d P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Leader election won for term 1
I20250113 02:51:43.509362 2596 raft_consensus.cc:695] T 283fc7f1c74e480c8a63c74d3ae9eb7d P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: Becoming Leader. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Running, Role: LEADER
I20250113 02:51:43.509670 2596 consensus_queue.cc:237] T 283fc7f1c74e480c8a63c74d3ae9eb7d P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:43.512367 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_51d4d166147f48b9b374db8c8a89e368"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:43.513052 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_51d4d166147f48b9b374db8c8a89e368 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:43.514664 736 tablet_service.cc:1514] Processing DeleteTablet for tablet e312e70334e04ecd843159ac72f4a2a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:43 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:43.515429 736 tablet_service.cc:1467] Processing CreateTablet for tablet 9572215fd2eb45faa99cc236fcc0d5ea (DEFAULT_TABLE table=default.Table_bbd8e889c0eb420fbda882e52e6beb0b [id=0277bf104f954c84b1dfb439d37da12b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:43.515836 736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9572215fd2eb45faa99cc236fcc0d5ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:43.516098 751 tablet_service.cc:1514] Processing DeleteTablet for tablet 77036be46b574f73a02ca656bf5cc0a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:43 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:43.516443 748 tablet_service.cc:1467] Processing CreateTablet for tablet 7de409bf5dda4de9a488f7b4ce0deba7 (DEFAULT_TABLE table=default.Table_48b46990944b44dfae31af6b82c145e3 [id=9f434a2091d1418c8daeae5430b33653]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:43.516765 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7de409bf5dda4de9a488f7b4ce0deba7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:43.520268 739 tablet_service.cc:1467] Processing CreateTablet for tablet 4310db82a66c4fe0963320dd6a84bbb2 (DEFAULT_TABLE table=default.Table_40a3924489b2446697bd846b8ab9d284 [id=c74125acebc74d9c9809bc30c7f49cb5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:43.520413 1191 tablet_bootstrap.cc:492] T 569d666c128a45b7a526c13be5090ae3 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:43.520524 1191 ts_tablet_manager.cc:1397] T 569d666c128a45b7a526c13be5090ae3 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.187s user 0.003s sys 0.015s
I20250113 02:51:43.520671 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4310db82a66c4fe0963320dd6a84bbb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:43.521175 1191 raft_consensus.cc:357] T 569d666c128a45b7a526c13be5090ae3 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:43.521354 1191 raft_consensus.cc:383] T 569d666c128a45b7a526c13be5090ae3 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:43.521400 1191 raft_consensus.cc:738] T 569d666c128a45b7a526c13be5090ae3 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:43.521574 1191 consensus_queue.cc:260] T 569d666c128a45b7a526c13be5090ae3 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:43.524175 1194 raft_consensus.cc:383] T b0113b16110e4787ac1c4676b2c8bd1e P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:43.524274 1194 raft_consensus.cc:738] T b0113b16110e4787ac1c4676b2c8bd1e P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:43.524477 1194 consensus_queue.cc:260] T b0113b16110e4787ac1c4676b2c8bd1e P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:43.525816 1035 tablet_service.cc:1467] Processing CreateTablet for tablet 7de409bf5dda4de9a488f7b4ce0deba7 (DEFAULT_TABLE table=default.Table_48b46990944b44dfae31af6b82c145e3 [id=9f434a2091d1418c8daeae5430b33653]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:43.528401 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7de409bf5dda4de9a488f7b4ce0deba7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:43.531854 1194 ts_tablet_manager.cc:1428] T b0113b16110e4787ac1c4676b2c8bd1e P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.059s user 0.001s sys 0.001s
I20250113 02:51:43.535442 1195 ts_tablet_manager.cc:1918] T 64aa8390b1b1475c8743b6491d40674c P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250113 02:51:43.536005 1194 tablet_bootstrap.cc:492] T c518f791b4d74c54ab0c0364d29b4a64 P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:43.540158 2474 leader_election.cc:290] T 21bcdd741104474bad7023426856d18f P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Requested vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:43.541090 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21bcdd741104474bad7023426856d18f" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f"
W20250113 02:51:43.541638 998 leader_election.cc:343] T 21bcdd741104474bad7023426856d18f P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:43.541891 1194 tablet_bootstrap.cc:654] T c518f791b4d74c54ab0c0364d29b4a64 P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:43.542922 883 tablet_service.cc:1467] Processing CreateTablet for tablet 4310db82a66c4fe0963320dd6a84bbb2 (DEFAULT_TABLE table=default.Table_40a3924489b2446697bd846b8ab9d284 [id=c74125acebc74d9c9809bc30c7f49cb5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:43.547137 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4310db82a66c4fe0963320dd6a84bbb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:43.547377 1192 tablet_bootstrap.cc:492] T f1237f7b04af4ebbb39b03aa3c78a499 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:43.552346 1192 ts_tablet_manager.cc:1397] T f1237f7b04af4ebbb39b03aa3c78a499 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.083s user 0.005s sys 0.000s
I20250113 02:51:43.545996 1196 ts_tablet_manager.cc:1918] T f1e32e09a8244957a99bdfea5812ff31 P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:43.553159 1196 log.cc:1198] T f1e32e09a8244957a99bdfea5812ff31 P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/f1e32e09a8244957a99bdfea5812ff31
I20250113 02:51:43.553579 1196 ts_tablet_manager.cc:1939] T f1e32e09a8244957a99bdfea5812ff31 P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:43.550472 2632 tablet.cc:1721] T 3f62f344c5a648328ca2de8ca1069c19 P 4216a7f853d642b7bc3e5cff458f3aca: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250113 02:51:43.556398 1198 ts_tablet_manager.cc:1918] T ba8980ad3a364c888a825affbcbc9b1f P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:43.559134 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet f1e32e09a8244957a99bdfea5812ff31 (table default.Table_12a092a4543e4b04a466945f4b694c53 [id=c84b256e3b5744edbbf4de12eeaff0a3]) successfully deleted
W20250113 02:51:43.560359 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet ef078624960843eead2edd1e39b10118 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ef078624960843eead2edd1e39b10118 already in progress: creating tablet
I20250113 02:51:43.539696 1191 ts_tablet_manager.cc:1428] T 569d666c128a45b7a526c13be5090ae3 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.019s user 0.002s sys 0.000s
I20250113 02:51:43.563481 2636 tablet.cc:1721] T 44d2c7d5334c476fac60fdd57e2c02b6 P df69f48656b44bb890f76587ee69cc0f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250113 02:51:43.563752 1191 tablet_bootstrap.cc:492] T 88a776aa820c44b3a3854ca5022bd353 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:43.539863 2588 raft_consensus.cc:491] T fb2458e2044d49029ed2d6851db9de25 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:43.564077 2588 raft_consensus.cc:513] T fb2458e2044d49029ed2d6851db9de25 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:43.564718 2588 leader_election.cc:290] T fb2458e2044d49029ed2d6851db9de25 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:43.542387 2501 raft_consensus.cc:491] T b217cb8c09fb41cab4f53e605c37efe8 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:43.565058 2501 raft_consensus.cc:513] T b217cb8c09fb41cab4f53e605c37efe8 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:43.565568 2501 leader_election.cc:290] T b217cb8c09fb41cab4f53e605c37efe8 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:43.566097 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb2458e2044d49029ed2d6851db9de25" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:43.568603 1191 tablet_bootstrap.cc:654] T 88a776aa820c44b3a3854ca5022bd353 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:43.568562 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b217cb8c09fb41cab4f53e605c37efe8" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:43.569532 711 leader_election.cc:343] T fb2458e2044d49029ed2d6851db9de25 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
W20250113 02:51:43.551724 995 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:40095, user_credentials={real_user=slave}} blocked reactor thread for 49685.7us
W20250113 02:51:43.569748 711 leader_election.cc:343] T b217cb8c09fb41cab4f53e605c37efe8 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:43.562289 2633 tablet.cc:1721] T 3f62f344c5a648328ca2de8ca1069c19 P 1186fa869eb14812832fa6562531f196: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250113 02:51:43.577594 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb2458e2044d49029ed2d6851db9de25" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:43.577818 1059 raft_consensus.cc:2463] T fb2458e2044d49029ed2d6851db9de25 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1186fa869eb14812832fa6562531f196 in term 0.
I20250113 02:51:43.578541 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b217cb8c09fb41cab4f53e605c37efe8" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:43.583595 709 leader_election.cc:304] T fb2458e2044d49029ed2d6851db9de25 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters: df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:43.592200 1198 log.cc:1198] T ba8980ad3a364c888a825affbcbc9b1f P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/ba8980ad3a364c888a825affbcbc9b1f
W20250113 02:51:43.591848 709 leader_election.cc:343] T b217cb8c09fb41cab4f53e605c37efe8 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:43.592576 1198 ts_tablet_manager.cc:1939] T ba8980ad3a364c888a825affbcbc9b1f P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:43.593351 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1345) took 2102 ms. Trace:
I20250113 02:51:43.592586 709 leader_election.cc:304] T b217cb8c09fb41cab4f53e605c37efe8 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:43.593439 1198 rpcz_store.cc:276] 0113 02:51:41.490997 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:41.491055 (+ 58us) service_pool.cc:225] Handling call
0113 02:51:43.427434 (+1936379us) ts_tablet_manager.cc:1145] Deleting tablet ba8980ad3a364c888a825affbcbc9b1f
0113 02:51:43.427445 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:43.431291 (+ 3846us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:43.556316 (+125025us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:43.593326 (+ 37010us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1935361}
I20250113 02:51:43.593760 1198 tablet_replica.cc:331] T 153b8f90b5a847b88944f683d306f579 P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:43.593974 1198 raft_consensus.cc:2238] T 153b8f90b5a847b88944f683d306f579 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:43.593997 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet ba8980ad3a364c888a825affbcbc9b1f (table default.Table_459ff6639996470cb94e898c9ef4880e [id=a1d8c84954c747adb2aa43726d7b5334]) successfully deleted
I20250113 02:51:43.594137 1198 raft_consensus.cc:2267] T 153b8f90b5a847b88944f683d306f579 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:43.594664 2501 raft_consensus.cc:2798] T fb2458e2044d49029ed2d6851db9de25 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 02:51:43.594731 1198 ts_tablet_manager.cc:1905] T 153b8f90b5a847b88944f683d306f579 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:43.594746 2501 raft_consensus.cc:491] T fb2458e2044d49029ed2d6851db9de25 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 02:51:43.594807 2501 raft_consensus.cc:3054] T fb2458e2044d49029ed2d6851db9de25 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:43.595670 1196 tablet_replica.cc:331] T 7527e6e0b65e4a43b7f48899a34cdf7f P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:43.595837 1196 raft_consensus.cc:2238] T 7527e6e0b65e4a43b7f48899a34cdf7f P df69f48656b44bb890f76587ee69cc0f [term 1 FOLLOWER]: Raft consensus shutting down.
W20250113 02:51:43.595842 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 7da6f8a4b89647968609f73f9ad5ba14 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7da6f8a4b89647968609f73f9ad5ba14 already in progress: creating tablet
I20250113 02:51:43.596040 1196 raft_consensus.cc:2267] T 7527e6e0b65e4a43b7f48899a34cdf7f P df69f48656b44bb890f76587ee69cc0f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:43.596560 1196 ts_tablet_manager.cc:1905] T 7527e6e0b65e4a43b7f48899a34cdf7f P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:43.597252 2501 raft_consensus.cc:513] T fb2458e2044d49029ed2d6851db9de25 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
W20250113 02:51:43.597967 995 consensus_peers.cc:487] T 438604e8859f422f9caccb61c5b5b125 P 4216a7f853d642b7bc3e5cff458f3aca -> Peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Couldn't send request to peer 1186fa869eb14812832fa6562531f196. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 438604e8859f422f9caccb61c5b5b125. This is attempt 1: this message will repeat every 5th retry.
I20250113 02:51:43.598093 2501 leader_election.cc:290] T fb2458e2044d49029ed2d6851db9de25 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 election: Requested vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:43.598241 2501 raft_consensus.cc:2743] T b217cb8c09fb41cab4f53e605c37efe8 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:43.598343 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb2458e2044d49029ed2d6851db9de25" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f"
W20250113 02:51:43.598850 711 leader_election.cc:343] T fb2458e2044d49029ed2d6851db9de25 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:43.558425 1192 raft_consensus.cc:357] T f1237f7b04af4ebbb39b03aa3c78a499 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:43.600410 1192 raft_consensus.cc:383] T f1237f7b04af4ebbb39b03aa3c78a499 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:43.600482 1192 raft_consensus.cc:738] T f1237f7b04af4ebbb39b03aa3c78a499 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:43.600695 1192 consensus_queue.cc:260] T f1237f7b04af4ebbb39b03aa3c78a499 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:43.603447 1192 ts_tablet_manager.cc:1428] T f1237f7b04af4ebbb39b03aa3c78a499 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.051s user 0.002s sys 0.000s
I20250113 02:51:43.603713 1192 tablet_bootstrap.cc:492] T cfa963a6df094ce0b9c54408f75ca6b2 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:43.605484 1192 tablet_bootstrap.cc:654] T cfa963a6df094ce0b9c54408f75ca6b2 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:43.607718 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb2458e2044d49029ed2d6851db9de25" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca"
I20250113 02:51:43.607995 1059 raft_consensus.cc:3054] T fb2458e2044d49029ed2d6851db9de25 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:43.611312 1059 raft_consensus.cc:2463] T fb2458e2044d49029ed2d6851db9de25 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1186fa869eb14812832fa6562531f196 in term 1.
I20250113 02:51:43.612309 2645 raft_consensus.cc:491] T 79aaab73797c474bae06831fa308b066 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:43.612797 2645 raft_consensus.cc:513] T 79aaab73797c474bae06831fa308b066 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:43.615933 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79aaab73797c474bae06831fa308b066" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:43.615993 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79aaab73797c474bae06831fa308b066" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:43.616650 711 leader_election.cc:343] T 79aaab73797c474bae06831fa308b066 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:43.616958 2645 leader_election.cc:290] T 79aaab73797c474bae06831fa308b066 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:43.617635 709 leader_election.cc:304] T fb2458e2044d49029ed2d6851db9de25 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters: df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:43.618742 2645 raft_consensus.cc:2798] T fb2458e2044d49029ed2d6851db9de25 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Leader election won for term 1
I20250113 02:51:43.619717 2645 raft_consensus.cc:695] T fb2458e2044d49029ed2d6851db9de25 P 1186fa869eb14812832fa6562531f196 [term 1 LEADER]: Becoming Leader. State: Replica: 1186fa869eb14812832fa6562531f196, State: Running, Role: LEADER
I20250113 02:51:43.620210 2645 consensus_queue.cc:237] T fb2458e2044d49029ed2d6851db9de25 P 1186fa869eb14812832fa6562531f196 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:43.634032 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_a9e430e1146e40d7945c1680ae7ef2ba"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:43.634670 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a9e430e1146e40d7945c1680ae7ef2ba in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:43.634994 449 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_2bcd0f2a378c4d2aaefbbbfa2c144431" } modify_external_catalogs: true
I20250113 02:51:43.636124 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_bfd9035770884f2eb446fae15640ab22"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:43.636772 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bfd9035770884f2eb446fae15640ab22 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:43.636145 454 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_7a15797b8bd04c2e911c0803d28cecf6" } modify_external_catalogs: true
W20250113 02:51:43.639117 709 leader_election.cc:343] T 79aaab73797c474bae06831fa308b066 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:43.640590 1196 ts_tablet_manager.cc:1918] T 7527e6e0b65e4a43b7f48899a34cdf7f P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250113 02:51:43.640720 1196 log.cc:1198] T 7527e6e0b65e4a43b7f48899a34cdf7f P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/7527e6e0b65e4a43b7f48899a34cdf7f
I20250113 02:51:43.641033 709 leader_election.cc:304] T 79aaab73797c474bae06831fa308b066 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:43.645218 1198 ts_tablet_manager.cc:1918] T 153b8f90b5a847b88944f683d306f579 P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:43.645356 1198 log.cc:1198] T 153b8f90b5a847b88944f683d306f579 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/153b8f90b5a847b88944f683d306f579
I20250113 02:51:43.645678 1198 ts_tablet_manager.cc:1939] T 153b8f90b5a847b88944f683d306f579 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:43.646522 1196 ts_tablet_manager.cc:1939] T 7527e6e0b65e4a43b7f48899a34cdf7f P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:43.642292 1195 log.cc:1198] T 64aa8390b1b1475c8743b6491d40674c P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/64aa8390b1b1475c8743b6491d40674c
I20250113 02:51:43.648521 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet 7527e6e0b65e4a43b7f48899a34cdf7f (table default.Table_c8b93363989c4e109540af6bef426f73 [id=b7846a8ebee84005b1b5d259560a26bd]) successfully deleted
I20250113 02:51:43.649559 1196 tablet_replica.cc:331] T 655b63ff61ff4f4aa08e0449df39352a P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:43.649762 1196 raft_consensus.cc:2238] T 655b63ff61ff4f4aa08e0449df39352a P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:43.649920 1196 raft_consensus.cc:2267] T 655b63ff61ff4f4aa08e0449df39352a P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:43.652338 1195 ts_tablet_manager.cc:1939] T 64aa8390b1b1475c8743b6491d40674c P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:43.652693 1196 ts_tablet_manager.cc:1905] T 655b63ff61ff4f4aa08e0449df39352a P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:43.653249 2501 raft_consensus.cc:2743] T 79aaab73797c474bae06831fa308b066 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:43.660486 2632 tablet.cc:1721] T 1a63f485caf54068abd6b50d7de3d14a P 4216a7f853d642b7bc3e5cff458f3aca: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250113 02:51:43.661283 2636 tablet.cc:1721] T 1a63f485caf54068abd6b50d7de3d14a P df69f48656b44bb890f76587ee69cc0f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250113 02:51:43.665181 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1128) took 5809 ms. Trace:
I20250113 02:51:43.665364 1195 rpcz_store.cc:276] 0113 02:51:37.856167 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:37.896145 (+ 39978us) service_pool.cc:225] Handling call
0113 02:51:43.344941 (+5448796us) ts_tablet_manager.cc:1145] Deleting tablet 64aa8390b1b1475c8743b6491d40674c
0113 02:51:43.344950 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:43.499551 (+154601us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:43.535335 (+ 35784us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:43.665156 (+129821us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5448499}
I20250113 02:51:43.666152 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet 64aa8390b1b1475c8743b6491d40674c (table default.Table_b89481a4a7be4e49b13f3260d8a31d5d [id=6c751a49967d448ba1432934782b08d3]) successfully deleted
I20250113 02:51:43.666344 1195 tablet_replica.cc:331] T 82505f1389184d0ca12ec0e2d6084d22 P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:43.666723 1195 raft_consensus.cc:2238] T 82505f1389184d0ca12ec0e2d6084d22 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:43.667088 1195 raft_consensus.cc:2267] T 82505f1389184d0ca12ec0e2d6084d22 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:43.668081 2501 raft_consensus.cc:491] T 65d9bbd8c15f40a4a50c1695fb5e648e P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:43.668202 2501 raft_consensus.cc:513] T 65d9bbd8c15f40a4a50c1695fb5e648e P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:43.668802 2501 leader_election.cc:290] T 65d9bbd8c15f40a4a50c1695fb5e648e P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:43.658365 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1344) took 2167 ms. Trace:
I20250113 02:51:43.669099 1198 rpcz_store.cc:276] 0113 02:51:41.490461 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:41.491815 (+ 1354us) service_pool.cc:225] Handling call
0113 02:51:43.593713 (+2101898us) ts_tablet_manager.cc:1145] Deleting tablet 153b8f90b5a847b88944f683d306f579
0113 02:51:43.593722 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:43.625802 (+ 32080us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:43.645104 (+ 19302us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:43.658336 (+ 13232us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2101462}
I20250113 02:51:43.669644 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet 153b8f90b5a847b88944f683d306f579 (table default.Table_a7c3697e25a94815ac03a3cb1a57bc0c [id=39d8412c30d24c44a01c24b3c9645a8d]) successfully deleted
I20250113 02:51:43.669665 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1343) took 2199 ms. Trace:
I20250113 02:51:43.669767 1198 rpcz_store.cc:276] 0113 02:51:41.470635 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:41.481432 (+ 10797us) service_pool.cc:225] Handling call
0113 02:51:43.669521 (+2188089us) ts_tablet_manager.cc:1145] Deleting tablet c8966b3b93144a30813bfdaf04f9c43a
0113 02:51:43.669532 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:43.669640 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2165275}
W20250113 02:51:43.670174 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet c8966b3b93144a30813bfdaf04f9c43a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c8966b3b93144a30813bfdaf04f9c43a already in progress: creating tablet
I20250113 02:51:43.670554 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1355) took 1896 ms. Trace:
I20250113 02:51:43.670643 1198 rpcz_store.cc:276] 0113 02:51:41.773724 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:41.773777 (+ 53us) service_pool.cc:225] Handling call
0113 02:51:43.670442 (+1896665us) ts_tablet_manager.cc:1145] Deleting tablet 316cb2863ed645158a8f72e3cb0a94fd
0113 02:51:43.670453 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:43.670536 (+ 83us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1896492}
W20250113 02:51:43.671036 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 316cb2863ed645158a8f72e3cb0a94fd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 316cb2863ed645158a8f72e3cb0a94fd already in progress: creating tablet
I20250113 02:51:43.671403 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1356) took 1885 ms. Trace:
I20250113 02:51:43.671491 1198 rpcz_store.cc:276] 0113 02:51:41.785697 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:41.785749 (+ 52us) service_pool.cc:225] Handling call
0113 02:51:43.671296 (+1885547us) ts_tablet_manager.cc:1145] Deleting tablet 4cc15487be474354b995c4e415276a18
0113 02:51:43.671307 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:43.671386 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1885372}
W20250113 02:51:43.671866 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 4cc15487be474354b995c4e415276a18 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4cc15487be474354b995c4e415276a18 already in progress: creating tablet
I20250113 02:51:43.672405 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1358) took 1847 ms. Trace:
I20250113 02:51:43.672485 1198 rpcz_store.cc:276] 0113 02:51:41.825273 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:41.825475 (+ 202us) service_pool.cc:225] Handling call
0113 02:51:43.672132 (+1846657us) ts_tablet_manager.cc:1145] Deleting tablet 85d13f8abbec484d82c4d0b86480531b
0113 02:51:43.672141 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:43.672389 (+ 248us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1846427}
W20250113 02:51:43.672861 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 85d13f8abbec484d82c4d0b86480531b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 85d13f8abbec484d82c4d0b86480531b already in progress: creating tablet
I20250113 02:51:43.674454 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d9bbd8c15f40a4a50c1695fb5e648e" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:43.675082 2501 consensus_queue.cc:1035] T 938cc1ccd8dd449996836188ba9e64ff P 1186fa869eb14812832fa6562531f196 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250113 02:51:43.676169 709 leader_election.cc:343] T 65d9bbd8c15f40a4a50c1695fb5e648e P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:43.676627 1196 ts_tablet_manager.cc:1918] T 655b63ff61ff4f4aa08e0449df39352a P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:43.677460 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1359) took 1840 ms. Trace:
I20250113 02:51:43.677635 1198 rpcz_store.cc:276] 0113 02:51:41.836577 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:41.837039 (+ 462us) service_pool.cc:225] Handling call
0113 02:51:43.676815 (+1839776us) ts_tablet_manager.cc:1145] Deleting tablet 8ee0b2e06dc74faab48912dd16fe1537
0113 02:51:43.676824 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:43.676907 (+ 83us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1839498}
W20250113 02:51:43.678315 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 8ee0b2e06dc74faab48912dd16fe1537 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8ee0b2e06dc74faab48912dd16fe1537 already in progress: creating tablet
I20250113 02:51:43.678766 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1360) took 1830 ms. Trace:
I20250113 02:51:43.679271 1198 rpcz_store.cc:276] 0113 02:51:41.848300 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:41.848450 (+ 150us) service_pool.cc:225] Handling call
0113 02:51:43.678654 (+1830204us) ts_tablet_manager.cc:1145] Deleting tablet 9b9e7deda7334ff4a3429f70d9b3a7fc
0113 02:51:43.678666 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:43.678749 (+ 83us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1829921}
W20250113 02:51:43.679948 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 9b9e7deda7334ff4a3429f70d9b3a7fc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9b9e7deda7334ff4a3429f70d9b3a7fc already in progress: creating tablet
I20250113 02:51:43.680395 1198 tablet_replica.cc:331] T 1d2fdc75d00f4e749d322ac6cd5fc410 P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:43.680825 1198 raft_consensus.cc:2238] T 1d2fdc75d00f4e749d322ac6cd5fc410 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:43.681214 1198 raft_consensus.cc:2267] T 1d2fdc75d00f4e749d322ac6cd5fc410 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:43.682410 1198 ts_tablet_manager.cc:1905] T 1d2fdc75d00f4e749d322ac6cd5fc410 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:43.687664 1191 tablet_bootstrap.cc:492] T 88a776aa820c44b3a3854ca5022bd353 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:43.687881 1191 ts_tablet_manager.cc:1397] T 88a776aa820c44b3a3854ca5022bd353 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.124s user 0.001s sys 0.008s
I20250113 02:51:43.653015 1194 tablet_bootstrap.cc:492] T c518f791b4d74c54ab0c0364d29b4a64 P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:43.688463 1194 ts_tablet_manager.cc:1397] T c518f791b4d74c54ab0c0364d29b4a64 P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.153s user 0.003s sys 0.000s
I20250113 02:51:43.688849 1191 raft_consensus.cc:357] T 88a776aa820c44b3a3854ca5022bd353 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:43.689102 1191 raft_consensus.cc:383] T 88a776aa820c44b3a3854ca5022bd353 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:43.683162 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d9bbd8c15f40a4a50c1695fb5e648e" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:43.689175 1191 raft_consensus.cc:738] T 88a776aa820c44b3a3854ca5022bd353 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:43.689431 1191 consensus_queue.cc:260] T 88a776aa820c44b3a3854ca5022bd353 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
W20250113 02:51:43.689769 711 leader_election.cc:343] T 65d9bbd8c15f40a4a50c1695fb5e648e P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:43.689886 711 leader_election.cc:304] T 65d9bbd8c15f40a4a50c1695fb5e648e P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:43.690208 2645 raft_consensus.cc:2743] T 65d9bbd8c15f40a4a50c1695fb5e648e P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:43.690646 1194 raft_consensus.cc:357] T c518f791b4d74c54ab0c0364d29b4a64 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:43.690935 1194 raft_consensus.cc:383] T c518f791b4d74c54ab0c0364d29b4a64 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:43.691046 1194 raft_consensus.cc:738] T c518f791b4d74c54ab0c0364d29b4a64 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:43.691215 1191 ts_tablet_manager.cc:1428] T 88a776aa820c44b3a3854ca5022bd353 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20250113 02:51:43.691330 1194 consensus_queue.cc:260] T c518f791b4d74c54ab0c0364d29b4a64 P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:43.692180 1194 ts_tablet_manager.cc:1428] T c518f791b4d74c54ab0c0364d29b4a64 P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250113 02:51:43.692643 1196 log.cc:1198] T 655b63ff61ff4f4aa08e0449df39352a P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/655b63ff61ff4f4aa08e0449df39352a
I20250113 02:51:43.693138 1196 ts_tablet_manager.cc:1939] T 655b63ff61ff4f4aa08e0449df39352a P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:43.693181 1191 tablet_bootstrap.cc:492] T 58053845187f466599750f2a62f4c3e1 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:43.693496 1194 tablet_bootstrap.cc:492] T c7842d6f2c074e3b9d1a3378db9856be P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:43.697415 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet 655b63ff61ff4f4aa08e0449df39352a (table default.Table_c6c64517b49e4843ba63567a871d22bd [id=ec4bdc424404427589852603bb6d8112]) successfully deleted
W20250113 02:51:43.698377 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 77036be46b574f73a02ca656bf5cc0a3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 77036be46b574f73a02ca656bf5cc0a3 already in progress: creating tablet
I20250113 02:51:43.698943 1194 tablet_bootstrap.cc:654] T c7842d6f2c074e3b9d1a3378db9856be P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
W20250113 02:51:43.699960 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 1c82cf9373d24fedad88b5df7d5a8b64 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1c82cf9373d24fedad88b5df7d5a8b64 already in progress: creating tablet
I20250113 02:51:43.697219 2652 tablet.cc:1721] T 1a63f485caf54068abd6b50d7de3d14a P 1186fa869eb14812832fa6562531f196: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20250113 02:51:43.701855 998 consensus_peers.cc:487] T 438604e8859f422f9caccb61c5b5b125 P 4216a7f853d642b7bc3e5cff458f3aca -> Peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Couldn't send request to peer df69f48656b44bb890f76587ee69cc0f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250113 02:51:43.702242 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet e312e70334e04ecd843159ac72f4a2a6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e312e70334e04ecd843159ac72f4a2a6 already in progress: creating tablet
W20250113 02:51:43.703019 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 559c2f8ea91d4e81bf303b3a0c61c143 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 559c2f8ea91d4e81bf303b3a0c61c143 already in progress: creating tablet
I20250113 02:51:43.705029 1194 tablet_bootstrap.cc:492] T c7842d6f2c074e3b9d1a3378db9856be P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:43.708155 1194 ts_tablet_manager.cc:1397] T c7842d6f2c074e3b9d1a3378db9856be P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.000s
I20250113 02:51:43.709746 1194 raft_consensus.cc:357] T c7842d6f2c074e3b9d1a3378db9856be P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:43.710834 1194 raft_consensus.cc:383] T c7842d6f2c074e3b9d1a3378db9856be P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:43.711038 1194 raft_consensus.cc:738] T c7842d6f2c074e3b9d1a3378db9856be P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:43.713920 1192 tablet_bootstrap.cc:492] T cfa963a6df094ce0b9c54408f75ca6b2 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:43.714097 1192 ts_tablet_manager.cc:1397] T cfa963a6df094ce0b9c54408f75ca6b2 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.110s user 0.012s sys 0.003s
I20250113 02:51:43.714459 2645 raft_consensus.cc:491] T 59fadc693f554854bc2874f8db7c64c8 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:43.714609 2645 raft_consensus.cc:513] T 59fadc693f554854bc2874f8db7c64c8 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:43.715263 2645 leader_election.cc:290] T 59fadc693f554854bc2874f8db7c64c8 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:43.715438 1192 raft_consensus.cc:357] T cfa963a6df094ce0b9c54408f75ca6b2 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:43.715674 1192 raft_consensus.cc:383] T cfa963a6df094ce0b9c54408f75ca6b2 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:43.715807 1192 raft_consensus.cc:738] T cfa963a6df094ce0b9c54408f75ca6b2 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:43.716106 1192 consensus_queue.cc:260] T cfa963a6df094ce0b9c54408f75ca6b2 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:43.716307 1191 tablet_bootstrap.cc:654] T 58053845187f466599750f2a62f4c3e1 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:43.716734 2643 consensus_queue.cc:1035] T c13e9a51c1e74b44b352bade78390ba5 P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 02:51:43.719350 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59fadc693f554854bc2874f8db7c64c8" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:43.719903 711 leader_election.cc:343] T 59fadc693f554854bc2874f8db7c64c8 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:43.721360 449 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_2bcd0f2a378c4d2aaefbbbfa2c144431" } modify_external_catalogs: true
I20250113 02:51:43.721776 454 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_7a15797b8bd04c2e911c0803d28cecf6" } modify_external_catalogs: true
I20250113 02:51:43.723501 1192 ts_tablet_manager.cc:1428] T cfa963a6df094ce0b9c54408f75ca6b2 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20250113 02:51:43.730070 1192 tablet_bootstrap.cc:492] T 79aaab73797c474bae06831fa308b066 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:43.730351 1194 consensus_queue.cc:260] T c7842d6f2c074e3b9d1a3378db9856be P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
W20250113 02:51:43.732362 448 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 119baabad10a488082f86f2a52ac3779 consensus state has no leader
I20250113 02:51:43.733794 453 catalog_manager.cc:5618] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467) does not have the latest schema for tablet 44d2c7d5334c476fac60fdd57e2c02b6 (table default.Table_6daec99b05744d0caed86e51e86f24c2 [id=2931f217041c46cc8ae52bd6ce8cfd17]). Expected version 1 got 0
I20250113 02:51:43.738065 452 catalog_manager.cc:5618] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095) does not have the latest schema for tablet 3f62f344c5a648328ca2de8ca1069c19 (table default.Table_9ca0c47692a840c39dbc1c73c7a6bc7c [id=7783df86a6ee404f87055036508adb20]). Expected version 1 got 0
I20250113 02:51:43.740211 1194 ts_tablet_manager.cc:1428] T c7842d6f2c074e3b9d1a3378db9856be P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.032s user 0.002s sys 0.000s
I20250113 02:51:43.740645 1194 tablet_bootstrap.cc:492] T bc533518faa24dffa3306ff092e8d507 P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:43.747337 1058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59fadc693f554854bc2874f8db7c64c8" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
W20250113 02:51:43.750427 998 consensus_peers.cc:487] T c13e9a51c1e74b44b352bade78390ba5 P 4216a7f853d642b7bc3e5cff458f3aca -> Peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Couldn't send request to peer df69f48656b44bb890f76587ee69cc0f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250113 02:51:43.750722 1198 ts_tablet_manager.cc:1918] T 1d2fdc75d00f4e749d322ac6cd5fc410 P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:43.751150 1198 log.cc:1198] T 1d2fdc75d00f4e749d322ac6cd5fc410 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/1d2fdc75d00f4e749d322ac6cd5fc410
I20250113 02:51:43.751608 1198 ts_tablet_manager.cc:1939] T 1d2fdc75d00f4e749d322ac6cd5fc410 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:43.751883 1192 tablet_bootstrap.cc:654] T 79aaab73797c474bae06831fa308b066 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:43.753702 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1366) took 1734 ms. Trace:
I20250113 02:51:43.754410 1194 tablet_bootstrap.cc:654] T bc533518faa24dffa3306ff092e8d507 P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:43.754551 1198 rpcz_store.cc:276] 0113 02:51:42.019454 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:42.019499 (+ 45us) service_pool.cc:225] Handling call
0113 02:51:43.680337 (+1660838us) ts_tablet_manager.cc:1145] Deleting tablet 1d2fdc75d00f4e749d322ac6cd5fc410
0113 02:51:43.680348 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:43.686472 (+ 6124us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:43.712740 (+ 26268us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:43.753665 (+ 40925us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1660646}
I20250113 02:51:43.756022 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet 1d2fdc75d00f4e749d322ac6cd5fc410 (table default.Table_3fa4f9f57be84226a01935b139545f6e [id=0ab8e0c80dda42fcb07f2fd6fbf01609]) successfully deleted
I20250113 02:51:43.756438 1198 tablet_replica.cc:331] T acd83024592e431898e9675c17f50d4f P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:43.756705 1198 raft_consensus.cc:2238] T acd83024592e431898e9675c17f50d4f P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: Raft consensus shutting down.
I20250113 02:51:43.757095 1198 raft_consensus.cc:2267] T acd83024592e431898e9675c17f50d4f P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:43.758517 1198 ts_tablet_manager.cc:1905] T acd83024592e431898e9675c17f50d4f P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250113 02:51:43.764254 998 consensus_peers.cc:487] T 6a4c6dc08df744e6856bdf99ab65c575 P 4216a7f853d642b7bc3e5cff458f3aca -> Peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Couldn't send request to peer df69f48656b44bb890f76587ee69cc0f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250113 02:51:43.769565 2643 consensus_queue.cc:1035] T aa1ebdb92a5f4c92a5195b614689c100 P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 02:51:43.769879 2547 raft_consensus.cc:491] T cfa963a6df094ce0b9c54408f75ca6b2 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:43.770023 2547 raft_consensus.cc:513] T cfa963a6df094ce0b9c54408f75ca6b2 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:43.770632 2547 leader_election.cc:290] T cfa963a6df094ce0b9c54408f75ca6b2 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
W20250113 02:51:43.770866 709 leader_election.cc:343] T 59fadc693f554854bc2874f8db7c64c8 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:43.771116 1194 tablet_bootstrap.cc:492] T bc533518faa24dffa3306ff092e8d507 P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:43.771323 1194 ts_tablet_manager.cc:1397] T bc533518faa24dffa3306ff092e8d507 P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.031s user 0.006s sys 0.000s
I20250113 02:51:43.771911 709 leader_election.cc:304] T 59fadc693f554854bc2874f8db7c64c8 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:43.772215 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfa963a6df094ce0b9c54408f75ca6b2" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
I20250113 02:51:43.772532 773 raft_consensus.cc:2463] T cfa963a6df094ce0b9c54408f75ca6b2 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 0.
I20250113 02:51:43.772485 1194 raft_consensus.cc:357] T bc533518faa24dffa3306ff092e8d507 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:43.772653 1194 raft_consensus.cc:383] T bc533518faa24dffa3306ff092e8d507 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:43.772703 1194 raft_consensus.cc:738] T bc533518faa24dffa3306ff092e8d507 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
W20250113 02:51:43.773180 998 consensus_peers.cc:487] T 8337f3c82d654ecdac2cbc1ffa88ef05 P 4216a7f853d642b7bc3e5cff458f3aca -> Peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Couldn't send request to peer df69f48656b44bb890f76587ee69cc0f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8337f3c82d654ecdac2cbc1ffa88ef05. This is attempt 1: this message will repeat every 5th retry.
I20250113 02:51:43.773308 995 leader_election.cc:304] T cfa963a6df094ce0b9c54408f75ca6b2 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters:
I20250113 02:51:43.773823 2547 raft_consensus.cc:2798] T cfa963a6df094ce0b9c54408f75ca6b2 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 02:51:43.773928 2547 raft_consensus.cc:491] T cfa963a6df094ce0b9c54408f75ca6b2 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 02:51:43.774004 2547 raft_consensus.cc:3054] T cfa963a6df094ce0b9c54408f75ca6b2 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:43.774902 2588 raft_consensus.cc:2743] T 59fadc693f554854bc2874f8db7c64c8 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:43.775504 1198 ts_tablet_manager.cc:1918] T acd83024592e431898e9675c17f50d4f P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250113 02:51:43.776304 1194 consensus_queue.cc:260] T bc533518faa24dffa3306ff092e8d507 P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:43.773624 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfa963a6df094ce0b9c54408f75ca6b2" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:43.771700 1195 ts_tablet_manager.cc:1905] T 82505f1389184d0ca12ec0e2d6084d22 P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250113 02:51:43.777215 998 leader_election.cc:343] T cfa963a6df094ce0b9c54408f75ca6b2 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:43.777320 1192 tablet_bootstrap.cc:492] T 79aaab73797c474bae06831fa308b066 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:43.777436 1192 ts_tablet_manager.cc:1397] T 79aaab73797c474bae06831fa308b066 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.047s user 0.005s sys 0.001s
I20250113 02:51:43.778381 1192 raft_consensus.cc:357] T 79aaab73797c474bae06831fa308b066 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:43.778573 1192 raft_consensus.cc:383] T 79aaab73797c474bae06831fa308b066 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:43.778648 1192 raft_consensus.cc:738] T 79aaab73797c474bae06831fa308b066 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:43.778889 1192 consensus_queue.cc:260] T 79aaab73797c474bae06831fa308b066 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:43.779649 1192 ts_tablet_manager.cc:1428] T 79aaab73797c474bae06831fa308b066 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250113 02:51:43.776346 2547 raft_consensus.cc:513] T cfa963a6df094ce0b9c54408f75ca6b2 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:43.781039 2547 leader_election.cc:290] T cfa963a6df094ce0b9c54408f75ca6b2 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Requested vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:43.781364 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfa963a6df094ce0b9c54408f75ca6b2" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196"
I20250113 02:51:43.781553 773 raft_consensus.cc:3054] T cfa963a6df094ce0b9c54408f75ca6b2 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:43.781733 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfa963a6df094ce0b9c54408f75ca6b2" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f"
W20250113 02:51:43.782244 998 leader_election.cc:343] T cfa963a6df094ce0b9c54408f75ca6b2 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:43.784253 773 raft_consensus.cc:2463] T cfa963a6df094ce0b9c54408f75ca6b2 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 1.
I20250113 02:51:43.784732 1192 tablet_bootstrap.cc:492] T 019eb21c661547cfa0b619d119eb024e P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:43.784991 995 leader_election.cc:304] T cfa963a6df094ce0b9c54408f75ca6b2 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters: df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:43.785516 2547 raft_consensus.cc:2798] T cfa963a6df094ce0b9c54408f75ca6b2 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Leader election won for term 1
I20250113 02:51:43.785696 1194 ts_tablet_manager.cc:1428] T bc533518faa24dffa3306ff092e8d507 P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.014s user 0.001s sys 0.000s
I20250113 02:51:43.786121 2547 raft_consensus.cc:695] T cfa963a6df094ce0b9c54408f75ca6b2 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: Becoming Leader. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Running, Role: LEADER
I20250113 02:51:43.788210 2547 consensus_queue.cc:237] T cfa963a6df094ce0b9c54408f75ca6b2 P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:43.791170 1194 tablet_bootstrap.cc:492] T b5c1882125b5491ea79a29a6dac6771e P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:43.786835 1192 tablet_bootstrap.cc:654] T 019eb21c661547cfa0b619d119eb024e P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:43.791611 2588 consensus_queue.cc:1035] T b38cd34d67834a3f9bf5b5636bb79217 P 1186fa869eb14812832fa6562531f196 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 02:51:43.796218 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_b7c9a4329c0a4aac98cdb053b3921b63" } modify_external_catalogs: true
I20250113 02:51:43.796492 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_b7c9a4329c0a4aac98cdb053b3921b63" } modify_external_catalogs: true
I20250113 02:51:43.797988 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_8d0905ada0b74b8fba4964abaf95a2b3" } new_table_name: "default.Table_28f8ce5ef2a343cabcea84369261064b" modify_external_catalogs: true
I20250113 02:51:43.799371 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_63be23fc2ed64476b299996a4175c5d5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:43.799861 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_63be23fc2ed64476b299996a4175c5d5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:43.799880 1192 tablet_bootstrap.cc:492] T 019eb21c661547cfa0b619d119eb024e P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:43.800034 1192 ts_tablet_manager.cc:1397] T 019eb21c661547cfa0b619d119eb024e P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.004s
I20250113 02:51:43.800913 1192 raft_consensus.cc:357] T 019eb21c661547cfa0b619d119eb024e P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:43.800992 446 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_ffa9bfdfd58f4159b17a5185a1d2192e" } modify_external_catalogs: true
I20250113 02:51:43.801096 1192 raft_consensus.cc:383] T 019eb21c661547cfa0b619d119eb024e P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:43.801136 1192 raft_consensus.cc:738] T 019eb21c661547cfa0b619d119eb024e P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:43.801062 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_3e1f7bdb9ecf47d0baa280c94babfc41"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:43.801142 446 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_ffa9bfdfd58f4159b17a5185a1d2192e" } modify_external_catalogs: true
W20250113 02:51:43.801506 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3e1f7bdb9ecf47d0baa280c94babfc41 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:43.801484 1192 consensus_queue.cc:260] T 019eb21c661547cfa0b619d119eb024e P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:43.802608 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_ac79159d146b433ebdea04d6746fb7d4" } new_table_name: "default.Table_008e2368031e40e79fb92b4fad5b41ed" modify_external_catalogs: true
I20250113 02:51:43.803102 1192 ts_tablet_manager.cc:1428] T 019eb21c661547cfa0b619d119eb024e P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250113 02:51:43.804203 1192 tablet_bootstrap.cc:492] T cf7b8bd6801743f49e9b9473747447c5 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:43.809778 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 4e59953509884a0f82b276b232bb00fb
I20250113 02:51:43.810259 1192 tablet_bootstrap.cc:654] T cf7b8bd6801743f49e9b9473747447c5 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:43.811919 736 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e59953509884a0f82b276b232bb00fb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:43 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:43.813123 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e59953509884a0f82b276b232bb00fb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:43 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:43.813606 883 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e59953509884a0f82b276b232bb00fb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:43 UTC) from {username='slave'} at 127.0.0.1:37366
W20250113 02:51:43.814285 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 4e59953509884a0f82b276b232bb00fb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4e59953509884a0f82b276b232bb00fb already in progress: creating tablet
I20250113 02:51:43.820338 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 8e4b74a1ef824f40a40c0b215b2b5711
I20250113 02:51:43.823298 454 catalog_manager.cc:5618] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587) does not have the latest schema for tablet f1237f7b04af4ebbb39b03aa3c78a499 (table default.Table_4c5e2e1008b54019a243ffe1ec72c020 [id=0817677719d045fd83bb85890f11df25]). Expected version 2 got 0
I20250113 02:51:43.823653 454 catalog_manager.cc:5618] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587) does not have the latest schema for tablet 938cc1ccd8dd449996836188ba9e64ff (table default.Table_e5b4d614e4b743d7982b025ca6d297e5 [id=5824acd778ed47d09f087bbf8e7822d6]). Expected version 1 got 0
I20250113 02:51:43.828624 883 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e4b74a1ef824f40a40c0b215b2b5711 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:43 UTC) from {username='slave'} at 127.0.0.1:37366
W20250113 02:51:43.829625 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 8e4b74a1ef824f40a40c0b215b2b5711 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8e4b74a1ef824f40a40c0b215b2b5711 already in progress: creating tablet
I20250113 02:51:43.830158 736 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e4b74a1ef824f40a40c0b215b2b5711 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:43 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:43.830943 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e4b74a1ef824f40a40c0b215b2b5711 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:43 UTC) from {username='slave'} at 127.0.0.1:60230
W20250113 02:51:43.844019 709 outbound_call.cc:321] RPC callback for RPC call kudu.master.MasterService.TSHeartbeat -> {remote=127.0.104.61:35271, user_credentials={real_user=slave}} blocked reactor thread for 51544.5us
W20250113 02:51:43.845866 711 consensus_peers.cc:487] T b38cd34d67834a3f9bf5b5636bb79217 P 1186fa869eb14812832fa6562531f196 -> Peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Couldn't send request to peer df69f48656b44bb890f76587ee69cc0f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250113 02:51:43.849355 2476 raft_consensus.cc:491] T c518f791b4d74c54ab0c0364d29b4a64 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:43.849503 2476 raft_consensus.cc:513] T c518f791b4d74c54ab0c0364d29b4a64 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:43.850144 2476 leader_election.cc:290] T c518f791b4d74c54ab0c0364d29b4a64 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
W20250113 02:51:43.851490 851 leader_election.cc:343] T c518f791b4d74c54ab0c0364d29b4a64 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Not found: Tablet not found: c518f791b4d74c54ab0c0364d29b4a64
I20250113 02:51:43.852977 1191 tablet_bootstrap.cc:492] T 58053845187f466599750f2a62f4c3e1 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:43.853129 1191 ts_tablet_manager.cc:1397] T 58053845187f466599750f2a62f4c3e1 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.160s user 0.002s sys 0.002s
W20250113 02:51:43.853516 852 leader_election.cc:343] T c518f791b4d74c54ab0c0364d29b4a64 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Not found: Tablet not found: c518f791b4d74c54ab0c0364d29b4a64
I20250113 02:51:43.853618 852 leader_election.cc:304] T c518f791b4d74c54ab0c0364d29b4a64 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df69f48656b44bb890f76587ee69cc0f; no voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca
I20250113 02:51:43.853958 2476 raft_consensus.cc:2743] T c518f791b4d74c54ab0c0364d29b4a64 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:43.853876 1191 raft_consensus.cc:357] T 58053845187f466599750f2a62f4c3e1 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:43.854059 1191 raft_consensus.cc:383] T 58053845187f466599750f2a62f4c3e1 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:43.854107 1191 raft_consensus.cc:738] T 58053845187f466599750f2a62f4c3e1 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:43.854298 1198 log.cc:1198] T acd83024592e431898e9675c17f50d4f P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/acd83024592e431898e9675c17f50d4f
I20250113 02:51:43.854312 1191 consensus_queue.cc:260] T 58053845187f466599750f2a62f4c3e1 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:43.854985 1198 ts_tablet_manager.cc:1939] T acd83024592e431898e9675c17f50d4f P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:43.855895 1191 ts_tablet_manager.cc:1428] T 58053845187f466599750f2a62f4c3e1 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250113 02:51:43.856333 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1367) took 1836 ms. Trace:
I20250113 02:51:43.856511 1198 rpcz_store.cc:276] 0113 02:51:42.020231 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:42.020282 (+ 51us) service_pool.cc:225] Handling call
0113 02:51:43.756370 (+1736088us) ts_tablet_manager.cc:1145] Deleting tablet acd83024592e431898e9675c17f50d4f
0113 02:51:43.756380 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:43.762592 (+ 6212us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:43.775378 (+ 12786us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:43.856291 (+ 80913us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1733119}
I20250113 02:51:43.857046 1191 tablet_bootstrap.cc:492] T f4379f7fc6b54e6785f27975d9030cce P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:43.857229 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet acd83024592e431898e9675c17f50d4f (table default.Table_25cb4fc43dc6485ca0e4ec0cf048105d [id=044304e1c4b44cc28b8990d6f28145a8]) successfully deleted
I20250113 02:51:43.857758 1198 tablet_replica.cc:331] T 64aa8390b1b1475c8743b6491d40674c P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:43.858105 1198 raft_consensus.cc:2238] T 64aa8390b1b1475c8743b6491d40674c P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: Raft consensus shutting down.
I20250113 02:51:43.858703 1198 raft_consensus.cc:2267] T 64aa8390b1b1475c8743b6491d40674c P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:43.859941 1198 ts_tablet_manager.cc:1905] T 64aa8390b1b1475c8743b6491d40674c P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250113 02:51:43.860908 998 consensus_peers.cc:487] T aa1ebdb92a5f4c92a5195b614689c100 P 4216a7f853d642b7bc3e5cff458f3aca -> Peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Couldn't send request to peer df69f48656b44bb890f76587ee69cc0f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250113 02:51:43.869724 1198 ts_tablet_manager.cc:1918] T 64aa8390b1b1475c8743b6491d40674c P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250113 02:51:43.870388 1198 log.cc:1198] T 64aa8390b1b1475c8743b6491d40674c P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/64aa8390b1b1475c8743b6491d40674c
I20250113 02:51:43.870946 1198 ts_tablet_manager.cc:1939] T 64aa8390b1b1475c8743b6491d40674c P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:43.872258 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1368) took 1851 ms. Trace:
I20250113 02:51:43.872395 1198 rpcz_store.cc:276] 0113 02:51:42.020763 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:42.020807 (+ 44us) service_pool.cc:225] Handling call
0113 02:51:43.857660 (+1836853us) ts_tablet_manager.cc:1145] Deleting tablet 64aa8390b1b1475c8743b6491d40674c
0113 02:51:43.857695 (+ 35us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:43.864974 (+ 7279us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:43.869557 (+ 4583us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:43.872216 (+ 2659us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1833347}
I20250113 02:51:43.872992 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet 64aa8390b1b1475c8743b6491d40674c (table default.Table_b89481a4a7be4e49b13f3260d8a31d5d [id=6c751a49967d448ba1432934782b08d3]) successfully deleted
I20250113 02:51:43.873883 1198 tablet_replica.cc:331] T 0c408921a09a463eb464e08edcf042b3 P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:43.874166 1198 raft_consensus.cc:2238] T 0c408921a09a463eb464e08edcf042b3 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:43.874408 1198 raft_consensus.cc:2267] T 0c408921a09a463eb464e08edcf042b3 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:43.875260 1198 ts_tablet_manager.cc:1905] T 0c408921a09a463eb464e08edcf042b3 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:43.880287 2547 raft_consensus.cc:491] T 9f8eb43a39774d2c8e9cb4e88a405b0f P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:43.880524 2547 raft_consensus.cc:513] T 9f8eb43a39774d2c8e9cb4e88a405b0f P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:43.881601 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f8eb43a39774d2c8e9cb4e88a405b0f" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
I20250113 02:51:43.881897 772 raft_consensus.cc:2463] T 9f8eb43a39774d2c8e9cb4e88a405b0f P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 0.
I20250113 02:51:43.882318 995 leader_election.cc:304] T 9f8eb43a39774d2c8e9cb4e88a405b0f P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters:
I20250113 02:51:43.882567 2643 raft_consensus.cc:2798] T 9f8eb43a39774d2c8e9cb4e88a405b0f P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 02:51:43.882633 2643 raft_consensus.cc:491] T 9f8eb43a39774d2c8e9cb4e88a405b0f P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 02:51:43.882688 2643 raft_consensus.cc:3054] T 9f8eb43a39774d2c8e9cb4e88a405b0f P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:43.885224 2643 raft_consensus.cc:513] T 9f8eb43a39774d2c8e9cb4e88a405b0f P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:43.886612 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f8eb43a39774d2c8e9cb4e88a405b0f" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f"
W20250113 02:51:43.887826 998 leader_election.cc:343] T 9f8eb43a39774d2c8e9cb4e88a405b0f P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:43.888017 2643 leader_election.cc:290] T 9f8eb43a39774d2c8e9cb4e88a405b0f P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Requested vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:43.889904 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f8eb43a39774d2c8e9cb4e88a405b0f" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196"
I20250113 02:51:43.890091 773 raft_consensus.cc:3054] T 9f8eb43a39774d2c8e9cb4e88a405b0f P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:43.892727 773 raft_consensus.cc:2463] T 9f8eb43a39774d2c8e9cb4e88a405b0f P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 1.
I20250113 02:51:43.893667 1198 ts_tablet_manager.cc:1918] T 0c408921a09a463eb464e08edcf042b3 P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:43.893852 1198 log.cc:1198] T 0c408921a09a463eb464e08edcf042b3 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/0c408921a09a463eb464e08edcf042b3
I20250113 02:51:43.894390 1198 ts_tablet_manager.cc:1939] T 0c408921a09a463eb464e08edcf042b3 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:43.894534 995 leader_election.cc:304] T 9f8eb43a39774d2c8e9cb4e88a405b0f P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters: df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:43.895064 2368 raft_consensus.cc:2798] T 9f8eb43a39774d2c8e9cb4e88a405b0f P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Leader election won for term 1
I20250113 02:51:43.895179 2368 raft_consensus.cc:695] T 9f8eb43a39774d2c8e9cb4e88a405b0f P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: Becoming Leader. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Running, Role: LEADER
I20250113 02:51:43.895298 455 catalog_manager.cc:5618] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095) does not have the latest schema for tablet c13e9a51c1e74b44b352bade78390ba5 (table default.Table_0468611ec68e47109f5357935e50874d [id=caf098e2b57849a997bd7c8eaf1f9326]). Expected version 1 got 0
I20250113 02:51:43.895390 2368 consensus_queue.cc:237] T 9f8eb43a39774d2c8e9cb4e88a405b0f P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:43.896136 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1369) took 1874 ms. Trace:
I20250113 02:51:43.898319 1194 tablet_bootstrap.cc:654] T b5c1882125b5491ea79a29a6dac6771e P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:43.896265 1198 rpcz_store.cc:276] 0113 02:51:42.021223 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:42.021279 (+ 56us) service_pool.cc:225] Handling call
0113 02:51:43.873786 (+1852507us) ts_tablet_manager.cc:1145] Deleting tablet 0c408921a09a463eb464e08edcf042b3
0113 02:51:43.873820 (+ 34us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:43.884499 (+ 10679us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:43.893517 (+ 9018us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:43.896097 (+ 2580us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1849417}
I20250113 02:51:43.903103 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet 0c408921a09a463eb464e08edcf042b3 (table default.Table_08def01561d44ec5a88f5b5a82d47e3b [id=c24af26d1a734152af598a5aa258cb06]) successfully deleted
I20250113 02:51:43.903391 1198 tablet_replica.cc:331] T 3d85539cf0074a03b167d3279e940079 P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:43.903383 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f8eb43a39774d2c8e9cb4e88a405b0f" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:43.903585 1198 raft_consensus.cc:2238] T 3d85539cf0074a03b167d3279e940079 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: Raft consensus shutting down.
I20250113 02:51:43.903890 1198 raft_consensus.cc:2267] T 3d85539cf0074a03b167d3279e940079 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Raft consensus is shut down!
W20250113 02:51:43.903963 998 leader_election.cc:343] T 9f8eb43a39774d2c8e9cb4e88a405b0f P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:43.904742 2547 leader_election.cc:290] T 9f8eb43a39774d2c8e9cb4e88a405b0f P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:43.905369 1198 ts_tablet_manager.cc:1905] T 3d85539cf0074a03b167d3279e940079 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:43.910773 2643 consensus_queue.cc:1035] T 6a4c6dc08df744e6856bdf99ab65c575 P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 02:51:43.916100 1198 ts_tablet_manager.cc:1918] T 3d85539cf0074a03b167d3279e940079 P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250113 02:51:43.916222 1198 log.cc:1198] T 3d85539cf0074a03b167d3279e940079 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/3d85539cf0074a03b167d3279e940079
W20250113 02:51:43.917726 998 consensus_peers.cc:487] T 7527e6e0b65e4a43b7f48899a34cdf7f P 4216a7f853d642b7bc3e5cff458f3aca -> Peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Couldn't send request to peer df69f48656b44bb890f76587ee69cc0f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7527e6e0b65e4a43b7f48899a34cdf7f. This is attempt 1: this message will repeat every 5th retry.
I20250113 02:51:43.920818 1198 ts_tablet_manager.cc:1939] T 3d85539cf0074a03b167d3279e940079 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:43.922369 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1370) took 1900 ms. Trace:
I20250113 02:51:43.922461 1198 rpcz_store.cc:276] 0113 02:51:42.021727 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:42.021787 (+ 60us) service_pool.cc:225] Handling call
0113 02:51:43.903342 (+1881555us) ts_tablet_manager.cc:1145] Deleting tablet 3d85539cf0074a03b167d3279e940079
0113 02:51:43.903352 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:43.912144 (+ 8792us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:43.915988 (+ 3844us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:43.922349 (+ 6361us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1878944}
I20250113 02:51:43.922885 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet 3d85539cf0074a03b167d3279e940079 (table default.Table_3a73b4672b154b99af1a24e962007ea0 [id=f742014cd0d44db99c4c1a031630d827]) successfully deleted
I20250113 02:51:43.924041 2547 raft_consensus.cc:491] T 4c759831773849f2bc52e484ffa0cf49 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:43.924141 1198 tablet_replica.cc:331] T 438604e8859f422f9caccb61c5b5b125 P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:43.924253 2547 raft_consensus.cc:513] T 4c759831773849f2bc52e484ffa0cf49 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:43.924503 1198 raft_consensus.cc:2238] T 438604e8859f422f9caccb61c5b5b125 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: Raft consensus shutting down.
I20250113 02:51:43.924752 1198 pending_rounds.cc:62] T 438604e8859f422f9caccb61c5b5b125 P 4216a7f853d642b7bc3e5cff458f3aca: Trying to abort 1 pending ops.
I20250113 02:51:43.924810 1198 pending_rounds.cc:69] T 438604e8859f422f9caccb61c5b5b125 P 4216a7f853d642b7bc3e5cff458f3aca: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113673533214212096 op_type: NO_OP noop_request { }
I20250113 02:51:43.924911 1198 raft_consensus.cc:2883] T 438604e8859f422f9caccb61c5b5b125 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250113 02:51:43.924991 1198 raft_consensus.cc:2267] T 438604e8859f422f9caccb61c5b5b125 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:43.925343 1195 ts_tablet_manager.cc:1918] T 82505f1389184d0ca12ec0e2d6084d22 P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250113 02:51:43.925514 1195 log.cc:1198] T 82505f1389184d0ca12ec0e2d6084d22 P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/82505f1389184d0ca12ec0e2d6084d22
I20250113 02:51:43.925535 1198 ts_tablet_manager.cc:1905] T 438604e8859f422f9caccb61c5b5b125 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:43.925928 1195 ts_tablet_manager.cc:1939] T 82505f1389184d0ca12ec0e2d6084d22 P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:43.926123 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c759831773849f2bc52e484ffa0cf49" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:43.926687 998 leader_election.cc:343] T 4c759831773849f2bc52e484ffa0cf49 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:43.926882 2547 leader_election.cc:290] T 4c759831773849f2bc52e484ffa0cf49 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:43.934235 1192 tablet_bootstrap.cc:492] T cf7b8bd6801743f49e9b9473747447c5 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:43.934437 1192 ts_tablet_manager.cc:1397] T cf7b8bd6801743f49e9b9473747447c5 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.131s user 0.013s sys 0.004s
I20250113 02:51:43.934345 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_f62a86c4159541ab9c2abe3d3492f790" } new_table_name: "default.Table_1ec47bba53df4a508134dbdc0e95b15b" modify_external_catalogs: true
I20250113 02:51:43.935302 1192 raft_consensus.cc:357] T cf7b8bd6801743f49e9b9473747447c5 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:43.935581 1192 raft_consensus.cc:383] T cf7b8bd6801743f49e9b9473747447c5 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:43.935642 1192 raft_consensus.cc:738] T cf7b8bd6801743f49e9b9473747447c5 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:43.935880 1192 consensus_queue.cc:260] T cf7b8bd6801743f49e9b9473747447c5 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:43.937063 1192 ts_tablet_manager.cc:1428] T cf7b8bd6801743f49e9b9473747447c5 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250113 02:51:43.937322 1192 tablet_bootstrap.cc:492] T e01e7f8db83d461bbfbb2e901419ce0b P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:43.937681 1198 ts_tablet_manager.cc:1918] T 438604e8859f422f9caccb61c5b5b125 P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250113 02:51:43.937767 1198 log.cc:1198] T 438604e8859f422f9caccb61c5b5b125 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/438604e8859f422f9caccb61c5b5b125
I20250113 02:51:43.938115 1198 ts_tablet_manager.cc:1939] T 438604e8859f422f9caccb61c5b5b125 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:43.938771 1192 tablet_bootstrap.cc:654] T e01e7f8db83d461bbfbb2e901419ce0b P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:43.941709 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 4a654e8085fd49729aef1d343ceaf789
I20250113 02:51:43.942430 883 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a654e8085fd49729aef1d343ceaf789 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:43 UTC) from {username='slave'} at 127.0.0.1:37366
W20250113 02:51:43.943154 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 4a654e8085fd49729aef1d343ceaf789 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4a654e8085fd49729aef1d343ceaf789 already in progress: creating tablet
I20250113 02:51:43.944141 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a654e8085fd49729aef1d343ceaf789 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:43 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:43.949162 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_9f1f5d4e13b44704a2e7edf981461d71"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:43.949893 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9f1f5d4e13b44704a2e7edf981461d71 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250113 02:51:43.955663 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c6d1dc31a8af4086ad069fba6fd22d26 consensus state has no leader
I20250113 02:51:43.958662 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1365) took 1939 ms. Trace:
I20250113 02:51:43.959024 2368 raft_consensus.cc:491] T 79aaab73797c474bae06831fa308b066 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:43.959231 2368 raft_consensus.cc:513] T 79aaab73797c474bae06831fa308b066 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:43.966051 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79aaab73797c474bae06831fa308b066" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:43.966681 998 leader_election.cc:343] T 79aaab73797c474bae06831fa308b066 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:43.967041 2368 leader_election.cc:290] T 79aaab73797c474bae06831fa308b066 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:43.969286 2368 raft_consensus.cc:491] T 67baaab1e36e415381579d7385d4ab5a P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:43.969506 2368 raft_consensus.cc:513] T 67baaab1e36e415381579d7385d4ab5a P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:43.972916 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1133) took 6060 ms. Trace:
I20250113 02:51:43.973060 1195 rpcz_store.cc:276] 0113 02:51:37.912848 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:37.912903 (+ 55us) service_pool.cc:225] Handling call
0113 02:51:43.666186 (+5753283us) ts_tablet_manager.cc:1145] Deleting tablet 82505f1389184d0ca12ec0e2d6084d22
0113 02:51:43.666196 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:43.922428 (+256232us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:43.925256 (+ 2828us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:43.972894 (+ 47638us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5753103}
I20250113 02:51:43.973654 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet 82505f1389184d0ca12ec0e2d6084d22 (table default.Table_855f1069eae9472998eb347f20b003b8 [id=210a731382514eb098fb9442e6c290ba]) successfully deleted
I20250113 02:51:43.973747 1195 tablet_replica.cc:331] T cfa963a6df094ce0b9c54408f75ca6b2 P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:43.973930 1195 raft_consensus.cc:2238] T cfa963a6df094ce0b9c54408f75ca6b2 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:43.974084 1195 raft_consensus.cc:2267] T cfa963a6df094ce0b9c54408f75ca6b2 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:43.974591 1195 ts_tablet_manager.cc:1905] T cfa963a6df094ce0b9c54408f75ca6b2 P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:43.927145 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c759831773849f2bc52e484ffa0cf49" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
I20250113 02:51:43.944020 736 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a654e8085fd49729aef1d343ceaf789 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:43 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:43.976373 772 raft_consensus.cc:2463] T 4c759831773849f2bc52e484ffa0cf49 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 0.
I20250113 02:51:43.948413 1191 tablet_bootstrap.cc:654] T f4379f7fc6b54e6785f27975d9030cce P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:43.951994 2501 raft_consensus.cc:491] T 88a776aa820c44b3a3854ca5022bd353 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:43.961143 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79aaab73797c474bae06831fa308b066" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
I20250113 02:51:43.976773 769 raft_consensus.cc:2463] T 79aaab73797c474bae06831fa308b066 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 0.
I20250113 02:51:43.977061 2501 raft_consensus.cc:513] T 88a776aa820c44b3a3854ca5022bd353 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:43.977737 2501 leader_election.cc:290] T 88a776aa820c44b3a3854ca5022bd353 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:43.978022 995 leader_election.cc:304] T 4c759831773849f2bc52e484ffa0cf49 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters: df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:43.978008 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88a776aa820c44b3a3854ca5022bd353" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:43.978480 995 leader_election.cc:304] T 79aaab73797c474bae06831fa308b066 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters: df69f48656b44bb890f76587ee69cc0f
W20250113 02:51:43.978637 711 leader_election.cc:343] T 88a776aa820c44b3a3854ca5022bd353 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:43.978813 2639 raft_consensus.cc:2798] T 4c759831773849f2bc52e484ffa0cf49 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 02:51:43.978879 2639 raft_consensus.cc:491] T 4c759831773849f2bc52e484ffa0cf49 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 02:51:43.978938 2639 raft_consensus.cc:3054] T 4c759831773849f2bc52e484ffa0cf49 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:43.979357 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88a776aa820c44b3a3854ca5022bd353" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
W20250113 02:51:43.979773 709 leader_election.cc:343] T 88a776aa820c44b3a3854ca5022bd353 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:43.979864 709 leader_election.cc:304] T 88a776aa820c44b3a3854ca5022bd353 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:43.980206 2501 raft_consensus.cc:2743] T 88a776aa820c44b3a3854ca5022bd353 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:43.981134 1194 tablet_bootstrap.cc:492] T b5c1882125b5491ea79a29a6dac6771e P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:43.981261 1194 ts_tablet_manager.cc:1397] T b5c1882125b5491ea79a29a6dac6771e P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.190s user 0.015s sys 0.026s
I20250113 02:51:43.981498 2639 raft_consensus.cc:513] T 4c759831773849f2bc52e484ffa0cf49 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:43.982035 1194 raft_consensus.cc:357] T b5c1882125b5491ea79a29a6dac6771e P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:43.982221 1194 raft_consensus.cc:383] T b5c1882125b5491ea79a29a6dac6771e P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:43.982249 2474 raft_consensus.cc:2798] T 79aaab73797c474bae06831fa308b066 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 02:51:43.982282 1194 raft_consensus.cc:738] T b5c1882125b5491ea79a29a6dac6771e P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:43.982314 2474 raft_consensus.cc:491] T 79aaab73797c474bae06831fa308b066 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 02:51:43.982374 2474 raft_consensus.cc:3054] T 79aaab73797c474bae06831fa308b066 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:43.982491 1194 consensus_queue.cc:260] T b5c1882125b5491ea79a29a6dac6771e P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:43.982952 1194 ts_tablet_manager.cc:1428] T b5c1882125b5491ea79a29a6dac6771e P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250113 02:51:43.984221 1194 tablet_bootstrap.cc:492] T eceed0f52c9f46bbaa78ca773a7afde5 P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:43.984624 2474 raft_consensus.cc:513] T 79aaab73797c474bae06831fa308b066 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:43.985178 2474 leader_election.cc:290] T 79aaab73797c474bae06831fa308b066 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Requested vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:43.985572 1192 tablet_bootstrap.cc:492] T e01e7f8db83d461bbfbb2e901419ce0b P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:43.985685 1192 ts_tablet_manager.cc:1397] T e01e7f8db83d461bbfbb2e901419ce0b P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.048s user 0.004s sys 0.003s
I20250113 02:51:43.986323 1192 raft_consensus.cc:357] T e01e7f8db83d461bbfbb2e901419ce0b P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:43.986462 1192 raft_consensus.cc:383] T e01e7f8db83d461bbfbb2e901419ce0b P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:43.986501 1192 raft_consensus.cc:738] T e01e7f8db83d461bbfbb2e901419ce0b P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:43.986644 1192 consensus_queue.cc:260] T e01e7f8db83d461bbfbb2e901419ce0b P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:43.987178 1192 ts_tablet_manager.cc:1428] T e01e7f8db83d461bbfbb2e901419ce0b P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20250113 02:51:43.987385 2368 leader_election.cc:290] T 67baaab1e36e415381579d7385d4ab5a P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:43.988543 2639 leader_election.cc:290] T 4c759831773849f2bc52e484ffa0cf49 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Requested vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:43.989287 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79aaab73797c474bae06831fa308b066" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f"
I20250113 02:51:43.970750 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67baaab1e36e415381579d7385d4ab5a" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
W20250113 02:51:43.990135 998 leader_election.cc:343] T 79aaab73797c474bae06831fa308b066 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:43.990208 773 raft_consensus.cc:2463] T 67baaab1e36e415381579d7385d4ab5a P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 0.
I20250113 02:51:43.990957 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c759831773849f2bc52e484ffa0cf49" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196"
I20250113 02:51:43.991189 769 raft_consensus.cc:3054] T 4c759831773849f2bc52e484ffa0cf49 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:43.992523 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c759831773849f2bc52e484ffa0cf49" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f"
W20250113 02:51:43.992996 998 leader_election.cc:343] T 4c759831773849f2bc52e484ffa0cf49 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:43.993777 769 raft_consensus.cc:2463] T 4c759831773849f2bc52e484ffa0cf49 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 1.
I20250113 02:51:43.994277 995 leader_election.cc:304] T 4c759831773849f2bc52e484ffa0cf49 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters: df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:43.994557 2363 raft_consensus.cc:2798] T 4c759831773849f2bc52e484ffa0cf49 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Leader election won for term 1
I20250113 02:51:43.994675 2363 raft_consensus.cc:695] T 4c759831773849f2bc52e484ffa0cf49 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: Becoming Leader. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Running, Role: LEADER
I20250113 02:51:43.994890 2363 consensus_queue.cc:237] T 4c759831773849f2bc52e484ffa0cf49 P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:43.995034 995 leader_election.cc:304] T 67baaab1e36e415381579d7385d4ab5a P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters:
I20250113 02:51:43.995368 2368 raft_consensus.cc:2798] T 67baaab1e36e415381579d7385d4ab5a P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 02:51:43.995430 2368 raft_consensus.cc:491] T 67baaab1e36e415381579d7385d4ab5a P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 02:51:43.995482 2368 raft_consensus.cc:3054] T 67baaab1e36e415381579d7385d4ab5a P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:43.998373 2368 raft_consensus.cc:513] T 67baaab1e36e415381579d7385d4ab5a P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:43.999404 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67baaab1e36e415381579d7385d4ab5a" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196"
I20250113 02:51:43.999600 769 raft_consensus.cc:3054] T 67baaab1e36e415381579d7385d4ab5a P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:44.000216 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67baaab1e36e415381579d7385d4ab5a" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:44.000730 998 leader_election.cc:343] T 67baaab1e36e415381579d7385d4ab5a P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:44.001979 1194 tablet_bootstrap.cc:654] T eceed0f52c9f46bbaa78ca773a7afde5 P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:44.002791 769 raft_consensus.cc:2463] T 67baaab1e36e415381579d7385d4ab5a P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 1.
I20250113 02:51:44.003309 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79aaab73797c474bae06831fa308b066" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196"
I20250113 02:51:44.003562 772 raft_consensus.cc:3054] T 79aaab73797c474bae06831fa308b066 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:44.004315 995 leader_election.cc:304] T 67baaab1e36e415381579d7385d4ab5a P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters:
I20250113 02:51:44.005231 2643 raft_consensus.cc:2798] T 67baaab1e36e415381579d7385d4ab5a P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Leader election won for term 1
I20250113 02:51:44.005573 2643 raft_consensus.cc:695] T 67baaab1e36e415381579d7385d4ab5a P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: Becoming Leader. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Running, Role: LEADER
I20250113 02:51:44.005837 2643 consensus_queue.cc:237] T 67baaab1e36e415381579d7385d4ab5a P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:44.006892 2501 raft_consensus.cc:491] T f2c6c8c17d6e499d84ad7c0c23889cc4 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:44.007016 2501 raft_consensus.cc:513] T f2c6c8c17d6e499d84ad7c0c23889cc4 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:44.007643 2501 leader_election.cc:290] T f2c6c8c17d6e499d84ad7c0c23889cc4 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:44.008030 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67baaab1e36e415381579d7385d4ab5a" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f"
I20250113 02:51:44.008291 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2c6c8c17d6e499d84ad7c0c23889cc4" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:44.008625 998 leader_election.cc:343] T 67baaab1e36e415381579d7385d4ab5a P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:44.008790 2368 leader_election.cc:290] T 67baaab1e36e415381579d7385d4ab5a P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Requested vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
W20250113 02:51:44.008865 711 leader_election.cc:343] T f2c6c8c17d6e499d84ad7c0c23889cc4 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:44.011210 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2c6c8c17d6e499d84ad7c0c23889cc4" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
W20250113 02:51:44.011804 709 leader_election.cc:343] T f2c6c8c17d6e499d84ad7c0c23889cc4 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:44.011926 709 leader_election.cc:304] T f2c6c8c17d6e499d84ad7c0c23889cc4 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:44.012625 2501 raft_consensus.cc:2743] T f2c6c8c17d6e499d84ad7c0c23889cc4 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:44.013262 1192 tablet_bootstrap.cc:492] T 8cc78bf7d6324b7ba10a1b1045923f2b P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:44.013866 772 raft_consensus.cc:2463] T 79aaab73797c474bae06831fa308b066 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 1.
I20250113 02:51:44.016074 1194 tablet_bootstrap.cc:492] T eceed0f52c9f46bbaa78ca773a7afde5 P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:44.017344 1192 tablet_bootstrap.cc:654] T 8cc78bf7d6324b7ba10a1b1045923f2b P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:44.014171 995 leader_election.cc:304] T 79aaab73797c474bae06831fa308b066 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters: df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:44.017845 2363 raft_consensus.cc:2798] T 79aaab73797c474bae06831fa308b066 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Leader election won for term 1
I20250113 02:51:44.018030 2363 raft_consensus.cc:695] T 79aaab73797c474bae06831fa308b066 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: Becoming Leader. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Running, Role: LEADER
I20250113 02:51:44.018786 1191 tablet_bootstrap.cc:492] T f4379f7fc6b54e6785f27975d9030cce P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:44.018961 1191 ts_tablet_manager.cc:1397] T f4379f7fc6b54e6785f27975d9030cce P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.162s user 0.001s sys 0.003s
I20250113 02:51:44.018919 2363 consensus_queue.cc:237] T 79aaab73797c474bae06831fa308b066 P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:44.019677 1191 raft_consensus.cc:357] T f4379f7fc6b54e6785f27975d9030cce P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:44.019856 1191 raft_consensus.cc:383] T f4379f7fc6b54e6785f27975d9030cce P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:44.020174 1191 raft_consensus.cc:738] T f4379f7fc6b54e6785f27975d9030cce P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:44.020460 1191 consensus_queue.cc:260] T f4379f7fc6b54e6785f27975d9030cce P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:44.021847 1191 ts_tablet_manager.cc:1428] T f4379f7fc6b54e6785f27975d9030cce P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20250113 02:51:44.023840 2501 raft_consensus.cc:491] T 58053845187f466599750f2a62f4c3e1 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:44.023957 2501 raft_consensus.cc:513] T 58053845187f466599750f2a62f4c3e1 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:44.024135 1194 ts_tablet_manager.cc:1397] T eceed0f52c9f46bbaa78ca773a7afde5 P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.040s user 0.002s sys 0.002s
I20250113 02:51:44.024502 2501 leader_election.cc:290] T 58053845187f466599750f2a62f4c3e1 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:44.024878 1194 raft_consensus.cc:357] T eceed0f52c9f46bbaa78ca773a7afde5 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:44.025071 1194 raft_consensus.cc:383] T eceed0f52c9f46bbaa78ca773a7afde5 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:44.025158 1194 raft_consensus.cc:738] T eceed0f52c9f46bbaa78ca773a7afde5 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:44.025324 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58053845187f466599750f2a62f4c3e1" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:44.025422 1194 consensus_queue.cc:260] T eceed0f52c9f46bbaa78ca773a7afde5 P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:44.025631 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58053845187f466599750f2a62f4c3e1" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
W20250113 02:51:44.025943 711 leader_election.cc:343] T 58053845187f466599750f2a62f4c3e1 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
W20250113 02:51:44.026429 709 leader_election.cc:343] T 58053845187f466599750f2a62f4c3e1 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:44.026670 709 leader_election.cc:304] T 58053845187f466599750f2a62f4c3e1 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:44.027017 2501 raft_consensus.cc:2743] T 58053845187f466599750f2a62f4c3e1 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:44.028425 1191 tablet_bootstrap.cc:492] T 883027967e0c442a91ac820210b9c6c7 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:44.029384 1194 ts_tablet_manager.cc:1428] T eceed0f52c9f46bbaa78ca773a7afde5 P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250113 02:51:44.030112 1194 tablet_bootstrap.cc:492] T 1014e51f1c184d729552a5b34292f232 P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:44.030167 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_fb2e08d7bab54382b45c211685c114db"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:44.030776 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fb2e08d7bab54382b45c211685c114db in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:44.032008 1191 tablet_bootstrap.cc:654] T 883027967e0c442a91ac820210b9c6c7 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:44.032063 1194 tablet_bootstrap.cc:654] T 1014e51f1c184d729552a5b34292f232 P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:44.032660 1192 tablet_bootstrap.cc:492] T 8cc78bf7d6324b7ba10a1b1045923f2b P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:44.032809 1192 ts_tablet_manager.cc:1397] T 8cc78bf7d6324b7ba10a1b1045923f2b P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.020s user 0.001s sys 0.001s
I20250113 02:51:44.033571 1192 raft_consensus.cc:357] T 8cc78bf7d6324b7ba10a1b1045923f2b P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:44.037330 1192 raft_consensus.cc:383] T 8cc78bf7d6324b7ba10a1b1045923f2b P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:44.037451 1192 raft_consensus.cc:738] T 8cc78bf7d6324b7ba10a1b1045923f2b P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:44.037696 1192 consensus_queue.cc:260] T 8cc78bf7d6324b7ba10a1b1045923f2b P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:44.038458 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_6d93783c61ff45a1a8908e77ca3e03a9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:44.039028 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6d93783c61ff45a1a8908e77ca3e03a9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:44.039112 1192 ts_tablet_manager.cc:1428] T 8cc78bf7d6324b7ba10a1b1045923f2b P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20250113 02:51:44.040292 1192 tablet_bootstrap.cc:492] T 0839a35fcab04f3c9d03e256553e01c8 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:43.958772 1198 rpcz_store.cc:276] 0113 02:51:42.019116 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:42.026077 (+ 6961us) service_pool.cc:225] Handling call
0113 02:51:43.924089 (+1898012us) ts_tablet_manager.cc:1145] Deleting tablet 438604e8859f422f9caccb61c5b5b125
0113 02:51:43.924100 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:43.928594 (+ 4494us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:43.937597 (+ 9003us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:43.958637 (+ 21040us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1897855}
I20250113 02:51:44.051888 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet 438604e8859f422f9caccb61c5b5b125 (table default.Table_2cb862a84f3f44c5bcfb646461a5af1c [id=c6e65e4bab5040cca1f0f55721639d2e]) successfully deleted
I20250113 02:51:44.052417 1198 tablet_replica.cc:331] T dc680b5680b84abbab219296bc7ee707 P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:44.052637 1198 raft_consensus.cc:2238] T dc680b5680b84abbab219296bc7ee707 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:44.052834 1198 raft_consensus.cc:2267] T dc680b5680b84abbab219296bc7ee707 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:44.053012 1192 tablet_bootstrap.cc:654] T 0839a35fcab04f3c9d03e256553e01c8 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:44.053149 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet ca144327ca8d48d48a072caaff74212f
I20250113 02:51:44.053925 883 tablet_service.cc:1514] Processing DeleteTablet for tablet ca144327ca8d48d48a072caaff74212f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:43 UTC) from {username='slave'} at 127.0.0.1:37366
W20250113 02:51:44.054610 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet cff2d65491ac47cea9aacd87e3a1f00d consensus state has no leader
I20250113 02:51:44.055385 736 tablet_service.cc:1514] Processing DeleteTablet for tablet ca144327ca8d48d48a072caaff74212f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:43 UTC) from {username='slave'} at 127.0.0.1:55398
W20250113 02:51:44.055809 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet ca144327ca8d48d48a072caaff74212f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ca144327ca8d48d48a072caaff74212f already in progress: creating tablet
I20250113 02:51:44.053462 1198 ts_tablet_manager.cc:1905] T dc680b5680b84abbab219296bc7ee707 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:44.054167 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet ca144327ca8d48d48a072caaff74212f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:43 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:44.066361 1191 tablet_bootstrap.cc:492] T 883027967e0c442a91ac820210b9c6c7 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:44.066633 1191 ts_tablet_manager.cc:1397] T 883027967e0c442a91ac820210b9c6c7 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.038s user 0.002s sys 0.002s
I20250113 02:51:44.067732 1191 raft_consensus.cc:357] T 883027967e0c442a91ac820210b9c6c7 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:44.068184 1191 raft_consensus.cc:383] T 883027967e0c442a91ac820210b9c6c7 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:44.068317 1191 raft_consensus.cc:738] T 883027967e0c442a91ac820210b9c6c7 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:44.068696 1191 consensus_queue.cc:260] T 883027967e0c442a91ac820210b9c6c7 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:44.069453 449 catalog_manager.cc:6071] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: default.Table_0ae4f97208974a97b60a2691c78ef43e [id=71508871e2d5469a91aaf0e19b02b638] alter complete (version 2)
I20250113 02:51:44.070816 883 tablet_service.cc:1514] Processing DeleteTablet for tablet b0113b16110e4787ac1c4676b2c8bd1e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:22 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:44.071033 891 tablet_service.cc:1514] Processing DeleteTablet for tablet c518f791b4d74c54ab0c0364d29b4a64 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:23 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:44.071180 885 tablet_service.cc:1514] Processing DeleteTablet for tablet c7842d6f2c074e3b9d1a3378db9856be with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:24 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:44.071401 1196 tablet_replica.cc:331] T b0113b16110e4787ac1c4676b2c8bd1e P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:44.071683 1196 raft_consensus.cc:2238] T b0113b16110e4787ac1c4676b2c8bd1e P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:44.071870 1196 raft_consensus.cc:2267] T b0113b16110e4787ac1c4676b2c8bd1e P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:44.072458 1196 ts_tablet_manager.cc:1905] T b0113b16110e4787ac1c4676b2c8bd1e P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:44.072703 1191 ts_tablet_manager.cc:1428] T 883027967e0c442a91ac820210b9c6c7 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20250113 02:51:44.073105 1191 tablet_bootstrap.cc:492] T 68c4a7e8c0fb4feb85498709b00d3258 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:44.083056 1191 tablet_bootstrap.cc:654] T 68c4a7e8c0fb4feb85498709b00d3258 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:44.092418 1196 ts_tablet_manager.cc:1918] T b0113b16110e4787ac1c4676b2c8bd1e P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:44.092669 1196 log.cc:1198] T b0113b16110e4787ac1c4676b2c8bd1e P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/b0113b16110e4787ac1c4676b2c8bd1e
I20250113 02:51:44.093837 1196 ts_tablet_manager.cc:1939] T b0113b16110e4787ac1c4676b2c8bd1e P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:44.095049 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet b0113b16110e4787ac1c4676b2c8bd1e (table default.Table_d5c2b805dfbf4227bb1ac190525b7dee [id=bf855203fc20403294cc733cdbc593e5]) successfully deleted
I20250113 02:51:44.095526 1196 tablet_replica.cc:331] T c518f791b4d74c54ab0c0364d29b4a64 P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:44.095774 1196 raft_consensus.cc:2238] T c518f791b4d74c54ab0c0364d29b4a64 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:44.096024 1196 raft_consensus.cc:2267] T c518f791b4d74c54ab0c0364d29b4a64 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:44.096824 1196 ts_tablet_manager.cc:1905] T c518f791b4d74c54ab0c0364d29b4a64 P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:44.098600 1198 ts_tablet_manager.cc:1918] T dc680b5680b84abbab219296bc7ee707 P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:44.098749 1198 log.cc:1198] T dc680b5680b84abbab219296bc7ee707 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/dc680b5680b84abbab219296bc7ee707
I20250113 02:51:44.099170 1198 ts_tablet_manager.cc:1939] T dc680b5680b84abbab219296bc7ee707 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:44.099437 1191 tablet_bootstrap.cc:492] T 68c4a7e8c0fb4feb85498709b00d3258 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:44.099584 1191 ts_tablet_manager.cc:1397] T 68c4a7e8c0fb4feb85498709b00d3258 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.027s user 0.003s sys 0.001s
I20250113 02:51:44.101104 1191 raft_consensus.cc:357] T 68c4a7e8c0fb4feb85498709b00d3258 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:44.104409 1191 raft_consensus.cc:383] T 68c4a7e8c0fb4feb85498709b00d3258 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:44.104526 1191 raft_consensus.cc:738] T 68c4a7e8c0fb4feb85498709b00d3258 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:44.104473 446 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_d65b7b94ce864ac6a57e6b151433a14d" } modify_external_catalogs: true
I20250113 02:51:44.104669 446 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_d65b7b94ce864ac6a57e6b151433a14d" } modify_external_catalogs: true
I20250113 02:51:44.104810 1191 consensus_queue.cc:260] T 68c4a7e8c0fb4feb85498709b00d3258 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:44.105424 1191 ts_tablet_manager.cc:1428] T 68c4a7e8c0fb4feb85498709b00d3258 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20250113 02:51:44.105737 1191 tablet_bootstrap.cc:492] T afcb5b1977c447d3a51de9cb72d7196a P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:44.106850 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_f410165a9c1c45229555f7b89641c3db" } new_table_name: "default.Table_0eba782647e84a61bda213d2ce05f0a4" modify_external_catalogs: true
I20250113 02:51:44.107012 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_f5b4d64c86db49829574aea1b88e3968" } new_table_name: "default.Table_168905c33c1e4fb18c72bce89c511483" modify_external_catalogs: true
I20250113 02:51:44.107806 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_28e6f68bf4304b12b107c8e71e66e1e3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:44.108314 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_28e6f68bf4304b12b107c8e71e66e1e3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:44.111101 1191 tablet_bootstrap.cc:654] T afcb5b1977c447d3a51de9cb72d7196a P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:44.117873 2363 consensus_queue.cc:1035] T 1968c7e9357c4c6e901d334ace0467fb P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 02:51:44.128916 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1364) took 2110 ms. Trace:
I20250113 02:51:44.130410 1198 rpcz_store.cc:276] 0113 02:51:42.018751 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:42.029025 (+ 10274us) service_pool.cc:225] Handling call
0113 02:51:44.052359 (+2023334us) ts_tablet_manager.cc:1145] Deleting tablet dc680b5680b84abbab219296bc7ee707
0113 02:51:44.052370 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:44.063448 (+ 11078us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:44.098475 (+ 35027us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:44.128859 (+ 30384us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2023153}
I20250113 02:51:44.131551 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet dc680b5680b84abbab219296bc7ee707 (table default.Table_25a2c280183748df92b064c40756f5ac [id=a1ac1dc79bce47258f1dbf33837b782c]) successfully deleted
I20250113 02:51:44.132182 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1381) took 1936 ms. Trace:
I20250113 02:51:44.132675 1198 rpcz_store.cc:276] 0113 02:51:42.195701 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:42.242962 (+ 47261us) service_pool.cc:225] Handling call
0113 02:51:44.131983 (+1889021us) ts_tablet_manager.cc:1145] Deleting tablet 1c153e47a7aa4bf5ae61dd19a91930b9
0113 02:51:44.131995 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:44.132162 (+ 167us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1884927}
W20250113 02:51:44.133852 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 1c153e47a7aa4bf5ae61dd19a91930b9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1c153e47a7aa4bf5ae61dd19a91930b9 already in progress: creating tablet
I20250113 02:51:44.134527 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1383) took 1592 ms. Trace:
I20250113 02:51:44.134778 2363 consensus_queue.cc:1035] T fd0b8592997d4a61b8ca854ecbb67c55 P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 02:51:44.137765 1198 rpcz_store.cc:276] 0113 02:51:42.541931 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:42.542334 (+ 403us) service_pool.cc:225] Handling call
0113 02:51:44.134409 (+1592075us) ts_tablet_manager.cc:1145] Deleting tablet 86c8c07f90d2470ebb2b3f1d16daf4cd
0113 02:51:44.134420 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:44.134509 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1591724}
I20250113 02:51:44.138427 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1386) took 1534 ms. Trace:
I20250113 02:51:44.138814 1198 rpcz_store.cc:276] 0113 02:51:42.603555 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:42.603613 (+ 58us) service_pool.cc:225] Handling call
0113 02:51:44.138303 (+1534690us) ts_tablet_manager.cc:1145] Deleting tablet 125ebde7a9f14918ba328a75e88a7746
0113 02:51:44.138314 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:44.138410 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1534512}
I20250113 02:51:44.139807 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1387) took 1534 ms. Trace:
W20250113 02:51:44.140146 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 86c8c07f90d2470ebb2b3f1d16daf4cd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 86c8c07f90d2470ebb2b3f1d16daf4cd already in progress: creating tablet
I20250113 02:51:44.140250 1198 rpcz_store.cc:276] 0113 02:51:42.604925 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:42.604988 (+ 63us) service_pool.cc:225] Handling call
0113 02:51:44.139671 (+1534683us) ts_tablet_manager.cc:1145] Deleting tablet 166f58d45b7b443ea54087e4ab48eaba
0113 02:51:44.139685 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:44.139787 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1534495}
W20250113 02:51:44.140466 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 125ebde7a9f14918ba328a75e88a7746 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 125ebde7a9f14918ba328a75e88a7746 already in progress: creating tablet
W20250113 02:51:44.140709 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 166f58d45b7b443ea54087e4ab48eaba because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 166f58d45b7b443ea54087e4ab48eaba already in progress: creating tablet
I20250113 02:51:44.141005 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1390) took 1417 ms. Trace:
I20250113 02:51:44.143950 1198 rpcz_store.cc:276] 0113 02:51:42.723793 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:42.723848 (+ 55us) service_pool.cc:225] Handling call
0113 02:51:44.140874 (+1417026us) ts_tablet_manager.cc:1145] Deleting tablet 495064a93dfb41ff999fc73b579baca0
0113 02:51:44.140887 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:44.140989 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1408619}
W20250113 02:51:44.144747 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 495064a93dfb41ff999fc73b579baca0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 495064a93dfb41ff999fc73b579baca0 already in progress: creating tablet
I20250113 02:51:44.145185 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1389) took 1421 ms. Trace:
I20250113 02:51:44.148108 1198 rpcz_store.cc:276] 0113 02:51:42.723651 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:42.723744 (+ 93us) service_pool.cc:225] Handling call
0113 02:51:44.145089 (+1421345us) ts_tablet_manager.cc:1145] Deleting tablet 4791a82b5e13429184d0c7192dd7aef6
0113 02:51:44.145100 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:44.145170 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1412684}
I20250113 02:51:44.150406 1194 tablet_bootstrap.cc:492] T 1014e51f1c184d729552a5b34292f232 P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:44.150629 1194 ts_tablet_manager.cc:1397] T 1014e51f1c184d729552a5b34292f232 P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.121s user 0.010s sys 0.001s
I20250113 02:51:44.155452 1194 raft_consensus.cc:357] T 1014e51f1c184d729552a5b34292f232 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:44.155799 1194 raft_consensus.cc:383] T 1014e51f1c184d729552a5b34292f232 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:44.155905 1194 raft_consensus.cc:738] T 1014e51f1c184d729552a5b34292f232 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:44.156273 1194 consensus_queue.cc:260] T 1014e51f1c184d729552a5b34292f232 P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
W20250113 02:51:44.165220 998 consensus_peers.cc:487] T fd0b8592997d4a61b8ca854ecbb67c55 P 4216a7f853d642b7bc3e5cff458f3aca -> Peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Couldn't send request to peer df69f48656b44bb890f76587ee69cc0f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250113 02:51:44.169674 1192 tablet_bootstrap.cc:492] T 0839a35fcab04f3c9d03e256553e01c8 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
W20250113 02:51:44.170564 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 4791a82b5e13429184d0c7192dd7aef6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4791a82b5e13429184d0c7192dd7aef6 already in progress: creating tablet
I20250113 02:51:44.170837 1192 ts_tablet_manager.cc:1397] T 0839a35fcab04f3c9d03e256553e01c8 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.131s user 0.023s sys 0.047s
I20250113 02:51:44.170960 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1391) took 1337 ms. Trace:
I20250113 02:51:44.169044 1194 ts_tablet_manager.cc:1428] T 1014e51f1c184d729552a5b34292f232 P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.014s user 0.001s sys 0.001s
W20250113 02:51:44.171801 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6a998cc32ff34ba190f4951851f46403 has no consensus state
I20250113 02:51:44.171461 455 catalog_manager.cc:6071] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: default.Table_9ca0c47692a840c39dbc1c73c7a6bc7c [id=7783df86a6ee404f87055036508adb20] alter complete (version 1)
I20250113 02:51:44.173303 450 master_service.cc:946] ReplaceTablet: received request to replace tablet 08180ab0274a40fd955dda5b08197a08 from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:44.174762 736 tablet_service.cc:1514] Processing DeleteTablet for tablet 569d666c128a45b7a526c13be5090ae3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:26 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:44.178728 739 tablet_service.cc:1514] Processing DeleteTablet for tablet 88a776aa820c44b3a3854ca5022bd353 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:31 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:44.182655 1195 ts_tablet_manager.cc:1918] T cfa963a6df094ce0b9c54408f75ca6b2 P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:44.183522 1195 log.cc:1198] T cfa963a6df094ce0b9c54408f75ca6b2 P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/cfa963a6df094ce0b9c54408f75ca6b2
I20250113 02:51:44.183702 1194 tablet_bootstrap.cc:492] T ee240d838d4a4bbba97f8ed61f69de78 P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:44.184115 1195 ts_tablet_manager.cc:1939] T cfa963a6df094ce0b9c54408f75ca6b2 P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:44.184031 1198 rpcz_store.cc:276] 0113 02:51:42.833529 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:42.834431 (+ 902us) service_pool.cc:225] Handling call
0113 02:51:44.170839 (+1336408us) ts_tablet_manager.cc:1145] Deleting tablet e3a1e65ec9874cc7b83485683aafff77
0113 02:51:44.170850 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:44.170945 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1336233}
W20250113 02:51:44.184477 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet e3a1e65ec9874cc7b83485683aafff77 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e3a1e65ec9874cc7b83485683aafff77 already in progress: creating tablet
I20250113 02:51:44.171902 1192 raft_consensus.cc:357] T 0839a35fcab04f3c9d03e256553e01c8 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:44.173897 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet cfa963a6df094ce0b9c54408f75ca6b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:37 UTC) from {username='slave'} at 127.0.0.1:60230
W20250113 02:51:44.174669 998 consensus_peers.cc:487] T 1968c7e9357c4c6e901d334ace0467fb P 4216a7f853d642b7bc3e5cff458f3aca -> Peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Couldn't send request to peer df69f48656b44bb890f76587ee69cc0f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250113 02:51:44.184741 1192 raft_consensus.cc:383] T 0839a35fcab04f3c9d03e256553e01c8 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:44.184805 1192 raft_consensus.cc:738] T 0839a35fcab04f3c9d03e256553e01c8 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:44.181135 2368 raft_consensus.cc:491] T fe8f075d4c9a4f9abe372bfdb81a829c P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:44.185032 2368 raft_consensus.cc:513] T fe8f075d4c9a4f9abe372bfdb81a829c P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:44.185573 2368 leader_election.cc:290] T fe8f075d4c9a4f9abe372bfdb81a829c P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:44.185834 1191 tablet_bootstrap.cc:492] T afcb5b1977c447d3a51de9cb72d7196a P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:44.185942 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1134) took 6201 ms. Trace:
I20250113 02:51:44.186625 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe8f075d4c9a4f9abe372bfdb81a829c" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:44.187112 998 leader_election.cc:343] T fe8f075d4c9a4f9abe372bfdb81a829c P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
W20250113 02:51:44.187572 995 leader_election.cc:343] T fe8f075d4c9a4f9abe372bfdb81a829c P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Not found: Tablet not found: fe8f075d4c9a4f9abe372bfdb81a829c
I20250113 02:51:44.187669 995 leader_election.cc:304] T fe8f075d4c9a4f9abe372bfdb81a829c P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4216a7f853d642b7bc3e5cff458f3aca; no voters: 1186fa869eb14812832fa6562531f196, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:44.184976 1192 consensus_queue.cc:260] T 0839a35fcab04f3c9d03e256553e01c8 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:44.188002 2368 raft_consensus.cc:2743] T fe8f075d4c9a4f9abe372bfdb81a829c P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:44.189574 1194 tablet_bootstrap.cc:654] T ee240d838d4a4bbba97f8ed61f69de78 P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:44.186013 1191 ts_tablet_manager.cc:1397] T afcb5b1977c447d3a51de9cb72d7196a P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.080s user 0.008s sys 0.008s
I20250113 02:51:44.189757 1195 rpcz_store.cc:276] 0113 02:51:37.984212 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:37.984483 (+ 271us) service_pool.cc:225] Handling call
0113 02:51:43.973699 (+5989216us) ts_tablet_manager.cc:1145] Deleting tablet cfa963a6df094ce0b9c54408f75ca6b2
0113 02:51:43.973709 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:44.016253 (+ 42544us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:44.150055 (+133802us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:44.185925 (+ 35870us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5988968}
I20250113 02:51:44.190567 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet cfa963a6df094ce0b9c54408f75ca6b2 (table default.Table_0874a35f3d5a44d2ac045a85227b148d [id=9dcef6fe27f247ff868d8e2ab2a37ead]) successfully deleted
I20250113 02:51:44.190627 1191 raft_consensus.cc:357] T afcb5b1977c447d3a51de9cb72d7196a P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:44.190841 1191 raft_consensus.cc:383] T afcb5b1977c447d3a51de9cb72d7196a P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:44.190935 1191 raft_consensus.cc:738] T afcb5b1977c447d3a51de9cb72d7196a P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:44.191134 1195 tablet_replica.cc:331] T 3d85539cf0074a03b167d3279e940079 P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:44.191190 1191 consensus_queue.cc:260] T afcb5b1977c447d3a51de9cb72d7196a P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:44.191565 1195 raft_consensus.cc:2238] T 3d85539cf0074a03b167d3279e940079 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:44.192215 1192 ts_tablet_manager.cc:1428] T 0839a35fcab04f3c9d03e256553e01c8 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.021s user 0.002s sys 0.000s
I20250113 02:51:44.192847 1192 tablet_bootstrap.cc:492] T f105f8b32a654e4ca154e8fc0cb96091 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:44.195093 1192 tablet_bootstrap.cc:654] T f105f8b32a654e4ca154e8fc0cb96091 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:44.195472 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 8f46d38f2c634b7d87df37e59ef4f5ef
I20250113 02:51:44.196307 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f46d38f2c634b7d87df37e59ef4f5ef with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:44 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:44.197609 885 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f46d38f2c634b7d87df37e59ef4f5ef with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:44 UTC) from {username='slave'} at 127.0.0.1:37366
W20250113 02:51:44.199658 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet bd48dbbfe20342569e6f153353f1ef9c consensus state has no leader
I20250113 02:51:44.202584 447 catalog_manager.cc:5618] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467) does not have the latest schema for tablet b5c1882125b5491ea79a29a6dac6771e (table default.Table_99031e0341fd4ad8a3bb04a6e222a1d0 [id=3237c1be7e1c48918ff095162d191dc1]). Expected version 3 got 0
I20250113 02:51:44.203701 455 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_a306bb67a3c44ddb86b707c1660cf369" } modify_external_catalogs: true
I20250113 02:51:44.203850 455 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_a306bb67a3c44ddb86b707c1660cf369" } modify_external_catalogs: true
I20250113 02:51:44.204140 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_d2fb23238827478fbe0d1afa409ba711"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:44.204689 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d2fb23238827478fbe0d1afa409ba711 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:44.207877 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1392) took 1344 ms. Trace:
I20250113 02:51:44.207989 1198 rpcz_store.cc:276] 0113 02:51:42.863860 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:42.863956 (+ 96us) service_pool.cc:225] Handling call
0113 02:51:44.207686 (+1343730us) ts_tablet_manager.cc:1145] Deleting tablet a446180f980640268170b4ab652d32cc
0113 02:51:44.207702 (+ 16us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:44.207848 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1343509}
W20250113 02:51:44.208583 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet a446180f980640268170b4ab652d32cc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a446180f980640268170b4ab652d32cc already in progress: creating tablet
I20250113 02:51:44.209601 449 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_a632f9b27ba74e3b947fd6371082dde3" } modify_external_catalogs: true
I20250113 02:51:44.209750 449 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_a632f9b27ba74e3b947fd6371082dde3" } modify_external_catalogs: true
I20250113 02:51:44.211647 891 tablet_service.cc:1514] Processing DeleteTablet for tablet bc533518faa24dffa3306ff092e8d507 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:24 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:44.211745 1198 tablet_replica.cc:331] T 0b119704bc774ca9b4155c9619d4765b P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:44.211849 885 tablet_service.cc:1514] Processing DeleteTablet for tablet eceed0f52c9f46bbaa78ca773a7afde5 with delete_type TABLET_DATA_DELETED (replaced by tablet 7d90e1689b164943b09f6cda5a6ae5cb) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:44.212091 1198 raft_consensus.cc:2238] T 0b119704bc774ca9b4155c9619d4765b P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: Raft consensus shutting down.
I20250113 02:51:44.212507 1198 raft_consensus.cc:2267] T 0b119704bc774ca9b4155c9619d4765b P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:44.213224 1198 ts_tablet_manager.cc:1905] T 0b119704bc774ca9b4155c9619d4765b P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:44.213408 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_e01a09e962074d5c87c15ad97b576d09"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:44.213907 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e01a09e962074d5c87c15ad97b576d09 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:44.217005 885 tablet_service.cc:1514] Processing DeleteTablet for tablet 1014e51f1c184d729552a5b34292f232 with delete_type TABLET_DATA_DELETED (replaced by tablet ec8196505dab4e05bb6487b566cf3144) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:44.218968 447 master_service.cc:946] ReplaceTablet: received request to replace tablet f3771f0fe7ff4d54b433640c76bc07ae from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:44.220544 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_f88977415dbf4f43abaa5fdfa7c46f4d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:44.221400 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f88977415dbf4f43abaa5fdfa7c46f4d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:44.237116 1192 tablet_bootstrap.cc:492] T f105f8b32a654e4ca154e8fc0cb96091 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:44.237329 1192 ts_tablet_manager.cc:1397] T f105f8b32a654e4ca154e8fc0cb96091 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.045s user 0.004s sys 0.000s
I20250113 02:51:44.237242 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_a4837fc75b6f4d2c8756d0967cce0b31"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:44.237867 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a4837fc75b6f4d2c8756d0967cce0b31 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:44.238096 1192 raft_consensus.cc:357] T f105f8b32a654e4ca154e8fc0cb96091 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:44.238269 1192 raft_consensus.cc:383] T f105f8b32a654e4ca154e8fc0cb96091 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:44.238312 1192 raft_consensus.cc:738] T f105f8b32a654e4ca154e8fc0cb96091 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:44.238503 1192 consensus_queue.cc:260] T f105f8b32a654e4ca154e8fc0cb96091 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:44.240885 1194 tablet_bootstrap.cc:492] T ee240d838d4a4bbba97f8ed61f69de78 P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:44.241068 1194 ts_tablet_manager.cc:1397] T ee240d838d4a4bbba97f8ed61f69de78 P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.057s user 0.007s sys 0.028s
I20250113 02:51:44.241911 1194 raft_consensus.cc:357] T ee240d838d4a4bbba97f8ed61f69de78 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:44.242184 1194 raft_consensus.cc:383] T ee240d838d4a4bbba97f8ed61f69de78 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:44.242254 1194 raft_consensus.cc:738] T ee240d838d4a4bbba97f8ed61f69de78 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:44.242465 1194 consensus_queue.cc:260] T ee240d838d4a4bbba97f8ed61f69de78 P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:44.242596 1192 ts_tablet_manager.cc:1428] T f105f8b32a654e4ca154e8fc0cb96091 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20250113 02:51:44.243692 1194 ts_tablet_manager.cc:1428] T ee240d838d4a4bbba97f8ed61f69de78 P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
W20250113 02:51:44.246564 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 59a447c6d0264ec38390fe688bc9555a consensus state has no leader
I20250113 02:51:44.247056 1192 tablet_bootstrap.cc:492] T 16a132384b0e4264a73d09a3ea133893 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:44.248760 1192 tablet_bootstrap.cc:654] T 16a132384b0e4264a73d09a3ea133893 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:44.252081 453 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_2aa1258a7cff4416b0682966ee8150e9" } modify_external_catalogs: true
I20250113 02:51:44.252238 453 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_2aa1258a7cff4416b0682966ee8150e9" } modify_external_catalogs: true
I20250113 02:51:44.253259 885 tablet_service.cc:1514] Processing DeleteTablet for tablet ee240d838d4a4bbba97f8ed61f69de78 with delete_type TABLET_DATA_DELETED (replaced by tablet 95e47db8a4d74615800ba6c049ec7131) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:44.255882 1194 tablet_bootstrap.cc:492] T 0644f5600c5b47e29f3fa73060ed96e3 P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:44.258560 1194 tablet_bootstrap.cc:654] T 0644f5600c5b47e29f3fa73060ed96e3 P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:44.259347 2476 raft_consensus.cc:491] T eceed0f52c9f46bbaa78ca773a7afde5 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:44.259476 2476 raft_consensus.cc:513] T eceed0f52c9f46bbaa78ca773a7afde5 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:44.260046 2476 leader_election.cc:290] T eceed0f52c9f46bbaa78ca773a7afde5 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
W20250113 02:51:44.261035 852 leader_election.cc:343] T eceed0f52c9f46bbaa78ca773a7afde5 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Not found: Tablet not found: eceed0f52c9f46bbaa78ca773a7afde5
I20250113 02:51:44.261229 2476 raft_consensus.cc:491] T c7842d6f2c074e3b9d1a3378db9856be P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:44.261340 2476 raft_consensus.cc:513] T c7842d6f2c074e3b9d1a3378db9856be P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:44.261518 1198 ts_tablet_manager.cc:1918] T 0b119704bc774ca9b4155c9619d4765b P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250113 02:51:44.261618 1198 log.cc:1198] T 0b119704bc774ca9b4155c9619d4765b P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/0b119704bc774ca9b4155c9619d4765b
I20250113 02:51:44.261894 2476 leader_election.cc:290] T c7842d6f2c074e3b9d1a3378db9856be P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:44.261956 1198 ts_tablet_manager.cc:1939] T 0b119704bc774ca9b4155c9619d4765b P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
W20250113 02:51:44.263048 851 leader_election.cc:343] T c7842d6f2c074e3b9d1a3378db9856be P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Not found: Tablet not found: c7842d6f2c074e3b9d1a3378db9856be
W20250113 02:51:44.263447 852 leader_election.cc:343] T c7842d6f2c074e3b9d1a3378db9856be P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Not found: Tablet not found: c7842d6f2c074e3b9d1a3378db9856be
I20250113 02:51:44.263557 852 leader_election.cc:304] T c7842d6f2c074e3b9d1a3378db9856be P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df69f48656b44bb890f76587ee69cc0f; no voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca
I20250113 02:51:44.264127 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet f3f6113ee3d94bb98dbe95d21b1478ef
I20250113 02:51:44.264269 2476 raft_consensus.cc:2743] T c7842d6f2c074e3b9d1a3378db9856be P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:44.266417 885 tablet_service.cc:1514] Processing DeleteTablet for tablet f3f6113ee3d94bb98dbe95d21b1478ef with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:44 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:44.267206 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet f3f6113ee3d94bb98dbe95d21b1478ef with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:44 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:44.267827 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1393) took 1370 ms. Trace:
I20250113 02:51:44.267944 1198 rpcz_store.cc:276] 0113 02:51:42.897199 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:42.897249 (+ 50us) service_pool.cc:225] Handling call
0113 02:51:44.211671 (+1314422us) ts_tablet_manager.cc:1145] Deleting tablet 0b119704bc774ca9b4155c9619d4765b
0113 02:51:44.211686 (+ 15us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:44.227628 (+ 15942us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:44.261394 (+ 33766us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:44.267806 (+ 6412us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1314231}
I20250113 02:51:44.268586 1192 tablet_bootstrap.cc:492] T 16a132384b0e4264a73d09a3ea133893 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:44.268628 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet 0b119704bc774ca9b4155c9619d4765b (table default.Table_b95a9b1d12284c9aac701a8ef9c93482 [id=843aabcff7074acd98e5801b21282faa]) successfully deleted
I20250113 02:51:44.268718 1192 ts_tablet_manager.cc:1397] T 16a132384b0e4264a73d09a3ea133893 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.022s user 0.004s sys 0.000s
I20250113 02:51:44.268754 1198 tablet_replica.cc:331] T 8337f3c82d654ecdac2cbc1ffa88ef05 P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:44.268968 1198 raft_consensus.cc:2238] T 8337f3c82d654ecdac2cbc1ffa88ef05 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: Raft consensus shutting down.
I20250113 02:51:44.269300 1198 raft_consensus.cc:2267] T 8337f3c82d654ecdac2cbc1ffa88ef05 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:44.191854 1191 ts_tablet_manager.cc:1428] T afcb5b1977c447d3a51de9cb72d7196a P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250113 02:51:44.198170 739 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f46d38f2c634b7d87df37e59ef4f5ef with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:44 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:44.260911 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eceed0f52c9f46bbaa78ca773a7afde5" candidate_uuid: "df69f48656b44bb890f76587ee69cc0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
I20250113 02:51:44.265479 736 tablet_service.cc:1514] Processing DeleteTablet for tablet f3f6113ee3d94bb98dbe95d21b1478ef with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:44 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:44.270007 1195 raft_consensus.cc:2267] T 3d85539cf0074a03b167d3279e940079 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:44.269636 1192 raft_consensus.cc:357] T 16a132384b0e4264a73d09a3ea133893 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:44.270190 1191 tablet_bootstrap.cc:492] T c04be975c8c049b892b16129ed47a142 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:44.270220 1192 raft_consensus.cc:383] T 16a132384b0e4264a73d09a3ea133893 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:44.270291 1192 raft_consensus.cc:738] T 16a132384b0e4264a73d09a3ea133893 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:44.270522 1192 consensus_queue.cc:260] T 16a132384b0e4264a73d09a3ea133893 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:44.270774 851 leader_election.cc:304] T eceed0f52c9f46bbaa78ca773a7afde5 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df69f48656b44bb890f76587ee69cc0f; no voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca
I20250113 02:51:44.270947 1195 ts_tablet_manager.cc:1905] T 3d85539cf0074a03b167d3279e940079 P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:44.271072 1198 ts_tablet_manager.cc:1905] T 8337f3c82d654ecdac2cbc1ffa88ef05 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:44.271304 2476 raft_consensus.cc:2743] T eceed0f52c9f46bbaa78ca773a7afde5 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:44.271832 1191 tablet_bootstrap.cc:654] T c04be975c8c049b892b16129ed47a142 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:44.281893 1192 ts_tablet_manager.cc:1428] T 16a132384b0e4264a73d09a3ea133893 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20250113 02:51:44.286396 1196 ts_tablet_manager.cc:1918] T c518f791b4d74c54ab0c0364d29b4a64 P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:44.286638 1196 log.cc:1198] T c518f791b4d74c54ab0c0364d29b4a64 P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/c518f791b4d74c54ab0c0364d29b4a64
I20250113 02:51:44.287127 1196 ts_tablet_manager.cc:1939] T c518f791b4d74c54ab0c0364d29b4a64 P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:44.287679 1196 tablet_replica.cc:331] T c7842d6f2c074e3b9d1a3378db9856be P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:44.287768 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet c518f791b4d74c54ab0c0364d29b4a64 (table default.Table_157d1350cce74502a990462ac9bd0e5f [id=f96a39bfcdf442f7965a7b16d9aa432f]) successfully deleted
W20250113 02:51:44.290726 998 consensus_peers.cc:487] T 283fc7f1c74e480c8a63c74d3ae9eb7d P 4216a7f853d642b7bc3e5cff458f3aca -> Peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Couldn't send request to peer df69f48656b44bb890f76587ee69cc0f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250113 02:51:44.291445 1192 tablet_bootstrap.cc:492] T 88a776aa820c44b3a3854ca5022bd353 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:44.293188 1192 tablet_bootstrap.cc:654] T 88a776aa820c44b3a3854ca5022bd353 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:44.298056 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_5a9dd13ac839443f862131a444ab842c" } new_table_name: "default.Table_70b5a5b3cec84af19415f09e22c1718c" modify_external_catalogs: true
I20250113 02:51:44.300114 1198 ts_tablet_manager.cc:1918] T 8337f3c82d654ecdac2cbc1ffa88ef05 P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250113 02:51:44.300693 1198 log.cc:1198] T 8337f3c82d654ecdac2cbc1ffa88ef05 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/8337f3c82d654ecdac2cbc1ffa88ef05
I20250113 02:51:44.301247 1198 ts_tablet_manager.cc:1939] T 8337f3c82d654ecdac2cbc1ffa88ef05 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:44.302536 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1395) took 1351 ms. Trace:
I20250113 02:51:44.303512 1198 rpcz_store.cc:276] 0113 02:51:42.951248 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:42.951513 (+ 265us) service_pool.cc:225] Handling call
0113 02:51:44.268704 (+1317191us) ts_tablet_manager.cc:1145] Deleting tablet 8337f3c82d654ecdac2cbc1ffa88ef05
0113 02:51:44.268714 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:44.294665 (+ 25951us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:44.299937 (+ 5272us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:44.302516 (+ 2579us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1316970}
I20250113 02:51:44.304157 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet 8337f3c82d654ecdac2cbc1ffa88ef05 (table default.Table_37d63a1b8622483a855edf2ce71c8d71 [id=885fa0a4645f4211bd874149829284c4]) successfully deleted
I20250113 02:51:44.305145 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1398) took 1212 ms. Trace:
I20250113 02:51:44.305502 1198 rpcz_store.cc:276] 0113 02:51:43.093049 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:43.093221 (+ 172us) service_pool.cc:225] Handling call
0113 02:51:44.305028 (+1211807us) ts_tablet_manager.cc:1145] Deleting tablet 1c3c3b7cf89e4f22b8a5dd7ba4f8f5ec
0113 02:51:44.305039 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:44.305130 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1211611}
W20250113 02:51:44.306035 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 1c3c3b7cf89e4f22b8a5dd7ba4f8f5ec because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1c3c3b7cf89e4f22b8a5dd7ba4f8f5ec already in progress: creating tablet
I20250113 02:51:44.306087 454 master_service.cc:946] ReplaceTablet: received request to replace tablet 7d0d6f8669dd41e39d402ad1d237418d from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:44.306401 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1399) took 1213 ms. Trace:
I20250113 02:51:44.306773 1198 rpcz_store.cc:276] 0113 02:51:43.093165 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:43.093213 (+ 48us) service_pool.cc:225] Handling call
0113 02:51:44.306304 (+1213091us) ts_tablet_manager.cc:1145] Deleting tablet d1f30873a02f4b61ac6940e44c332ad3
0113 02:51:44.306312 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:44.306389 (+ 77us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1212776}
W20250113 02:51:44.307296 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet d1f30873a02f4b61ac6940e44c332ad3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d1f30873a02f4b61ac6940e44c332ad3 already in progress: creating tablet
I20250113 02:51:44.307863 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1400) took 1213 ms. Trace:
I20250113 02:51:44.308164 1198 rpcz_store.cc:276] 0113 02:51:43.093923 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:43.093969 (+ 46us) service_pool.cc:225] Handling call
0113 02:51:44.307776 (+1213807us) ts_tablet_manager.cc:1145] Deleting tablet f86318637d5b4e378ce47e89a5d52b07
0113 02:51:44.307784 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:44.307849 (+ 65us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1213642}
W20250113 02:51:44.308871 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet f86318637d5b4e378ce47e89a5d52b07 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f86318637d5b4e378ce47e89a5d52b07 already in progress: creating tablet
I20250113 02:51:44.309302 1198 tablet_replica.cc:331] T aa1ebdb92a5f4c92a5195b614689c100 P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:44.309790 1198 raft_consensus.cc:2238] T aa1ebdb92a5f4c92a5195b614689c100 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: Raft consensus shutting down.
I20250113 02:51:44.310387 1198 raft_consensus.cc:2267] T aa1ebdb92a5f4c92a5195b614689c100 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:44.311575 1198 ts_tablet_manager.cc:1905] T aa1ebdb92a5f4c92a5195b614689c100 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:44.313364 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet ff645eb0f8fa4744b7ba5cd03ce21729
I20250113 02:51:44.315389 736 tablet_service.cc:1514] Processing DeleteTablet for tablet ff645eb0f8fa4744b7ba5cd03ce21729 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:44 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:44.316108 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet ff645eb0f8fa4744b7ba5cd03ce21729 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:44 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:44.317970 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_d7c36054e0c84d26a64741fa4e91cd8e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:44.318468 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d7c36054e0c84d26a64741fa4e91cd8e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:44.321107 1198 ts_tablet_manager.cc:1918] T aa1ebdb92a5f4c92a5195b614689c100 P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250113 02:51:44.324573 1198 log.cc:1198] T aa1ebdb92a5f4c92a5195b614689c100 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/aa1ebdb92a5f4c92a5195b614689c100
I20250113 02:51:44.325186 1198 ts_tablet_manager.cc:1939] T aa1ebdb92a5f4c92a5195b614689c100 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:44.326673 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1402) took 1231 ms. Trace:
I20250113 02:51:44.327950 1198 rpcz_store.cc:276] 0113 02:51:43.094946 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:43.094997 (+ 51us) service_pool.cc:225] Handling call
0113 02:51:44.309244 (+1214247us) ts_tablet_manager.cc:1145] Deleting tablet aa1ebdb92a5f4c92a5195b614689c100
0113 02:51:44.309257 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:44.318098 (+ 8841us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:44.321027 (+ 2929us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:44.326645 (+ 5618us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1214061}
I20250113 02:51:44.328733 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet aa1ebdb92a5f4c92a5195b614689c100 (table default.Table_74ce8ff3e2c64e088ac82c48ec722b39 [id=4a5048f6224642deafdc335e17679294]) successfully deleted
I20250113 02:51:44.329160 1198 tablet_replica.cc:331] T 6a4c6dc08df744e6856bdf99ab65c575 P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:44.329546 1198 raft_consensus.cc:2238] T 6a4c6dc08df744e6856bdf99ab65c575 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: Raft consensus shutting down.
I20250113 02:51:44.330052 1198 raft_consensus.cc:2267] T 6a4c6dc08df744e6856bdf99ab65c575 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:44.331046 1198 ts_tablet_manager.cc:1905] T 6a4c6dc08df744e6856bdf99ab65c575 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:44.331249 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_2d36ffc5be5043fcb20b054d5283120d" } modify_external_catalogs: true
I20250113 02:51:44.331462 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_2d36ffc5be5043fcb20b054d5283120d" } modify_external_catalogs: true
I20250113 02:51:44.334415 1191 tablet_bootstrap.cc:492] T c04be975c8c049b892b16129ed47a142 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:44.334569 1191 ts_tablet_manager.cc:1397] T c04be975c8c049b892b16129ed47a142 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.064s user 0.010s sys 0.002s
I20250113 02:51:44.335368 1191 raft_consensus.cc:357] T c04be975c8c049b892b16129ed47a142 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:44.335582 1191 raft_consensus.cc:383] T c04be975c8c049b892b16129ed47a142 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:44.335659 1191 raft_consensus.cc:738] T c04be975c8c049b892b16129ed47a142 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:44.335925 1191 consensus_queue.cc:260] T c04be975c8c049b892b16129ed47a142 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:44.339836 1191 ts_tablet_manager.cc:1428] T c04be975c8c049b892b16129ed47a142 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250113 02:51:44.340233 1191 tablet_bootstrap.cc:492] T 01a455eda0ee488fb3d45a6a09c43833 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
W20250113 02:51:44.341800 441 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet be5d6907a02f41be8eab71de051a6d3b (table default.Table_d6208e261d8b4ccaae513508a6ff1e72 [id=3e6a7094710a444aac74e1919ba52ac8], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet be5d6907a02f41be8eab71de051a6d3b consensus state has no leader
I20250113 02:51:44.341877 1191 tablet_bootstrap.cc:654] T 01a455eda0ee488fb3d45a6a09c43833 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:44.344296 1198 ts_tablet_manager.cc:1918] T 6a4c6dc08df744e6856bdf99ab65c575 P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250113 02:51:44.315101 885 tablet_service.cc:1514] Processing DeleteTablet for tablet ff645eb0f8fa4744b7ba5cd03ce21729 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:44 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:44.331701 1194 tablet_bootstrap.cc:492] T 0644f5600c5b47e29f3fa73060ed96e3 P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:44.344641 1194 ts_tablet_manager.cc:1397] T 0644f5600c5b47e29f3fa73060ed96e3 P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.089s user 0.012s sys 0.009s
I20250113 02:51:44.344789 1196 raft_consensus.cc:2238] T c7842d6f2c074e3b9d1a3378db9856be P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:44.344967 1196 raft_consensus.cc:2267] T c7842d6f2c074e3b9d1a3378db9856be P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:44.345233 1198 log.cc:1198] T 6a4c6dc08df744e6856bdf99ab65c575 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/6a4c6dc08df744e6856bdf99ab65c575
I20250113 02:51:44.345489 1196 ts_tablet_manager.cc:1905] T c7842d6f2c074e3b9d1a3378db9856be P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:44.345595 1198 ts_tablet_manager.cc:1939] T 6a4c6dc08df744e6856bdf99ab65c575 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:44.345575 1194 raft_consensus.cc:357] T 0644f5600c5b47e29f3fa73060ed96e3 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:44.345767 1194 raft_consensus.cc:383] T 0644f5600c5b47e29f3fa73060ed96e3 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:44.345815 1194 raft_consensus.cc:738] T 0644f5600c5b47e29f3fa73060ed96e3 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:44.346001 1194 consensus_queue.cc:260] T 0644f5600c5b47e29f3fa73060ed96e3 P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:44.346345 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1401) took 1251 ms. Trace:
I20250113 02:51:44.346473 1194 ts_tablet_manager.cc:1428] T 0644f5600c5b47e29f3fa73060ed96e3 P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250113 02:51:44.346431 1198 rpcz_store.cc:276] 0113 02:51:43.094821 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:43.095240 (+ 419us) service_pool.cc:225] Handling call
0113 02:51:44.329096 (+1233856us) ts_tablet_manager.cc:1145] Deleting tablet 6a4c6dc08df744e6856bdf99ab65c575
0113 02:51:44.329107 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:44.335603 (+ 6496us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:44.344200 (+ 8597us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:44.346313 (+ 2113us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1233712}
I20250113 02:51:44.346731 1194 tablet_bootstrap.cc:492] T bc6a7322c0c34adb85457750526d8955 P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:44.346848 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet 6a4c6dc08df744e6856bdf99ab65c575 (table default.Table_82a014a9f48c43b4b14d63522bac7124 [id=c489a295f02b4f00ac7ea90e62d8ed04]) successfully deleted
I20250113 02:51:44.349254 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1407) took 1161 ms. Trace:
I20250113 02:51:44.349352 1198 rpcz_store.cc:276] 0113 02:51:43.187642 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:43.188110 (+ 468us) service_pool.cc:225] Handling call
0113 02:51:44.349121 (+1161011us) ts_tablet_manager.cc:1145] Deleting tablet ef078624960843eead2edd1e39b10118
0113 02:51:44.349131 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:44.349234 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1160835}
W20250113 02:51:44.349788 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet ef078624960843eead2edd1e39b10118 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ef078624960843eead2edd1e39b10118 already in progress: creating tablet
I20250113 02:51:44.351254 1191 tablet_bootstrap.cc:492] T 01a455eda0ee488fb3d45a6a09c43833 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:44.351395 1191 ts_tablet_manager.cc:1397] T 01a455eda0ee488fb3d45a6a09c43833 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.004s
I20250113 02:51:44.352237 1191 raft_consensus.cc:357] T 01a455eda0ee488fb3d45a6a09c43833 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:44.352438 1191 raft_consensus.cc:383] T 01a455eda0ee488fb3d45a6a09c43833 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:44.352511 1191 raft_consensus.cc:738] T 01a455eda0ee488fb3d45a6a09c43833 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:44.352734 1191 consensus_queue.cc:260] T 01a455eda0ee488fb3d45a6a09c43833 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:44.353235 1191 ts_tablet_manager.cc:1428] T 01a455eda0ee488fb3d45a6a09c43833 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250113 02:51:44.353521 1191 tablet_bootstrap.cc:492] T 98bfe9620e284ab1954433ba9a036392 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:44.355201 1191 tablet_bootstrap.cc:654] T 98bfe9620e284ab1954433ba9a036392 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:44.357137 1194 tablet_bootstrap.cc:654] T bc6a7322c0c34adb85457750526d8955 P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:44.362489 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_6ea04b79ee2c446c986176fbd2cad259"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:44.363181 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6ea04b79ee2c446c986176fbd2cad259 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:44.363734 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1406) took 1176 ms. Trace:
I20250113 02:51:44.363850 1198 rpcz_store.cc:276] 0113 02:51:43.187549 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:43.189328 (+ 1779us) service_pool.cc:225] Handling call
0113 02:51:44.363552 (+1174224us) ts_tablet_manager.cc:1145] Deleting tablet 7da6f8a4b89647968609f73f9ad5ba14
0113 02:51:44.363569 (+ 17us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:44.363705 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1174010}
W20250113 02:51:44.367086 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 7da6f8a4b89647968609f73f9ad5ba14 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7da6f8a4b89647968609f73f9ad5ba14 already in progress: creating tablet
I20250113 02:51:44.367509 1198 tablet_replica.cc:331] T 7527e6e0b65e4a43b7f48899a34cdf7f P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:44.368310 1198 raft_consensus.cc:2238] T 7527e6e0b65e4a43b7f48899a34cdf7f P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: Raft consensus shutting down.
I20250113 02:51:44.370112 1191 tablet_bootstrap.cc:492] T 98bfe9620e284ab1954433ba9a036392 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:44.370283 1191 ts_tablet_manager.cc:1397] T 98bfe9620e284ab1954433ba9a036392 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.002s
I20250113 02:51:44.371003 1191 raft_consensus.cc:357] T 98bfe9620e284ab1954433ba9a036392 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:44.371237 1191 raft_consensus.cc:383] T 98bfe9620e284ab1954433ba9a036392 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:44.371335 1191 raft_consensus.cc:738] T 98bfe9620e284ab1954433ba9a036392 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:44.371575 1191 consensus_queue.cc:260] T 98bfe9620e284ab1954433ba9a036392 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:44.372326 1191 ts_tablet_manager.cc:1428] T 98bfe9620e284ab1954433ba9a036392 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250113 02:51:44.372685 1194 tablet_bootstrap.cc:492] T bc6a7322c0c34adb85457750526d8955 P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:44.372844 1194 ts_tablet_manager.cc:1397] T bc6a7322c0c34adb85457750526d8955 P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.026s user 0.002s sys 0.005s
I20250113 02:51:44.373416 2368 raft_consensus.cc:491] T ca75e7cae4f6423191a17bb5a8f150a9 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:44.373548 2368 raft_consensus.cc:513] T ca75e7cae4f6423191a17bb5a8f150a9 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:44.374187 2368 leader_election.cc:290] T ca75e7cae4f6423191a17bb5a8f150a9 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:44.374752 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca75e7cae4f6423191a17bb5a8f150a9" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:44.375162 1191 tablet_bootstrap.cc:492] T 415a1fa4b2f14cacb43937c8a372849a P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:44.375531 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca75e7cae4f6423191a17bb5a8f150a9" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
I20250113 02:51:44.375735 772 raft_consensus.cc:2463] T ca75e7cae4f6423191a17bb5a8f150a9 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 0.
W20250113 02:51:44.376399 998 leader_election.cc:343] T ca75e7cae4f6423191a17bb5a8f150a9 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:44.376469 2501 raft_consensus.cc:491] T afcb5b1977c447d3a51de9cb72d7196a P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:44.376608 2501 raft_consensus.cc:513] T afcb5b1977c447d3a51de9cb72d7196a P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:44.376888 1191 tablet_bootstrap.cc:654] T 415a1fa4b2f14cacb43937c8a372849a P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:44.376854 995 leader_election.cc:304] T ca75e7cae4f6423191a17bb5a8f150a9 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters: df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:44.377236 2501 leader_election.cc:290] T afcb5b1977c447d3a51de9cb72d7196a P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:44.377562 2368 raft_consensus.cc:2798] T ca75e7cae4f6423191a17bb5a8f150a9 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 02:51:44.377629 2368 raft_consensus.cc:491] T ca75e7cae4f6423191a17bb5a8f150a9 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 02:51:44.377689 2368 raft_consensus.cc:3054] T ca75e7cae4f6423191a17bb5a8f150a9 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:44.378585 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afcb5b1977c447d3a51de9cb72d7196a" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
W20250113 02:51:44.379108 709 leader_election.cc:343] T afcb5b1977c447d3a51de9cb72d7196a P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:44.379441 1198 raft_consensus.cc:2267] T 7527e6e0b65e4a43b7f48899a34cdf7f P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:44.380041 1198 ts_tablet_manager.cc:1905] T 7527e6e0b65e4a43b7f48899a34cdf7f P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:44.380174 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afcb5b1977c447d3a51de9cb72d7196a" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:44.380673 711 leader_election.cc:343] T afcb5b1977c447d3a51de9cb72d7196a P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:44.380807 711 leader_election.cc:304] T afcb5b1977c447d3a51de9cb72d7196a P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:44.381403 2501 raft_consensus.cc:2743] T afcb5b1977c447d3a51de9cb72d7196a P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:44.381729 2368 raft_consensus.cc:513] T ca75e7cae4f6423191a17bb5a8f150a9 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:44.382304 2363 raft_consensus.cc:491] T 8cc78bf7d6324b7ba10a1b1045923f2b P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:44.382408 2363 raft_consensus.cc:513] T 8cc78bf7d6324b7ba10a1b1045923f2b P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:44.382891 2363 leader_election.cc:290] T 8cc78bf7d6324b7ba10a1b1045923f2b P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:44.383267 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cc78bf7d6324b7ba10a1b1045923f2b" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:44.383455 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cc78bf7d6324b7ba10a1b1045923f2b" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
I20250113 02:51:44.383636 772 raft_consensus.cc:2463] T 8cc78bf7d6324b7ba10a1b1045923f2b P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 0.
W20250113 02:51:44.383795 998 leader_election.cc:343] T 8cc78bf7d6324b7ba10a1b1045923f2b P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:44.384002 995 leader_election.cc:304] T 8cc78bf7d6324b7ba10a1b1045923f2b P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters: df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:44.384756 2639 raft_consensus.cc:2798] T 8cc78bf7d6324b7ba10a1b1045923f2b P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 02:51:44.384829 2639 raft_consensus.cc:491] T 8cc78bf7d6324b7ba10a1b1045923f2b P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 02:51:44.384899 2639 raft_consensus.cc:3054] T 8cc78bf7d6324b7ba10a1b1045923f2b P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:44.387305 2639 raft_consensus.cc:513] T 8cc78bf7d6324b7ba10a1b1045923f2b P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:44.387701 1198 ts_tablet_manager.cc:1918] T 7527e6e0b65e4a43b7f48899a34cdf7f P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250113 02:51:44.387789 1198 log.cc:1198] T 7527e6e0b65e4a43b7f48899a34cdf7f P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/7527e6e0b65e4a43b7f48899a34cdf7f
I20250113 02:51:44.387838 2639 leader_election.cc:290] T 8cc78bf7d6324b7ba10a1b1045923f2b P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Requested vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:44.388142 1198 ts_tablet_manager.cc:1939] T 7527e6e0b65e4a43b7f48899a34cdf7f P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:44.388628 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cc78bf7d6324b7ba10a1b1045923f2b" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f"
I20250113 02:51:44.388881 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1408) took 1180 ms. Trace:
I20250113 02:51:44.388988 1198 rpcz_store.cc:276] 0113 02:51:43.208178 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:43.208228 (+ 50us) service_pool.cc:225] Handling call
0113 02:51:44.367439 (+1159211us) ts_tablet_manager.cc:1145] Deleting tablet 7527e6e0b65e4a43b7f48899a34cdf7f
0113 02:51:44.367452 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:44.384973 (+ 17521us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:44.387619 (+ 2646us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:44.388863 (+ 1244us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1159037}
I20250113 02:51:44.389417 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet 7527e6e0b65e4a43b7f48899a34cdf7f (table default.Table_c8b93363989c4e109540af6bef426f73 [id=b7846a8ebee84005b1b5d259560a26bd]) successfully deleted
W20250113 02:51:44.389636 998 leader_election.cc:343] T 8cc78bf7d6324b7ba10a1b1045923f2b P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:44.390311 2501 raft_consensus.cc:491] T f1237f7b04af4ebbb39b03aa3c78a499 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:44.390493 2501 raft_consensus.cc:513] T f1237f7b04af4ebbb39b03aa3c78a499 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:44.391309 2501 leader_election.cc:290] T f1237f7b04af4ebbb39b03aa3c78a499 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:44.392059 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1237f7b04af4ebbb39b03aa3c78a499" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:44.392689 711 leader_election.cc:343] T f1237f7b04af4ebbb39b03aa3c78a499 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:44.393335 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1237f7b04af4ebbb39b03aa3c78a499" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:44.393559 1059 raft_consensus.cc:2463] T f1237f7b04af4ebbb39b03aa3c78a499 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1186fa869eb14812832fa6562531f196 in term 0.
I20250113 02:51:44.394372 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cc78bf7d6324b7ba10a1b1045923f2b" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196"
I20250113 02:51:44.394593 772 raft_consensus.cc:3054] T 8cc78bf7d6324b7ba10a1b1045923f2b P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:44.394804 1198 tablet_replica.cc:331] T e570465a3d4046b1ac19a3a034fe91f1 P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:44.395157 1198 raft_consensus.cc:2238] T e570465a3d4046b1ac19a3a034fe91f1 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: Raft consensus shutting down.
I20250113 02:51:44.395634 1198 pending_rounds.cc:62] T e570465a3d4046b1ac19a3a034fe91f1 P 4216a7f853d642b7bc3e5cff458f3aca: Trying to abort 1 pending ops.
I20250113 02:51:44.396046 1198 pending_rounds.cc:69] T e570465a3d4046b1ac19a3a034fe91f1 P 4216a7f853d642b7bc3e5cff458f3aca: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113673537396518912 op_type: NO_OP noop_request { }
I20250113 02:51:44.396337 1198 raft_consensus.cc:2883] T e570465a3d4046b1ac19a3a034fe91f1 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250113 02:51:44.396559 1198 raft_consensus.cc:2267] T e570465a3d4046b1ac19a3a034fe91f1 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:44.397403 1198 ts_tablet_manager.cc:1905] T e570465a3d4046b1ac19a3a034fe91f1 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:44.398218 772 raft_consensus.cc:2463] T 8cc78bf7d6324b7ba10a1b1045923f2b P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 1.
I20250113 02:51:44.398820 709 leader_election.cc:304] T f1237f7b04af4ebbb39b03aa3c78a499 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters: df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:44.398959 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_c569a6d1ad344d33920376cb1ba96eba" } new_table_name: "default.Table_d4a9d5f02f1243249011f1f7fa787487" modify_external_catalogs: true
I20250113 02:51:44.399559 2588 raft_consensus.cc:2798] T f1237f7b04af4ebbb39b03aa3c78a499 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 02:51:44.399670 2588 raft_consensus.cc:491] T f1237f7b04af4ebbb39b03aa3c78a499 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 02:51:44.399778 2588 raft_consensus.cc:3054] T f1237f7b04af4ebbb39b03aa3c78a499 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:44.400933 885 tablet_service.cc:1514] Processing DeleteTablet for tablet 0644f5600c5b47e29f3fa73060ed96e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:22 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:44.402673 2588 raft_consensus.cc:513] T f1237f7b04af4ebbb39b03aa3c78a499 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:44.403378 2588 leader_election.cc:290] T f1237f7b04af4ebbb39b03aa3c78a499 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 election: Requested vote from peers 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:44.404109 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1237f7b04af4ebbb39b03aa3c78a499" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f"
I20250113 02:51:44.404748 995 leader_election.cc:304] T 8cc78bf7d6324b7ba10a1b1045923f2b P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters: df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:44.405225 2363 raft_consensus.cc:2798] T 8cc78bf7d6324b7ba10a1b1045923f2b P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Leader election won for term 1
W20250113 02:51:44.405230 711 leader_election.cc:343] T f1237f7b04af4ebbb39b03aa3c78a499 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:44.405352 2363 raft_consensus.cc:695] T 8cc78bf7d6324b7ba10a1b1045923f2b P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: Becoming Leader. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Running, Role: LEADER
I20250113 02:51:44.405586 2363 consensus_queue.cc:237] T 8cc78bf7d6324b7ba10a1b1045923f2b P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:44.406666 1058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1237f7b04af4ebbb39b03aa3c78a499" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca"
I20250113 02:51:44.406863 1058 raft_consensus.cc:3054] T f1237f7b04af4ebbb39b03aa3c78a499 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:44.410262 1058 raft_consensus.cc:2463] T f1237f7b04af4ebbb39b03aa3c78a499 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1186fa869eb14812832fa6562531f196 in term 1.
I20250113 02:51:44.410305 1195 ts_tablet_manager.cc:1918] T 3d85539cf0074a03b167d3279e940079 P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250113 02:51:44.410423 1195 log.cc:1198] T 3d85539cf0074a03b167d3279e940079 P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/3d85539cf0074a03b167d3279e940079
I20250113 02:51:44.410823 1195 ts_tablet_manager.cc:1939] T 3d85539cf0074a03b167d3279e940079 P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:44.411638 709 leader_election.cc:304] T f1237f7b04af4ebbb39b03aa3c78a499 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters: df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:44.412739 2501 raft_consensus.cc:2798] T f1237f7b04af4ebbb39b03aa3c78a499 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Leader election won for term 1
I20250113 02:51:44.413686 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca75e7cae4f6423191a17bb5a8f150a9" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196"
I20250113 02:51:44.413866 769 raft_consensus.cc:3054] T ca75e7cae4f6423191a17bb5a8f150a9 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:44.413177 2368 leader_election.cc:290] T ca75e7cae4f6423191a17bb5a8f150a9 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Requested vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:44.417032 769 raft_consensus.cc:2463] T ca75e7cae4f6423191a17bb5a8f150a9 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 1.
I20250113 02:51:44.417250 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca75e7cae4f6423191a17bb5a8f150a9" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f"
I20250113 02:51:44.417734 995 leader_election.cc:304] T ca75e7cae4f6423191a17bb5a8f150a9 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters:
W20250113 02:51:44.417970 998 leader_election.cc:343] T ca75e7cae4f6423191a17bb5a8f150a9 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:44.420199 2368 raft_consensus.cc:2798] T ca75e7cae4f6423191a17bb5a8f150a9 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Leader election won for term 1
I20250113 02:51:44.420279 449 master_service.cc:946] ReplaceTablet: received request to replace tablet 99843ce7f1bf41da9b7cb5bd7ac2e875 from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:44.414127 1192 tablet_bootstrap.cc:492] T 88a776aa820c44b3a3854ca5022bd353 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:44.421623 1192 ts_tablet_manager.cc:1397] T 88a776aa820c44b3a3854ca5022bd353 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.130s user 0.014s sys 0.024s
I20250113 02:51:44.422041 447 master_service.cc:946] ReplaceTablet: received request to replace tablet 936b8ceaedd34a019a4fb89a80dd2708 from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:44.422593 1192 raft_consensus.cc:357] T 88a776aa820c44b3a3854ca5022bd353 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:44.422755 1192 raft_consensus.cc:383] T 88a776aa820c44b3a3854ca5022bd353 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:44.422797 1192 raft_consensus.cc:738] T 88a776aa820c44b3a3854ca5022bd353 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:44.422814 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_3fda809831ba43319e2ee62fb49dde92"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:44.423333 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3fda809831ba43319e2ee62fb49dde92 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:44.424172 2368 raft_consensus.cc:695] T ca75e7cae4f6423191a17bb5a8f150a9 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: Becoming Leader. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Running, Role: LEADER
I20250113 02:51:44.424403 2368 consensus_queue.cc:237] T ca75e7cae4f6423191a17bb5a8f150a9 P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:44.424825 1192 consensus_queue.cc:260] T 88a776aa820c44b3a3854ca5022bd353 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:44.425448 1192 ts_tablet_manager.cc:1428] T 88a776aa820c44b3a3854ca5022bd353 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
W20250113 02:51:44.426156 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 045ff4572f3347fc890b439d90cb0bbe consensus state has no leader
I20250113 02:51:44.426450 1192 tablet_bootstrap.cc:492] T b217cb8c09fb41cab4f53e605c37efe8 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:44.426548 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_4fcb7bb3569b4eddafedff2f3dec10f8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:44.427032 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4fcb7bb3569b4eddafedff2f3dec10f8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:44.427502 2501 raft_consensus.cc:695] T f1237f7b04af4ebbb39b03aa3c78a499 P 1186fa869eb14812832fa6562531f196 [term 1 LEADER]: Becoming Leader. State: Replica: 1186fa869eb14812832fa6562531f196, State: Running, Role: LEADER
I20250113 02:51:44.427814 2501 consensus_queue.cc:237] T f1237f7b04af4ebbb39b03aa3c78a499 P 1186fa869eb14812832fa6562531f196 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:44.433686 1192 tablet_bootstrap.cc:654] T b217cb8c09fb41cab4f53e605c37efe8 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:44.437764 1198 ts_tablet_manager.cc:1918] T e570465a3d4046b1ac19a3a034fe91f1 P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250113 02:51:44.438156 1198 log.cc:1198] T e570465a3d4046b1ac19a3a034fe91f1 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/e570465a3d4046b1ac19a3a034fe91f1
I20250113 02:51:44.438733 1198 ts_tablet_manager.cc:1939] T e570465a3d4046b1ac19a3a034fe91f1 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:44.439556 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1410) took 1224 ms. Trace:
I20250113 02:51:44.439640 1194 raft_consensus.cc:357] T bc6a7322c0c34adb85457750526d8955 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:44.440006 1194 raft_consensus.cc:383] T bc6a7322c0c34adb85457750526d8955 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:44.440122 1194 raft_consensus.cc:738] T bc6a7322c0c34adb85457750526d8955 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:44.440060 1198 rpcz_store.cc:276] 0113 02:51:43.214830 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:43.214888 (+ 58us) service_pool.cc:225] Handling call
0113 02:51:44.394754 (+1179866us) ts_tablet_manager.cc:1145] Deleting tablet e570465a3d4046b1ac19a3a034fe91f1
0113 02:51:44.394766 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:44.430603 (+ 35837us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:44.437650 (+ 7047us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:44.439538 (+ 1888us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1175316}
I20250113 02:51:44.440433 1194 consensus_queue.cc:260] T bc6a7322c0c34adb85457750526d8955 P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:44.440784 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet e570465a3d4046b1ac19a3a034fe91f1 (table default.Table_766a63106d8a4fd29bb338e76bd96089 [id=3042221d269a4bd88c04c1b6bb1bcd26]) successfully deleted
I20250113 02:51:44.441133 1198 tablet_replica.cc:331] T 1968c7e9357c4c6e901d334ace0467fb P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:44.441335 1198 raft_consensus.cc:2238] T 1968c7e9357c4c6e901d334ace0467fb P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: Raft consensus shutting down.
I20250113 02:51:44.441601 1198 raft_consensus.cc:2267] T 1968c7e9357c4c6e901d334ace0467fb P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:44.443550 1198 ts_tablet_manager.cc:1905] T 1968c7e9357c4c6e901d334ace0467fb P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:44.447045 1194 ts_tablet_manager.cc:1428] T bc6a7322c0c34adb85457750526d8955 P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.074s user 0.002s sys 0.000s
W20250113 02:51:44.454124 998 consensus_peers.cc:487] T 21bcdd741104474bad7023426856d18f P 4216a7f853d642b7bc3e5cff458f3aca -> Peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Couldn't send request to peer df69f48656b44bb890f76587ee69cc0f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250113 02:51:44.455243 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5da22bf39b4b43d080165893cbf75a97 has no consensus state
I20250113 02:51:44.456851 1194 tablet_bootstrap.cc:492] T 788ab82626424b6e833e31761a15b9b2 P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:44.457988 2476 raft_consensus.cc:491] T bc533518faa24dffa3306ff092e8d507 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:44.458120 2476 raft_consensus.cc:513] T bc533518faa24dffa3306ff092e8d507 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:44.458881 2476 leader_election.cc:290] T bc533518faa24dffa3306ff092e8d507 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:44.459379 1198 ts_tablet_manager.cc:1918] T 1968c7e9357c4c6e901d334ace0467fb P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250113 02:51:44.459308 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_21fe260f30524201a0367153afecc2d8" } new_table_name: "default.Table_b3926f9a1b6a4f249cf1e20b1129438e" modify_external_catalogs: true
I20250113 02:51:44.460009 1198 log.cc:1198] T 1968c7e9357c4c6e901d334ace0467fb P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/1968c7e9357c4c6e901d334ace0467fb
W20250113 02:51:44.460417 852 leader_election.cc:343] T bc533518faa24dffa3306ff092e8d507 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Not found: Tablet not found: bc533518faa24dffa3306ff092e8d507
I20250113 02:51:44.460909 1198 ts_tablet_manager.cc:1939] T 1968c7e9357c4c6e901d334ace0467fb P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
W20250113 02:51:44.461105 851 leader_election.cc:343] T bc533518faa24dffa3306ff092e8d507 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Not found: Tablet not found: bc533518faa24dffa3306ff092e8d507
I20250113 02:51:44.461205 851 leader_election.cc:304] T bc533518faa24dffa3306ff092e8d507 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df69f48656b44bb890f76587ee69cc0f; no voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca
I20250113 02:51:44.461584 2476 raft_consensus.cc:2743] T bc533518faa24dffa3306ff092e8d507 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:44.461907 2474 raft_consensus.cc:491] T f105f8b32a654e4ca154e8fc0cb96091 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:44.462047 2474 raft_consensus.cc:513] T f105f8b32a654e4ca154e8fc0cb96091 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:44.462642 2474 leader_election.cc:290] T f105f8b32a654e4ca154e8fc0cb96091 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:44.463073 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f105f8b32a654e4ca154e8fc0cb96091" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:44.463332 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f105f8b32a654e4ca154e8fc0cb96091" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
I20250113 02:51:44.463557 772 raft_consensus.cc:2463] T f105f8b32a654e4ca154e8fc0cb96091 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 0.
W20250113 02:51:44.463649 998 leader_election.cc:343] T f105f8b32a654e4ca154e8fc0cb96091 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:44.463932 995 leader_election.cc:304] T f105f8b32a654e4ca154e8fc0cb96091 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters: df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:44.464167 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1411) took 1248 ms. Trace:
I20250113 02:51:44.464372 2474 raft_consensus.cc:2798] T f105f8b32a654e4ca154e8fc0cb96091 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 02:51:44.464450 2474 raft_consensus.cc:491] T f105f8b32a654e4ca154e8fc0cb96091 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 02:51:44.464591 2474 raft_consensus.cc:3054] T f105f8b32a654e4ca154e8fc0cb96091 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:44.467929 2363 raft_consensus.cc:491] T e01e7f8db83d461bbfbb2e901419ce0b P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:44.468057 2363 raft_consensus.cc:513] T e01e7f8db83d461bbfbb2e901419ce0b P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:44.469033 2363 leader_election.cc:290] T e01e7f8db83d461bbfbb2e901419ce0b P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:44.469213 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e01e7f8db83d461bbfbb2e901419ce0b" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
I20250113 02:51:44.469410 772 raft_consensus.cc:2463] T e01e7f8db83d461bbfbb2e901419ce0b P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 0.
I20250113 02:51:44.469799 995 leader_election.cc:304] T e01e7f8db83d461bbfbb2e901419ce0b P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters:
I20250113 02:51:44.469938 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e01e7f8db83d461bbfbb2e901419ce0b" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:44.470551 998 leader_election.cc:343] T e01e7f8db83d461bbfbb2e901419ce0b P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:44.470924 2363 raft_consensus.cc:2798] T e01e7f8db83d461bbfbb2e901419ce0b P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 02:51:44.470993 2363 raft_consensus.cc:491] T e01e7f8db83d461bbfbb2e901419ce0b P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 02:51:44.471045 2363 raft_consensus.cc:3054] T e01e7f8db83d461bbfbb2e901419ce0b P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:44.471621 1194 tablet_bootstrap.cc:654] T 788ab82626424b6e833e31761a15b9b2 P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:44.473815 2363 raft_consensus.cc:513] T e01e7f8db83d461bbfbb2e901419ce0b P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:44.475508 2476 raft_consensus.cc:491] T ee240d838d4a4bbba97f8ed61f69de78 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:44.475641 2476 raft_consensus.cc:513] T ee240d838d4a4bbba97f8ed61f69de78 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:44.476217 2476 leader_election.cc:290] T ee240d838d4a4bbba97f8ed61f69de78 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
W20250113 02:51:44.477917 852 leader_election.cc:343] T ee240d838d4a4bbba97f8ed61f69de78 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Not found: Tablet not found: ee240d838d4a4bbba97f8ed61f69de78
W20250113 02:51:44.478257 851 leader_election.cc:343] T ee240d838d4a4bbba97f8ed61f69de78 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Not found: Tablet not found: ee240d838d4a4bbba97f8ed61f69de78
I20250113 02:51:44.478384 851 leader_election.cc:304] T ee240d838d4a4bbba97f8ed61f69de78 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df69f48656b44bb890f76587ee69cc0f; no voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca
I20250113 02:51:44.478829 2476 raft_consensus.cc:2743] T ee240d838d4a4bbba97f8ed61f69de78 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:44.479296 2700 consensus_queue.cc:1035] T fb2458e2044d49029ed2d6851db9de25 P 1186fa869eb14812832fa6562531f196 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 02:51:44.481444 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e01e7f8db83d461bbfbb2e901419ce0b" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f"
W20250113 02:51:44.481971 998 leader_election.cc:343] T e01e7f8db83d461bbfbb2e901419ce0b P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:44.482146 2363 leader_election.cc:290] T e01e7f8db83d461bbfbb2e901419ce0b P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Requested vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:44.483948 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e01e7f8db83d461bbfbb2e901419ce0b" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196"
I20250113 02:51:44.484207 772 raft_consensus.cc:3054] T e01e7f8db83d461bbfbb2e901419ce0b P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:44.467679 2474 raft_consensus.cc:513] T f105f8b32a654e4ca154e8fc0cb96091 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:44.487327 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f105f8b32a654e4ca154e8fc0cb96091" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196"
I20250113 02:51:44.487301 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f105f8b32a654e4ca154e8fc0cb96091" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f"
I20250113 02:51:44.487509 769 raft_consensus.cc:3054] T f105f8b32a654e4ca154e8fc0cb96091 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Advancing to term 1
W20250113 02:51:44.488010 998 leader_election.cc:343] T f105f8b32a654e4ca154e8fc0cb96091 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:44.488622 2474 leader_election.cc:290] T f105f8b32a654e4ca154e8fc0cb96091 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Requested vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:44.490995 447 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_99031e0341fd4ad8a3bb04a6e222a1d0" } modify_external_catalogs: true
I20250113 02:51:44.491137 447 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_99031e0341fd4ad8a3bb04a6e222a1d0" } modify_external_catalogs: true
I20250113 02:51:44.491799 2700 raft_consensus.cc:491] T 569d666c128a45b7a526c13be5090ae3 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:44.492000 2700 raft_consensus.cc:513] T 569d666c128a45b7a526c13be5090ae3 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:44.492800 2700 leader_election.cc:290] T 569d666c128a45b7a526c13be5090ae3 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:44.493734 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "569d666c128a45b7a526c13be5090ae3" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:44.494370 711 leader_election.cc:343] T 569d666c128a45b7a526c13be5090ae3 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:44.495097 1058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "569d666c128a45b7a526c13be5090ae3" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
W20250113 02:51:44.495798 709 leader_election.cc:343] T 569d666c128a45b7a526c13be5090ae3 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:44.495962 709 leader_election.cc:304] T 569d666c128a45b7a526c13be5090ae3 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:44.496488 2700 raft_consensus.cc:491] T c04be975c8c049b892b16129ed47a142 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:44.496647 2700 raft_consensus.cc:513] T c04be975c8c049b892b16129ed47a142 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:44.497368 2700 leader_election.cc:290] T c04be975c8c049b892b16129ed47a142 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:44.497529 2700 raft_consensus.cc:2743] T 569d666c128a45b7a526c13be5090ae3 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:44.464426 1198 rpcz_store.cc:276] 0113 02:51:43.215315 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:43.215369 (+ 54us) service_pool.cc:225] Handling call
0113 02:51:44.441087 (+1225718us) ts_tablet_manager.cc:1145] Deleting tablet 1968c7e9357c4c6e901d334ace0467fb
0113 02:51:44.441096 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:44.451677 (+ 10581us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:44.459265 (+ 7588us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:44.464150 (+ 4885us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1221598}
I20250113 02:51:44.498190 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c04be975c8c049b892b16129ed47a142" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:44.498725 711 leader_election.cc:343] T c04be975c8c049b892b16129ed47a142 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:44.499004 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet 1968c7e9357c4c6e901d334ace0467fb (table default.Table_f2da084a4aee4d19b45e784e56f926a6 [id=fa9bcd8c66c34d65b43561572abb8890]) successfully deleted
I20250113 02:51:44.499258 1058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c04be975c8c049b892b16129ed47a142" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:44.499645 1198 tablet_replica.cc:331] T fd0b8592997d4a61b8ca854ecbb67c55 P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
W20250113 02:51:44.499763 709 leader_election.cc:343] T c04be975c8c049b892b16129ed47a142 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:44.499888 709 leader_election.cc:304] T c04be975c8c049b892b16129ed47a142 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:44.500182 1198 raft_consensus.cc:2238] T fd0b8592997d4a61b8ca854ecbb67c55 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: Raft consensus shutting down.
I20250113 02:51:44.500919 1198 raft_consensus.cc:2267] T fd0b8592997d4a61b8ca854ecbb67c55 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:44.502542 1198 ts_tablet_manager.cc:1905] T fd0b8592997d4a61b8ca854ecbb67c55 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:44.504052 2588 raft_consensus.cc:2743] T c04be975c8c049b892b16129ed47a142 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:44.506700 769 raft_consensus.cc:2463] T f105f8b32a654e4ca154e8fc0cb96091 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 1.
I20250113 02:51:44.507438 995 leader_election.cc:304] T f105f8b32a654e4ca154e8fc0cb96091 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters: df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:44.508622 2363 raft_consensus.cc:2798] T f105f8b32a654e4ca154e8fc0cb96091 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Leader election won for term 1
I20250113 02:51:44.508747 2363 raft_consensus.cc:695] T f105f8b32a654e4ca154e8fc0cb96091 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: Becoming Leader. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Running, Role: LEADER
I20250113 02:51:44.509042 2363 consensus_queue.cc:237] T f105f8b32a654e4ca154e8fc0cb96091 P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:44.513808 772 raft_consensus.cc:2463] T e01e7f8db83d461bbfbb2e901419ce0b P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 1.
I20250113 02:51:44.514396 995 leader_election.cc:304] T e01e7f8db83d461bbfbb2e901419ce0b P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters: df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:44.514775 2474 raft_consensus.cc:2798] T e01e7f8db83d461bbfbb2e901419ce0b P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Leader election won for term 1
I20250113 02:51:44.515046 2474 raft_consensus.cc:695] T e01e7f8db83d461bbfbb2e901419ce0b P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: Becoming Leader. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Running, Role: LEADER
I20250113 02:51:44.515374 2474 consensus_queue.cc:237] T e01e7f8db83d461bbfbb2e901419ce0b P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:44.518335 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_9bdc99d09f2d405b88dcb4a48144ca4b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:44.519097 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9bdc99d09f2d405b88dcb4a48144ca4b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250113 02:51:44.522346 711 consensus_peers.cc:487] T fb2458e2044d49029ed2d6851db9de25 P 1186fa869eb14812832fa6562531f196 -> Peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Couldn't send request to peer df69f48656b44bb890f76587ee69cc0f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250113 02:51:44.523329 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1135) took 6467 ms. Trace:
I20250113 02:51:44.523479 1195 rpcz_store.cc:276] 0113 02:51:38.055793 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:38.055850 (+ 57us) service_pool.cc:225] Handling call
0113 02:51:44.191083 (+6135233us) ts_tablet_manager.cc:1145] Deleting tablet 3d85539cf0074a03b167d3279e940079
0113 02:51:44.191098 (+ 15us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:44.274461 (+ 83363us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:44.410170 (+135709us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:44.523283 (+113113us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":110953,"tablet-delete.queue_time_us":6134956}
I20250113 02:51:44.524134 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet 3d85539cf0074a03b167d3279e940079 (table default.Table_3a73b4672b154b99af1a24e962007ea0 [id=f742014cd0d44db99c4c1a031630d827]) successfully deleted
I20250113 02:51:44.524572 1195 tablet_replica.cc:331] T 42dbd37e20fa47318bc6497423dbe565 P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:44.524863 1195 raft_consensus.cc:2238] T 42dbd37e20fa47318bc6497423dbe565 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:44.525105 1195 raft_consensus.cc:2267] T 42dbd37e20fa47318bc6497423dbe565 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:44.525804 1195 ts_tablet_manager.cc:1905] T 42dbd37e20fa47318bc6497423dbe565 P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250113 02:51:44.529417 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 247af1a17df04b27898f86642336d51e consensus state has no leader
I20250113 02:51:44.535584 1194 tablet_bootstrap.cc:492] T 788ab82626424b6e833e31761a15b9b2 P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:44.535861 1194 ts_tablet_manager.cc:1397] T 788ab82626424b6e833e31761a15b9b2 P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.079s user 0.004s sys 0.000s
I20250113 02:51:44.536273 1196 ts_tablet_manager.cc:1918] T c7842d6f2c074e3b9d1a3378db9856be P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:44.536491 1196 log.cc:1198] T c7842d6f2c074e3b9d1a3378db9856be P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/c7842d6f2c074e3b9d1a3378db9856be
I20250113 02:51:44.536880 1196 ts_tablet_manager.cc:1939] T c7842d6f2c074e3b9d1a3378db9856be P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:44.536945 1194 raft_consensus.cc:357] T 788ab82626424b6e833e31761a15b9b2 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:44.537285 1194 raft_consensus.cc:383] T 788ab82626424b6e833e31761a15b9b2 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:44.537395 1194 raft_consensus.cc:738] T 788ab82626424b6e833e31761a15b9b2 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:44.537532 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet c7842d6f2c074e3b9d1a3378db9856be (table default.Table_59401ddb940e4e789a5bd54f96548ba5 [id=0ac370ed485e4d64a3e1aa1e8d67ea23]) successfully deleted
I20250113 02:51:44.537745 1194 consensus_queue.cc:260] T 788ab82626424b6e833e31761a15b9b2 P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
W20250113 02:51:44.538381 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 8f46d38f2c634b7d87df37e59ef4f5ef because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8f46d38f2c634b7d87df37e59ef4f5ef already in progress: creating tablet
I20250113 02:51:44.538686 1196 tablet_replica.cc:331] T bc533518faa24dffa3306ff092e8d507 P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:44.538911 1196 raft_consensus.cc:2238] T bc533518faa24dffa3306ff092e8d507 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:44.539119 1196 raft_consensus.cc:2267] T bc533518faa24dffa3306ff092e8d507 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:44.540194 1196 ts_tablet_manager.cc:1905] T bc533518faa24dffa3306ff092e8d507 P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:44.542155 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_ff6df4afcf4d4f9a964d079ee24f02b5" } new_table_name: "default.Table_f061e077ed234496a8d398619d0eaf01" modify_external_catalogs: true
I20250113 02:51:44.546540 1194 ts_tablet_manager.cc:1428] T 788ab82626424b6e833e31761a15b9b2 P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20250113 02:51:44.553277 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet b5c1882125b5491ea79a29a6dac6771e
I20250113 02:51:44.553622 1198 ts_tablet_manager.cc:1918] T fd0b8592997d4a61b8ca854ecbb67c55 P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250113 02:51:44.554160 885 tablet_service.cc:1514] Processing DeleteTablet for tablet b5c1882125b5491ea79a29a6dac6771e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:44 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:44.555280 736 tablet_service.cc:1514] Processing DeleteTablet for tablet b5c1882125b5491ea79a29a6dac6771e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:44 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:44.556308 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet b5c1882125b5491ea79a29a6dac6771e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:44 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:44.562060 1198 log.cc:1198] T fd0b8592997d4a61b8ca854ecbb67c55 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/fd0b8592997d4a61b8ca854ecbb67c55
I20250113 02:51:44.562494 1198 ts_tablet_manager.cc:1939] T fd0b8592997d4a61b8ca854ecbb67c55 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:44.563267 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1412) took 1347 ms. Trace:
I20250113 02:51:44.563308 885 tablet_service.cc:1514] Processing DeleteTablet for tablet 788ab82626424b6e833e31761a15b9b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:26 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:44.563405 1198 rpcz_store.cc:276] 0113 02:51:43.215774 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:43.215819 (+ 45us) service_pool.cc:225] Handling call
0113 02:51:44.499595 (+1283776us) ts_tablet_manager.cc:1145] Deleting tablet fd0b8592997d4a61b8ca854ecbb67c55
0113 02:51:44.499605 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:44.541893 (+ 42288us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:44.553486 (+ 11593us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:44.563226 (+ 9740us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1279868}
I20250113 02:51:44.563742 1198 tablet_replica.cc:331] T 67baaab1e36e415381579d7385d4ab5a P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:44.563897 1194 tablet_bootstrap.cc:492] T bbc7832c50114d6f890fec45a3d994e9 P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:44.563959 1198 raft_consensus.cc:2238] T 67baaab1e36e415381579d7385d4ab5a P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: Raft consensus shutting down.
I20250113 02:51:44.564301 1198 pending_rounds.cc:62] T 67baaab1e36e415381579d7385d4ab5a P 4216a7f853d642b7bc3e5cff458f3aca: Trying to abort 1 pending ops.
I20250113 02:51:44.564500 1198 pending_rounds.cc:69] T 67baaab1e36e415381579d7385d4ab5a P 4216a7f853d642b7bc3e5cff458f3aca: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113673539616772096 op_type: NO_OP noop_request { }
I20250113 02:51:44.565079 1198 raft_consensus.cc:2883] T 67baaab1e36e415381579d7385d4ab5a P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250113 02:51:44.565294 1198 raft_consensus.cc:2267] T 67baaab1e36e415381579d7385d4ab5a P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:44.565939 1191 tablet_bootstrap.cc:492] T 415a1fa4b2f14cacb43937c8a372849a P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:44.566126 1191 ts_tablet_manager.cc:1397] T 415a1fa4b2f14cacb43937c8a372849a P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.191s user 0.034s sys 0.044s
I20250113 02:51:44.566560 1198 ts_tablet_manager.cc:1905] T 67baaab1e36e415381579d7385d4ab5a P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:44.566913 1191 raft_consensus.cc:357] T 415a1fa4b2f14cacb43937c8a372849a P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:44.567516 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_6f8ebd124a0e4bfb83502fc9f0f97130"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:44.568396 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6f8ebd124a0e4bfb83502fc9f0f97130 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:44.571259 1194 tablet_bootstrap.cc:654] T bbc7832c50114d6f890fec45a3d994e9 P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:44.567135 1191 raft_consensus.cc:383] T 415a1fa4b2f14cacb43937c8a372849a P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:44.571810 1191 raft_consensus.cc:738] T 415a1fa4b2f14cacb43937c8a372849a P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:44.572145 1191 consensus_queue.cc:260] T 415a1fa4b2f14cacb43937c8a372849a P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:44.573988 1191 ts_tablet_manager.cc:1428] T 415a1fa4b2f14cacb43937c8a372849a P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.008s user 0.000s sys 0.002s
I20250113 02:51:44.574641 1191 tablet_bootstrap.cc:492] T 6001369444894b48b51a48fb60ddc52a P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:44.576062 1191 tablet_bootstrap.cc:654] T 6001369444894b48b51a48fb60ddc52a P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:44.585990 913 raft_consensus.cc:3054] T b5c1882125b5491ea79a29a6dac6771e P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:44.595546 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_f167644bb9424d71af7befe561e5f3f5" } new_table_name: "default.Table_5a12a9665e634f778ea9191a4b31f2aa" modify_external_catalogs: true
I20250113 02:51:44.596089 1192 tablet_bootstrap.cc:492] T b217cb8c09fb41cab4f53e605c37efe8 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:44.596247 1192 ts_tablet_manager.cc:1397] T b217cb8c09fb41cab4f53e605c37efe8 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.170s user 0.006s sys 0.017s
I20250113 02:51:44.597043 1192 raft_consensus.cc:357] T b217cb8c09fb41cab4f53e605c37efe8 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:44.597234 2501 raft_consensus.cc:491] T 01a455eda0ee488fb3d45a6a09c43833 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:44.597330 1192 raft_consensus.cc:383] T b217cb8c09fb41cab4f53e605c37efe8 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:44.597430 1192 raft_consensus.cc:738] T b217cb8c09fb41cab4f53e605c37efe8 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:44.597373 2501 raft_consensus.cc:513] T 01a455eda0ee488fb3d45a6a09c43833 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:44.597651 1192 consensus_queue.cc:260] T b217cb8c09fb41cab4f53e605c37efe8 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:44.597932 2501 leader_election.cc:290] T 01a455eda0ee488fb3d45a6a09c43833 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:44.598275 1192 ts_tablet_manager.cc:1428] T b217cb8c09fb41cab4f53e605c37efe8 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250113 02:51:44.598567 1192 tablet_bootstrap.cc:492] T 883027967e0c442a91ac820210b9c6c7 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:44.598484 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01a455eda0ee488fb3d45a6a09c43833" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:44.599187 711 leader_election.cc:343] T 01a455eda0ee488fb3d45a6a09c43833 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:44.600167 1192 tablet_bootstrap.cc:654] T 883027967e0c442a91ac820210b9c6c7 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:44.602025 1192 tablet_bootstrap.cc:492] T 883027967e0c442a91ac820210b9c6c7 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:44.602237 1192 ts_tablet_manager.cc:1397] T 883027967e0c442a91ac820210b9c6c7 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20250113 02:51:44.602944 1192 raft_consensus.cc:357] T 883027967e0c442a91ac820210b9c6c7 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:44.604281 1192 raft_consensus.cc:383] T 883027967e0c442a91ac820210b9c6c7 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:44.604637 1192 raft_consensus.cc:738] T 883027967e0c442a91ac820210b9c6c7 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:44.605109 1195 ts_tablet_manager.cc:1918] T 42dbd37e20fa47318bc6497423dbe565 P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250113 02:51:44.605867 1192 consensus_queue.cc:260] T 883027967e0c442a91ac820210b9c6c7 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:44.606163 1195 log.cc:1198] T 42dbd37e20fa47318bc6497423dbe565 P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/42dbd37e20fa47318bc6497423dbe565
I20250113 02:51:44.607110 1192 ts_tablet_manager.cc:1428] T 883027967e0c442a91ac820210b9c6c7 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250113 02:51:44.607426 1192 tablet_bootstrap.cc:492] T f2c6c8c17d6e499d84ad7c0c23889cc4 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:44.608333 1198 ts_tablet_manager.cc:1918] T 67baaab1e36e415381579d7385d4ab5a P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20250113 02:51:44.608471 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7c443454994a47f2bfa7e2ec7ac576e0 consensus state has no leader
I20250113 02:51:44.608632 1195 ts_tablet_manager.cc:1939] T 42dbd37e20fa47318bc6497423dbe565 P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:44.613533 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_82bda2c55fed4358ba3769c07352f440" } modify_external_catalogs: true
I20250113 02:51:44.614353 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_82bda2c55fed4358ba3769c07352f440" } modify_external_catalogs: true
I20250113 02:51:44.617620 1194 tablet_bootstrap.cc:492] T bbc7832c50114d6f890fec45a3d994e9 P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:44.617765 1194 ts_tablet_manager.cc:1397] T bbc7832c50114d6f890fec45a3d994e9 P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.054s user 0.009s sys 0.009s
I20250113 02:51:44.618438 1194 raft_consensus.cc:357] T bbc7832c50114d6f890fec45a3d994e9 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:44.618618 1194 raft_consensus.cc:383] T bbc7832c50114d6f890fec45a3d994e9 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:44.618674 1194 raft_consensus.cc:738] T bbc7832c50114d6f890fec45a3d994e9 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:44.618862 1194 consensus_queue.cc:260] T bbc7832c50114d6f890fec45a3d994e9 P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:44.620908 1198 log.cc:1198] T 67baaab1e36e415381579d7385d4ab5a P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/67baaab1e36e415381579d7385d4ab5a
I20250113 02:51:44.612821 1192 tablet_bootstrap.cc:654] T f2c6c8c17d6e499d84ad7c0c23889cc4 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:44.620251 2474 consensus_queue.cc:1035] T 21bcdd741104474bad7023426856d18f P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 02:51:44.624128 1194 ts_tablet_manager.cc:1428] T bbc7832c50114d6f890fec45a3d994e9 P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
W20250113 02:51:44.624244 2603 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
I20250113 02:51:44.624580 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1136) took 6536 ms. Trace:
I20250113 02:51:44.625360 1194 tablet_bootstrap.cc:492] T 42dbd37e20fa47318bc6497423dbe565 P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:44.624827 1195 rpcz_store.cc:276] 0113 02:51:38.088462 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:38.088514 (+ 52us) service_pool.cc:225] Handling call
0113 02:51:44.524506 (+6435992us) ts_tablet_manager.cc:1145] Deleting tablet 42dbd37e20fa47318bc6497423dbe565
0113 02:51:44.524521 (+ 15us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:44.529749 (+ 5228us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:44.604987 (+ 75238us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:44.624559 (+ 19572us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6435810}
I20250113 02:51:44.627473 1194 tablet_bootstrap.cc:654] T 42dbd37e20fa47318bc6497423dbe565 P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
W20250113 02:51:44.629235 709 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.3:42587, user_credentials={real_user=slave}} blocked reactor thread for 129497us
I20250113 02:51:44.625404 1198 ts_tablet_manager.cc:1939] T 67baaab1e36e415381579d7385d4ab5a P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:44.630795 2474 raft_consensus.cc:491] T 019eb21c661547cfa0b619d119eb024e P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:44.630970 2474 raft_consensus.cc:513] T 019eb21c661547cfa0b619d119eb024e P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:44.631261 1191 tablet_bootstrap.cc:492] T 6001369444894b48b51a48fb60ddc52a P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:44.631384 1191 ts_tablet_manager.cc:1397] T 6001369444894b48b51a48fb60ddc52a P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.057s user 0.006s sys 0.012s
I20250113 02:51:44.631736 2474 leader_election.cc:290] T 019eb21c661547cfa0b619d119eb024e P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:44.632103 1191 raft_consensus.cc:357] T 6001369444894b48b51a48fb60ddc52a P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:44.632294 1191 raft_consensus.cc:383] T 6001369444894b48b51a48fb60ddc52a P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:44.632350 1191 raft_consensus.cc:738] T 6001369444894b48b51a48fb60ddc52a P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:44.632560 1191 consensus_queue.cc:260] T 6001369444894b48b51a48fb60ddc52a P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:44.632645 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "019eb21c661547cfa0b619d119eb024e" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:44.633311 1191 ts_tablet_manager.cc:1428] T 6001369444894b48b51a48fb60ddc52a P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20250113 02:51:44.633373 998 leader_election.cc:343] T 019eb21c661547cfa0b619d119eb024e P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:44.633571 1191 tablet_bootstrap.cc:492] T 39a014ac69a2435a9132a75125063093 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:44.634215 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "019eb21c661547cfa0b619d119eb024e" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
I20250113 02:51:44.634416 769 raft_consensus.cc:2463] T 019eb21c661547cfa0b619d119eb024e P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 0.
I20250113 02:51:44.634294 1195 tablet_replica.cc:331] T b33025a462e74f2aa26cc3779f36615b P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:44.634864 1195 raft_consensus.cc:2238] T b33025a462e74f2aa26cc3779f36615b P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:44.634837 995 leader_election.cc:304] T 019eb21c661547cfa0b619d119eb024e P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters: df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:44.635085 2474 raft_consensus.cc:2798] T 019eb21c661547cfa0b619d119eb024e P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 02:51:44.635119 1195 raft_consensus.cc:2267] T b33025a462e74f2aa26cc3779f36615b P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:44.635195 2474 raft_consensus.cc:491] T 019eb21c661547cfa0b619d119eb024e P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 02:51:44.635277 2474 raft_consensus.cc:3054] T 019eb21c661547cfa0b619d119eb024e P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:44.635640 1195 ts_tablet_manager.cc:1905] T b33025a462e74f2aa26cc3779f36615b P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250113 02:51:44.635689 2603 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250113 02:51:44.635856 2603 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20250113 02:51:44.634945 1191 tablet_bootstrap.cc:654] T 39a014ac69a2435a9132a75125063093 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:44.638183 2474 raft_consensus.cc:513] T 019eb21c661547cfa0b619d119eb024e P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:44.639958 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "019eb21c661547cfa0b619d119eb024e" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f"
W20250113 02:51:44.640568 998 leader_election.cc:343] T 019eb21c661547cfa0b619d119eb024e P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:44.640769 2474 leader_election.cc:290] T 019eb21c661547cfa0b619d119eb024e P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Requested vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:44.641315 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "019eb21c661547cfa0b619d119eb024e" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196"
I20250113 02:51:44.641489 769 raft_consensus.cc:3054] T 019eb21c661547cfa0b619d119eb024e P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:44.642509 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1409) took 1427 ms. Trace:
I20250113 02:51:44.642872 1198 rpcz_store.cc:276] 0113 02:51:43.214557 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:43.214611 (+ 54us) service_pool.cc:225] Handling call
0113 02:51:44.563677 (+1349066us) ts_tablet_manager.cc:1145] Deleting tablet 67baaab1e36e415381579d7385d4ab5a
0113 02:51:44.563688 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:44.584931 (+ 21243us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:44.608173 (+ 23242us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:44.642469 (+ 34296us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1344070}
I20250113 02:51:44.644125 769 raft_consensus.cc:2463] T 019eb21c661547cfa0b619d119eb024e P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 1.
I20250113 02:51:44.644362 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet 67baaab1e36e415381579d7385d4ab5a (table default.Table_c3ba3f6047274ab885a4c853b06e1417 [id=6666a614063942c99a78b95d35add037]) successfully deleted
I20250113 02:51:44.644620 995 leader_election.cc:304] T 019eb21c661547cfa0b619d119eb024e P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters: df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:44.645156 2474 raft_consensus.cc:2798] T 019eb21c661547cfa0b619d119eb024e P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Leader election won for term 1
I20250113 02:51:44.645650 2474 raft_consensus.cc:695] T 019eb21c661547cfa0b619d119eb024e P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: Becoming Leader. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Running, Role: LEADER
I20250113 02:51:44.646023 2474 consensus_queue.cc:237] T 019eb21c661547cfa0b619d119eb024e P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:44.647004 1191 tablet_bootstrap.cc:492] T 39a014ac69a2435a9132a75125063093 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:44.647156 1191 ts_tablet_manager.cc:1397] T 39a014ac69a2435a9132a75125063093 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20250113 02:51:44.647873 1191 raft_consensus.cc:357] T 39a014ac69a2435a9132a75125063093 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:44.648119 1191 raft_consensus.cc:383] T 39a014ac69a2435a9132a75125063093 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:44.648188 1191 raft_consensus.cc:738] T 39a014ac69a2435a9132a75125063093 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:44.648394 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_1416b17502f140db98544370a86bdcb6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:44.649674 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1416b17502f140db98544370a86bdcb6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:44.654102 1191 consensus_queue.cc:260] T 39a014ac69a2435a9132a75125063093 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:44.655681 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01a455eda0ee488fb3d45a6a09c43833" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
W20250113 02:51:44.659376 439 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.60:42067, user_credentials={real_user=slave}} blocked reactor thread for 56699.9us
W20250113 02:51:44.659587 709 leader_election.cc:343] T 01a455eda0ee488fb3d45a6a09c43833 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:44.659744 709 leader_election.cc:304] T 01a455eda0ee488fb3d45a6a09c43833 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:44.660033 2700 raft_consensus.cc:2743] T 01a455eda0ee488fb3d45a6a09c43833 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:44.660519 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet 42dbd37e20fa47318bc6497423dbe565 (table default.Table_37b52d3cfe0f4b0fb1ef1468117512dd [id=45aa63632c914a4e99eb69759b56c842]) successfully deleted
I20250113 02:51:44.669298 2639 raft_consensus.cc:491] T cf7b8bd6801743f49e9b9473747447c5 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:44.669461 2639 raft_consensus.cc:513] T cf7b8bd6801743f49e9b9473747447c5 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:44.670073 2639 leader_election.cc:290] T cf7b8bd6801743f49e9b9473747447c5 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:44.670168 1191 ts_tablet_manager.cc:1428] T 39a014ac69a2435a9132a75125063093 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.023s user 0.000s sys 0.002s
I20250113 02:51:44.670418 1191 tablet_bootstrap.cc:492] T c73c649a4ca44471af2cf6a77500b065 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:44.671876 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf7b8bd6801743f49e9b9473747447c5" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:44.672104 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1376cf72589d43048cb829e03240c8da consensus state has no leader
W20250113 02:51:44.672963 998 leader_election.cc:343] T cf7b8bd6801743f49e9b9473747447c5 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:44.673592 455 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_dc9db684670a46229bcca7df696bb04c" } modify_external_catalogs: true
I20250113 02:51:44.673708 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf7b8bd6801743f49e9b9473747447c5" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
I20250113 02:51:44.673794 455 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_dc9db684670a46229bcca7df696bb04c" } modify_external_catalogs: true
I20250113 02:51:44.673924 769 raft_consensus.cc:2463] T cf7b8bd6801743f49e9b9473747447c5 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 0.
I20250113 02:51:44.674603 1195 ts_tablet_manager.cc:1918] T b33025a462e74f2aa26cc3779f36615b P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250113 02:51:44.675451 995 leader_election.cc:304] T cf7b8bd6801743f49e9b9473747447c5 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters: df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:44.675747 2639 raft_consensus.cc:2798] T cf7b8bd6801743f49e9b9473747447c5 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 02:51:44.675797 1195 log.cc:1198] T b33025a462e74f2aa26cc3779f36615b P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/b33025a462e74f2aa26cc3779f36615b
I20250113 02:51:44.675850 2639 raft_consensus.cc:491] T cf7b8bd6801743f49e9b9473747447c5 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 02:51:44.675940 2639 raft_consensus.cc:3054] T cf7b8bd6801743f49e9b9473747447c5 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:44.676236 1195 ts_tablet_manager.cc:1939] T b33025a462e74f2aa26cc3779f36615b P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:44.676899 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1137) took 6471 ms. Trace:
I20250113 02:51:44.676967 1195 rpcz_store.cc:276] 0113 02:51:38.205280 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:38.205339 (+ 59us) service_pool.cc:225] Handling call
0113 02:51:44.634255 (+6428916us) ts_tablet_manager.cc:1145] Deleting tablet b33025a462e74f2aa26cc3779f36615b
0113 02:51:44.634262 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:44.640648 (+ 6386us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:44.674496 (+ 33848us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:44.676883 (+ 2387us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6427645}
I20250113 02:51:44.677326 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet b33025a462e74f2aa26cc3779f36615b (table default.Table_23c1655ab4cf4a30b67185ccf4fbb76d [id=a8e428598ca94e15b04f1df8112a4abc]) successfully deleted
I20250113 02:51:44.675343 1191 tablet_bootstrap.cc:654] T c73c649a4ca44471af2cf6a77500b065 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:44.678596 2639 raft_consensus.cc:513] T cf7b8bd6801743f49e9b9473747447c5 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:44.679294 2639 leader_election.cc:290] T cf7b8bd6801743f49e9b9473747447c5 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Requested vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:44.679838 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf7b8bd6801743f49e9b9473747447c5" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f"
W20250113 02:51:44.680508 998 leader_election.cc:343] T cf7b8bd6801743f49e9b9473747447c5 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:44.681300 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf7b8bd6801743f49e9b9473747447c5" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196"
I20250113 02:51:44.681499 769 raft_consensus.cc:3054] T cf7b8bd6801743f49e9b9473747447c5 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:44.681915 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1138) took 6466 ms. Trace:
I20250113 02:51:44.681977 1195 rpcz_store.cc:276] 0113 02:51:38.215344 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:38.215397 (+ 53us) service_pool.cc:225] Handling call
0113 02:51:44.681802 (+6466405us) ts_tablet_manager.cc:1145] Deleting tablet ed9e6ba0729b483fa6599dd6dde016e7
0113 02:51:44.681810 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:44.681900 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6466244}
W20250113 02:51:44.682349 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet ed9e6ba0729b483fa6599dd6dde016e7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ed9e6ba0729b483fa6599dd6dde016e7 already in progress: creating tablet
I20250113 02:51:44.683935 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1141) took 5933 ms. Trace:
I20250113 02:51:44.684029 1195 rpcz_store.cc:276] 0113 02:51:38.750876 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:38.750928 (+ 52us) service_pool.cc:225] Handling call
0113 02:51:44.683829 (+5932901us) ts_tablet_manager.cc:1145] Deleting tablet 13f9711b16ae4e36be00db07d9bde7fc
0113 02:51:44.683838 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:44.683920 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5932738}
I20250113 02:51:44.644264 1198 tablet_replica.cc:331] T 21bcdd741104474bad7023426856d18f P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:44.684427 769 raft_consensus.cc:2463] T cf7b8bd6801743f49e9b9473747447c5 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 1.
W20250113 02:51:44.685714 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 13f9711b16ae4e36be00db07d9bde7fc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 13f9711b16ae4e36be00db07d9bde7fc already in progress: creating tablet
W20250113 02:51:44.679008 2603 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
I20250113 02:51:44.685948 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1143) took 5924 ms. Trace:
I20250113 02:51:44.686009 1195 rpcz_store.cc:276] 0113 02:51:38.761906 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:38.761959 (+ 53us) service_pool.cc:225] Handling call
0113 02:51:44.685850 (+5923891us) ts_tablet_manager.cc:1145] Deleting tablet 368e878e89b54694823863a49074c11e
0113 02:51:44.685858 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:44.685935 (+ 77us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5923712}
W20250113 02:51:44.686203 2603 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
I20250113 02:51:44.686163 995 leader_election.cc:304] T cf7b8bd6801743f49e9b9473747447c5 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters: df69f48656b44bb890f76587ee69cc0f
W20250113 02:51:44.686305 2603 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250113 02:51:44.686409 2603 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250113 02:51:44.686599 2639 raft_consensus.cc:2798] T cf7b8bd6801743f49e9b9473747447c5 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Leader election won for term 1
I20250113 02:51:44.686786 2639 raft_consensus.cc:695] T cf7b8bd6801743f49e9b9473747447c5 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: Becoming Leader. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Running, Role: LEADER
W20250113 02:51:44.687621 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 368e878e89b54694823863a49074c11e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 368e878e89b54694823863a49074c11e already in progress: creating tablet
I20250113 02:51:44.687677 2639 consensus_queue.cc:237] T cf7b8bd6801743f49e9b9473747447c5 P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:44.687855 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1144) took 5924 ms. Trace:
I20250113 02:51:44.687928 1195 rpcz_store.cc:276] 0113 02:51:38.763402 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:38.763449 (+ 47us) service_pool.cc:225] Handling call
0113 02:51:44.687746 (+5924297us) ts_tablet_manager.cc:1145] Deleting tablet 71befed8b5c74b41b199919b82325891
0113 02:51:44.687755 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:44.687837 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5924150}
W20250113 02:51:44.689466 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 71befed8b5c74b41b199919b82325891 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 71befed8b5c74b41b199919b82325891 already in progress: creating tablet
I20250113 02:51:44.689741 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1142) took 5929 ms. Trace:
I20250113 02:51:44.690654 1195 rpcz_store.cc:276] 0113 02:51:38.760260 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:38.824610 (+ 64350us) service_pool.cc:225] Handling call
0113 02:51:44.689637 (+5865027us) ts_tablet_manager.cc:1145] Deleting tablet 49c2e648a49f4034a521f38e7d98edc4
0113 02:51:44.689646 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:44.689726 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5864703}
W20250113 02:51:44.692406 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 49c2e648a49f4034a521f38e7d98edc4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 49c2e648a49f4034a521f38e7d98edc4 already in progress: creating tablet
I20250113 02:51:44.692495 2723 raft_consensus.cc:491] T 883027967e0c442a91ac820210b9c6c7 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:44.692668 2723 raft_consensus.cc:513] T 883027967e0c442a91ac820210b9c6c7 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:44.693485 2723 leader_election.cc:290] T 883027967e0c442a91ac820210b9c6c7 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:44.694010 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1150) took 5860 ms. Trace:
I20250113 02:51:44.693939 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "883027967e0c442a91ac820210b9c6c7" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:44.695035 1198 raft_consensus.cc:2238] T 21bcdd741104474bad7023426856d18f P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: Raft consensus shutting down.
I20250113 02:51:44.695426 1198 raft_consensus.cc:2267] T 21bcdd741104474bad7023426856d18f P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Raft consensus is shut down!
W20250113 02:51:44.694443 711 leader_election.cc:343] T 883027967e0c442a91ac820210b9c6c7 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:44.695582 1195 rpcz_store.cc:276] 0113 02:51:38.833933 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:38.834190 (+ 257us) service_pool.cc:225] Handling call
0113 02:51:44.693736 (+5859546us) ts_tablet_manager.cc:1145] Deleting tablet a6d0f9a0a7104b8fb3d22aac976b7d99
0113 02:51:44.693901 (+ 165us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:44.693995 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5859359}
I20250113 02:51:44.691903 1191 tablet_bootstrap.cc:492] T c73c649a4ca44471af2cf6a77500b065 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:44.696561 1191 ts_tablet_manager.cc:1397] T c73c649a4ca44471af2cf6a77500b065 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.026s user 0.004s sys 0.001s
W20250113 02:51:44.696558 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet a6d0f9a0a7104b8fb3d22aac976b7d99 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a6d0f9a0a7104b8fb3d22aac976b7d99 already in progress: creating tablet
I20250113 02:51:44.698040 1191 raft_consensus.cc:357] T c73c649a4ca44471af2cf6a77500b065 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:44.700419 2476 raft_consensus.cc:491] T 788ab82626424b6e833e31761a15b9b2 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:44.700538 2476 raft_consensus.cc:513] T 788ab82626424b6e833e31761a15b9b2 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:44.701002 1191 raft_consensus.cc:383] T c73c649a4ca44471af2cf6a77500b065 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:44.701030 2476 leader_election.cc:290] T 788ab82626424b6e833e31761a15b9b2 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:44.701083 1191 raft_consensus.cc:738] T c73c649a4ca44471af2cf6a77500b065 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:44.701262 1191 consensus_queue.cc:260] T c73c649a4ca44471af2cf6a77500b065 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:44.699651 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1151) took 5864 ms. Trace:
I20250113 02:51:44.701474 1195 rpcz_store.cc:276] 0113 02:51:38.834836 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:38.837549 (+ 2713us) service_pool.cc:225] Handling call
0113 02:51:44.697263 (+5859714us) ts_tablet_manager.cc:1145] Deleting tablet f4acbe59400247678d2441b1b844d976
0113 02:51:44.699544 (+ 2281us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:44.699634 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5859522}
W20250113 02:51:44.701869 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet f4acbe59400247678d2441b1b844d976 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f4acbe59400247678d2441b1b844d976 already in progress: creating tablet
I20250113 02:51:44.701998 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1148) took 5870 ms. Trace:
I20250113 02:51:44.702126 1195 rpcz_store.cc:276] 0113 02:51:38.831166 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:38.856278 (+ 25112us) service_pool.cc:225] Handling call
0113 02:51:44.701887 (+5845609us) ts_tablet_manager.cc:1145] Deleting tablet 9fad58cccf1744b0a476523e0af0036e
0113 02:51:44.701896 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:44.701986 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5845317}
W20250113 02:51:44.702497 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 9fad58cccf1744b0a476523e0af0036e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9fad58cccf1744b0a476523e0af0036e already in progress: creating tablet
I20250113 02:51:44.702687 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1155) took 5812 ms. Trace:
I20250113 02:51:44.703243 1195 rpcz_store.cc:276] 0113 02:51:38.890451 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:38.905366 (+ 14915us) service_pool.cc:225] Handling call
0113 02:51:44.702591 (+5797225us) ts_tablet_manager.cc:1145] Deleting tablet 50c4d4783e6d4a55bd3bd355613e1ea0
0113 02:51:44.702599 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:44.702671 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5796747}
I20250113 02:51:44.706472 1060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "883027967e0c442a91ac820210b9c6c7" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
W20250113 02:51:44.706557 852 leader_election.cc:343] T 788ab82626424b6e833e31761a15b9b2 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Not found: Tablet not found: 788ab82626424b6e833e31761a15b9b2
I20250113 02:51:44.706705 1060 raft_consensus.cc:2463] T 883027967e0c442a91ac820210b9c6c7 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1186fa869eb14812832fa6562531f196 in term 0.
I20250113 02:51:44.707369 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1158) took 5779 ms. Trace:
I20250113 02:51:44.708698 1195 rpcz_store.cc:276] 0113 02:51:38.927517 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:38.927566 (+ 49us) service_pool.cc:225] Handling call
0113 02:51:44.707276 (+5779710us) ts_tablet_manager.cc:1145] Deleting tablet 17fc320eb01f4ac2b9361aeab5b452fc
0113 02:51:44.707286 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:44.707356 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5779540}
I20250113 02:51:44.709605 1195 tablet_replica.cc:331] T 6a4c6dc08df744e6856bdf99ab65c575 P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:44.709594 2363 consensus_queue.cc:1035] T 283fc7f1c74e480c8a63c74d3ae9eb7d P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250113 02:51:44.709712 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 50c4d4783e6d4a55bd3bd355613e1ea0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 50c4d4783e6d4a55bd3bd355613e1ea0 already in progress: creating tablet
W20250113 02:51:44.709939 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 17fc320eb01f4ac2b9361aeab5b452fc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 17fc320eb01f4ac2b9361aeab5b452fc already in progress: creating tablet
I20250113 02:51:44.710204 1195 raft_consensus.cc:2238] T 6a4c6dc08df744e6856bdf99ab65c575 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:44.710902 1195 raft_consensus.cc:2267] T 6a4c6dc08df744e6856bdf99ab65c575 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:44.708213 1191 ts_tablet_manager.cc:1428] T c73c649a4ca44471af2cf6a77500b065 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.011s user 0.001s sys 0.000s
I20250113 02:51:44.711745 1191 tablet_bootstrap.cc:492] T a2781ba5061a4225b5636366857a229e P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
W20250113 02:51:44.712215 851 leader_election.cc:343] T 788ab82626424b6e833e31761a15b9b2 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Not found: Tablet not found: 788ab82626424b6e833e31761a15b9b2
I20250113 02:51:44.712363 851 leader_election.cc:304] T 788ab82626424b6e833e31761a15b9b2 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df69f48656b44bb890f76587ee69cc0f; no voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca
I20250113 02:51:44.712800 2476 raft_consensus.cc:2743] T 788ab82626424b6e833e31761a15b9b2 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:44.713378 1191 tablet_bootstrap.cc:654] T a2781ba5061a4225b5636366857a229e P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:44.714834 1195 ts_tablet_manager.cc:1905] T 6a4c6dc08df744e6856bdf99ab65c575 P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:44.720541 1198 ts_tablet_manager.cc:1905] T 21bcdd741104474bad7023426856d18f P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:44.727826 709 leader_election.cc:304] T 883027967e0c442a91ac820210b9c6c7 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters: df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:44.728139 2723 raft_consensus.cc:2798] T 883027967e0c442a91ac820210b9c6c7 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 02:51:44.728230 2723 raft_consensus.cc:491] T 883027967e0c442a91ac820210b9c6c7 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 02:51:44.728305 2723 raft_consensus.cc:3054] T 883027967e0c442a91ac820210b9c6c7 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:44.729693 1191 tablet_bootstrap.cc:492] T a2781ba5061a4225b5636366857a229e P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:44.730888 1191 ts_tablet_manager.cc:1397] T a2781ba5061a4225b5636366857a229e P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.000s
I20250113 02:51:44.731125 2723 raft_consensus.cc:513] T 883027967e0c442a91ac820210b9c6c7 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:44.731917 2723 leader_election.cc:290] T 883027967e0c442a91ac820210b9c6c7 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 election: Requested vote from peers 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:44.732620 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "883027967e0c442a91ac820210b9c6c7" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f"
I20250113 02:51:44.733531 1191 raft_consensus.cc:357] T a2781ba5061a4225b5636366857a229e P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:44.733709 1191 raft_consensus.cc:383] T a2781ba5061a4225b5636366857a229e P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:44.733764 1191 raft_consensus.cc:738] T a2781ba5061a4225b5636366857a229e P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:44.735767 1191 consensus_queue.cc:260] T a2781ba5061a4225b5636366857a229e P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:44.736377 1191 ts_tablet_manager.cc:1428] T a2781ba5061a4225b5636366857a229e P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20250113 02:51:44.736632 1191 tablet_bootstrap.cc:492] T d47f544f723542bea1d396f586d8d658 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:44.738938 1191 tablet_bootstrap.cc:654] T d47f544f723542bea1d396f586d8d658 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
W20250113 02:51:44.739368 711 leader_election.cc:343] T 883027967e0c442a91ac820210b9c6c7 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:44.741214 1195 ts_tablet_manager.cc:1918] T 6a4c6dc08df744e6856bdf99ab65c575 P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250113 02:51:44.742949 1195 log.cc:1198] T 6a4c6dc08df744e6856bdf99ab65c575 P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/6a4c6dc08df744e6856bdf99ab65c575
I20250113 02:51:44.743623 1195 ts_tablet_manager.cc:1939] T 6a4c6dc08df744e6856bdf99ab65c575 P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:44.745479 1060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "883027967e0c442a91ac820210b9c6c7" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca"
I20250113 02:51:44.745698 1060 raft_consensus.cc:3054] T 883027967e0c442a91ac820210b9c6c7 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:44.746402 2474 raft_consensus.cc:491] T 88a776aa820c44b3a3854ca5022bd353 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:44.746795 2474 raft_consensus.cc:513] T 88a776aa820c44b3a3854ca5022bd353 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:44.748059 2474 leader_election.cc:290] T 88a776aa820c44b3a3854ca5022bd353 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:44.748272 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88a776aa820c44b3a3854ca5022bd353" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
I20250113 02:51:44.748414 1060 raft_consensus.cc:2463] T 883027967e0c442a91ac820210b9c6c7 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1186fa869eb14812832fa6562531f196 in term 1.
I20250113 02:51:44.748454 769 raft_consensus.cc:2463] T 88a776aa820c44b3a3854ca5022bd353 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 0.
I20250113 02:51:44.748869 995 leader_election.cc:304] T 88a776aa820c44b3a3854ca5022bd353 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters:
I20250113 02:51:44.749202 2474 raft_consensus.cc:2798] T 88a776aa820c44b3a3854ca5022bd353 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 02:51:44.749296 2474 raft_consensus.cc:491] T 88a776aa820c44b3a3854ca5022bd353 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 02:51:44.749419 2474 raft_consensus.cc:3054] T 88a776aa820c44b3a3854ca5022bd353 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:44.749787 1192 tablet_bootstrap.cc:492] T f2c6c8c17d6e499d84ad7c0c23889cc4 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:44.750209 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1159) took 5817 ms. Trace:
I20250113 02:51:44.750432 1192 ts_tablet_manager.cc:1397] T f2c6c8c17d6e499d84ad7c0c23889cc4 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.143s user 0.010s sys 0.018s
I20250113 02:51:44.750535 1195 rpcz_store.cc:276] 0113 02:51:38.933101 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:38.933160 (+ 59us) service_pool.cc:225] Handling call
0113 02:51:44.709560 (+5776400us) ts_tablet_manager.cc:1145] Deleting tablet 6a4c6dc08df744e6856bdf99ab65c575
0113 02:51:44.709571 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:44.718864 (+ 9293us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:44.741117 (+ 22253us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:44.750192 (+ 9075us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1,"tablet-delete.queue_time_us":5776206}
I20250113 02:51:44.751601 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet 6a4c6dc08df744e6856bdf99ab65c575 (table default.Table_82a014a9f48c43b4b14d63522bac7124 [id=c489a295f02b4f00ac7ea90e62d8ed04]) successfully deleted
I20250113 02:51:44.751857 2474 raft_consensus.cc:513] T 88a776aa820c44b3a3854ca5022bd353 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:44.752532 1192 raft_consensus.cc:357] T f2c6c8c17d6e499d84ad7c0c23889cc4 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:44.752725 1192 raft_consensus.cc:383] T f2c6c8c17d6e499d84ad7c0c23889cc4 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:44.752835 1192 raft_consensus.cc:738] T f2c6c8c17d6e499d84ad7c0c23889cc4 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:44.753016 1195 tablet_replica.cc:331] T 1968c7e9357c4c6e901d334ace0467fb P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:44.753388 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88a776aa820c44b3a3854ca5022bd353" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:44.753451 1195 raft_consensus.cc:2238] T 1968c7e9357c4c6e901d334ace0467fb P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:44.753610 1195 raft_consensus.cc:2267] T 1968c7e9357c4c6e901d334ace0467fb P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:44.753158 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88a776aa820c44b3a3854ca5022bd353" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196"
W20250113 02:51:44.753921 998 leader_election.cc:343] T 88a776aa820c44b3a3854ca5022bd353 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:44.753930 772 raft_consensus.cc:3054] T 88a776aa820c44b3a3854ca5022bd353 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:44.754920 709 leader_election.cc:304] T 883027967e0c442a91ac820210b9c6c7 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters: df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:44.755149 2723 raft_consensus.cc:2798] T 883027967e0c442a91ac820210b9c6c7 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Leader election won for term 1
I20250113 02:51:44.755254 2723 raft_consensus.cc:695] T 883027967e0c442a91ac820210b9c6c7 P 1186fa869eb14812832fa6562531f196 [term 1 LEADER]: Becoming Leader. State: Replica: 1186fa869eb14812832fa6562531f196, State: Running, Role: LEADER
I20250113 02:51:44.755489 2723 consensus_queue.cc:237] T 883027967e0c442a91ac820210b9c6c7 P 1186fa869eb14812832fa6562531f196 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:44.757154 1195 ts_tablet_manager.cc:1905] T 1968c7e9357c4c6e901d334ace0467fb P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:44.761018 1196 ts_tablet_manager.cc:1918] T bc533518faa24dffa3306ff092e8d507 P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:44.762166 2474 leader_election.cc:290] T 88a776aa820c44b3a3854ca5022bd353 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Requested vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:44.764813 1196 log.cc:1198] T bc533518faa24dffa3306ff092e8d507 P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/bc533518faa24dffa3306ff092e8d507
I20250113 02:51:44.764659 885 tablet_service.cc:1514] Processing DeleteTablet for tablet bbc7832c50114d6f890fec45a3d994e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:22 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:44.761816 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88a776aa820c44b3a3854ca5022bd353" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f"
W20250113 02:51:44.766194 998 leader_election.cc:343] T 88a776aa820c44b3a3854ca5022bd353 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:44.766809 1196 ts_tablet_manager.cc:1939] T bc533518faa24dffa3306ff092e8d507 P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:44.767134 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_7f54426882784252bf8e26b7028d253a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:44.767863 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet bc533518faa24dffa3306ff092e8d507 (table default.Table_03babf194cc448f296512a09b2e3fe3e [id=64b42698782f41a589577af20b33501e]) successfully deleted
W20250113 02:51:44.768483 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7f54426882784252bf8e26b7028d253a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:44.768780 1196 tablet_replica.cc:331] T eceed0f52c9f46bbaa78ca773a7afde5 P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:44.769042 1196 raft_consensus.cc:2238] T eceed0f52c9f46bbaa78ca773a7afde5 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:44.769129 772 raft_consensus.cc:2463] T 88a776aa820c44b3a3854ca5022bd353 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 1.
I20250113 02:51:44.769379 1196 raft_consensus.cc:2267] T eceed0f52c9f46bbaa78ca773a7afde5 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:44.769528 995 leader_election.cc:304] T 88a776aa820c44b3a3854ca5022bd353 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters: df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:44.769990 2474 raft_consensus.cc:2798] T 88a776aa820c44b3a3854ca5022bd353 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Leader election won for term 1
I20250113 02:51:44.770256 2474 raft_consensus.cc:695] T 88a776aa820c44b3a3854ca5022bd353 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: Becoming Leader. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Running, Role: LEADER
I20250113 02:51:44.765985 2603 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:45573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/master-2/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:44217,127.0.104.61:35271,127.0.104.60:42067
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.60:42067
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.104.60
--webserver_port=38411
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 13 Jan 2025 02:43:23 UTC on 24a791456cd2
build id 4151
ASAN enabled
I20250113 02:51:44.770849 2474 consensus_queue.cc:237] T 88a776aa820c44b3a3854ca5022bd353 P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:44.771692 1196 ts_tablet_manager.cc:1905] T eceed0f52c9f46bbaa78ca773a7afde5 P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:44.772553 2603 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250113 02:51:44.774773 2603 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250113 02:51:44.774941 1195 ts_tablet_manager.cc:1918] T 1968c7e9357c4c6e901d334ace0467fb P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250113 02:51:44.775031 1195 log.cc:1198] T 1968c7e9357c4c6e901d334ace0467fb P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/1968c7e9357c4c6e901d334ace0467fb
I20250113 02:51:44.775333 1195 ts_tablet_manager.cc:1939] T 1968c7e9357c4c6e901d334ace0467fb P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:44.776014 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1163) took 5781 ms. Trace:
I20250113 02:51:44.776104 1195 rpcz_store.cc:276] 0113 02:51:38.994478 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:38.994530 (+ 52us) service_pool.cc:225] Handling call
0113 02:51:44.751623 (+5757093us) ts_tablet_manager.cc:1145] Deleting tablet 1968c7e9357c4c6e901d334ace0467fb
0113 02:51:44.751633 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:44.772144 (+ 20511us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:44.774855 (+ 2711us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:44.775968 (+ 1113us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5756894}
I20250113 02:51:44.776522 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet 1968c7e9357c4c6e901d334ace0467fb (table default.Table_f2da084a4aee4d19b45e784e56f926a6 [id=fa9bcd8c66c34d65b43561572abb8890]) successfully deleted
I20250113 02:51:44.776957 1195 tablet_replica.cc:331] T aa1ebdb92a5f4c92a5195b614689c100 P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:44.777128 1195 raft_consensus.cc:2238] T aa1ebdb92a5f4c92a5195b614689c100 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:44.777274 1195 raft_consensus.cc:2267] T aa1ebdb92a5f4c92a5195b614689c100 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:44.778100 1195 ts_tablet_manager.cc:1905] T aa1ebdb92a5f4c92a5195b614689c100 P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:44.779407 1198 ts_tablet_manager.cc:1918] T 21bcdd741104474bad7023426856d18f P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250113 02:51:44.779515 1198 log.cc:1198] T 21bcdd741104474bad7023426856d18f P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/21bcdd741104474bad7023426856d18f
I20250113 02:51:44.779851 1198 ts_tablet_manager.cc:1939] T 21bcdd741104474bad7023426856d18f P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:44.780138 1192 consensus_queue.cc:260] T f2c6c8c17d6e499d84ad7c0c23889cc4 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:44.780660 1192 ts_tablet_manager.cc:1428] T f2c6c8c17d6e499d84ad7c0c23889cc4 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.030s user 0.000s sys 0.001s
I20250113 02:51:44.783304 1191 tablet_bootstrap.cc:492] T d47f544f723542bea1d396f586d8d658 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:44.783429 1191 ts_tablet_manager.cc:1397] T d47f544f723542bea1d396f586d8d658 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.047s user 0.003s sys 0.000s
I20250113 02:51:44.785318 1195 ts_tablet_manager.cc:1918] T aa1ebdb92a5f4c92a5195b614689c100 P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250113 02:51:44.785462 1195 log.cc:1198] T aa1ebdb92a5f4c92a5195b614689c100 P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/aa1ebdb92a5f4c92a5195b614689c100
I20250113 02:51:44.785822 1195 ts_tablet_manager.cc:1939] T aa1ebdb92a5f4c92a5195b614689c100 P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:44.786577 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1164) took 5792 ms. Trace:
I20250113 02:51:44.786657 1191 raft_consensus.cc:357] T d47f544f723542bea1d396f586d8d658 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:44.786832 1191 raft_consensus.cc:383] T d47f544f723542bea1d396f586d8d658 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:44.786888 1191 raft_consensus.cc:738] T d47f544f723542bea1d396f586d8d658 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:44.786705 1195 rpcz_store.cc:276] 0113 02:51:38.994602 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:38.994650 (+ 48us) service_pool.cc:225] Handling call
0113 02:51:44.776909 (+5782259us) ts_tablet_manager.cc:1145] Deleting tablet aa1ebdb92a5f4c92a5195b614689c100
0113 02:51:44.776918 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:44.781891 (+ 4973us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:44.785255 (+ 3364us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:44.786560 (+ 1305us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5782105}
I20250113 02:51:44.787096 1191 consensus_queue.cc:260] T d47f544f723542bea1d396f586d8d658 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:44.787339 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet aa1ebdb92a5f4c92a5195b614689c100 (table default.Table_74ce8ff3e2c64e088ac82c48ec722b39 [id=4a5048f6224642deafdc335e17679294]) successfully deleted
I20250113 02:51:44.787616 1191 ts_tablet_manager.cc:1428] T d47f544f723542bea1d396f586d8d658 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20250113 02:51:44.787832 1191 tablet_bootstrap.cc:492] T fe3d35d9aaf64188aaa3c1ac38d279d8 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:44.789407 1191 tablet_bootstrap.cc:654] T fe3d35d9aaf64188aaa3c1ac38d279d8 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:44.790863 1194 tablet_bootstrap.cc:492] T 42dbd37e20fa47318bc6497423dbe565 P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:44.791016 1194 ts_tablet_manager.cc:1397] T 42dbd37e20fa47318bc6497423dbe565 P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.166s user 0.007s sys 0.016s
I20250113 02:51:44.791810 1194 raft_consensus.cc:357] T 42dbd37e20fa47318bc6497423dbe565 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:44.792052 1194 raft_consensus.cc:383] T 42dbd37e20fa47318bc6497423dbe565 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:44.792284 1194 raft_consensus.cc:738] T 42dbd37e20fa47318bc6497423dbe565 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:44.792492 1194 consensus_queue.cc:260] T 42dbd37e20fa47318bc6497423dbe565 P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:44.792912 1194 ts_tablet_manager.cc:1428] T 42dbd37e20fa47318bc6497423dbe565 P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250113 02:51:44.793191 1194 tablet_bootstrap.cc:492] T ba8980ad3a364c888a825affbcbc9b1f P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:44.794868 1194 tablet_bootstrap.cc:654] T ba8980ad3a364c888a825affbcbc9b1f P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:44.794920 1195 tablet_replica.cc:331] T acd83024592e431898e9675c17f50d4f P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:44.795084 1195 raft_consensus.cc:2238] T acd83024592e431898e9675c17f50d4f P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:44.795243 1195 raft_consensus.cc:2267] T acd83024592e431898e9675c17f50d4f P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:44.780929 1192 tablet_bootstrap.cc:492] T 65d9bbd8c15f40a4a50c1695fb5e648e P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:44.796079 1195 ts_tablet_manager.cc:1905] T acd83024592e431898e9675c17f50d4f P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:44.797390 1192 tablet_bootstrap.cc:654] T 65d9bbd8c15f40a4a50c1695fb5e648e P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:44.802409 2723 raft_consensus.cc:491] T 68c4a7e8c0fb4feb85498709b00d3258 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:44.802533 2723 raft_consensus.cc:513] T 68c4a7e8c0fb4feb85498709b00d3258 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:44.803119 2723 leader_election.cc:290] T 68c4a7e8c0fb4feb85498709b00d3258 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:44.804190 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68c4a7e8c0fb4feb85498709b00d3258" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
W20250113 02:51:44.804749 709 leader_election.cc:343] T 68c4a7e8c0fb4feb85498709b00d3258 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:44.805042 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68c4a7e8c0fb4feb85498709b00d3258" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:44.805500 711 leader_election.cc:343] T 68c4a7e8c0fb4feb85498709b00d3258 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:44.805598 711 leader_election.cc:304] T 68c4a7e8c0fb4feb85498709b00d3258 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:44.805946 2723 raft_consensus.cc:2743] T 68c4a7e8c0fb4feb85498709b00d3258 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:44.807955 1195 ts_tablet_manager.cc:1918] T acd83024592e431898e9675c17f50d4f P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250113 02:51:44.808215 1195 log.cc:1198] T acd83024592e431898e9675c17f50d4f P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/acd83024592e431898e9675c17f50d4f
I20250113 02:51:44.808761 1195 ts_tablet_manager.cc:1939] T acd83024592e431898e9675c17f50d4f P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:44.810158 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1162) took 5815 ms. Trace:
I20250113 02:51:44.810350 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1413) took 1578 ms. Trace:
I20250113 02:51:44.810285 1195 rpcz_store.cc:276] 0113 02:51:38.994371 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:38.995122 (+ 751us) service_pool.cc:225] Handling call
0113 02:51:44.794865 (+5799743us) ts_tablet_manager.cc:1145] Deleting tablet acd83024592e431898e9675c17f50d4f
0113 02:51:44.794881 (+ 16us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:44.801295 (+ 6414us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:44.807868 (+ 6573us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:44.810136 (+ 2268us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5799509}
I20250113 02:51:44.810428 1198 rpcz_store.cc:276] 0113 02:51:43.232349 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:43.232415 (+ 66us) service_pool.cc:225] Handling call
0113 02:51:44.644190 (+1411775us) ts_tablet_manager.cc:1145] Deleting tablet 21bcdd741104474bad7023426856d18f
0113 02:51:44.644201 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:44.776506 (+132305us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:44.779316 (+ 2810us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:44.810332 (+ 31016us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1411603}
I20250113 02:51:44.810848 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet acd83024592e431898e9675c17f50d4f (table default.Table_25cb4fc43dc6485ca0e4ec0cf048105d [id=044304e1c4b44cc28b8990d6f28145a8]) successfully deleted
I20250113 02:51:44.811028 1198 tablet_replica.cc:331] T 283fc7f1c74e480c8a63c74d3ae9eb7d P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:44.810848 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet 21bcdd741104474bad7023426856d18f (table default.Table_4ef3edc5b07c4e0881456bfe23741c90 [id=46a732d6c22e472d82163d1295b34dcd]) successfully deleted
I20250113 02:51:44.811813 1191 tablet_bootstrap.cc:492] T fe3d35d9aaf64188aaa3c1ac38d279d8 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:44.811949 1191 ts_tablet_manager.cc:1397] T fe3d35d9aaf64188aaa3c1ac38d279d8 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.024s user 0.002s sys 0.000s
I20250113 02:51:44.812667 1191 raft_consensus.cc:357] T fe3d35d9aaf64188aaa3c1ac38d279d8 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:44.812853 1191 raft_consensus.cc:383] T fe3d35d9aaf64188aaa3c1ac38d279d8 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:44.812918 1191 raft_consensus.cc:738] T fe3d35d9aaf64188aaa3c1ac38d279d8 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:44.813601 1191 consensus_queue.cc:260] T fe3d35d9aaf64188aaa3c1ac38d279d8 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:44.813874 2476 raft_consensus.cc:491] T 0644f5600c5b47e29f3fa73060ed96e3 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:44.814066 2476 raft_consensus.cc:513] T 0644f5600c5b47e29f3fa73060ed96e3 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:44.814724 2476 leader_election.cc:290] T 0644f5600c5b47e29f3fa73060ed96e3 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:44.814872 1191 ts_tablet_manager.cc:1428] T fe3d35d9aaf64188aaa3c1ac38d279d8 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
W20250113 02:51:44.816490 852 leader_election.cc:343] T 0644f5600c5b47e29f3fa73060ed96e3 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Not found: Tablet not found: 0644f5600c5b47e29f3fa73060ed96e3
W20250113 02:51:44.818693 998 consensus_peers.cc:487] T cfa963a6df094ce0b9c54408f75ca6b2 P 4216a7f853d642b7bc3e5cff458f3aca -> Peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Couldn't send request to peer df69f48656b44bb890f76587ee69cc0f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250113 02:51:44.821111 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 08180ab0274a40fd955dda5b08197a08
I20250113 02:51:44.821604 450 catalog_manager.cc:6596] ReplaceTablet: tablet 08180ab0274a40fd955dda5b08197a08 deleted and replaced by tablet bc88415f52a0424db18574c886dd6f1a
I20250113 02:51:44.821990 885 tablet_service.cc:1514] Processing DeleteTablet for tablet 08180ab0274a40fd955dda5b08197a08 with delete_type TABLET_DATA_DELETED (replaced by tablet bc88415f52a0424db18574c886dd6f1a) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:44.822536 736 tablet_service.cc:1514] Processing DeleteTablet for tablet 08180ab0274a40fd955dda5b08197a08 with delete_type TABLET_DATA_DELETED (replaced by tablet bc88415f52a0424db18574c886dd6f1a) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:44.822683 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet 08180ab0274a40fd955dda5b08197a08 with delete_type TABLET_DATA_DELETED (replaced by tablet bc88415f52a0424db18574c886dd6f1a) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:44.823231 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_5b9ead21b07d41b2ad99e51e4c522ac2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:44.823761 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5b9ead21b07d41b2ad99e51e4c522ac2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:44.825946 885 tablet_service.cc:1514] Processing DeleteTablet for tablet 42dbd37e20fa47318bc6497423dbe565 with delete_type TABLET_DATA_DELETED (replaced by tablet 9174df876fe64adc805fba0a36fb4d38) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:44.827636 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_6441bd25946b4757af31d88201d6e3f2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:44.828212 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6441bd25946b4757af31d88201d6e3f2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:44.828805 1198 raft_consensus.cc:2238] T 283fc7f1c74e480c8a63c74d3ae9eb7d P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: Raft consensus shutting down.
I20250113 02:51:44.829190 1198 raft_consensus.cc:2267] T 283fc7f1c74e480c8a63c74d3ae9eb7d P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:44.829550 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 3a89e991ebee4b4f8c639ea95e1d1ea3
I20250113 02:51:44.830116 1198 ts_tablet_manager.cc:1905] T 283fc7f1c74e480c8a63c74d3ae9eb7d P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:44.830227 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_50f8312d0d834e25ab560f9c59fa3ce0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:44.830639 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_50f8312d0d834e25ab560f9c59fa3ce0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:44.831302 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1176) took 5516 ms. Trace:
I20250113 02:51:44.831420 1195 rpcz_store.cc:276] 0113 02:51:39.315185 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:39.315242 (+ 57us) service_pool.cc:225] Handling call
0113 02:51:44.812758 (+5497516us) ts_tablet_manager.cc:1145] Deleting tablet 960a7078181f4621a5ff4f3eb3050de3
0113 02:51:44.812767 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:44.831279 (+ 18512us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5490291}
I20250113 02:51:44.832338 1192 tablet_bootstrap.cc:492] T 65d9bbd8c15f40a4a50c1695fb5e648e P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:44.832496 1192 ts_tablet_manager.cc:1397] T 65d9bbd8c15f40a4a50c1695fb5e648e P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.052s user 0.003s sys 0.000s
I20250113 02:51:44.833381 885 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a89e991ebee4b4f8c639ea95e1d1ea3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:44 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:44.833576 1192 raft_consensus.cc:357] T 65d9bbd8c15f40a4a50c1695fb5e648e P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:44.833755 1192 raft_consensus.cc:383] T 65d9bbd8c15f40a4a50c1695fb5e648e P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:44.833806 1192 raft_consensus.cc:738] T 65d9bbd8c15f40a4a50c1695fb5e648e P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
W20250113 02:51:44.833998 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 960a7078181f4621a5ff4f3eb3050de3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 960a7078181f4621a5ff4f3eb3050de3 already in progress: creating tablet
I20250113 02:51:44.833999 1192 consensus_queue.cc:260] T 65d9bbd8c15f40a4a50c1695fb5e648e P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:44.834707 2723 raft_consensus.cc:491] T f4379f7fc6b54e6785f27975d9030cce P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:44.834738 736 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a89e991ebee4b4f8c639ea95e1d1ea3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:44 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:44.834823 2723 raft_consensus.cc:513] T f4379f7fc6b54e6785f27975d9030cce P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:44.835155 1192 ts_tablet_manager.cc:1428] T 65d9bbd8c15f40a4a50c1695fb5e648e P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250113 02:51:44.835420 2723 leader_election.cc:290] T f4379f7fc6b54e6785f27975d9030cce P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:44.835556 1192 tablet_bootstrap.cc:492] T 9b2d0aa6f0e9456d95f4e48c360ab6e8 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:44.836069 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4379f7fc6b54e6785f27975d9030cce" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:44.836401 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a89e991ebee4b4f8c639ea95e1d1ea3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:44 UTC) from {username='slave'} at 127.0.0.1:60230
W20250113 02:51:44.836903 709 leader_election.cc:343] T f4379f7fc6b54e6785f27975d9030cce P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:44.837230 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4379f7fc6b54e6785f27975d9030cce" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:44.837694 711 leader_election.cc:343] T f4379f7fc6b54e6785f27975d9030cce P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:44.837782 711 leader_election.cc:304] T f4379f7fc6b54e6785f27975d9030cce P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:44.838203 2723 raft_consensus.cc:2743] T f4379f7fc6b54e6785f27975d9030cce P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:44.839733 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_28f9de083c724932a0a03761e6148593"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:44.840226 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_28f9de083c724932a0a03761e6148593 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:44.840341 1198 ts_tablet_manager.cc:1918] T 283fc7f1c74e480c8a63c74d3ae9eb7d P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250113 02:51:44.840436 1198 log.cc:1198] T 283fc7f1c74e480c8a63c74d3ae9eb7d P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/283fc7f1c74e480c8a63c74d3ae9eb7d
I20250113 02:51:44.840804 1198 ts_tablet_manager.cc:1939] T 283fc7f1c74e480c8a63c74d3ae9eb7d P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:44.841073 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1414) took 1608 ms. Trace:
I20250113 02:51:44.841150 1198 rpcz_store.cc:276] 0113 02:51:43.232631 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:43.232679 (+ 48us) service_pool.cc:225] Handling call
0113 02:51:44.810984 (+1578305us) ts_tablet_manager.cc:1145] Deleting tablet 283fc7f1c74e480c8a63c74d3ae9eb7d
0113 02:51:44.810992 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:44.837508 (+ 26516us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:44.840266 (+ 2758us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:44.841058 (+ 792us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":17580,"tablet-delete.queue_time_us":1578177}
I20250113 02:51:44.841631 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet 283fc7f1c74e480c8a63c74d3ae9eb7d (table default.Table_1505767419694e5ba79a90c46780bbb7 [id=7128c928ad5b40aa910aa297049c4944]) successfully deleted
I20250113 02:51:44.842253 2723 raft_consensus.cc:491] T 6001369444894b48b51a48fb60ddc52a P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:44.842367 2723 raft_consensus.cc:513] T 6001369444894b48b51a48fb60ddc52a P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:44.842895 2723 leader_election.cc:290] T 6001369444894b48b51a48fb60ddc52a P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:44.843396 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6001369444894b48b51a48fb60ddc52a" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:44.844120 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6001369444894b48b51a48fb60ddc52a" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:44.844583 709 leader_election.cc:343] T 6001369444894b48b51a48fb60ddc52a P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
W20250113 02:51:44.844897 711 leader_election.cc:343] T 6001369444894b48b51a48fb60ddc52a P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:44.844985 711 leader_election.cc:304] T 6001369444894b48b51a48fb60ddc52a P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:44.845315 2723 raft_consensus.cc:2743] T 6001369444894b48b51a48fb60ddc52a P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:44.845611 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_bbd8e889c0eb420fbda882e52e6beb0b" } new_table_name: "default.Table_8cf41dedb3ca4e848b36d1e6e19e5126" modify_external_catalogs: true
I20250113 02:51:44.846429 2723 raft_consensus.cc:491] T 39a014ac69a2435a9132a75125063093 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250113 02:51:44.846562 998 consensus_peers.cc:487] T 79aaab73797c474bae06831fa308b066 P 4216a7f853d642b7bc3e5cff458f3aca -> Peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Couldn't send request to peer df69f48656b44bb890f76587ee69cc0f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250113 02:51:44.846542 2723 raft_consensus.cc:513] T 39a014ac69a2435a9132a75125063093 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:44.846762 1198 tablet_replica.cc:331] T 9f8eb43a39774d2c8e9cb4e88a405b0f P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:44.846905 1198 raft_consensus.cc:2238] T 9f8eb43a39774d2c8e9cb4e88a405b0f P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: Raft consensus shutting down.
I20250113 02:51:44.847090 1198 pending_rounds.cc:62] T 9f8eb43a39774d2c8e9cb4e88a405b0f P 4216a7f853d642b7bc3e5cff458f3aca: Trying to abort 1 pending ops.
I20250113 02:51:44.847090 2723 leader_election.cc:290] T 39a014ac69a2435a9132a75125063093 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:44.847141 1198 pending_rounds.cc:69] T 9f8eb43a39774d2c8e9cb4e88a405b0f P 4216a7f853d642b7bc3e5cff458f3aca: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113673539157614592 op_type: NO_OP noop_request { }
I20250113 02:51:44.847220 1198 raft_consensus.cc:2883] T 9f8eb43a39774d2c8e9cb4e88a405b0f P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250113 02:51:44.847548 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39a014ac69a2435a9132a75125063093" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:44.847895 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39a014ac69a2435a9132a75125063093" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:44.848388 711 leader_election.cc:343] T 39a014ac69a2435a9132a75125063093 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
W20250113 02:51:44.848781 709 leader_election.cc:343] T 39a014ac69a2435a9132a75125063093 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:44.848863 709 leader_election.cc:304] T 39a014ac69a2435a9132a75125063093 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:44.849328 2723 raft_consensus.cc:2743] T 39a014ac69a2435a9132a75125063093 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250113 02:51:44.850237 851 leader_election.cc:343] T 0644f5600c5b47e29f3fa73060ed96e3 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Not found: Tablet not found: 0644f5600c5b47e29f3fa73060ed96e3
I20250113 02:51:44.850445 851 leader_election.cc:304] T 0644f5600c5b47e29f3fa73060ed96e3 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df69f48656b44bb890f76587ee69cc0f; no voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca
I20250113 02:51:44.850482 1198 raft_consensus.cc:2267] T 9f8eb43a39774d2c8e9cb4e88a405b0f P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:44.850914 2476 raft_consensus.cc:2743] T 0644f5600c5b47e29f3fa73060ed96e3 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:44.851267 1198 ts_tablet_manager.cc:1905] T 9f8eb43a39774d2c8e9cb4e88a405b0f P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:44.852001 1195 tablet_replica.cc:331] T 283fc7f1c74e480c8a63c74d3ae9eb7d P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:44.852187 1195 raft_consensus.cc:2238] T 283fc7f1c74e480c8a63c74d3ae9eb7d P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:44.852309 1195 raft_consensus.cc:2267] T 283fc7f1c74e480c8a63c74d3ae9eb7d P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:44.852492 1191 tablet_bootstrap.cc:492] T 936b8ceaedd34a019a4fb89a80dd2708 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:44.853133 1195 ts_tablet_manager.cc:1905] T 283fc7f1c74e480c8a63c74d3ae9eb7d P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:44.854004 1191 tablet_bootstrap.cc:654] T 936b8ceaedd34a019a4fb89a80dd2708 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:44.854970 1192 tablet_bootstrap.cc:654] T 9b2d0aa6f0e9456d95f4e48c360ab6e8 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:44.855568 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_73371ec3d0f94477b851fc6ce3028d8e" } new_table_name: "default.Table_6614ddab2a14402f8904c042817148aa" modify_external_catalogs: true
W20250113 02:51:44.857735 995 consensus_peers.cc:487] T cfa963a6df094ce0b9c54408f75ca6b2 P 4216a7f853d642b7bc3e5cff458f3aca -> Peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Couldn't send request to peer 1186fa869eb14812832fa6562531f196. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cfa963a6df094ce0b9c54408f75ca6b2. This is attempt 1: this message will repeat every 5th retry.
I20250113 02:51:44.858724 1195 ts_tablet_manager.cc:1918] T 283fc7f1c74e480c8a63c74d3ae9eb7d P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250113 02:51:44.860440 1195 log.cc:1198] T 283fc7f1c74e480c8a63c74d3ae9eb7d P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/283fc7f1c74e480c8a63c74d3ae9eb7d
I20250113 02:51:44.860947 1195 ts_tablet_manager.cc:1939] T 283fc7f1c74e480c8a63c74d3ae9eb7d P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:44.862272 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1180) took 5524 ms. Trace:
I20250113 02:51:44.866070 1195 rpcz_store.cc:276] 0113 02:51:39.338260 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:39.338536 (+ 276us) service_pool.cc:225] Handling call
0113 02:51:44.832996 (+5494460us) ts_tablet_manager.cc:1145] Deleting tablet 283fc7f1c74e480c8a63c74d3ae9eb7d
0113 02:51:44.851946 (+ 18950us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:44.856060 (+ 4114us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:44.858650 (+ 2590us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:44.862256 (+ 3606us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5492798}
I20250113 02:51:44.866789 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet 283fc7f1c74e480c8a63c74d3ae9eb7d (table default.Table_1505767419694e5ba79a90c46780bbb7 [id=7128c928ad5b40aa910aa297049c4944]) successfully deleted
I20250113 02:51:44.866791 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1197) took 5327 ms. Trace:
I20250113 02:51:44.866881 1195 rpcz_store.cc:276] 0113 02:51:39.539103 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:39.539307 (+ 204us) service_pool.cc:225] Handling call
0113 02:51:44.866662 (+5327355us) ts_tablet_manager.cc:1145] Deleting tablet 16f547e3cec54baf86b96ad40e3afb6a
0113 02:51:44.866674 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:44.866774 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5321269}
I20250113 02:51:44.867123 446 catalog_manager.cc:5618] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587) does not have the latest schema for tablet 938cc1ccd8dd449996836188ba9e64ff (table default.Table_e5b4d614e4b743d7982b025ca6d297e5 [id=5824acd778ed47d09f087bbf8e7822d6]). Expected version 1 got 0
I20250113 02:51:44.867224 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1199) took 5327 ms. Trace:
I20250113 02:51:44.867281 1195 rpcz_store.cc:276] 0113 02:51:39.539662 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:39.539774 (+ 112us) service_pool.cc:225] Handling call
0113 02:51:44.867130 (+5327356us) ts_tablet_manager.cc:1145] Deleting tablet e5ed7016af184b86be4c234f42e2a6ef
0113 02:51:44.867139 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:44.867211 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5321616}
W20250113 02:51:44.867370 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 16f547e3cec54baf86b96ad40e3afb6a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 16f547e3cec54baf86b96ad40e3afb6a already in progress: creating tablet
I20250113 02:51:44.867380 446 catalog_manager.cc:5618] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587) does not have the latest schema for tablet c13e9a51c1e74b44b352bade78390ba5 (table default.Table_0468611ec68e47109f5357935e50874d [id=caf098e2b57849a997bd7c8eaf1f9326]). Expected version 1 got 0
I20250113 02:51:44.867597 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1198) took 5328 ms. Trace:
I20250113 02:51:44.867646 1195 rpcz_store.cc:276] 0113 02:51:39.539558 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:39.566724 (+ 27166us) service_pool.cc:225] Handling call
0113 02:51:44.867501 (+5300777us) ts_tablet_manager.cc:1145] Deleting tablet 128f74fdc98f4135b39dbf23f1d987b0
0113 02:51:44.867508 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:44.867585 (+ 77us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5300366}
W20250113 02:51:44.867765 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet e5ed7016af184b86be4c234f42e2a6ef because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e5ed7016af184b86be4c234f42e2a6ef already in progress: creating tablet
I20250113 02:51:44.867833 446 catalog_manager.cc:5618] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587) does not have the latest schema for tablet 019eb21c661547cfa0b619d119eb024e (table default.Table_022a7c61a6864393acd739f562893b7c [id=97bfbd46eecd4e57b7f9757c413f6795]). Expected version 1 got 0
I20250113 02:51:44.867937 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1223) took 4846 ms. Trace:
I20250113 02:51:44.868072 446 catalog_manager.cc:5618] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587) does not have the latest schema for tablet e01e7f8db83d461bbfbb2e901419ce0b (table default.Table_e27f8b6f8232415388a5e53ab15494b7 [id=efc4ebba97f74c9ba12ab1dcc6577da0]). Expected version 1 got 0
I20250113 02:51:44.868008 1195 rpcz_store.cc:276] 0113 02:51:40.020993 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:40.021038 (+ 45us) service_pool.cc:225] Handling call
0113 02:51:44.867837 (+4846799us) ts_tablet_manager.cc:1145] Deleting tablet 045167a6122a4560b298aae2358a485c
0113 02:51:44.867847 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:44.867913 (+ 66us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4846634}
W20250113 02:51:44.868109 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 128f74fdc98f4135b39dbf23f1d987b0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 128f74fdc98f4135b39dbf23f1d987b0 already in progress: creating tablet
W20250113 02:51:44.868299 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 045167a6122a4560b298aae2358a485c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 045167a6122a4560b298aae2358a485c already in progress: creating tablet
I20250113 02:51:44.870105 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1222) took 4849 ms. Trace:
I20250113 02:51:44.870187 1195 rpcz_store.cc:276] 0113 02:51:40.020899 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:40.024151 (+ 3252us) service_pool.cc:225] Handling call
0113 02:51:44.869980 (+4845829us) ts_tablet_manager.cc:1145] Deleting tablet 61c79029dfb74f01aaac0f7f828880d0
0113 02:51:44.869992 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:44.870086 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4845580}
W20250113 02:51:44.870581 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 61c79029dfb74f01aaac0f7f828880d0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 61c79029dfb74f01aaac0f7f828880d0 already in progress: creating tablet
I20250113 02:51:44.871165 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet 79aaab73797c474bae06831fa308b066 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:30 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:44.871403 1032 tablet_service.cc:1514] Processing DeleteTablet for tablet aa1ebdb92a5f4c92a5195b614689c100 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:29 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:44.871563 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1225) took 4773 ms. Trace:
I20250113 02:51:44.871626 1195 rpcz_store.cc:276] 0113 02:51:40.098199 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:40.101339 (+ 3140us) service_pool.cc:225] Handling call
0113 02:51:44.871478 (+4770139us) ts_tablet_manager.cc:1145] Deleting tablet 797d91d4aae240bd82b00021717772ae
0113 02:51:44.871486 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:44.871549 (+ 63us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4768938}
W20250113 02:51:44.872361 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 797d91d4aae240bd82b00021717772ae because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 797d91d4aae240bd82b00021717772ae already in progress: creating tablet
I20250113 02:51:44.872838 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet 8cc78bf7d6324b7ba10a1b1045923f2b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:32 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:44.873034 1039 tablet_service.cc:1514] Processing DeleteTablet for tablet fd0b8592997d4a61b8ca854ecbb67c55 with delete_type TABLET_DATA_DELETED (replaced by tablet e312e70334e04ecd843159ac72f4a2a6) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:44.873050 448 catalog_manager.cc:5618] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095) does not have the latest schema for tablet 938cc1ccd8dd449996836188ba9e64ff (table default.Table_e5b4d614e4b743d7982b025ca6d297e5 [id=5824acd778ed47d09f087bbf8e7822d6]). Expected version 1 got 0
I20250113 02:51:44.873188 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet 67baaab1e36e415381579d7385d4ab5a with delete_type TABLET_DATA_DELETED (replaced by tablet c368b5b0c3a940c4b4efb64c946375c3) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:44.873217 448 catalog_manager.cc:5618] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095) does not have the latest schema for tablet c13e9a51c1e74b44b352bade78390ba5 (table default.Table_0468611ec68e47109f5357935e50874d [id=caf098e2b57849a997bd7c8eaf1f9326]). Expected version 1 got 0
I20250113 02:51:44.873378 448 catalog_manager.cc:5618] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095) does not have the latest schema for tablet 68c4a7e8c0fb4feb85498709b00d3258 (table default.Table_c90f3a5c35c842f6ad2cb74cf29dac13 [id=8d2f84cd670a4e8d9fa17b8313d3fab4]). Expected version 2 got 0
I20250113 02:51:44.873575 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1226) took 4772 ms. Trace:
I20250113 02:51:44.873639 1195 rpcz_store.cc:276] 0113 02:51:40.101590 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:40.101865 (+ 275us) service_pool.cc:225] Handling call
0113 02:51:44.873483 (+4771618us) ts_tablet_manager.cc:1145] Deleting tablet 67fe2f52d98048dab9f37dd820b7f712
0113 02:51:44.873492 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:44.873561 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4770889}
I20250113 02:51:44.873719 1032 tablet_service.cc:1514] Processing DeleteTablet for tablet cf7b8bd6801743f49e9b9473747447c5 with delete_type TABLET_DATA_DELETED (replaced by tablet a8aedbc8d7bb4183bb1ab9902f07bea7) from {username='slave'} at 127.0.0.1:60230
W20250113 02:51:44.874018 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 67fe2f52d98048dab9f37dd820b7f712 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 67fe2f52d98048dab9f37dd820b7f712 already in progress: creating tablet
I20250113 02:51:44.875514 736 tablet_service.cc:1514] Processing DeleteTablet for tablet aa1ebdb92a5f4c92a5195b614689c100 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:29 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:44.875916 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a4c6dc08df744e6856bdf99ab65c575 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:26 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:44.876168 1195 tablet_replica.cc:331] T fd0b8592997d4a61b8ca854ecbb67c55 P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:44.876255 736 tablet_service.cc:1514] Processing DeleteTablet for tablet 1968c7e9357c4c6e901d334ace0467fb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:26 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:44.876348 1195 raft_consensus.cc:2238] T fd0b8592997d4a61b8ca854ecbb67c55 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:44.876825 1191 tablet_bootstrap.cc:492] T 936b8ceaedd34a019a4fb89a80dd2708 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:44.876885 736 tablet_service.cc:1514] Processing DeleteTablet for tablet 883027967e0c442a91ac820210b9c6c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:31 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:44.876981 739 tablet_service.cc:1514] Processing DeleteTablet for tablet f4379f7fc6b54e6785f27975d9030cce with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:31 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:44.877028 1191 ts_tablet_manager.cc:1397] T 936b8ceaedd34a019a4fb89a80dd2708 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.025s user 0.006s sys 0.000s
I20250113 02:51:44.877444 739 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a4c6dc08df744e6856bdf99ab65c575 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:26 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:44.877668 736 tablet_service.cc:1514] Processing DeleteTablet for tablet 58053845187f466599750f2a62f4c3e1 with delete_type TABLET_DATA_DELETED (replaced by tablet a04cec5ed9f34a16a80e112e5c9671e6) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:44.877720 1191 raft_consensus.cc:357] T 936b8ceaedd34a019a4fb89a80dd2708 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:44.877910 1191 raft_consensus.cc:383] T 936b8ceaedd34a019a4fb89a80dd2708 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:44.877909 1195 raft_consensus.cc:2267] T fd0b8592997d4a61b8ca854ecbb67c55 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:44.877995 1191 raft_consensus.cc:738] T 936b8ceaedd34a019a4fb89a80dd2708 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:44.871906 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_41ea2489c272425292a6a6562964690f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:44.880589 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_41ea2489c272425292a6a6562964690f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:44.880856 1191 consensus_queue.cc:260] T 936b8ceaedd34a019a4fb89a80dd2708 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:44.881798 1192 tablet_bootstrap.cc:492] T 9b2d0aa6f0e9456d95f4e48c360ab6e8 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:44.881927 1192 ts_tablet_manager.cc:1397] T 9b2d0aa6f0e9456d95f4e48c360ab6e8 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.046s user 0.005s sys 0.000s
I20250113 02:51:44.882462 1195 ts_tablet_manager.cc:1905] T fd0b8592997d4a61b8ca854ecbb67c55 P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:44.882548 1192 raft_consensus.cc:357] T 9b2d0aa6f0e9456d95f4e48c360ab6e8 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:44.882725 1192 raft_consensus.cc:383] T 9b2d0aa6f0e9456d95f4e48c360ab6e8 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:44.882773 1192 raft_consensus.cc:738] T 9b2d0aa6f0e9456d95f4e48c360ab6e8 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:44.882969 1192 consensus_queue.cc:260] T 9b2d0aa6f0e9456d95f4e48c360ab6e8 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:44.883427 1192 ts_tablet_manager.cc:1428] T 9b2d0aa6f0e9456d95f4e48c360ab6e8 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20250113 02:51:44.884616 1192 tablet_bootstrap.cc:492] T 569d666c128a45b7a526c13be5090ae3 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:44.886020 1192 tablet_bootstrap.cc:654] T 569d666c128a45b7a526c13be5090ae3 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:44.894089 1198 ts_tablet_manager.cc:1918] T 9f8eb43a39774d2c8e9cb4e88a405b0f P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250113 02:51:44.894209 1198 log.cc:1198] T 9f8eb43a39774d2c8e9cb4e88a405b0f P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/9f8eb43a39774d2c8e9cb4e88a405b0f
I20250113 02:51:44.894559 1198 ts_tablet_manager.cc:1939] T 9f8eb43a39774d2c8e9cb4e88a405b0f P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:44.895310 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1415) took 1599 ms. Trace:
I20250113 02:51:44.895386 1198 rpcz_store.cc:276] 0113 02:51:43.296109 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:43.296171 (+ 62us) service_pool.cc:225] Handling call
0113 02:51:44.846723 (+1550552us) ts_tablet_manager.cc:1145] Deleting tablet 9f8eb43a39774d2c8e9cb4e88a405b0f
0113 02:51:44.846732 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:44.854088 (+ 7356us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:44.893974 (+ 39886us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:44.895292 (+ 1318us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1550358}
I20250113 02:51:44.895550 1191 ts_tablet_manager.cc:1428] T 936b8ceaedd34a019a4fb89a80dd2708 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.018s user 0.002s sys 0.000s
I20250113 02:51:44.896968 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet 9f8eb43a39774d2c8e9cb4e88a405b0f (table default.Table_bfda83bbe8bb4025b01edf1e38b1b73c [id=e900f4da41bf46bcace34b132f3ba977]) successfully deleted
I20250113 02:51:44.898272 1195 ts_tablet_manager.cc:1918] T fd0b8592997d4a61b8ca854ecbb67c55 P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250113 02:51:44.898375 1195 log.cc:1198] T fd0b8592997d4a61b8ca854ecbb67c55 P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/fd0b8592997d4a61b8ca854ecbb67c55
W20250113 02:51:44.898371 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9572215fd2eb45faa99cc236fcc0d5ea consensus state has no leader
I20250113 02:51:44.898742 1195 ts_tablet_manager.cc:1939] T fd0b8592997d4a61b8ca854ecbb67c55 P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:44.899035 1191 tablet_bootstrap.cc:492] T 7fb282707196444690352dea3eb1387f P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:44.899451 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1227) took 4797 ms. Trace:
I20250113 02:51:44.899520 1195 rpcz_store.cc:276] 0113 02:51:40.102060 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:40.102341 (+ 281us) service_pool.cc:225] Handling call
0113 02:51:44.876128 (+4773787us) ts_tablet_manager.cc:1145] Deleting tablet fd0b8592997d4a61b8ca854ecbb67c55
0113 02:51:44.876135 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:44.885407 (+ 9272us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:44.898178 (+ 12771us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:44.899432 (+ 1254us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4773494}
I20250113 02:51:44.900861 1191 tablet_bootstrap.cc:654] T 7fb282707196444690352dea3eb1387f P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:44.901450 2474 raft_consensus.cc:491] T 9b2d0aa6f0e9456d95f4e48c360ab6e8 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:44.901638 2474 raft_consensus.cc:513] T 9b2d0aa6f0e9456d95f4e48c360ab6e8 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:44.902679 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b2d0aa6f0e9456d95f4e48c360ab6e8" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
I20250113 02:51:44.902895 769 raft_consensus.cc:2463] T 9b2d0aa6f0e9456d95f4e48c360ab6e8 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 0.
I20250113 02:51:44.903333 995 leader_election.cc:304] T 9b2d0aa6f0e9456d95f4e48c360ab6e8 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters:
I20250113 02:51:44.903352 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1224) took 4805 ms. Trace:
I20250113 02:51:44.903451 1195 rpcz_store.cc:276] 0113 02:51:40.098095 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:40.098162 (+ 67us) service_pool.cc:225] Handling call
0113 02:51:44.903200 (+4805038us) ts_tablet_manager.cc:1145] Deleting tablet 432bcbc7ba6a44ba98cfe588f8292e32
0113 02:51:44.903212 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:44.903331 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4800351}
I20250113 02:51:44.903622 2363 raft_consensus.cc:2798] T 9b2d0aa6f0e9456d95f4e48c360ab6e8 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 02:51:44.903698 2363 raft_consensus.cc:491] T 9b2d0aa6f0e9456d95f4e48c360ab6e8 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 02:51:44.903770 2363 raft_consensus.cc:3054] T 9b2d0aa6f0e9456d95f4e48c360ab6e8 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Advancing to term 1
W20250113 02:51:44.903863 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 432bcbc7ba6a44ba98cfe588f8292e32 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 432bcbc7ba6a44ba98cfe588f8292e32 already in progress: creating tablet
I20250113 02:51:44.906920 2363 raft_consensus.cc:513] T 9b2d0aa6f0e9456d95f4e48c360ab6e8 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:44.908434 1195 tablet_replica.cc:331] T e570465a3d4046b1ac19a3a034fe91f1 P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:44.908629 1195 raft_consensus.cc:2238] T e570465a3d4046b1ac19a3a034fe91f1 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:44.908772 1195 raft_consensus.cc:2267] T e570465a3d4046b1ac19a3a034fe91f1 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:44.909348 1195 ts_tablet_manager.cc:1905] T e570465a3d4046b1ac19a3a034fe91f1 P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:44.909435 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b2d0aa6f0e9456d95f4e48c360ab6e8" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196"
I20250113 02:51:44.909534 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 7d0d6f8669dd41e39d402ad1d237418d
I20250113 02:51:44.909627 769 raft_consensus.cc:3054] T 9b2d0aa6f0e9456d95f4e48c360ab6e8 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:44.910015 454 catalog_manager.cc:6596] ReplaceTablet: tablet 7d0d6f8669dd41e39d402ad1d237418d deleted and replaced by tablet 9842ee1a8e3c4011add4b117a874f620
I20250113 02:51:44.911186 736 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d0d6f8669dd41e39d402ad1d237418d with delete_type TABLET_DATA_DELETED (replaced by tablet 9842ee1a8e3c4011add4b117a874f620) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:44.912396 885 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d0d6f8669dd41e39d402ad1d237418d with delete_type TABLET_DATA_DELETED (replaced by tablet 9842ee1a8e3c4011add4b117a874f620) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:44.912533 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d0d6f8669dd41e39d402ad1d237418d with delete_type TABLET_DATA_DELETED (replaced by tablet 9842ee1a8e3c4011add4b117a874f620) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:44.913542 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b2d0aa6f0e9456d95f4e48c360ab6e8" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f"
W20250113 02:51:44.914045 998 leader_election.cc:343] T 9b2d0aa6f0e9456d95f4e48c360ab6e8 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:44.914464 2363 leader_election.cc:290] T 9b2d0aa6f0e9456d95f4e48c360ab6e8 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Requested vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:44.914863 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_e0d35592e5c842afbe63a4fb305cae3a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:44.915421 769 raft_consensus.cc:2463] T 9b2d0aa6f0e9456d95f4e48c360ab6e8 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 1.
W20250113 02:51:44.915481 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e0d35592e5c842afbe63a4fb305cae3a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:44.915840 995 leader_election.cc:304] T 9b2d0aa6f0e9456d95f4e48c360ab6e8 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters: df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:44.916501 2639 raft_consensus.cc:2798] T 9b2d0aa6f0e9456d95f4e48c360ab6e8 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Leader election won for term 1
I20250113 02:51:44.916640 2639 raft_consensus.cc:695] T 9b2d0aa6f0e9456d95f4e48c360ab6e8 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: Becoming Leader. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Running, Role: LEADER
I20250113 02:51:44.916932 2639 consensus_queue.cc:237] T 9b2d0aa6f0e9456d95f4e48c360ab6e8 P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:44.920563 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b2d0aa6f0e9456d95f4e48c360ab6e8" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:44.921063 998 leader_election.cc:343] T 9b2d0aa6f0e9456d95f4e48c360ab6e8 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:44.921454 2474 leader_election.cc:290] T 9b2d0aa6f0e9456d95f4e48c360ab6e8 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:44.922686 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 83762997f3bd47e6966898ccad38abbb
I20250113 02:51:44.924259 736 tablet_service.cc:1514] Processing DeleteTablet for tablet 83762997f3bd47e6966898ccad38abbb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:44 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:44.924819 885 tablet_service.cc:1514] Processing DeleteTablet for tablet 83762997f3bd47e6966898ccad38abbb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:44 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:44.925660 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet 83762997f3bd47e6966898ccad38abbb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:44 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:44.925827 1198 tablet_replica.cc:331] T fe8f075d4c9a4f9abe372bfdb81a829c P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:44.926077 1198 raft_consensus.cc:2238] T fe8f075d4c9a4f9abe372bfdb81a829c P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:44.926234 1198 raft_consensus.cc:2267] T fe8f075d4c9a4f9abe372bfdb81a829c P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:44.926601 1195 ts_tablet_manager.cc:1918] T e570465a3d4046b1ac19a3a034fe91f1 P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:44.926856 1198 ts_tablet_manager.cc:1905] T fe8f075d4c9a4f9abe372bfdb81a829c P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:44.928627 1195 log.cc:1198] T e570465a3d4046b1ac19a3a034fe91f1 P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/e570465a3d4046b1ac19a3a034fe91f1
I20250113 02:51:44.929227 1195 ts_tablet_manager.cc:1939] T e570465a3d4046b1ac19a3a034fe91f1 P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:44.930543 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1228) took 4827 ms. Trace:
I20250113 02:51:44.931747 1194 tablet_bootstrap.cc:492] T ba8980ad3a364c888a825affbcbc9b1f P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:44.931967 1194 ts_tablet_manager.cc:1397] T ba8980ad3a364c888a825affbcbc9b1f P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.139s user 0.002s sys 0.013s
I20250113 02:51:44.932829 1195 rpcz_store.cc:276] 0113 02:51:40.102731 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:40.102919 (+ 188us) service_pool.cc:225] Handling call
0113 02:51:44.908375 (+4805456us) ts_tablet_manager.cc:1145] Deleting tablet e570465a3d4046b1ac19a3a034fe91f1
0113 02:51:44.908386 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:44.914758 (+ 6372us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:44.926498 (+ 11740us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:44.930524 (+ 4026us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4805257}
I20250113 02:51:44.932981 1194 raft_consensus.cc:357] T ba8980ad3a364c888a825affbcbc9b1f P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:44.933303 1194 raft_consensus.cc:383] T ba8980ad3a364c888a825affbcbc9b1f P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:44.933472 1194 raft_consensus.cc:738] T ba8980ad3a364c888a825affbcbc9b1f P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:44.933585 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet e570465a3d4046b1ac19a3a034fe91f1 (table default.Table_766a63106d8a4fd29bb338e76bd96089 [id=3042221d269a4bd88c04c1b6bb1bcd26]) successfully deleted
I20250113 02:51:44.933547 448 catalog_manager.cc:5526] T 019eb21c661547cfa0b619d119eb024e P 4216a7f853d642b7bc3e5cff458f3aca reported cstate change: term changed from 0 to 1, leader changed from <none> to 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3). New cstate: current_term: 1 leader_uuid: "4216a7f853d642b7bc3e5cff458f3aca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } health_report { overall_health: UNKNOWN } } }
I20250113 02:51:44.933974 1195 tablet_replica.cc:331] T 67baaab1e36e415381579d7385d4ab5a P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:44.933832 1194 consensus_queue.cc:260] T ba8980ad3a364c888a825affbcbc9b1f P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:44.934084 448 catalog_manager.cc:5618] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587) does not have the latest schema for tablet 019eb21c661547cfa0b619d119eb024e (table default.Table_022a7c61a6864393acd739f562893b7c [id=97bfbd46eecd4e57b7f9757c413f6795]). Expected version 1 got 0
I20250113 02:51:44.934410 1195 raft_consensus.cc:2238] T 67baaab1e36e415381579d7385d4ab5a P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:44.934564 1195 raft_consensus.cc:2267] T 67baaab1e36e415381579d7385d4ab5a P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:44.934448 448 catalog_manager.cc:5526] T e01e7f8db83d461bbfbb2e901419ce0b P 4216a7f853d642b7bc3e5cff458f3aca reported cstate change: term changed from 0 to 1, leader changed from <none> to 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3). New cstate: current_term: 1 leader_uuid: "4216a7f853d642b7bc3e5cff458f3aca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } health_report { overall_health: UNKNOWN } } }
I20250113 02:51:44.935087 1195 ts_tablet_manager.cc:1905] T 67baaab1e36e415381579d7385d4ab5a P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:44.935479 448 catalog_manager.cc:5618] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587) does not have the latest schema for tablet e01e7f8db83d461bbfbb2e901419ce0b (table default.Table_e27f8b6f8232415388a5e53ab15494b7 [id=efc4ebba97f74c9ba12ab1dcc6577da0]). Expected version 1 got 0
I20250113 02:51:44.935882 448 catalog_manager.cc:5526] T ca75e7cae4f6423191a17bb5a8f150a9 P 4216a7f853d642b7bc3e5cff458f3aca reported cstate change: term changed from 0 to 1, leader changed from <none> to 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3). New cstate: current_term: 1 leader_uuid: "4216a7f853d642b7bc3e5cff458f3aca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } health_report { overall_health: UNKNOWN } } }
I20250113 02:51:44.936447 448 catalog_manager.cc:5618] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587) does not have the latest schema for tablet f2c6c8c17d6e499d84ad7c0c23889cc4 (table default.Table_92ea541283614df981d96c5b74362f1d [id=9e58548d1a184cc0b362f148c44005ed]). Expected version 2 got 0
I20250113 02:51:44.937968 653 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 83762997f3bd47e6966898ccad38abbb
I20250113 02:51:44.939069 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet 83762997f3bd47e6966898ccad38abbb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:44 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:44.940109 736 tablet_service.cc:1514] Processing DeleteTablet for tablet 83762997f3bd47e6966898ccad38abbb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:44 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:44.941538 1028 tablet_service.cc:1467] Processing CreateTablet for tablet 7bf4d76e35fb4ed39434818adcc76ead (DEFAULT_TABLE table=default.Table_33d141799a774a28887f4c91cbac6ad6 [id=06b368f0c8ca4368a0ec2790259c9166]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:44.941953 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7bf4d76e35fb4ed39434818adcc76ead. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:44.945443 885 tablet_service.cc:1514] Processing DeleteTablet for tablet 83762997f3bd47e6966898ccad38abbb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:44 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:44.945910 891 tablet_service.cc:1467] Processing CreateTablet for tablet 7bf4d76e35fb4ed39434818adcc76ead (DEFAULT_TABLE table=default.Table_33d141799a774a28887f4c91cbac6ad6 [id=06b368f0c8ca4368a0ec2790259c9166]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:44.951961 1195 ts_tablet_manager.cc:1918] T 67baaab1e36e415381579d7385d4ab5a P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:44.952104 1195 log.cc:1198] T 67baaab1e36e415381579d7385d4ab5a P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/67baaab1e36e415381579d7385d4ab5a
I20250113 02:51:44.952451 1195 ts_tablet_manager.cc:1939] T 67baaab1e36e415381579d7385d4ab5a P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:44.955567 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_ba61ec17b9374461a322a08a44edecf7" } new_table_name: "default.Table_9708a2361fca4506b08138704bc8ff7b" modify_external_catalogs: true
I20250113 02:51:44.957432 1194 ts_tablet_manager.cc:1428] T ba8980ad3a364c888a825affbcbc9b1f P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.025s user 0.002s sys 0.000s
I20250113 02:51:44.958829 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1229) took 4855 ms. Trace:
I20250113 02:51:44.958932 1195 rpcz_store.cc:276] 0113 02:51:40.103039 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:40.103152 (+ 113us) service_pool.cc:225] Handling call
0113 02:51:44.933920 (+4830768us) ts_tablet_manager.cc:1145] Deleting tablet 67baaab1e36e415381579d7385d4ab5a
0113 02:51:44.933933 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:44.944833 (+ 10900us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:44.951871 (+ 7038us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:44.958807 (+ 6936us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4830575}
I20250113 02:51:44.959379 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet 67baaab1e36e415381579d7385d4ab5a (table default.Table_c3ba3f6047274ab885a4c853b06e1417 [id=6666a614063942c99a78b95d35add037]) successfully deleted
I20250113 02:51:44.960462 1195 tablet_replica.cc:331] T 9f8eb43a39774d2c8e9cb4e88a405b0f P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:44.960654 1195 raft_consensus.cc:2238] T 9f8eb43a39774d2c8e9cb4e88a405b0f P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:44.960790 1195 raft_consensus.cc:2267] T 9f8eb43a39774d2c8e9cb4e88a405b0f P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:44.961333 1195 ts_tablet_manager.cc:1905] T 9f8eb43a39774d2c8e9cb4e88a405b0f P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250113 02:51:44.963688 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d95efbdafa354795835cc7a48e4b3052 consensus state has no leader
I20250113 02:51:44.963992 1194 tablet_bootstrap.cc:492] T 0c408921a09a463eb464e08edcf042b3 P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:44.966116 1194 tablet_bootstrap.cc:654] T 0c408921a09a463eb464e08edcf042b3 P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:44.966462 2723 raft_consensus.cc:491] T a2781ba5061a4225b5636366857a229e P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:44.966602 2723 raft_consensus.cc:513] T a2781ba5061a4225b5636366857a229e P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:44.967288 2723 leader_election.cc:290] T a2781ba5061a4225b5636366857a229e P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:44.967892 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2781ba5061a4225b5636366857a229e" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
W20250113 02:51:44.968569 709 leader_election.cc:343] T a2781ba5061a4225b5636366857a229e P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:44.969434 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2781ba5061a4225b5636366857a229e" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:44.970103 711 leader_election.cc:343] T a2781ba5061a4225b5636366857a229e P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:44.970209 711 leader_election.cc:304] T a2781ba5061a4225b5636366857a229e P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:44.970595 2723 raft_consensus.cc:2743] T a2781ba5061a4225b5636366857a229e P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:44.971014 2474 raft_consensus.cc:491] T 65d9bbd8c15f40a4a50c1695fb5e648e P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:44.971310 2474 raft_consensus.cc:513] T 65d9bbd8c15f40a4a50c1695fb5e648e P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:44.971428 736 tablet_service.cc:1467] Processing CreateTablet for tablet 7bf4d76e35fb4ed39434818adcc76ead (DEFAULT_TABLE table=default.Table_33d141799a774a28887f4c91cbac6ad6 [id=06b368f0c8ca4368a0ec2790259c9166]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:44.971827 736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7bf4d76e35fb4ed39434818adcc76ead. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:44.973088 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d9bbd8c15f40a4a50c1695fb5e648e" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:44.973659 998 leader_election.cc:343] T 65d9bbd8c15f40a4a50c1695fb5e648e P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:44.973829 2474 leader_election.cc:290] T 65d9bbd8c15f40a4a50c1695fb5e648e P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:44.974058 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d9bbd8c15f40a4a50c1695fb5e648e" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
I20250113 02:51:44.974306 769 raft_consensus.cc:2463] T 65d9bbd8c15f40a4a50c1695fb5e648e P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 0.
I20250113 02:51:44.974676 995 leader_election.cc:304] T 65d9bbd8c15f40a4a50c1695fb5e648e P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters: df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:44.975173 2474 raft_consensus.cc:2798] T 65d9bbd8c15f40a4a50c1695fb5e648e P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 02:51:44.975250 2474 raft_consensus.cc:491] T 65d9bbd8c15f40a4a50c1695fb5e648e P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 02:51:44.975346 2474 raft_consensus.cc:3054] T 65d9bbd8c15f40a4a50c1695fb5e648e P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:44.977495 1195 ts_tablet_manager.cc:1918] T 9f8eb43a39774d2c8e9cb4e88a405b0f P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:44.977631 1195 log.cc:1198] T 9f8eb43a39774d2c8e9cb4e88a405b0f P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/9f8eb43a39774d2c8e9cb4e88a405b0f
I20250113 02:51:44.977993 1195 ts_tablet_manager.cc:1939] T 9f8eb43a39774d2c8e9cb4e88a405b0f P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:44.978869 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7bf4d76e35fb4ed39434818adcc76ead. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:44.980265 2368 raft_consensus.cc:491] T 0839a35fcab04f3c9d03e256553e01c8 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:44.980399 2368 raft_consensus.cc:513] T 0839a35fcab04f3c9d03e256553e01c8 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:44.981024 2368 leader_election.cc:290] T 0839a35fcab04f3c9d03e256553e01c8 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:44.981810 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0839a35fcab04f3c9d03e256553e01c8" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:44.982319 998 leader_election.cc:343] T 0839a35fcab04f3c9d03e256553e01c8 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:44.982882 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0839a35fcab04f3c9d03e256553e01c8" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
I20250113 02:51:44.983107 769 raft_consensus.cc:2463] T 0839a35fcab04f3c9d03e256553e01c8 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 0.
I20250113 02:51:44.983624 995 leader_election.cc:304] T 0839a35fcab04f3c9d03e256553e01c8 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters: df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:44.984103 2363 raft_consensus.cc:2798] T 0839a35fcab04f3c9d03e256553e01c8 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 02:51:44.984191 2363 raft_consensus.cc:491] T 0839a35fcab04f3c9d03e256553e01c8 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 02:51:44.984275 2363 raft_consensus.cc:3054] T 0839a35fcab04f3c9d03e256553e01c8 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:44.985719 2368 consensus_queue.cc:1035] T 4c759831773849f2bc52e484ffa0cf49 P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 02:51:44.986347 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1230) took 4883 ms. Trace:
I20250113 02:51:44.987011 2363 raft_consensus.cc:513] T 0839a35fcab04f3c9d03e256553e01c8 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:44.987901 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0839a35fcab04f3c9d03e256553e01c8" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196"
I20250113 02:51:44.988260 772 raft_consensus.cc:3054] T 0839a35fcab04f3c9d03e256553e01c8 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:44.988610 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0839a35fcab04f3c9d03e256553e01c8" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f"
W20250113 02:51:44.989269 998 leader_election.cc:343] T 0839a35fcab04f3c9d03e256553e01c8 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:44.989660 2363 leader_election.cc:290] T 0839a35fcab04f3c9d03e256553e01c8 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Requested vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:44.978453 2474 raft_consensus.cc:513] T 65d9bbd8c15f40a4a50c1695fb5e648e P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:44.990765 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d9bbd8c15f40a4a50c1695fb5e648e" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196"
I20250113 02:51:44.991207 771 raft_consensus.cc:3054] T 65d9bbd8c15f40a4a50c1695fb5e648e P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:44.991189 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d9bbd8c15f40a4a50c1695fb5e648e" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f"
I20250113 02:51:44.991755 772 raft_consensus.cc:2463] T 0839a35fcab04f3c9d03e256553e01c8 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 1.
W20250113 02:51:44.991786 998 leader_election.cc:343] T 65d9bbd8c15f40a4a50c1695fb5e648e P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:44.991964 2474 leader_election.cc:290] T 65d9bbd8c15f40a4a50c1695fb5e648e P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Requested vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:44.992331 995 leader_election.cc:304] T 0839a35fcab04f3c9d03e256553e01c8 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters: df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:44.992703 2474 raft_consensus.cc:2798] T 0839a35fcab04f3c9d03e256553e01c8 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Leader election won for term 1
I20250113 02:51:44.992822 2474 raft_consensus.cc:695] T 0839a35fcab04f3c9d03e256553e01c8 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: Becoming Leader. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Running, Role: LEADER
I20250113 02:51:44.993073 2474 consensus_queue.cc:237] T 0839a35fcab04f3c9d03e256553e01c8 P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:44.997146 1195 rpcz_store.cc:276] 0113 02:51:40.103116 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:40.103485 (+ 369us) service_pool.cc:225] Handling call
0113 02:51:44.960411 (+4856926us) ts_tablet_manager.cc:1145] Deleting tablet 9f8eb43a39774d2c8e9cb4e88a405b0f
0113 02:51:44.960422 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:44.974630 (+ 14208us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:44.977423 (+ 2793us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:44.986298 (+ 8875us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4856795}
I20250113 02:51:44.997658 771 raft_consensus.cc:2463] T 65d9bbd8c15f40a4a50c1695fb5e648e P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 1.
I20250113 02:51:44.997821 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet 9f8eb43a39774d2c8e9cb4e88a405b0f (table default.Table_bfda83bbe8bb4025b01edf1e38b1b73c [id=e900f4da41bf46bcace34b132f3ba977]) successfully deleted
I20250113 02:51:44.998402 995 leader_election.cc:304] T 65d9bbd8c15f40a4a50c1695fb5e648e P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters: df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:44.998893 2363 raft_consensus.cc:2798] T 65d9bbd8c15f40a4a50c1695fb5e648e P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Leader election won for term 1
I20250113 02:51:44.999418 736 tablet_service.cc:1467] Processing CreateTablet for tablet 17eb8f7d6fa3440a97e45455a46a405a (DEFAULT_TABLE table=default.Table_e85020a361f343dab7ddc0e6b11037ce [id=897543dc2397414cbb04e5cab4c6ca1c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:44.999927 736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 17eb8f7d6fa3440a97e45455a46a405a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:44.999936 2363 raft_consensus.cc:695] T 65d9bbd8c15f40a4a50c1695fb5e648e P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: Becoming Leader. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Running, Role: LEADER
I20250113 02:51:45.000520 2363 consensus_queue.cc:237] T 65d9bbd8c15f40a4a50c1695fb5e648e P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:45.003580 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1231) took 4833 ms. Trace:
I20250113 02:51:45.005654 1195 rpcz_store.cc:276] 0113 02:51:40.169690 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:40.169761 (+ 71us) service_pool.cc:225] Handling call
0113 02:51:45.003403 (+4833642us) ts_tablet_manager.cc:1145] Deleting tablet efe5f5ffd2c8423686b4007ded447466
0113 02:51:45.003416 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:45.003521 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4833429}
I20250113 02:51:45.005770 1028 tablet_service.cc:1467] Processing CreateTablet for tablet 17eb8f7d6fa3440a97e45455a46a405a (DEFAULT_TABLE table=default.Table_e85020a361f343dab7ddc0e6b11037ce [id=897543dc2397414cbb04e5cab4c6ca1c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:45.006220 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 17eb8f7d6fa3440a97e45455a46a405a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:45.007194 1195 tablet_replica.cc:331] T 21bcdd741104474bad7023426856d18f P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:45.007455 1195 raft_consensus.cc:2238] T 21bcdd741104474bad7023426856d18f P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:45.007652 1195 raft_consensus.cc:2267] T 21bcdd741104474bad7023426856d18f P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:45.010419 1195 ts_tablet_manager.cc:1905] T 21bcdd741104474bad7023426856d18f P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:45.010800 2368 raft_consensus.cc:491] T f2c6c8c17d6e499d84ad7c0c23889cc4 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:45.010962 2368 raft_consensus.cc:513] T f2c6c8c17d6e499d84ad7c0c23889cc4 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:45.011721 2368 leader_election.cc:290] T f2c6c8c17d6e499d84ad7c0c23889cc4 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:45.012835 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2c6c8c17d6e499d84ad7c0c23889cc4" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:45.013449 998 leader_election.cc:343] T f2c6c8c17d6e499d84ad7c0c23889cc4 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:45.015168 891 tablet_service.cc:1467] Processing CreateTablet for tablet 17eb8f7d6fa3440a97e45455a46a405a (DEFAULT_TABLE table=default.Table_e85020a361f343dab7ddc0e6b11037ce [id=897543dc2397414cbb04e5cab4c6ca1c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:45.016427 1195 ts_tablet_manager.cc:1918] T 21bcdd741104474bad7023426856d18f P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250113 02:51:45.016603 1195 log.cc:1198] T 21bcdd741104474bad7023426856d18f P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/21bcdd741104474bad7023426856d18f
I20250113 02:51:45.017488 1195 ts_tablet_manager.cc:1939] T 21bcdd741104474bad7023426856d18f P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:45.018379 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1232) took 4798 ms. Trace:
I20250113 02:51:45.018540 1195 rpcz_store.cc:276] 0113 02:51:40.219775 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:40.219831 (+ 56us) service_pool.cc:225] Handling call
0113 02:51:45.007107 (+4787276us) ts_tablet_manager.cc:1145] Deleting tablet 21bcdd741104474bad7023426856d18f
0113 02:51:45.007121 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:45.013430 (+ 6309us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:45.016317 (+ 2887us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:45.018338 (+ 2021us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4786964}
I20250113 02:51:45.018698 885 tablet_service.cc:1467] Processing CreateTablet for tablet 5721eb4d349546019613d6ac02c26a05 (DEFAULT_TABLE table=default.Table_2452909212ee4e0e9d228074c38e6949 [id=6b9d5eabe3da45bfbf9c6fa849a30f58]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:45.019057 1195 tablet_replica.cc:331] T fb2458e2044d49029ed2d6851db9de25 P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:45.019347 1195 raft_consensus.cc:2238] T fb2458e2044d49029ed2d6851db9de25 P 1186fa869eb14812832fa6562531f196 [term 1 LEADER]: Raft consensus shutting down.
I20250113 02:51:45.019779 1195 raft_consensus.cc:2267] T fb2458e2044d49029ed2d6851db9de25 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:45.020556 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet 21bcdd741104474bad7023426856d18f (table default.Table_4ef3edc5b07c4e0881456bfe23741c90 [id=46a732d6c22e472d82163d1295b34dcd]) successfully deleted
I20250113 02:51:45.021554 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2c6c8c17d6e499d84ad7c0c23889cc4" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
I20250113 02:51:45.021839 769 raft_consensus.cc:2463] T f2c6c8c17d6e499d84ad7c0c23889cc4 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 0.
I20250113 02:51:45.022305 995 leader_election.cc:304] T f2c6c8c17d6e499d84ad7c0c23889cc4 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters: df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:45.022888 2639 raft_consensus.cc:2798] T f2c6c8c17d6e499d84ad7c0c23889cc4 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 02:51:45.022857 1032 tablet_service.cc:1467] Processing CreateTablet for tablet 5721eb4d349546019613d6ac02c26a05 (DEFAULT_TABLE table=default.Table_2452909212ee4e0e9d228074c38e6949 [id=6b9d5eabe3da45bfbf9c6fa849a30f58]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:45.023523 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5721eb4d349546019613d6ac02c26a05. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:45.023763 883 tablet_service.cc:1514] Processing DeleteTablet for tablet ba8980ad3a364c888a825affbcbc9b1f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:36 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:45.023707 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_67b9f9f43ada4495a29ed9b8e8a6bf63" } modify_external_catalogs: true
I20250113 02:51:45.023878 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_67b9f9f43ada4495a29ed9b8e8a6bf63" } modify_external_catalogs: true
I20250113 02:51:45.030723 2476 raft_consensus.cc:491] T 42dbd37e20fa47318bc6497423dbe565 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:45.030866 2476 raft_consensus.cc:513] T 42dbd37e20fa47318bc6497423dbe565 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:45.031718 2476 leader_election.cc:290] T 42dbd37e20fa47318bc6497423dbe565 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:45.037557 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 99843ce7f1bf41da9b7cb5bd7ac2e875
I20250113 02:51:45.038044 449 catalog_manager.cc:6596] ReplaceTablet: tablet 99843ce7f1bf41da9b7cb5bd7ac2e875 deleted and replaced by tablet d664e48a65ae4e53aedbff0d55bfc17b
I20250113 02:51:45.038269 449 master_service.cc:946] ReplaceTablet: received request to replace tablet c7da2c406e64426faf123c4bddfaf213 from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:45.038795 739 tablet_service.cc:1467] Processing CreateTablet for tablet 5721eb4d349546019613d6ac02c26a05 (DEFAULT_TABLE table=default.Table_2452909212ee4e0e9d228074c38e6949 [id=6b9d5eabe3da45bfbf9c6fa849a30f58]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:45.039263 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5721eb4d349546019613d6ac02c26a05. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:45.039506 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet b6293a3b355b4c409b1158b6ba51f727
I20250113 02:51:45.043764 883 tablet_service.cc:1514] Processing DeleteTablet for tablet 99843ce7f1bf41da9b7cb5bd7ac2e875 with delete_type TABLET_DATA_DELETED (replaced by tablet d664e48a65ae4e53aedbff0d55bfc17b) from {username='slave'} at 127.0.0.1:37366
W20250113 02:51:45.043892 852 leader_election.cc:343] T 42dbd37e20fa47318bc6497423dbe565 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Not found: Tablet not found: 42dbd37e20fa47318bc6497423dbe565
I20250113 02:51:45.044128 882 tablet_service.cc:1514] Processing DeleteTablet for tablet b6293a3b355b4c409b1158b6ba51f727 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:44 UTC) from {username='slave'} at 127.0.0.1:37366
W20250113 02:51:45.044885 851 leader_election.cc:343] T 42dbd37e20fa47318bc6497423dbe565 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Not found: Tablet not found: 42dbd37e20fa47318bc6497423dbe565
I20250113 02:51:45.045048 851 leader_election.cc:304] T 42dbd37e20fa47318bc6497423dbe565 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df69f48656b44bb890f76587ee69cc0f; no voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca
I20250113 02:51:45.045140 748 tablet_service.cc:1514] Processing DeleteTablet for tablet 99843ce7f1bf41da9b7cb5bd7ac2e875 with delete_type TABLET_DATA_DELETED (replaced by tablet d664e48a65ae4e53aedbff0d55bfc17b) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:45.045838 2476 raft_consensus.cc:2743] T 42dbd37e20fa47318bc6497423dbe565 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:45.047116 2639 raft_consensus.cc:491] T f2c6c8c17d6e499d84ad7c0c23889cc4 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 02:51:45.047281 2639 raft_consensus.cc:3054] T f2c6c8c17d6e499d84ad7c0c23889cc4 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:45.047549 751 tablet_service.cc:1514] Processing DeleteTablet for tablet b6293a3b355b4c409b1158b6ba51f727 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:44 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:45.049491 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet b6293a3b355b4c409b1158b6ba51f727 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:44 UTC) from {username='slave'} at 127.0.0.1:60230
W20250113 02:51:45.050009 998 consensus_peers.cc:487] T 4c759831773849f2bc52e484ffa0cf49 P 4216a7f853d642b7bc3e5cff458f3aca -> Peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Couldn't send request to peer df69f48656b44bb890f76587ee69cc0f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250113 02:51:45.051061 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_36b33b166f834edbb7a465fc17ac3720"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:45.051501 1032 tablet_service.cc:1514] Processing DeleteTablet for tablet 99843ce7f1bf41da9b7cb5bd7ac2e875 with delete_type TABLET_DATA_DELETED (replaced by tablet d664e48a65ae4e53aedbff0d55bfc17b) from {username='slave'} at 127.0.0.1:60230
W20250113 02:51:45.051730 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_36b33b166f834edbb7a465fc17ac3720 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250113 02:51:45.052956 2747 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 02:51:45.059111 2368 raft_consensus.cc:491] T 16a132384b0e4264a73d09a3ea133893 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:45.059278 2368 raft_consensus.cc:513] T 16a132384b0e4264a73d09a3ea133893 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:45.059597 882 tablet_service.cc:1467] Processing CreateTablet for tablet 9991270a3d9c4344bb9e53a04d9e707e (DEFAULT_TABLE table=default.Table_27443429037f40faa2bad5d4eff2a571 [id=f9e6f7bd74264ba1a7304e54ea9cb6d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:45.060031 2368 leader_election.cc:290] T 16a132384b0e4264a73d09a3ea133893 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:45.060964 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16a132384b0e4264a73d09a3ea133893" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:45.061734 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16a132384b0e4264a73d09a3ea133893" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
W20250113 02:51:45.061832 998 leader_election.cc:343] T 16a132384b0e4264a73d09a3ea133893 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:45.061946 771 raft_consensus.cc:2463] T 16a132384b0e4264a73d09a3ea133893 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 0.
I20250113 02:51:45.062448 995 leader_election.cc:304] T 16a132384b0e4264a73d09a3ea133893 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters: df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:45.063027 2368 raft_consensus.cc:2798] T 16a132384b0e4264a73d09a3ea133893 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 02:51:45.063098 2368 raft_consensus.cc:491] T 16a132384b0e4264a73d09a3ea133893 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 02:51:45.063169 2368 raft_consensus.cc:3054] T 16a132384b0e4264a73d09a3ea133893 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:45.069151 2368 raft_consensus.cc:513] T 16a132384b0e4264a73d09a3ea133893 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:45.070744 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16a132384b0e4264a73d09a3ea133893" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f"
W20250113 02:51:45.071589 998 leader_election.cc:343] T 16a132384b0e4264a73d09a3ea133893 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:45.072069 739 tablet_service.cc:1467] Processing CreateTablet for tablet 9991270a3d9c4344bb9e53a04d9e707e (DEFAULT_TABLE table=default.Table_27443429037f40faa2bad5d4eff2a571 [id=f9e6f7bd74264ba1a7304e54ea9cb6d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:45.072454 2368 leader_election.cc:290] T 16a132384b0e4264a73d09a3ea133893 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Requested vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:45.072965 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9991270a3d9c4344bb9e53a04d9e707e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:45.072821 1032 tablet_service.cc:1467] Processing CreateTablet for tablet 9991270a3d9c4344bb9e53a04d9e707e (DEFAULT_TABLE table=default.Table_27443429037f40faa2bad5d4eff2a571 [id=f9e6f7bd74264ba1a7304e54ea9cb6d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:45.074954 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9991270a3d9c4344bb9e53a04d9e707e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:45.076615 2476 raft_consensus.cc:491] T 1014e51f1c184d729552a5b34292f232 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:45.076793 2476 raft_consensus.cc:513] T 1014e51f1c184d729552a5b34292f232 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:45.077057 1194 tablet_bootstrap.cc:492] T 0c408921a09a463eb464e08edcf042b3 P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:45.077245 1194 ts_tablet_manager.cc:1397] T 0c408921a09a463eb464e08edcf042b3 P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.113s user 0.003s sys 0.002s
I20250113 02:51:45.077543 2476 leader_election.cc:290] T 1014e51f1c184d729552a5b34292f232 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:45.077955 1194 raft_consensus.cc:357] T 0c408921a09a463eb464e08edcf042b3 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:45.078156 1194 raft_consensus.cc:383] T 0c408921a09a463eb464e08edcf042b3 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:45.078224 1194 raft_consensus.cc:738] T 0c408921a09a463eb464e08edcf042b3 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:45.078445 1194 consensus_queue.cc:260] T 0c408921a09a463eb464e08edcf042b3 P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
W20250113 02:51:45.078667 851 leader_election.cc:343] T 1014e51f1c184d729552a5b34292f232 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Not found: Tablet not found: 1014e51f1c184d729552a5b34292f232
I20250113 02:51:45.078975 1194 ts_tablet_manager.cc:1428] T 0c408921a09a463eb464e08edcf042b3 P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250113 02:51:45.086400 1192 tablet_bootstrap.cc:492] T 569d666c128a45b7a526c13be5090ae3 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:45.086314 1024 tablet_service.cc:1467] Processing CreateTablet for tablet 8ce7a241642e4701acea8206d4befab7 (DEFAULT_TABLE table=default.Table_fb3b54ceea934445932e1192983f3e27 [id=1af877cc00174792bf6f0f4ad6dbfbbc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:45.086611 1192 ts_tablet_manager.cc:1397] T 569d666c128a45b7a526c13be5090ae3 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.202s user 0.007s sys 0.051s
I20250113 02:51:45.086830 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ce7a241642e4701acea8206d4befab7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:45.086863 751 tablet_service.cc:1467] Processing CreateTablet for tablet 8ce7a241642e4701acea8206d4befab7 (DEFAULT_TABLE table=default.Table_fb3b54ceea934445932e1192983f3e27 [id=1af877cc00174792bf6f0f4ad6dbfbbc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:45.087239 1192 raft_consensus.cc:357] T 569d666c128a45b7a526c13be5090ae3 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:45.087352 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ce7a241642e4701acea8206d4befab7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:45.087409 1192 raft_consensus.cc:383] T 569d666c128a45b7a526c13be5090ae3 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:45.087455 1192 raft_consensus.cc:738] T 569d666c128a45b7a526c13be5090ae3 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:45.088130 2639 raft_consensus.cc:513] T f2c6c8c17d6e499d84ad7c0c23889cc4 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:45.089983 1194 tablet_bootstrap.cc:492] T b33025a462e74f2aa26cc3779f36615b P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:45.090768 1192 consensus_queue.cc:260] T 569d666c128a45b7a526c13be5090ae3 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:45.091504 1192 ts_tablet_manager.cc:1428] T 569d666c128a45b7a526c13be5090ae3 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250113 02:51:45.092460 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2c6c8c17d6e499d84ad7c0c23889cc4" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196"
I20250113 02:51:45.092665 771 raft_consensus.cc:3054] T f2c6c8c17d6e499d84ad7c0c23889cc4 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:45.092941 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16a132384b0e4264a73d09a3ea133893" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196"
I20250113 02:51:45.093128 773 raft_consensus.cc:3054] T 16a132384b0e4264a73d09a3ea133893 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Advancing to term 1
W20250113 02:51:45.093967 852 leader_election.cc:343] T 1014e51f1c184d729552a5b34292f232 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Not found: Tablet not found: 1014e51f1c184d729552a5b34292f232
I20250113 02:51:45.094074 852 leader_election.cc:304] T 1014e51f1c184d729552a5b34292f232 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df69f48656b44bb890f76587ee69cc0f; no voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca
I20250113 02:51:45.094422 2476 raft_consensus.cc:2743] T 1014e51f1c184d729552a5b34292f232 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:45.094915 1192 tablet_bootstrap.cc:492] T 68c4a7e8c0fb4feb85498709b00d3258 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:45.093689 2639 leader_election.cc:290] T f2c6c8c17d6e499d84ad7c0c23889cc4 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Requested vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:45.096283 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2c6c8c17d6e499d84ad7c0c23889cc4" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f"
W20250113 02:51:45.097344 998 leader_election.cc:343] T f2c6c8c17d6e499d84ad7c0c23889cc4 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:45.098063 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_fb2e08d7bab54382b45c211685c114db" } new_table_name: "default.Table_f166593d31ee4c56965f2c6eda65615b" modify_external_catalogs: true
I20250113 02:51:45.100124 1195 ts_tablet_manager.cc:1905] T fb2458e2044d49029ed2d6851db9de25 P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:45.101208 1198 ts_tablet_manager.cc:1918] T fe8f075d4c9a4f9abe372bfdb81a829c P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:45.102048 1191 tablet_bootstrap.cc:492] T 7fb282707196444690352dea3eb1387f P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:45.102246 1191 ts_tablet_manager.cc:1397] T 7fb282707196444690352dea3eb1387f P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.206s user 0.024s sys 0.033s
I20250113 02:51:45.103056 1191 raft_consensus.cc:357] T 7fb282707196444690352dea3eb1387f P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:45.103329 1191 raft_consensus.cc:383] T 7fb282707196444690352dea3eb1387f P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:45.103451 1191 raft_consensus.cc:738] T 7fb282707196444690352dea3eb1387f P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:45.103744 1191 consensus_queue.cc:260] T 7fb282707196444690352dea3eb1387f P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:45.081478 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 17eb8f7d6fa3440a97e45455a46a405a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:45.085405 883 tablet_service.cc:1467] Processing CreateTablet for tablet 8ce7a241642e4701acea8206d4befab7 (DEFAULT_TABLE table=default.Table_fb3b54ceea934445932e1192983f3e27 [id=1af877cc00174792bf6f0f4ad6dbfbbc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:45.106271 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ce7a241642e4701acea8206d4befab7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:45.107620 1191 ts_tablet_manager.cc:1428] T 7fb282707196444690352dea3eb1387f P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250113 02:51:45.107899 1191 tablet_bootstrap.cc:492] T 8f2fb58fc24d4692a3fe9a652e076aed P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:45.109519 1191 tablet_bootstrap.cc:654] T 8f2fb58fc24d4692a3fe9a652e076aed P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:45.110679 1198 log.cc:1198] T fe8f075d4c9a4f9abe372bfdb81a829c P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/fe8f075d4c9a4f9abe372bfdb81a829c
I20250113 02:51:45.111006 1198 ts_tablet_manager.cc:1939] T fe8f075d4c9a4f9abe372bfdb81a829c P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:45.111521 1194 tablet_bootstrap.cc:654] T b33025a462e74f2aa26cc3779f36615b P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:45.111833 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1416) took 1805 ms. Trace:
I20250113 02:51:45.111961 1198 rpcz_store.cc:276] 0113 02:51:43.306289 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:43.306343 (+ 54us) service_pool.cc:225] Handling call
0113 02:51:44.925763 (+1619420us) ts_tablet_manager.cc:1145] Deleting tablet fe8f075d4c9a4f9abe372bfdb81a829c
0113 02:51:44.925778 (+ 15us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:45.083629 (+157851us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:45.101121 (+ 17492us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:45.111811 (+ 10690us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1619251}
I20250113 02:51:45.112481 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet fe8f075d4c9a4f9abe372bfdb81a829c (table default.Table_ce5d67ac15bc4d89a173d58c766a259c [id=d37981b613bd494685de3e48d382f485]) successfully deleted
I20250113 02:51:45.112895 1198 tablet_replica.cc:331] T fb2458e2044d49029ed2d6851db9de25 P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:45.113108 1198 raft_consensus.cc:2238] T fb2458e2044d49029ed2d6851db9de25 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:45.113070 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9991270a3d9c4344bb9e53a04d9e707e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:45.113370 1198 raft_consensus.cc:2267] T fb2458e2044d49029ed2d6851db9de25 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:45.114082 1198 ts_tablet_manager.cc:1905] T fb2458e2044d49029ed2d6851db9de25 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:45.118841 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5721eb4d349546019613d6ac02c26a05. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:45.120709 1192 tablet_bootstrap.cc:654] T 68c4a7e8c0fb4feb85498709b00d3258 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:45.125910 1191 tablet_bootstrap.cc:492] T 8f2fb58fc24d4692a3fe9a652e076aed P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:45.126081 1191 ts_tablet_manager.cc:1397] T 8f2fb58fc24d4692a3fe9a652e076aed P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.000s
I20250113 02:51:45.126827 1191 raft_consensus.cc:357] T 8f2fb58fc24d4692a3fe9a652e076aed P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:45.127035 1191 raft_consensus.cc:383] T 8f2fb58fc24d4692a3fe9a652e076aed P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:45.127128 1191 raft_consensus.cc:738] T 8f2fb58fc24d4692a3fe9a652e076aed P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:45.127388 1191 consensus_queue.cc:260] T 8f2fb58fc24d4692a3fe9a652e076aed P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:45.130365 1191 ts_tablet_manager.cc:1428] T 8f2fb58fc24d4692a3fe9a652e076aed P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250113 02:51:45.130753 1191 tablet_bootstrap.cc:492] T d565b77f90784d87b64996772c938085 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:45.130960 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_22efb860611f468c98e2b9df40e31e83"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:45.131507 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_22efb860611f468c98e2b9df40e31e83 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:45.136049 883 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c408921a09a463eb464e08edcf042b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:24 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:45.150525 1198 ts_tablet_manager.cc:1918] T fb2458e2044d49029ed2d6851db9de25 P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250113 02:51:45.150934 1198 log.cc:1198] T fb2458e2044d49029ed2d6851db9de25 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/fb2458e2044d49029ed2d6851db9de25
I20250113 02:51:45.151166 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet e9767ab120c04e2694c913b951020a13
I20250113 02:51:45.151482 1198 ts_tablet_manager.cc:1939] T fb2458e2044d49029ed2d6851db9de25 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:45.151928 736 tablet_service.cc:1514] Processing DeleteTablet for tablet e9767ab120c04e2694c913b951020a13 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:44 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:45.153497 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_93aa0e80539449feacb8085adce4b4ea" } new_table_name: "default.Table_78ffe47bd98f4721b7a3b9f82ef45ac4" modify_external_catalogs: true
I20250113 02:51:45.155405 1032 tablet_service.cc:1514] Processing DeleteTablet for tablet e9767ab120c04e2694c913b951020a13 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:44 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:45.155877 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1417) took 1829 ms. Trace:
I20250113 02:51:45.156277 1198 rpcz_store.cc:276] 0113 02:51:43.326313 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:43.326470 (+ 157us) service_pool.cc:225] Handling call
0113 02:51:45.112851 (+1786381us) ts_tablet_manager.cc:1145] Deleting tablet fb2458e2044d49029ed2d6851db9de25
0113 02:51:45.112861 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:45.129028 (+ 16167us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:45.149774 (+ 20746us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:45.155861 (+ 6087us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1786204}
I20250113 02:51:45.157063 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet fb2458e2044d49029ed2d6851db9de25 (table default.Table_53a85211f49241c4af531e479ecdc0cb [id=ff8b884b04954ff48a7c8a72df232e8a]) successfully deleted
I20250113 02:51:45.158524 1191 tablet_bootstrap.cc:654] T d565b77f90784d87b64996772c938085 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:45.160085 885 tablet_service.cc:1514] Processing DeleteTablet for tablet e9767ab120c04e2694c913b951020a13 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:44 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:45.161510 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1423) took 1719 ms. Trace:
I20250113 02:51:45.161907 1198 rpcz_store.cc:276] 0113 02:51:43.441679 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:43.441759 (+ 80us) service_pool.cc:225] Handling call
0113 02:51:45.161388 (+1719629us) ts_tablet_manager.cc:1145] Deleting tablet 77036be46b574f73a02ca656bf5cc0a3
0113 02:51:45.161400 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:45.161494 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1719290}
W20250113 02:51:45.162539 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 77036be46b574f73a02ca656bf5cc0a3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 77036be46b574f73a02ca656bf5cc0a3 already in progress: creating tablet
I20250113 02:51:45.163007 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1425) took 1709 ms. Trace:
I20250113 02:51:45.163442 1198 rpcz_store.cc:276] 0113 02:51:43.453480 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:43.453533 (+ 53us) service_pool.cc:225] Handling call
0113 02:51:45.162899 (+1709366us) ts_tablet_manager.cc:1145] Deleting tablet 559c2f8ea91d4e81bf303b3a0c61c143
0113 02:51:45.162909 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:45.162990 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1709204}
W20250113 02:51:45.164954 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 559c2f8ea91d4e81bf303b3a0c61c143 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 559c2f8ea91d4e81bf303b3a0c61c143 already in progress: creating tablet
I20250113 02:51:45.165146 2723 raft_consensus.cc:491] T 415a1fa4b2f14cacb43937c8a372849a P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:45.165304 2723 raft_consensus.cc:513] T 415a1fa4b2f14cacb43937c8a372849a P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:45.165745 771 raft_consensus.cc:2463] T f2c6c8c17d6e499d84ad7c0c23889cc4 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 1.
I20250113 02:51:45.166059 2723 leader_election.cc:290] T 415a1fa4b2f14cacb43937c8a372849a P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:45.166249 1191 tablet_bootstrap.cc:492] T d565b77f90784d87b64996772c938085 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:45.166414 1191 ts_tablet_manager.cc:1397] T d565b77f90784d87b64996772c938085 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.036s user 0.003s sys 0.000s
I20250113 02:51:45.167080 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "415a1fa4b2f14cacb43937c8a372849a" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:45.167160 1191 raft_consensus.cc:357] T d565b77f90784d87b64996772c938085 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:45.167341 1191 raft_consensus.cc:383] T d565b77f90784d87b64996772c938085 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:45.167402 1191 raft_consensus.cc:738] T d565b77f90784d87b64996772c938085 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
W20250113 02:51:45.167656 711 leader_election.cc:343] T 415a1fa4b2f14cacb43937c8a372849a P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:45.168196 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "415a1fa4b2f14cacb43937c8a372849a" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:45.168720 1198 tablet_replica.cc:331] T 4c759831773849f2bc52e484ffa0cf49 P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
W20250113 02:51:45.168697 709 leader_election.cc:343] T 415a1fa4b2f14cacb43937c8a372849a P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:45.168794 709 leader_election.cc:304] T 415a1fa4b2f14cacb43937c8a372849a P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:45.168933 1198 raft_consensus.cc:2238] T 4c759831773849f2bc52e484ffa0cf49 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: Raft consensus shutting down.
I20250113 02:51:45.169241 1198 raft_consensus.cc:2267] T 4c759831773849f2bc52e484ffa0cf49 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Raft consensus is shut down!
W20250113 02:51:45.169340 2748 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 02:51:45.169397 2723 raft_consensus.cc:2743] T 415a1fa4b2f14cacb43937c8a372849a P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:45.169948 2476 raft_consensus.cc:491] T bbc7832c50114d6f890fec45a3d994e9 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:45.170104 2476 raft_consensus.cc:513] T bbc7832c50114d6f890fec45a3d994e9 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:45.170684 2476 leader_election.cc:290] T bbc7832c50114d6f890fec45a3d994e9 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
W20250113 02:51:45.171809 852 leader_election.cc:343] T bbc7832c50114d6f890fec45a3d994e9 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Not found: Tablet not found: bbc7832c50114d6f890fec45a3d994e9
I20250113 02:51:45.172133 1198 ts_tablet_manager.cc:1905] T 4c759831773849f2bc52e484ffa0cf49 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250113 02:51:45.173202 851 leader_election.cc:343] T bbc7832c50114d6f890fec45a3d994e9 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Not found: Tablet not found: bbc7832c50114d6f890fec45a3d994e9
I20250113 02:51:45.173319 851 leader_election.cc:304] T bbc7832c50114d6f890fec45a3d994e9 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df69f48656b44bb890f76587ee69cc0f; no voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca
I20250113 02:51:45.172617 995 leader_election.cc:304] T f2c6c8c17d6e499d84ad7c0c23889cc4 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters: df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:45.173678 2476 raft_consensus.cc:2743] T bbc7832c50114d6f890fec45a3d994e9 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:45.174041 2749 raft_consensus.cc:2798] T f2c6c8c17d6e499d84ad7c0c23889cc4 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Leader election won for term 1
I20250113 02:51:45.174161 2749 raft_consensus.cc:695] T f2c6c8c17d6e499d84ad7c0c23889cc4 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: Becoming Leader. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Running, Role: LEADER
I20250113 02:51:45.174424 2749 consensus_queue.cc:237] T f2c6c8c17d6e499d84ad7c0c23889cc4 P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:45.177605 2639 consensus_queue.cc:1035] T 79aaab73797c474bae06831fa308b066 P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 02:51:45.178617 1191 consensus_queue.cc:260] T d565b77f90784d87b64996772c938085 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:45.181371 455 catalog_manager.cc:5618] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095) does not have the latest schema for tablet c04be975c8c049b892b16129ed47a142 (table default.Table_f98f15e1cb6a43348c4438096a7f9ea4 [id=36184fc3b73745c487d1d7a9a69e85ef]). Expected version 1 got 0
I20250113 02:51:45.181707 455 catalog_manager.cc:5618] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095) does not have the latest schema for tablet afcb5b1977c447d3a51de9cb72d7196a (table default.Table_1eb2867afbff468cbbca66486afd7075 [id=d0ffa26e65894ab89facfccd75985d51]). Expected version 1 got 0
I20250113 02:51:45.181965 455 catalog_manager.cc:5618] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095) does not have the latest schema for tablet 938cc1ccd8dd449996836188ba9e64ff (table default.Table_e5b4d614e4b743d7982b025ca6d297e5 [id=5824acd778ed47d09f087bbf8e7822d6]). Expected version 1 got 0
I20250113 02:51:45.182227 455 catalog_manager.cc:5526] T f1237f7b04af4ebbb39b03aa3c78a499 P 1186fa869eb14812832fa6562531f196 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1186fa869eb14812832fa6562531f196 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "1186fa869eb14812832fa6562531f196" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } health_report { overall_health: UNKNOWN } } }
I20250113 02:51:45.182619 455 catalog_manager.cc:5618] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095) does not have the latest schema for tablet f1237f7b04af4ebbb39b03aa3c78a499 (table default.Table_4c5e2e1008b54019a243ffe1ec72c020 [id=0817677719d045fd83bb85890f11df25]). Expected version 2 got 0
I20250113 02:51:45.185753 1032 tablet_service.cc:1514] Processing DeleteTablet for tablet fb2458e2044d49029ed2d6851db9de25 with delete_type TABLET_DATA_DELETED (replaced by tablet 32e3dc0b7c044415a6343368ffa08552) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:45.186180 1039 tablet_service.cc:1514] Processing DeleteTablet for tablet f105f8b32a654e4ca154e8fc0cb96091 with delete_type TABLET_DATA_DELETED (replaced by tablet 65e09181ee4c4061a7b5a5353be440cf) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:45.186434 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet 883027967e0c442a91ac820210b9c6c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:31 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:45.186620 1039 tablet_service.cc:1514] Processing DeleteTablet for tablet 16a132384b0e4264a73d09a3ea133893 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:31 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:45.187036 1039 tablet_service.cc:1514] Processing DeleteTablet for tablet 65d9bbd8c15f40a4a50c1695fb5e648e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:24 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:45.187556 1039 tablet_service.cc:1514] Processing DeleteTablet for tablet b217cb8c09fb41cab4f53e605c37efe8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:26 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:45.188832 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet 88a776aa820c44b3a3854ca5022bd353 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:31 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:45.189549 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_f6c8709d5ec649f782b85cbbc6517f5b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:45.190289 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f6c8709d5ec649f782b85cbbc6517f5b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:45.190497 885 tablet_service.cc:1467] Processing CreateTablet for tablet 1cc28dd1eb034817b62bd9d8fd78d053 (DEFAULT_TABLE table=default.Table_b69c092ae6994928ad1eb5c10421a027 [id=14539bade7774fafa1f17ee4e2307405]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:45.190927 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1cc28dd1eb034817b62bd9d8fd78d053. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:45.191190 2474 raft_consensus.cc:491] T b217cb8c09fb41cab4f53e605c37efe8 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:45.191402 2474 raft_consensus.cc:513] T b217cb8c09fb41cab4f53e605c37efe8 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:45.192133 2476 raft_consensus.cc:491] T bc6a7322c0c34adb85457750526d8955 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:45.192271 2476 raft_consensus.cc:513] T bc6a7322c0c34adb85457750526d8955 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:45.192407 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b217cb8c09fb41cab4f53e605c37efe8" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
I20250113 02:51:45.192588 769 raft_consensus.cc:2463] T b217cb8c09fb41cab4f53e605c37efe8 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 0.
I20250113 02:51:45.192899 2476 leader_election.cc:290] T bc6a7322c0c34adb85457750526d8955 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:45.192960 995 leader_election.cc:304] T b217cb8c09fb41cab4f53e605c37efe8 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters:
I20250113 02:51:45.193373 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc6a7322c0c34adb85457750526d8955" candidate_uuid: "df69f48656b44bb890f76587ee69cc0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:45.193822 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc6a7322c0c34adb85457750526d8955" candidate_uuid: "df69f48656b44bb890f76587ee69cc0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
I20250113 02:51:45.194012 2749 raft_consensus.cc:2798] T b217cb8c09fb41cab4f53e605c37efe8 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 02:51:45.194082 2749 raft_consensus.cc:491] T b217cb8c09fb41cab4f53e605c37efe8 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 02:51:45.194139 2749 raft_consensus.cc:3054] T b217cb8c09fb41cab4f53e605c37efe8 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:45.194476 851 leader_election.cc:304] T bc6a7322c0c34adb85457750526d8955 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df69f48656b44bb890f76587ee69cc0f; no voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca
I20250113 02:51:45.195163 2476 raft_consensus.cc:2743] T bc6a7322c0c34adb85457750526d8955 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:45.196616 2749 raft_consensus.cc:513] T b217cb8c09fb41cab4f53e605c37efe8 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:45.197336 2749 leader_election.cc:290] T b217cb8c09fb41cab4f53e605c37efe8 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Requested vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:45.197758 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b217cb8c09fb41cab4f53e605c37efe8" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f"
I20250113 02:51:45.198038 1024 tablet_service.cc:1467] Processing CreateTablet for tablet 1cc28dd1eb034817b62bd9d8fd78d053 (DEFAULT_TABLE table=default.Table_b69c092ae6994928ad1eb5c10421a027 [id=14539bade7774fafa1f17ee4e2307405]), partition=RANGE (key) PARTITION UNBOUNDED
W20250113 02:51:45.198357 998 leader_election.cc:343] T b217cb8c09fb41cab4f53e605c37efe8 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:45.198575 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1cc28dd1eb034817b62bd9d8fd78d053. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:45.198602 1191 ts_tablet_manager.cc:1428] T d565b77f90784d87b64996772c938085 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.032s user 0.002s sys 0.000s
I20250113 02:51:45.198899 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b217cb8c09fb41cab4f53e605c37efe8" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196"
I20250113 02:51:45.199095 769 raft_consensus.cc:3054] T b217cb8c09fb41cab4f53e605c37efe8 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:45.202191 769 raft_consensus.cc:2463] T b217cb8c09fb41cab4f53e605c37efe8 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 1.
W20250113 02:51:45.205142 998 consensus_peers.cc:487] T 8cc78bf7d6324b7ba10a1b1045923f2b P 4216a7f853d642b7bc3e5cff458f3aca -> Peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Couldn't send request to peer df69f48656b44bb890f76587ee69cc0f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250113 02:51:45.206461 995 leader_election.cc:304] T b217cb8c09fb41cab4f53e605c37efe8 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters: df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:45.207073 2363 raft_consensus.cc:2798] T b217cb8c09fb41cab4f53e605c37efe8 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Leader election won for term 1
I20250113 02:51:45.207181 2363 raft_consensus.cc:695] T b217cb8c09fb41cab4f53e605c37efe8 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: Becoming Leader. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Running, Role: LEADER
I20250113 02:51:45.207412 2363 consensus_queue.cc:237] T b217cb8c09fb41cab4f53e605c37efe8 P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:45.218016 1194 tablet_bootstrap.cc:492] T b33025a462e74f2aa26cc3779f36615b P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:45.221130 1194 ts_tablet_manager.cc:1397] T b33025a462e74f2aa26cc3779f36615b P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.131s user 0.007s sys 0.007s
I20250113 02:51:45.222548 1194 raft_consensus.cc:357] T b33025a462e74f2aa26cc3779f36615b P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:45.227005 1195 ts_tablet_manager.cc:1918] T fb2458e2044d49029ed2d6851db9de25 P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250113 02:51:45.227432 1194 raft_consensus.cc:383] T b33025a462e74f2aa26cc3779f36615b P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:45.227952 1194 raft_consensus.cc:738] T b33025a462e74f2aa26cc3779f36615b P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:45.228704 1194 consensus_queue.cc:260] T b33025a462e74f2aa26cc3779f36615b P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:45.232688 1194 ts_tablet_manager.cc:1428] T b33025a462e74f2aa26cc3779f36615b P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.011s user 0.000s sys 0.002s
I20250113 02:51:45.237629 1195 log.cc:1198] T fb2458e2044d49029ed2d6851db9de25 P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/fb2458e2044d49029ed2d6851db9de25
I20250113 02:51:45.240597 1195 ts_tablet_manager.cc:1939] T fb2458e2044d49029ed2d6851db9de25 P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
W20250113 02:51:45.243566 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 306163486d294a7e8348ec6506ed3ed7 consensus state has no leader
I20250113 02:51:45.244263 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet be2ca8d2ac6547189a0edeff12413330
I20250113 02:51:45.245241 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet c7da2c406e64426faf123c4bddfaf213
I20250113 02:51:45.245620 449 catalog_manager.cc:6596] ReplaceTablet: tablet c7da2c406e64426faf123c4bddfaf213 deleted and replaced by tablet 423d5d1a57484806b76565aa2292c2a1
I20250113 02:51:45.247550 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1233) took 5027 ms. Trace:
I20250113 02:51:45.247653 1195 rpcz_store.cc:276] 0113 02:51:40.220269 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:40.220312 (+ 43us) service_pool.cc:225] Handling call
0113 02:51:45.018986 (+4798674us) ts_tablet_manager.cc:1145] Deleting tablet fb2458e2044d49029ed2d6851db9de25
0113 02:51:45.018998 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:45.193740 (+174742us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:45.226865 (+ 33125us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:45.247523 (+ 20658us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4798519}
I20250113 02:51:45.248760 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet be2ca8d2ac6547189a0edeff12413330 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:45 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:45.248965 1039 tablet_service.cc:1514] Processing DeleteTablet for tablet c7da2c406e64426faf123c4bddfaf213 with delete_type TABLET_DATA_DELETED (replaced by tablet 423d5d1a57484806b76565aa2292c2a1) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:45.254395 882 tablet_service.cc:1514] Processing DeleteTablet for tablet be2ca8d2ac6547189a0edeff12413330 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:45 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:45.255736 1192 tablet_bootstrap.cc:492] T 68c4a7e8c0fb4feb85498709b00d3258 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:45.256023 1192 ts_tablet_manager.cc:1397] T 68c4a7e8c0fb4feb85498709b00d3258 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.161s user 0.013s sys 0.000s
I20250113 02:51:45.257246 1192 raft_consensus.cc:357] T 68c4a7e8c0fb4feb85498709b00d3258 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:45.257649 1192 raft_consensus.cc:383] T 68c4a7e8c0fb4feb85498709b00d3258 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:45.257776 1192 raft_consensus.cc:738] T 68c4a7e8c0fb4feb85498709b00d3258 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:45.258153 1192 consensus_queue.cc:260] T 68c4a7e8c0fb4feb85498709b00d3258 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:45.258854 1192 ts_tablet_manager.cc:1428] T 68c4a7e8c0fb4feb85498709b00d3258 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250113 02:51:45.198899 1191 tablet_bootstrap.cc:492] T 9174df876fe64adc805fba0a36fb4d38 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:45.263679 1191 tablet_bootstrap.cc:654] T 9174df876fe64adc805fba0a36fb4d38 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
W20250113 02:51:45.265455 2763 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 02:51:45.267225 882 tablet_service.cc:1467] Processing CreateTablet for tablet b09082d6672344778dd319005e9fbe8b (DEFAULT_TABLE table=default.Table_37bcc8921e964a6391e0c8adde667af8 [id=4d1c6a9d2bb140188df4565f5bd2cf75]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:45.268982 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1234) took 5048 ms. Trace:
I20250113 02:51:45.269465 1195 rpcz_store.cc:276] 0113 02:51:40.220636 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:40.220674 (+ 38us) service_pool.cc:225] Handling call
0113 02:51:45.268634 (+5047960us) ts_tablet_manager.cc:1145] Deleting tablet fe8f075d4c9a4f9abe372bfdb81a829c
0113 02:51:45.268648 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:45.268957 (+ 309us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5047834}
I20250113 02:51:45.269627 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b217cb8c09fb41cab4f53e605c37efe8" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:45.269762 2603 server_base.cc:1034] running on GCE node
I20250113 02:51:45.271152 2603 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250113 02:51:45.272161 1195 tablet_replica.cc:331] T 79aaab73797c474bae06831fa308b066 P 1186fa869eb14812832fa6562531f196: stopping tablet replica
W20250113 02:51:45.270431 998 leader_election.cc:343] T b217cb8c09fb41cab4f53e605c37efe8 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
W20250113 02:51:45.273136 995 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.1:40095, user_credentials={real_user=slave}} blocked reactor thread for 51763.1us
I20250113 02:51:45.273830 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_83eaa3dec71e42e2921fc5957777f78e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:45.273268 2474 leader_election.cc:290] T b217cb8c09fb41cab4f53e605c37efe8 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
W20250113 02:51:45.274446 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_83eaa3dec71e42e2921fc5957777f78e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:45.277640 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_597caad30c254169ad25df814c309739" } modify_external_catalogs: true
I20250113 02:51:45.277825 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_597caad30c254169ad25df814c309739" } modify_external_catalogs: true
I20250113 02:51:45.279247 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_9a6c1e2713d74f7186906076d1f1212e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:45.279757 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9a6c1e2713d74f7186906076d1f1212e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:45.283452 447 catalog_manager.cc:5526] T 0839a35fcab04f3c9d03e256553e01c8 P 4216a7f853d642b7bc3e5cff458f3aca reported cstate change: term changed from 0 to 1, leader changed from <none> to 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3). New cstate: current_term: 1 leader_uuid: "4216a7f853d642b7bc3e5cff458f3aca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } health_report { overall_health: UNKNOWN } } }
I20250113 02:51:45.285941 1192 tablet_bootstrap.cc:492] T 58053845187f466599750f2a62f4c3e1 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:45.285635 447 catalog_manager.cc:5526] T f2c6c8c17d6e499d84ad7c0c23889cc4 P 4216a7f853d642b7bc3e5cff458f3aca reported cstate change: term changed from 0 to 1, leader changed from <none> to 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3). New cstate: current_term: 1 leader_uuid: "4216a7f853d642b7bc3e5cff458f3aca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } health_report { overall_health: HEALTHY } } }
I20250113 02:51:45.286926 447 catalog_manager.cc:5618] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587) does not have the latest schema for tablet f2c6c8c17d6e499d84ad7c0c23889cc4 (table default.Table_92ea541283614df981d96c5b74362f1d [id=9e58548d1a184cc0b362f148c44005ed]). Expected version 2 got 0
I20250113 02:51:45.287294 2769 consensus_queue.cc:1035] T f105f8b32a654e4ca154e8fc0cb96091 P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250113 02:51:45.289701 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 52e2b698832844baa0edfe732909e075 has no consensus state
I20250113 02:51:45.291555 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_da4eeeb904054a9aa6807387f76b1d8f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:45.292096 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_da4eeeb904054a9aa6807387f76b1d8f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250113 02:51:45.293447 439 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.60:42067, user_credentials={real_user=slave}} blocked reactor thread for 83442.4us
I20250113 02:51:45.293503 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_880629d9242b4654a306a4b3843a636a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:45.293948 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_880629d9242b4654a306a4b3843a636a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:45.294119 2588 raft_consensus.cc:491] T 98bfe9620e284ab1954433ba9a036392 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:45.294306 751 tablet_service.cc:1514] Processing DeleteTablet for tablet be2ca8d2ac6547189a0edeff12413330 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:45 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:45.294262 2588 raft_consensus.cc:513] T 98bfe9620e284ab1954433ba9a036392 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:45.294889 2588 leader_election.cc:290] T 98bfe9620e284ab1954433ba9a036392 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:45.295926 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98bfe9620e284ab1954433ba9a036392" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:45.296217 1191 tablet_bootstrap.cc:492] T 9174df876fe64adc805fba0a36fb4d38 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:45.296345 1191 ts_tablet_manager.cc:1397] T 9174df876fe64adc805fba0a36fb4d38 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.098s user 0.003s sys 0.001s
I20250113 02:51:45.296548 1060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98bfe9620e284ab1954433ba9a036392" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
W20250113 02:51:45.296919 711 leader_election.cc:343] T 98bfe9620e284ab1954433ba9a036392 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
W20250113 02:51:45.297391 709 leader_election.cc:343] T 98bfe9620e284ab1954433ba9a036392 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:45.297497 709 leader_election.cc:304] T 98bfe9620e284ab1954433ba9a036392 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:45.298087 1191 raft_consensus.cc:357] T 9174df876fe64adc805fba0a36fb4d38 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:45.298275 1191 raft_consensus.cc:383] T 9174df876fe64adc805fba0a36fb4d38 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:45.298326 1191 raft_consensus.cc:738] T 9174df876fe64adc805fba0a36fb4d38 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:45.298804 2588 raft_consensus.cc:2743] T 98bfe9620e284ab1954433ba9a036392 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:45.300509 1195 raft_consensus.cc:2238] T 79aaab73797c474bae06831fa308b066 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:45.301810 2777 tablet.cc:1721] T b5c1882125b5491ea79a29a6dac6771e P 4216a7f853d642b7bc3e5cff458f3aca: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250113 02:51:45.303138 2780 tablet.cc:1721] T 938cc1ccd8dd449996836188ba9e64ff P 4216a7f853d642b7bc3e5cff458f3aca: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250113 02:51:45.306087 1039 tablet_service.cc:1467] Processing CreateTablet for tablet b09082d6672344778dd319005e9fbe8b (DEFAULT_TABLE table=default.Table_37bcc8921e964a6391e0c8adde667af8 [id=4d1c6a9d2bb140188df4565f5bd2cf75]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:45.306456 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b09082d6672344778dd319005e9fbe8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:45.307305 1192 tablet_bootstrap.cc:654] T 58053845187f466599750f2a62f4c3e1 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:45.308614 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet fb2458e2044d49029ed2d6851db9de25 (table default.Table_53a85211f49241c4af531e479ecdc0cb [id=ff8b884b04954ff48a7c8a72df232e8a]) successfully deleted
W20250113 02:51:45.308877 439 catalog_manager.cc:4839] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet fe8f075d4c9a4f9abe372bfdb81a829c because the tablet was not found. No further retry: Not found: Tablet not found: fe8f075d4c9a4f9abe372bfdb81a829c
W20250113 02:51:45.309119 439 consensus_peers.cc:487] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9 -> Peer 45e9d43ecac0401fb2c18fe5eaf089b5 (127.0.104.60:42067): Couldn't send request to peer 45e9d43ecac0401fb2c18fe5eaf089b5. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.60:42067: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250113 02:51:45.300731 751 tablet_service.cc:1514] Processing DeleteTablet for tablet c7da2c406e64426faf123c4bddfaf213 with delete_type TABLET_DATA_DELETED (replaced by tablet 423d5d1a57484806b76565aa2292c2a1) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:45.304203 736 tablet_service.cc:1467] Processing CreateTablet for tablet 1cc28dd1eb034817b62bd9d8fd78d053 (DEFAULT_TABLE table=default.Table_b69c092ae6994928ad1eb5c10421a027 [id=14539bade7774fafa1f17ee4e2307405]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:45.312338 1195 raft_consensus.cc:2267] T 79aaab73797c474bae06831fa308b066 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:45.312618 736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1cc28dd1eb034817b62bd9d8fd78d053. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:45.313031 1194 tablet_bootstrap.cc:492] T b815003a27f64809baaa2d43d6293cef P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:45.313519 1195 ts_tablet_manager.cc:1905] T 79aaab73797c474bae06831fa308b066 P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:45.314375 1194 tablet_bootstrap.cc:654] T b815003a27f64809baaa2d43d6293cef P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:45.316570 883 tablet_service.cc:1467] Processing CreateTablet for tablet d89bcd0ceba04fd999b07e9c90b72252 (DEFAULT_TABLE table=default.Table_d9aa4ecc3d6143ce8ba0e6b1f5f87df0 [id=92b8260962ce455688807ef0b58e351a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:45.316934 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d89bcd0ceba04fd999b07e9c90b72252. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:45.319226 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b09082d6672344778dd319005e9fbe8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:45.322911 2603 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250113 02:51:45.325879 1196 ts_tablet_manager.cc:1918] T eceed0f52c9f46bbaa78ca773a7afde5 P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:45.308456 1191 consensus_queue.cc:260] T 9174df876fe64adc805fba0a36fb4d38 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:45.311079 2776 tablet.cc:1721] T 938cc1ccd8dd449996836188ba9e64ff P 1186fa869eb14812832fa6562531f196: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250113 02:51:45.330408 1191 ts_tablet_manager.cc:1428] T 9174df876fe64adc805fba0a36fb4d38 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.033s user 0.001s sys 0.000s
I20250113 02:51:45.332376 2603 hybrid_clock.cc:648] HybridClock initialized: now 1736736705332244 us; error 97 us; skew 500 ppm
I20250113 02:51:45.332736 1191 tablet_bootstrap.cc:492] T d59134b029a143038837366f51f00d28 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:45.332888 2603 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250113 02:51:45.334654 1191 tablet_bootstrap.cc:654] T d59134b029a143038837366f51f00d28 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:45.334579 1039 tablet_service.cc:1467] Processing CreateTablet for tablet d89bcd0ceba04fd999b07e9c90b72252 (DEFAULT_TABLE table=default.Table_d9aa4ecc3d6143ce8ba0e6b1f5f87df0 [id=92b8260962ce455688807ef0b58e351a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:45.334985 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d89bcd0ceba04fd999b07e9c90b72252. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:45.336989 739 tablet_service.cc:1467] Processing CreateTablet for tablet b09082d6672344778dd319005e9fbe8b (DEFAULT_TABLE table=default.Table_37bcc8921e964a6391e0c8adde667af8 [id=4d1c6a9d2bb140188df4565f5bd2cf75]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:45.339656 751 tablet_service.cc:1467] Processing CreateTablet for tablet d89bcd0ceba04fd999b07e9c90b72252 (DEFAULT_TABLE table=default.Table_d9aa4ecc3d6143ce8ba0e6b1f5f87df0 [id=92b8260962ce455688807ef0b58e351a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:45.344651 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d89bcd0ceba04fd999b07e9c90b72252. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:45.344888 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b09082d6672344778dd319005e9fbe8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:45.349612 2776 tablet.cc:1721] T b5c1882125b5491ea79a29a6dac6771e P 1186fa869eb14812832fa6562531f196: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
W20250113 02:51:45.352382 439 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet bd48dbbfe20342569e6f153353f1ef9c (table default.Table_0eba782647e84a61bda213d2ce05f0a4 [id=4496eba45fa043bc91553e1d77c53234], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet bd48dbbfe20342569e6f153353f1ef9c consensus state has no leader
I20250113 02:51:45.352535 1024 tablet_service.cc:1467] Processing CreateTablet for tablet 96218af997124404aaa550418b260cb6 (DEFAULT_TABLE table=default.Table_db479ab5faf8447d825ff655bfed0245 [id=8b2363e6f3ae4c17aa0ca80af5c83b3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:45.352962 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96218af997124404aaa550418b260cb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:45.354382 1028 tablet_service.cc:1467] Processing CreateTablet for tablet b862b0035e2845e4886ed8cdc6f9c948 (DEFAULT_TABLE table=default.Table_07901047d0824d36a72c975873c14d3a [id=2782f401f878441989c5209cb081a2bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:45.354740 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b862b0035e2845e4886ed8cdc6f9c948. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:45.352427 1198 ts_tablet_manager.cc:1918] T 4c759831773849f2bc52e484ffa0cf49 P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250113 02:51:45.357817 1198 log.cc:1198] T 4c759831773849f2bc52e484ffa0cf49 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/4c759831773849f2bc52e484ffa0cf49
I20250113 02:51:45.358465 1198 ts_tablet_manager.cc:1939] T 4c759831773849f2bc52e484ffa0cf49 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:45.332995 773 raft_consensus.cc:2463] T 16a132384b0e4264a73d09a3ea133893 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 1.
I20250113 02:51:45.360495 995 leader_election.cc:304] T 16a132384b0e4264a73d09a3ea133893 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters: df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:45.361075 2768 raft_consensus.cc:2798] T 16a132384b0e4264a73d09a3ea133893 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Leader election won for term 1
I20250113 02:51:45.364290 2768 raft_consensus.cc:695] T 16a132384b0e4264a73d09a3ea133893 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: Becoming Leader. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Running, Role: LEADER
I20250113 02:51:45.364516 2768 consensus_queue.cc:237] T 16a132384b0e4264a73d09a3ea133893 P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
W20250113 02:51:45.362439 998 consensus_peers.cc:487] T f105f8b32a654e4ca154e8fc0cb96091 P 4216a7f853d642b7bc3e5cff458f3aca -> Peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Couldn't send request to peer df69f48656b44bb890f76587ee69cc0f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250113 02:51:45.366400 739 tablet_service.cc:1467] Processing CreateTablet for tablet 96218af997124404aaa550418b260cb6 (DEFAULT_TABLE table=default.Table_db479ab5faf8447d825ff655bfed0245 [id=8b2363e6f3ae4c17aa0ca80af5c83b3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:45.366863 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96218af997124404aaa550418b260cb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:45.367748 1195 ts_tablet_manager.cc:1918] T 79aaab73797c474bae06831fa308b066 P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250113 02:51:45.367914 1195 log.cc:1198] T 79aaab73797c474bae06831fa308b066 P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/79aaab73797c474bae06831fa308b066
W20250113 02:51:45.373651 451 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 305e7a5304804580b7d302d82fdb5d6f consensus state has no leader
I20250113 02:51:45.360741 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1426) took 1907 ms. Trace:
I20250113 02:51:45.377439 1198 rpcz_store.cc:276] 0113 02:51:43.453695 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:43.453734 (+ 39us) service_pool.cc:225] Handling call
0113 02:51:45.168114 (+1714380us) ts_tablet_manager.cc:1145] Deleting tablet 4c759831773849f2bc52e484ffa0cf49
0113 02:51:45.168127 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:45.300915 (+132788us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:45.352347 (+ 51432us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:45.360726 (+ 8379us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1714260}
I20250113 02:51:45.380877 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1427) took 1926 ms. Trace:
I20250113 02:51:45.380872 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet 4c759831773849f2bc52e484ffa0cf49 (table default.Table_0fcdb63a92324044972d1e69630f524d [id=cc7909c1a59e460a8db262923636f9f1]) successfully deleted
I20250113 02:51:45.378558 1192 tablet_bootstrap.cc:492] T 58053845187f466599750f2a62f4c3e1 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:45.381250 1192 ts_tablet_manager.cc:1397] T 58053845187f466599750f2a62f4c3e1 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.095s user 0.003s sys 0.005s
I20250113 02:51:45.383695 2772 consensus_queue.cc:1035] T e01e7f8db83d461bbfbb2e901419ce0b P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 02:51:45.387343 748 tablet_service.cc:1514] Processing DeleteTablet for tablet fe3d35d9aaf64188aaa3c1ac38d279d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:36 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:45.387722 752 tablet_service.cc:1514] Processing DeleteTablet for tablet fb2458e2044d49029ed2d6851db9de25 with delete_type TABLET_DATA_DELETED (replaced by tablet 32e3dc0b7c044415a6343368ffa08552) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:45.387812 738 tablet_service.cc:1514] Processing DeleteTablet for tablet 415a1fa4b2f14cacb43937c8a372849a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:26 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:45.387888 740 tablet_service.cc:1514] Processing DeleteTablet for tablet 98bfe9620e284ab1954433ba9a036392 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:28 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:45.388041 743 tablet_service.cc:1514] Processing DeleteTablet for tablet a2781ba5061a4225b5636366857a229e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:24 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:45.390508 1196 log.cc:1198] T eceed0f52c9f46bbaa78ca773a7afde5 P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/eceed0f52c9f46bbaa78ca773a7afde5
I20250113 02:51:45.391368 1196 ts_tablet_manager.cc:1939] T eceed0f52c9f46bbaa78ca773a7afde5 P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:45.382264 1192 raft_consensus.cc:357] T 58053845187f466599750f2a62f4c3e1 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:45.391769 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 28000f78e5dd408c8de58c674ad95f6c
I20250113 02:51:45.391785 1196 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37366 (request call id 1345) took 1180 ms. Trace:
I20250113 02:51:45.391844 1192 raft_consensus.cc:383] T 58053845187f466599750f2a62f4c3e1 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:45.392210 1192 raft_consensus.cc:738] T 58053845187f466599750f2a62f4c3e1 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:45.392772 896 tablet_service.cc:1514] Processing DeleteTablet for tablet 28000f78e5dd408c8de58c674ad95f6c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:45 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:45.391865 1196 rpcz_store.cc:276] 0113 02:51:44.211400 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:44.211816 (+ 416us) service_pool.cc:225] Handling call
0113 02:51:44.768719 (+556903us) ts_tablet_manager.cc:1145] Deleting tablet eceed0f52c9f46bbaa78ca773a7afde5
0113 02:51:44.768728 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:44.841177 (+ 72449us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:45.325793 (+484616us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:45.391767 (+ 65974us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":556788}
I20250113 02:51:45.393244 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet eceed0f52c9f46bbaa78ca773a7afde5 (table default.Table_6a0195b054a043269e133f6ed8dd5979 [id=4b2bd999368c4b628c70cb3ee8c2b1f3]) successfully deleted
I20250113 02:51:45.393633 1196 tablet_replica.cc:331] T 1014e51f1c184d729552a5b34292f232 P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:45.393824 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet 28000f78e5dd408c8de58c674ad95f6c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:45 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:45.327764 891 tablet_service.cc:1514] Processing DeleteTablet for tablet c7da2c406e64426faf123c4bddfaf213 with delete_type TABLET_DATA_DELETED (replaced by tablet 423d5d1a57484806b76565aa2292c2a1) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:45.339053 882 tablet_service.cc:1467] Processing CreateTablet for tablet 96218af997124404aaa550418b260cb6 (DEFAULT_TABLE table=default.Table_db479ab5faf8447d825ff655bfed0245 [id=8b2363e6f3ae4c17aa0ca80af5c83b3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:45.394260 743 tablet_service.cc:1514] Processing DeleteTablet for tablet 28000f78e5dd408c8de58c674ad95f6c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:45 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:45.353446 883 tablet_service.cc:1467] Processing CreateTablet for tablet b862b0035e2845e4886ed8cdc6f9c948 (DEFAULT_TABLE table=default.Table_07901047d0824d36a72c975873c14d3a [id=2782f401f878441989c5209cb081a2bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:45.356953 1194 tablet_bootstrap.cc:492] T b815003a27f64809baaa2d43d6293cef P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:45.394743 1194 ts_tablet_manager.cc:1397] T b815003a27f64809baaa2d43d6293cef P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.082s user 0.003s sys 0.009s
I20250113 02:51:45.395032 1196 raft_consensus.cc:2238] T 1014e51f1c184d729552a5b34292f232 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:45.395327 1196 raft_consensus.cc:2267] T 1014e51f1c184d729552a5b34292f232 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:45.395562 1194 raft_consensus.cc:357] T b815003a27f64809baaa2d43d6293cef P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:45.395779 1194 raft_consensus.cc:383] T b815003a27f64809baaa2d43d6293cef P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:45.395835 1194 raft_consensus.cc:738] T b815003a27f64809baaa2d43d6293cef P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:45.395887 448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_f4d4f601c3c44c599304d8eb7ceb30c9" } new_table_name: "default.Table_37549a31a7dc4e7fabc47418eeb084b8" modify_external_catalogs: true
I20250113 02:51:45.396267 1194 consensus_queue.cc:260] T b815003a27f64809baaa2d43d6293cef P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:45.397675 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_2cdb0197edfc4223a764898c3f226cd8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:45.397935 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_b1bd611e962141f19470cb3296e4bd00" } modify_external_catalogs: true
I20250113 02:51:45.398062 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_b1bd611e962141f19470cb3296e4bd00" } modify_external_catalogs: true
W20250113 02:51:45.398209 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2cdb0197edfc4223a764898c3f226cd8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:45.399456 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_6d93783c61ff45a1a8908e77ca3e03a9" } new_table_name: "default.Table_186f88df8da44d0c87ac23e9406895d6" modify_external_catalogs: true
I20250113 02:51:45.400686 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_a0736ea5f5ac4ab1a819a6c08d410086"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:45.401099 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a0736ea5f5ac4ab1a819a6c08d410086 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:45.402248 891 tablet_service.cc:1514] Processing DeleteTablet for tablet b33025a462e74f2aa26cc3779f36615b with delete_type TABLET_DATA_DELETED (replaced by tablet f9269967dea94fc3a337546ad0842e11) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:45.404358 455 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_5a12a9665e634f778ea9191a4b31f2aa" } modify_external_catalogs: true
I20250113 02:51:45.404577 455 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_5a12a9665e634f778ea9191a4b31f2aa" } modify_external_catalogs: true
W20250113 02:51:45.405066 711 consensus_peers.cc:487] T f1237f7b04af4ebbb39b03aa3c78a499 P 1186fa869eb14812832fa6562531f196 -> Peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Couldn't send request to peer df69f48656b44bb890f76587ee69cc0f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250113 02:51:45.408175 1195 ts_tablet_manager.cc:1939] T 79aaab73797c474bae06831fa308b066 P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:45.408371 1194 ts_tablet_manager.cc:1428] T b815003a27f64809baaa2d43d6293cef P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.014s user 0.001s sys 0.001s
I20250113 02:51:45.408603 449 catalog_manager.cc:5618] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467) does not have the latest schema for tablet b815003a27f64809baaa2d43d6293cef (table default.Table_c2e8fd7d2cc14fa39c0abfece783c6c9 [id=42e19138c08f422a92ba9621e9609957]). Expected version 1 got 0
I20250113 02:51:45.409413 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1235) took 5178 ms. Trace:
I20250113 02:51:45.409617 449 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_ec0ab13bb2604f4da00a662572ae1734" } modify_external_catalogs: true
I20250113 02:51:45.409750 449 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_ec0ab13bb2604f4da00a662572ae1734" } modify_external_catalogs: true
I20250113 02:51:45.409572 1195 rpcz_store.cc:276] 0113 02:51:40.231360 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:40.231414 (+ 54us) service_pool.cc:225] Handling call
0113 02:51:45.272103 (+5040689us) ts_tablet_manager.cc:1145] Deleting tablet 79aaab73797c474bae06831fa308b066
0113 02:51:45.272116 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:45.316451 (+ 44335us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:45.367647 (+ 51196us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:45.409360 (+ 41713us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5040504}
I20250113 02:51:45.410331 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet 79aaab73797c474bae06831fa308b066 (table default.Table_f7097c6cf1fd4839872c42a30fa5f64f [id=89b7f6f853ca4e3c87cbccd41698009a]) successfully deleted
I20250113 02:51:45.410806 1195 tablet_replica.cc:331] T 4c759831773849f2bc52e484ffa0cf49 P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:45.411221 1195 raft_consensus.cc:2238] T 4c759831773849f2bc52e484ffa0cf49 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:45.411616 1195 raft_consensus.cc:2267] T 4c759831773849f2bc52e484ffa0cf49 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:45.411831 751 tablet_service.cc:1514] Processing DeleteTablet for tablet 39a014ac69a2435a9132a75125063093 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:26 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:45.415004 739 tablet_service.cc:1514] Processing DeleteTablet for tablet 21bcdd741104474bad7023426856d18f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:33 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:45.415625 1191 tablet_bootstrap.cc:492] T d59134b029a143038837366f51f00d28 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:45.415755 1191 ts_tablet_manager.cc:1397] T d59134b029a143038837366f51f00d28 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.083s user 0.001s sys 0.014s
I20250113 02:51:45.416054 1194 tablet_bootstrap.cc:492] T 64aa8390b1b1475c8743b6491d40674c P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:45.416519 1191 raft_consensus.cc:357] T d59134b029a143038837366f51f00d28 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:45.416707 1191 raft_consensus.cc:383] T d59134b029a143038837366f51f00d28 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:45.416770 1191 raft_consensus.cc:738] T d59134b029a143038837366f51f00d28 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:45.418269 1194 tablet_bootstrap.cc:654] T 64aa8390b1b1475c8743b6491d40674c P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:45.418622 1196 ts_tablet_manager.cc:1905] T 1014e51f1c184d729552a5b34292f232 P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:45.418769 1191 consensus_queue.cc:260] T d59134b029a143038837366f51f00d28 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:45.419687 2769 consensus_queue.cc:1035] T 8cc78bf7d6324b7ba10a1b1045923f2b P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 02:51:45.366632 2779 tablet.cc:1721] T b5c1882125b5491ea79a29a6dac6771e P df69f48656b44bb890f76587ee69cc0f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250113 02:51:45.426190 2769 consensus_queue.cc:1035] T ca75e7cae4f6423191a17bb5a8f150a9 P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 02:51:45.428579 1195 ts_tablet_manager.cc:1905] T 4c759831773849f2bc52e484ffa0cf49 P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:45.430662 1024 tablet_service.cc:1467] Processing CreateTablet for tablet 5afd8ccebd9e440cbecb60de81334050 (DEFAULT_TABLE table=default.Table_4600561619ef45df810b4bb624864f7d [id=bbc1fdbc1add440bbb41fd62a7bf1f63]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:45.431097 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5afd8ccebd9e440cbecb60de81334050. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:45.431401 751 tablet_service.cc:1467] Processing CreateTablet for tablet 5afd8ccebd9e440cbecb60de81334050 (DEFAULT_TABLE table=default.Table_4600561619ef45df810b4bb624864f7d [id=bbc1fdbc1add440bbb41fd62a7bf1f63]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:45.431936 739 tablet_service.cc:1467] Processing CreateTablet for tablet b862b0035e2845e4886ed8cdc6f9c948 (DEFAULT_TABLE table=default.Table_07901047d0824d36a72c975873c14d3a [id=2782f401f878441989c5209cb081a2bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:45.392781 1192 consensus_queue.cc:260] T 58053845187f466599750f2a62f4c3e1 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:45.433838 1192 ts_tablet_manager.cc:1428] T 58053845187f466599750f2a62f4c3e1 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.052s user 0.002s sys 0.000s
I20250113 02:51:45.434260 885 tablet_service.cc:1467] Processing CreateTablet for tablet 5afd8ccebd9e440cbecb60de81334050 (DEFAULT_TABLE table=default.Table_4600561619ef45df810b4bb624864f7d [id=bbc1fdbc1add440bbb41fd62a7bf1f63]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:45.434880 1192 tablet_bootstrap.cc:492] T afcb5b1977c447d3a51de9cb72d7196a P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:45.435442 1191 ts_tablet_manager.cc:1428] T d59134b029a143038837366f51f00d28 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.020s user 0.001s sys 0.001s
I20250113 02:51:45.436777 1191 tablet_bootstrap.cc:492] T 36726b40ac5145bfb371871b3fff079a P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:45.436745 891 tablet_service.cc:1467] Processing CreateTablet for tablet e386a99e7a0446a58f83f4cac0dc2e3a (DEFAULT_TABLE table=default.Table_534ccaa08cbf4ee6adaacca75de576bf [id=537aeb13a331411ea83eb203cb6a96fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:45.437464 743 tablet_service.cc:1467] Processing CreateTablet for tablet e386a99e7a0446a58f83f4cac0dc2e3a (DEFAULT_TABLE table=default.Table_534ccaa08cbf4ee6adaacca75de576bf [id=537aeb13a331411ea83eb203cb6a96fd]), partition=RANGE (key) PARTITION UNBOUNDED
W20250113 02:51:45.438112 998 consensus_peers.cc:487] T ca75e7cae4f6423191a17bb5a8f150a9 P 4216a7f853d642b7bc3e5cff458f3aca -> Peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Couldn't send request to peer df69f48656b44bb890f76587ee69cc0f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250113 02:51:45.438548 1028 tablet_service.cc:1467] Processing CreateTablet for tablet e386a99e7a0446a58f83f4cac0dc2e3a (DEFAULT_TABLE table=default.Table_534ccaa08cbf4ee6adaacca75de576bf [id=537aeb13a331411ea83eb203cb6a96fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:45.439365 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e386a99e7a0446a58f83f4cac0dc2e3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:45.440047 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e386a99e7a0446a58f83f4cac0dc2e3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:45.441097 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96218af997124404aaa550418b260cb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:45.444077 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b862b0035e2845e4886ed8cdc6f9c948. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:45.444283 1191 tablet_bootstrap.cc:654] T 36726b40ac5145bfb371871b3fff079a P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:45.380962 1198 rpcz_store.cc:276] 0113 02:51:43.454116 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:43.454257 (+ 141us) service_pool.cc:225] Handling call
0113 02:51:45.380752 (+1926495us) ts_tablet_manager.cc:1145] Deleting tablet 1c82cf9373d24fedad88b5df7d5a8b64
0113 02:51:45.380762 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:45.380860 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1926361}
W20250113 02:51:45.446167 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 1c82cf9373d24fedad88b5df7d5a8b64 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1c82cf9373d24fedad88b5df7d5a8b64 already in progress: creating tablet
I20250113 02:51:45.446247 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1429) took 1978 ms. Trace:
I20250113 02:51:45.446336 1198 rpcz_store.cc:276] 0113 02:51:43.468204 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:43.468410 (+ 206us) service_pool.cc:225] Handling call
0113 02:51:45.446133 (+1977723us) ts_tablet_manager.cc:1145] Deleting tablet e312e70334e04ecd843159ac72f4a2a6
0113 02:51:45.446143 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:45.446231 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1977546}
W20250113 02:51:45.446688 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet e312e70334e04ecd843159ac72f4a2a6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e312e70334e04ecd843159ac72f4a2a6 already in progress: creating tablet
I20250113 02:51:45.447536 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1434) took 1634 ms. Trace:
I20250113 02:51:45.448130 1198 rpcz_store.cc:276] 0113 02:51:43.813024 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:43.813076 (+ 52us) service_pool.cc:225] Handling call
0113 02:51:45.447444 (+1634368us) ts_tablet_manager.cc:1145] Deleting tablet 4e59953509884a0f82b276b232bb00fb
0113 02:51:45.447453 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:45.447522 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1634201}
W20250113 02:51:45.448825 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 4e59953509884a0f82b276b232bb00fb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4e59953509884a0f82b276b232bb00fb already in progress: creating tablet
I20250113 02:51:45.449002 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1435) took 1618 ms. Trace:
I20250113 02:51:45.449172 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e386a99e7a0446a58f83f4cac0dc2e3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:45.449635 1198 rpcz_store.cc:276] 0113 02:51:43.830849 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:43.830896 (+ 47us) service_pool.cc:225] Handling call
0113 02:51:45.448876 (+1617980us) ts_tablet_manager.cc:1145] Deleting tablet 8e4b74a1ef824f40a40c0b215b2b5711
0113 02:51:45.448884 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:45.448983 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1617809}
I20250113 02:51:45.449999 2782 raft_consensus.cc:491] T 7fb282707196444690352dea3eb1387f P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:45.450171 2782 raft_consensus.cc:513] T 7fb282707196444690352dea3eb1387f P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:45.450894 2782 leader_election.cc:290] T 7fb282707196444690352dea3eb1387f P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:45.451905 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fb282707196444690352dea3eb1387f" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:45.452543 711 leader_election.cc:343] T 7fb282707196444690352dea3eb1387f P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
W20250113 02:51:45.452839 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 8e4b74a1ef824f40a40c0b215b2b5711 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8e4b74a1ef824f40a40c0b215b2b5711 already in progress: creating tablet
I20250113 02:51:45.453161 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fb282707196444690352dea3eb1387f" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
W20250113 02:51:45.453704 709 leader_election.cc:343] T 7fb282707196444690352dea3eb1387f P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:45.453823 709 leader_election.cc:304] T 7fb282707196444690352dea3eb1387f P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:45.454269 2782 raft_consensus.cc:2743] T 7fb282707196444690352dea3eb1387f P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:45.455389 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1436) took 1511 ms. Trace:
I20250113 02:51:45.456950 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5afd8ccebd9e440cbecb60de81334050. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:45.460107 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b862b0035e2845e4886ed8cdc6f9c948. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:45.459712 998 consensus_peers.cc:487] T cf7b8bd6801743f49e9b9473747447c5 P 4216a7f853d642b7bc3e5cff458f3aca -> Peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Couldn't send request to peer df69f48656b44bb890f76587ee69cc0f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250113 02:51:45.462850 1198 rpcz_store.cc:276] 0113 02:51:43.943969 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:43.944089 (+ 120us) service_pool.cc:225] Handling call
0113 02:51:45.454871 (+1510782us) ts_tablet_manager.cc:1145] Deleting tablet 4a654e8085fd49729aef1d343ceaf789
0113 02:51:45.454883 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:45.455369 (+ 486us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1510606}
W20250113 02:51:45.463429 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 4a654e8085fd49729aef1d343ceaf789 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4a654e8085fd49729aef1d343ceaf789 already in progress: creating tablet
I20250113 02:51:45.463889 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1437) took 1410 ms. Trace:
I20250113 02:51:45.464015 1198 rpcz_store.cc:276] 0113 02:51:44.053831 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:44.054119 (+ 288us) service_pool.cc:225] Handling call
0113 02:51:45.463768 (+1409649us) ts_tablet_manager.cc:1145] Deleting tablet ca144327ca8d48d48a072caaff74212f
0113 02:51:45.463781 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:45.463870 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1405065}
W20250113 02:51:45.464465 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet ca144327ca8d48d48a072caaff74212f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ca144327ca8d48d48a072caaff74212f already in progress: creating tablet
I20250113 02:51:45.465560 1194 tablet_bootstrap.cc:492] T 64aa8390b1b1475c8743b6491d40674c P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:45.465852 1194 ts_tablet_manager.cc:1397] T 64aa8390b1b1475c8743b6491d40674c P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.050s user 0.000s sys 0.006s
I20250113 02:51:45.466509 1194 raft_consensus.cc:357] T 64aa8390b1b1475c8743b6491d40674c P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:45.466724 1194 raft_consensus.cc:383] T 64aa8390b1b1475c8743b6491d40674c P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:45.466815 1194 raft_consensus.cc:738] T 64aa8390b1b1475c8743b6491d40674c P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:45.467028 1194 consensus_queue.cc:260] T 64aa8390b1b1475c8743b6491d40674c P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:45.474617 1192 tablet_bootstrap.cc:654] T afcb5b1977c447d3a51de9cb72d7196a P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
W20250113 02:51:45.476361 998 consensus_peers.cc:487] T e01e7f8db83d461bbfbb2e901419ce0b P 4216a7f853d642b7bc3e5cff458f3aca -> Peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Couldn't send request to peer df69f48656b44bb890f76587ee69cc0f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250113 02:51:45.475307 1198 tablet_replica.cc:331] T cfa963a6df094ce0b9c54408f75ca6b2 P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:45.477916 1198 raft_consensus.cc:2238] T cfa963a6df094ce0b9c54408f75ca6b2 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: Raft consensus shutting down.
I20250113 02:51:45.478251 1198 pending_rounds.cc:62] T cfa963a6df094ce0b9c54408f75ca6b2 P 4216a7f853d642b7bc3e5cff458f3aca: Trying to abort 1 pending ops.
I20250113 02:51:45.478351 1198 pending_rounds.cc:69] T cfa963a6df094ce0b9c54408f75ca6b2 P 4216a7f853d642b7bc3e5cff458f3aca: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113673538721976320 op_type: NO_OP noop_request { }
I20250113 02:51:45.478515 1198 raft_consensus.cc:2883] T cfa963a6df094ce0b9c54408f75ca6b2 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250113 02:51:45.478668 1198 raft_consensus.cc:2267] T cfa963a6df094ce0b9c54408f75ca6b2 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:45.487264 1024 tablet_service.cc:1467] Processing CreateTablet for tablet f3e968f39d6e48afa5f9b2b8ab0ffe1d (DEFAULT_TABLE table=default.Table_3263b752318740a49b84af744eac2002 [id=a1ec8f20998f4118948cfacb77bcebcd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:45.487715 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3e968f39d6e48afa5f9b2b8ab0ffe1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:45.467509 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5afd8ccebd9e440cbecb60de81334050. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:45.486112 883 tablet_service.cc:1467] Processing CreateTablet for tablet f3e968f39d6e48afa5f9b2b8ab0ffe1d (DEFAULT_TABLE table=default.Table_3263b752318740a49b84af744eac2002 [id=a1ec8f20998f4118948cfacb77bcebcd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:45.497375 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3e968f39d6e48afa5f9b2b8ab0ffe1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:45.498662 1194 ts_tablet_manager.cc:1428] T 64aa8390b1b1475c8743b6491d40674c P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.033s user 0.000s sys 0.001s
I20250113 02:51:45.499322 1198 ts_tablet_manager.cc:1905] T cfa963a6df094ce0b9c54408f75ca6b2 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:45.501804 1194 tablet_bootstrap.cc:492] T 438604e8859f422f9caccb61c5b5b125 P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:45.503954 2782 raft_consensus.cc:491] T d565b77f90784d87b64996772c938085 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:45.504169 2782 raft_consensus.cc:513] T d565b77f90784d87b64996772c938085 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:45.504971 2782 leader_election.cc:290] T d565b77f90784d87b64996772c938085 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:45.506742 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d565b77f90784d87b64996772c938085" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
W20250113 02:51:45.507525 709 leader_election.cc:343] T d565b77f90784d87b64996772c938085 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:45.501503 1192 tablet_bootstrap.cc:492] T afcb5b1977c447d3a51de9cb72d7196a P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:45.510671 1192 ts_tablet_manager.cc:1397] T afcb5b1977c447d3a51de9cb72d7196a P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.076s user 0.004s sys 0.000s
I20250113 02:51:45.503566 913 raft_consensus.cc:3054] T bc6a7322c0c34adb85457750526d8955 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:45.512869 882 tablet_service.cc:1467] Processing CreateTablet for tablet 6a56c3979f90498c9c6b7966e1255c5a (DEFAULT_TABLE table=default.Table_5979f36091aa4a0c89338e504b22b891 [id=731d0911e8594c73b7d067721e5d5de9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:45.513250 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a56c3979f90498c9c6b7966e1255c5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:45.505981 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d565b77f90784d87b64996772c938085" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:45.515059 711 leader_election.cc:343] T d565b77f90784d87b64996772c938085 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:45.515221 711 leader_election.cc:304] T d565b77f90784d87b64996772c938085 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:45.517455 2782 raft_consensus.cc:2743] T d565b77f90784d87b64996772c938085 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:45.519444 736 tablet_service.cc:1467] Processing CreateTablet for tablet 28000f78e5dd408c8de58c674ad95f6c (DEFAULT_TABLE table=default.Table_597caad30c254169ad25df814c309739 [id=e3cc2b0a437c44afb7ce2026283b4d8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:45.519918 736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 28000f78e5dd408c8de58c674ad95f6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:45.510001 883 tablet_service.cc:1467] Processing CreateTablet for tablet 28000f78e5dd408c8de58c674ad95f6c (DEFAULT_TABLE table=default.Table_597caad30c254169ad25df814c309739 [id=e3cc2b0a437c44afb7ce2026283b4d8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:45.521281 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 28000f78e5dd408c8de58c674ad95f6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:45.522012 2476 raft_consensus.cc:491] T ba8980ad3a364c888a825affbcbc9b1f P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:45.522156 2476 raft_consensus.cc:513] T ba8980ad3a364c888a825affbcbc9b1f P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:45.522795 2476 leader_election.cc:290] T ba8980ad3a364c888a825affbcbc9b1f P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:45.522823 1196 ts_tablet_manager.cc:1918] T 1014e51f1c184d729552a5b34292f232 P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:45.522912 1196 log.cc:1198] T 1014e51f1c184d729552a5b34292f232 P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/1014e51f1c184d729552a5b34292f232
I20250113 02:51:45.523212 1196 ts_tablet_manager.cc:1939] T 1014e51f1c184d729552a5b34292f232 P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:45.523526 1196 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37366 (request call id 1347) took 1306 ms. Trace:
I20250113 02:51:45.523823 2782 consensus_queue.cc:1035] T f1237f7b04af4ebbb39b03aa3c78a499 P 1186fa869eb14812832fa6562531f196 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 02:51:45.536382 1192 raft_consensus.cc:357] T afcb5b1977c447d3a51de9cb72d7196a P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:45.537640 2782 raft_consensus.cc:491] T 438d1d2300e044f39a15586f335935ba P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4216a7f853d642b7bc3e5cff458f3aca)
I20250113 02:51:45.537825 2782 raft_consensus.cc:513] T 438d1d2300e044f39a15586f335935ba P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:45.538651 2782 leader_election.cc:290] T 438d1d2300e044f39a15586f335935ba P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:45.539248 1195 ts_tablet_manager.cc:1918] T 4c759831773849f2bc52e484ffa0cf49 P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250113 02:51:45.539441 1195 log.cc:1198] T 4c759831773849f2bc52e484ffa0cf49 P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/4c759831773849f2bc52e484ffa0cf49
I20250113 02:51:45.539884 1195 ts_tablet_manager.cc:1939] T 4c759831773849f2bc52e484ffa0cf49 P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
W20250113 02:51:45.540127 709 leader_election.cc:343] T 438d1d2300e044f39a15586f335935ba P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Not found: Tablet not found: 438d1d2300e044f39a15586f335935ba
I20250113 02:51:45.540774 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1236) took 5289 ms. Trace:
I20250113 02:51:45.544015 1195 rpcz_store.cc:276] 0113 02:51:40.251256 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:40.251494 (+ 238us) service_pool.cc:225] Handling call
0113 02:51:45.410681 (+5159187us) ts_tablet_manager.cc:1145] Deleting tablet 4c759831773849f2bc52e484ffa0cf49
0113 02:51:45.410730 (+ 49us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:45.515898 (+105168us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:45.539144 (+ 23246us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:45.540750 (+ 1606us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5158995}
I20250113 02:51:45.544621 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet 4c759831773849f2bc52e484ffa0cf49 (table default.Table_0fcdb63a92324044972d1e69630f524d [id=cc7909c1a59e460a8db262923636f9f1]) successfully deleted
I20250113 02:51:45.545128 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1237) took 5264 ms. Trace:
W20250113 02:51:45.524001 852 leader_election.cc:343] T ba8980ad3a364c888a825affbcbc9b1f P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Not found: Tablet not found: ba8980ad3a364c888a825affbcbc9b1f
I20250113 02:51:45.526466 914 raft_consensus.cc:3054] T b815003a27f64809baaa2d43d6293cef P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Advancing to term 1
W20250113 02:51:45.545729 851 leader_election.cc:343] T ba8980ad3a364c888a825affbcbc9b1f P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Not found: Tablet not found: ba8980ad3a364c888a825affbcbc9b1f
I20250113 02:51:45.546084 851 leader_election.cc:304] T ba8980ad3a364c888a825affbcbc9b1f P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df69f48656b44bb890f76587ee69cc0f; no voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca
I20250113 02:51:45.546833 2476 raft_consensus.cc:2743] T ba8980ad3a364c888a825affbcbc9b1f P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:45.547899 1196 rpcz_store.cc:276] 0113 02:51:44.216909 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:44.216960 (+ 51us) service_pool.cc:225] Handling call
0113 02:51:45.393577 (+1176617us) ts_tablet_manager.cc:1145] Deleting tablet 1014e51f1c184d729552a5b34292f232
0113 02:51:45.393590 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:45.502754 (+109164us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:45.522731 (+ 19977us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:45.523510 (+ 779us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1176444}
I20250113 02:51:45.548621 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet 1014e51f1c184d729552a5b34292f232 (table default.Table_2abef959c5c34459a8dfa3034e207995 [id=f7e15a7c1fe848d591ba3232281c45ed]) successfully deleted
I20250113 02:51:45.549042 1196 tablet_replica.cc:331] T ee240d838d4a4bbba97f8ed61f69de78 P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:45.549304 1196 raft_consensus.cc:2238] T ee240d838d4a4bbba97f8ed61f69de78 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:45.549669 1196 raft_consensus.cc:2267] T ee240d838d4a4bbba97f8ed61f69de78 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:45.550359 1196 ts_tablet_manager.cc:1905] T ee240d838d4a4bbba97f8ed61f69de78 P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:45.557972 453 catalog_manager.cc:5618] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467) does not have the latest schema for tablet b815003a27f64809baaa2d43d6293cef (table default.Table_c2e8fd7d2cc14fa39c0abfece783c6c9 [id=42e19138c08f422a92ba9621e9609957]). Expected version 1 got 0
I20250113 02:51:45.558765 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_168905c33c1e4fb18c72bce89c511483" } new_table_name: "default.Table_d467386709824eb68cc352a120812d3e" modify_external_catalogs: true
I20250113 02:51:45.560504 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_469a767a55b84a8b930b57666c06575b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:45.561013 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_469a767a55b84a8b930b57666c06575b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:45.568646 447 catalog_manager.cc:5618] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095) does not have the latest schema for tablet e01e7f8db83d461bbfbb2e901419ce0b (table default.Table_e27f8b6f8232415388a5e53ab15494b7 [id=efc4ebba97f74c9ba12ab1dcc6577da0]). Expected version 1 got 0
W20250113 02:51:45.548600 711 leader_election.cc:343] T 438d1d2300e044f39a15586f335935ba P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Not found: Tablet not found: 438d1d2300e044f39a15586f335935ba
I20250113 02:51:45.570056 711 leader_election.cc:304] T 438d1d2300e044f39a15586f335935ba P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:45.570138 447 catalog_manager.cc:5618] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095) does not have the latest schema for tablet 8f2fb58fc24d4692a3fe9a652e076aed (table default.Table_a752553e5fcc4ed1a56d9d5197d6b885 [id=8edee52f05e24c03b088ce58f4b3f4d9]). Expected version 1 got 0
I20250113 02:51:45.551712 2782 raft_consensus.cc:491] T 9174df876fe64adc805fba0a36fb4d38 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:45.570571 2588 raft_consensus.cc:2743] T 438d1d2300e044f39a15586f335935ba P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250113 02:51:45.570639 2782 raft_consensus.cc:513] T 9174df876fe64adc805fba0a36fb4d38 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:45.571442 2782 leader_election.cc:290] T 9174df876fe64adc805fba0a36fb4d38 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:45.572453 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9174df876fe64adc805fba0a36fb4d38" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:45.573016 711 leader_election.cc:343] T 9174df876fe64adc805fba0a36fb4d38 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:45.569981 1195 rpcz_store.cc:276] 0113 02:51:40.280285 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:40.280332 (+ 47us) service_pool.cc:225] Handling call
0113 02:51:45.544992 (+5264660us) ts_tablet_manager.cc:1145] Deleting tablet 368e5f00861b40ffb76e79376245b885
0113 02:51:45.545003 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:45.545108 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5256834}
W20250113 02:51:45.574689 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 368e5f00861b40ffb76e79376245b885 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 368e5f00861b40ffb76e79376245b885 already in progress: creating tablet
I20250113 02:51:45.575112 1195 tablet_replica.cc:331] T 59fadc693f554854bc2874f8db7c64c8 P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:45.575395 1195 raft_consensus.cc:2238] T 59fadc693f554854bc2874f8db7c64c8 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:45.575641 1195 raft_consensus.cc:2267] T 59fadc693f554854bc2874f8db7c64c8 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250113 02:51:45.578672 448 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 874e4392e5d946acae0bf9e3b3994735 consensus state has no leader
I20250113 02:51:45.583279 1195 ts_tablet_manager.cc:1905] T 59fadc693f554854bc2874f8db7c64c8 P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:45.585541 1194 tablet_bootstrap.cc:654] T 438604e8859f422f9caccb61c5b5b125 P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
W20250113 02:51:45.585915 451 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4200521dc098474fb5f498676d70f023 has no consensus state
I20250113 02:51:45.586405 1192 raft_consensus.cc:383] T afcb5b1977c447d3a51de9cb72d7196a P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:45.586544 1192 raft_consensus.cc:738] T afcb5b1977c447d3a51de9cb72d7196a P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:45.586807 1192 consensus_queue.cc:260] T afcb5b1977c447d3a51de9cb72d7196a P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:45.587471 1192 ts_tablet_manager.cc:1428] T afcb5b1977c447d3a51de9cb72d7196a P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.067s user 0.002s sys 0.000s
I20250113 02:51:45.588014 1192 tablet_bootstrap.cc:492] T 542df00229f645788f1b821c0ce00c46 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:45.554168 1039 tablet_service.cc:1514] Processing DeleteTablet for tablet 569d666c128a45b7a526c13be5090ae3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:26 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:45.554420 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c759831773849f2bc52e484ffa0cf49 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:30 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:45.554980 1032 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b2d0aa6f0e9456d95f4e48c360ab6e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:33 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:45.572950 1060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9174df876fe64adc805fba0a36fb4d38" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
W20250113 02:51:45.589767 709 leader_election.cc:343] T 9174df876fe64adc805fba0a36fb4d38 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:45.589883 709 leader_election.cc:304] T 9174df876fe64adc805fba0a36fb4d38 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:45.590090 2723 raft_consensus.cc:2743] T 9174df876fe64adc805fba0a36fb4d38 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:45.590672 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 593c54696ea7473d8450744a7c4a394d
I20250113 02:51:45.509835 1024 tablet_service.cc:1467] Processing CreateTablet for tablet 28000f78e5dd408c8de58c674ad95f6c (DEFAULT_TABLE table=default.Table_597caad30c254169ad25df814c309739 [id=e3cc2b0a437c44afb7ce2026283b4d8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:45.543548 2772 consensus_queue.cc:1035] T cf7b8bd6801743f49e9b9473747447c5 P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 02:51:45.594455 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 1376cf72589d43048cb829e03240c8da
I20250113 02:51:45.594436 1032 tablet_service.cc:1514] Processing DeleteTablet for tablet 593c54696ea7473d8450744a7c4a394d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:45 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:45.595847 891 tablet_service.cc:1514] Processing DeleteTablet for tablet 1376cf72589d43048cb829e03240c8da with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:45 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:45.596454 743 tablet_service.cc:1514] Processing DeleteTablet for tablet 1376cf72589d43048cb829e03240c8da with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:45 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:45.596762 882 tablet_service.cc:1514] Processing DeleteTablet for tablet 593c54696ea7473d8450744a7c4a394d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:45 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:45.597581 1032 tablet_service.cc:1514] Processing DeleteTablet for tablet 1376cf72589d43048cb829e03240c8da with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:45 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:45.598382 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 28000f78e5dd408c8de58c674ad95f6c. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:45.598640 711 consensus_peers.cc:487] T 883027967e0c442a91ac820210b9c6c7 P 1186fa869eb14812832fa6562531f196 -> Peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Couldn't send request to peer df69f48656b44bb890f76587ee69cc0f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250113 02:51:45.601644 1196 ts_tablet_manager.cc:1918] T ee240d838d4a4bbba97f8ed61f69de78 P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:45.601816 1196 log.cc:1198] T ee240d838d4a4bbba97f8ed61f69de78 P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/ee240d838d4a4bbba97f8ed61f69de78
I20250113 02:51:45.602193 1196 ts_tablet_manager.cc:1939] T ee240d838d4a4bbba97f8ed61f69de78 P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:45.602586 1196 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37366 (request call id 1348) took 1349 ms. Trace:
I20250113 02:51:45.602715 1196 rpcz_store.cc:276] 0113 02:51:44.253071 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:44.253198 (+ 127us) service_pool.cc:225] Handling call
0113 02:51:45.548987 (+1295789us) ts_tablet_manager.cc:1145] Deleting tablet ee240d838d4a4bbba97f8ed61f69de78
0113 02:51:45.548998 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:45.597660 (+ 48662us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:45.601541 (+ 3881us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:45.602570 (+ 1029us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1295559}
I20250113 02:51:45.603219 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet ee240d838d4a4bbba97f8ed61f69de78 (table default.Table_1b71e9ea2fca48daa60e8d692eab8e49 [id=383148ed836144c1bc5403dae46aad50]) successfully deleted
I20250113 02:51:45.603682 2603 webserver.cc:458] Webserver started at http://127.0.104.60:38411/ using document root <none> and password file <none>
I20250113 02:51:45.603672 1196 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37366 (request call id 1349) took 1337 ms. Trace:
I20250113 02:51:45.605141 2603 fs_manager.cc:362] Metadata directory not provided
I20250113 02:51:45.605321 2603 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 02:51:45.607730 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_54119752ebc147b8b34429f1c38b71d9" } new_table_name: "default.Table_61f58a3d8b674f55babbce38513edc2e" modify_external_catalogs: true
I20250113 02:51:45.609362 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_c2fe3752f4a1482fb35693231429d1df"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:45.609889 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c2fe3752f4a1482fb35693231429d1df in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:45.611167 1194 tablet_bootstrap.cc:492] T 438604e8859f422f9caccb61c5b5b125 P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:45.611354 1194 ts_tablet_manager.cc:1397] T 438604e8859f422f9caccb61c5b5b125 P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.110s user 0.003s sys 0.000s
I20250113 02:51:45.611747 454 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_1a482bb1752b4f4a8fc993b60179f50b" } new_table_name: "default.Table_34a0130d81dd4e9783f91e654822a399" modify_external_catalogs: true
I20250113 02:51:45.612083 1194 raft_consensus.cc:357] T 438604e8859f422f9caccb61c5b5b125 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:45.612279 1194 raft_consensus.cc:383] T 438604e8859f422f9caccb61c5b5b125 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:45.612325 1194 raft_consensus.cc:738] T 438604e8859f422f9caccb61c5b5b125 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:45.612497 1194 consensus_queue.cc:260] T 438604e8859f422f9caccb61c5b5b125 P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:45.613225 1194 ts_tablet_manager.cc:1428] T 438604e8859f422f9caccb61c5b5b125 P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250113 02:51:45.613544 1194 tablet_bootstrap.cc:492] T 1d2fdc75d00f4e749d322ac6cd5fc410 P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:45.613536 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_484890185d6b4dea9fe5ed3102b5b625"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:45.614022 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_484890185d6b4dea9fe5ed3102b5b625 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:45.615299 448 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_d0642c34ba7d40c9888ec7521d3cb1e3" } modify_external_catalogs: true
I20250113 02:51:45.615469 448 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_d0642c34ba7d40c9888ec7521d3cb1e3" } modify_external_catalogs: true
I20250113 02:51:45.616761 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_b88ada2171ff424baba4ef155ad02b02"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:45.617305 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b88ada2171ff424baba4ef155ad02b02 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:45.620015 1192 tablet_bootstrap.cc:654] T 542df00229f645788f1b821c0ce00c46 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:45.623126 1198 ts_tablet_manager.cc:1918] T cfa963a6df094ce0b9c54408f75ca6b2 P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250113 02:51:45.623128 2723 consensus_queue.cc:1035] T 883027967e0c442a91ac820210b9c6c7 P 1186fa869eb14812832fa6562531f196 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 02:51:45.623241 1198 log.cc:1198] T cfa963a6df094ce0b9c54408f75ca6b2 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/cfa963a6df094ce0b9c54408f75ca6b2
I20250113 02:51:45.623584 1198 ts_tablet_manager.cc:1939] T cfa963a6df094ce0b9c54408f75ca6b2 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:45.625391 1195 ts_tablet_manager.cc:1918] T 59fadc693f554854bc2874f8db7c64c8 P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:45.625506 1195 log.cc:1198] T 59fadc693f554854bc2874f8db7c64c8 P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/59fadc693f554854bc2874f8db7c64c8
I20250113 02:51:45.625811 1195 ts_tablet_manager.cc:1939] T 59fadc693f554854bc2874f8db7c64c8 P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:45.603838 1196 rpcz_store.cc:276] 0113 02:51:44.266312 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:44.266369 (+ 57us) service_pool.cc:225] Handling call
0113 02:51:45.603540 (+1337171us) ts_tablet_manager.cc:1145] Deleting tablet f3f6113ee3d94bb98dbe95d21b1478ef
0113 02:51:45.603550 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:45.603651 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1336989}
I20250113 02:51:45.626910 1192 tablet_bootstrap.cc:492] T 542df00229f645788f1b821c0ce00c46 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
W20250113 02:51:45.626977 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet f3f6113ee3d94bb98dbe95d21b1478ef because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f3f6113ee3d94bb98dbe95d21b1478ef already in progress: creating tablet
I20250113 02:51:45.627033 1192 ts_tablet_manager.cc:1397] T 542df00229f645788f1b821c0ce00c46 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.039s user 0.001s sys 0.001s
I20250113 02:51:45.627508 1196 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37366 (request call id 1350) took 1313 ms. Trace:
I20250113 02:51:45.627674 1196 rpcz_store.cc:276] 0113 02:51:44.314563 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:44.315041 (+ 478us) service_pool.cc:225] Handling call
0113 02:51:45.627368 (+1312327us) ts_tablet_manager.cc:1145] Deleting tablet ff645eb0f8fa4744b7ba5cd03ce21729
0113 02:51:45.627381 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:45.627487 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1282827}
W20250113 02:51:45.628551 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet ff645eb0f8fa4744b7ba5cd03ce21729 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ff645eb0f8fa4744b7ba5cd03ce21729 already in progress: creating tablet
I20250113 02:51:45.628893 1196 tablet_replica.cc:331] T 0644f5600c5b47e29f3fa73060ed96e3 P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:45.629477 1196 raft_consensus.cc:2238] T 0644f5600c5b47e29f3fa73060ed96e3 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:45.633472 1032 tablet_service.cc:1467] Processing CreateTablet for tablet 6a56c3979f90498c9c6b7966e1255c5a (DEFAULT_TABLE table=default.Table_5979f36091aa4a0c89338e504b22b891 [id=731d0911e8594c73b7d067721e5d5de9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:45.633832 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a56c3979f90498c9c6b7966e1255c5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:45.635895 882 tablet_service.cc:1467] Processing CreateTablet for tablet 2c7ce64e3f8f45018d6b4f552a1392e1 (DEFAULT_TABLE table=default.Table_906514aed3bf4e5e87a1748d6fffc261 [id=96cb0686ce4c4f1da81c955d99cd9438]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:45.636446 743 tablet_service.cc:1467] Processing CreateTablet for tablet 6a56c3979f90498c9c6b7966e1255c5a (DEFAULT_TABLE table=default.Table_5979f36091aa4a0c89338e504b22b891 [id=731d0911e8594c73b7d067721e5d5de9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:45.636636 1035 tablet_service.cc:1467] Processing CreateTablet for tablet a6603e63f3d643b2a2525adf9c7180c8 (DEFAULT_TABLE table=default.Table_f02ddc5b50b94468a206c5bb80525f28 [id=39085e6258b148c8b9ac8cd1a1c87309]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:45.636793 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c7ce64e3f8f45018d6b4f552a1392e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:45.636891 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a56c3979f90498c9c6b7966e1255c5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:45.636952 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6603e63f3d643b2a2525adf9c7180c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:45.640206 740 tablet_service.cc:1467] Processing CreateTablet for tablet a6603e63f3d643b2a2525adf9c7180c8 (DEFAULT_TABLE table=default.Table_f02ddc5b50b94468a206c5bb80525f28 [id=39085e6258b148c8b9ac8cd1a1c87309]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:45.640520 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6603e63f3d643b2a2525adf9c7180c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:45.642125 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1438) took 1469 ms. Trace:
I20250113 02:51:45.642204 1198 rpcz_store.cc:276] 0113 02:51:44.172584 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:44.173826 (+ 1242us) service_pool.cc:225] Handling call
0113 02:51:45.464706 (+1290880us) ts_tablet_manager.cc:1145] Deleting tablet cfa963a6df094ce0b9c54408f75ca6b2
0113 02:51:45.475257 (+ 10551us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:45.581123 (+105866us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:45.623014 (+ 41891us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:45.642096 (+ 19082us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1279900}
I20250113 02:51:45.642585 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet cfa963a6df094ce0b9c54408f75ca6b2 (table default.Table_0874a35f3d5a44d2ac045a85227b148d [id=9dcef6fe27f247ff868d8e2ab2a37ead]) successfully deleted
I20250113 02:51:45.642951 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1439) took 1446 ms. Trace:
I20250113 02:51:45.642971 1196 raft_consensus.cc:2267] T 0644f5600c5b47e29f3fa73060ed96e3 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:45.643013 1198 rpcz_store.cc:276] 0113 02:51:44.196186 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:44.196254 (+ 68us) service_pool.cc:225] Handling call
0113 02:51:45.642847 (+1446593us) ts_tablet_manager.cc:1145] Deleting tablet 8f46d38f2c634b7d87df37e59ef4f5ef
0113 02:51:45.642856 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:45.642937 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1446425}
I20250113 02:51:45.643244 751 tablet_service.cc:1467] Processing CreateTablet for tablet f3e968f39d6e48afa5f9b2b8ab0ffe1d (DEFAULT_TABLE table=default.Table_3263b752318740a49b84af744eac2002 [id=a1ec8f20998f4118948cfacb77bcebcd]), partition=RANGE (key) PARTITION UNBOUNDED
W20250113 02:51:45.643378 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 8f46d38f2c634b7d87df37e59ef4f5ef because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8f46d38f2c634b7d87df37e59ef4f5ef already in progress: creating tablet
I20250113 02:51:45.643566 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3e968f39d6e48afa5f9b2b8ab0ffe1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:45.643541 1196 ts_tablet_manager.cc:1905] T 0644f5600c5b47e29f3fa73060ed96e3 P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:45.643695 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1440) took 1376 ms. Trace:
I20250113 02:51:45.643759 1198 rpcz_store.cc:276] 0113 02:51:44.267078 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:44.267162 (+ 84us) service_pool.cc:225] Handling call
0113 02:51:45.643603 (+1376441us) ts_tablet_manager.cc:1145] Deleting tablet f3f6113ee3d94bb98dbe95d21b1478ef
0113 02:51:45.643612 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:45.643681 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1376273}
W20250113 02:51:45.644126 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet f3f6113ee3d94bb98dbe95d21b1478ef because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f3f6113ee3d94bb98dbe95d21b1478ef already in progress: creating tablet
I20250113 02:51:45.647866 1192 raft_consensus.cc:357] T 542df00229f645788f1b821c0ce00c46 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:45.648094 1192 raft_consensus.cc:383] T 542df00229f645788f1b821c0ce00c46 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:45.648169 1192 raft_consensus.cc:738] T 542df00229f645788f1b821c0ce00c46 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:45.648371 1192 consensus_queue.cc:260] T 542df00229f645788f1b821c0ce00c46 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:45.649408 1192 ts_tablet_manager.cc:1428] T 542df00229f645788f1b821c0ce00c46 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.022s user 0.002s sys 0.000s
I20250113 02:51:45.649832 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1238) took 5356 ms. Trace:
I20250113 02:51:45.649998 1192 tablet_bootstrap.cc:492] T 247af1a17df04b27898f86642336d51e P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:45.652259 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 33ccb3f7d2024f03b23d1e1c1046072a
I20250113 02:51:45.654613 1192 tablet_bootstrap.cc:654] T 247af1a17df04b27898f86642336d51e P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:45.655484 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1441) took 1335 ms. Trace:
I20250113 02:51:45.655573 1198 rpcz_store.cc:276] 0113 02:51:44.315965 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:44.316049 (+ 84us) service_pool.cc:225] Handling call
0113 02:51:45.651170 (+1335121us) ts_tablet_manager.cc:1145] Deleting tablet ff645eb0f8fa4744b7ba5cd03ce21729
0113 02:51:45.651182 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:45.651263 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1334938}
I20250113 02:51:45.655927 2368 raft_consensus.cc:491] T 569d666c128a45b7a526c13be5090ae3 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:45.656214 2368 raft_consensus.cc:513] T 569d666c128a45b7a526c13be5090ae3 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:45.656786 2368 leader_election.cc:290] T 569d666c128a45b7a526c13be5090ae3 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:45.657516 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "569d666c128a45b7a526c13be5090ae3" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
I20250113 02:51:45.657683 769 raft_consensus.cc:2463] T 569d666c128a45b7a526c13be5090ae3 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 0.
I20250113 02:51:45.657936 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "569d666c128a45b7a526c13be5090ae3" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:45.658488 998 leader_election.cc:343] T 569d666c128a45b7a526c13be5090ae3 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:45.649924 1195 rpcz_store.cc:276] 0113 02:51:40.293749 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:40.293799 (+ 50us) service_pool.cc:225] Handling call
0113 02:51:45.575049 (+5281250us) ts_tablet_manager.cc:1145] Deleting tablet 59fadc693f554854bc2874f8db7c64c8
0113 02:51:45.575063 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:45.586329 (+ 11266us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:45.625280 (+ 38951us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:45.649810 (+ 24530us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3,"tablet-delete.queue_time_us":5281075}
I20250113 02:51:45.658891 995 leader_election.cc:304] T 569d666c128a45b7a526c13be5090ae3 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters: df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:45.659173 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1239) took 5351 ms. Trace:
I20250113 02:51:45.659253 1195 rpcz_store.cc:276] 0113 02:51:40.307307 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:40.307353 (+ 46us) service_pool.cc:225] Handling call
0113 02:51:45.659052 (+5351699us) ts_tablet_manager.cc:1145] Deleting tablet 348c21d783ef4103a37409ef8634eb35
0113 02:51:45.659062 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:45.659156 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5351517}
I20250113 02:51:45.659389 2368 raft_consensus.cc:2798] T 569d666c128a45b7a526c13be5090ae3 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 02:51:45.659487 2368 raft_consensus.cc:491] T 569d666c128a45b7a526c13be5090ae3 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250113 02:51:45.659550 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet ff645eb0f8fa4744b7ba5cd03ce21729 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ff645eb0f8fa4744b7ba5cd03ce21729 already in progress: creating tablet
I20250113 02:51:45.659622 2368 raft_consensus.cc:3054] T 569d666c128a45b7a526c13be5090ae3 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:45.659654 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1240) took 5340 ms. Trace:
I20250113 02:51:45.659832 1195 rpcz_store.cc:276] 0113 02:51:40.318961 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:40.319016 (+ 55us) service_pool.cc:225] Handling call
0113 02:51:45.659543 (+5340527us) ts_tablet_manager.cc:1145] Deleting tablet 348c21d783ef4103a37409ef8634eb35
0113 02:51:45.659553 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:45.659640 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5340356}
I20250113 02:51:45.660210 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1242) took 5302 ms. Trace:
I20250113 02:51:45.660321 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet 59fadc693f554854bc2874f8db7c64c8 (table default.Table_17489ca6f295469eb97e9a3a5c7cb1aa [id=0e77d79275d44f37b11395f03d737c59]) successfully deleted
I20250113 02:51:45.660300 1195 rpcz_store.cc:276] 0113 02:51:40.357331 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:40.358535 (+ 1204us) service_pool.cc:225] Handling call
0113 02:51:45.660110 (+5301575us) ts_tablet_manager.cc:1145] Deleting tablet c368b5b0c3a940c4b4efb64c946375c3
0113 02:51:45.660119 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:45.660195 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5301337}
W20250113 02:51:45.660514 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 348c21d783ef4103a37409ef8634eb35 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 348c21d783ef4103a37409ef8634eb35 already in progress: creating tablet
I20250113 02:51:45.660601 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1243) took 5280 ms. Trace:
W20250113 02:51:45.660708 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 348c21d783ef4103a37409ef8634eb35 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 348c21d783ef4103a37409ef8634eb35 already in progress: creating tablet
I20250113 02:51:45.660668 1195 rpcz_store.cc:276] 0113 02:51:40.380641 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:40.381755 (+ 1114us) service_pool.cc:225] Handling call
0113 02:51:45.660500 (+5278745us) ts_tablet_manager.cc:1145] Deleting tablet e4998b86d32040ec959ff6fd94b1e9eb
0113 02:51:45.660509 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:45.660588 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5278563}
I20250113 02:51:45.660917 1195 tablet_replica.cc:331] T cf7b8bd6801743f49e9b9473747447c5 P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:45.661108 1195 raft_consensus.cc:2238] T cf7b8bd6801743f49e9b9473747447c5 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:45.661268 1195 raft_consensus.cc:2267] T cf7b8bd6801743f49e9b9473747447c5 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250113 02:51:45.661914 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet c368b5b0c3a940c4b4efb64c946375c3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c368b5b0c3a940c4b4efb64c946375c3 already in progress: creating tablet
W20250113 02:51:45.662171 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet e4998b86d32040ec959ff6fd94b1e9eb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e4998b86d32040ec959ff6fd94b1e9eb already in progress: creating tablet
I20250113 02:51:45.662303 740 tablet_service.cc:1514] Processing DeleteTablet for tablet 33ccb3f7d2024f03b23d1e1c1046072a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:45 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:45.662761 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet 33ccb3f7d2024f03b23d1e1c1046072a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:45 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:45.664206 1198 tablet_replica.cc:331] T b5c1882125b5491ea79a29a6dac6771e P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:45.664472 1198 raft_consensus.cc:2238] T b5c1882125b5491ea79a29a6dac6771e P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: Raft consensus shutting down.
I20250113 02:51:45.664891 1198 raft_consensus.cc:2267] T b5c1882125b5491ea79a29a6dac6771e P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:45.665167 2368 raft_consensus.cc:513] T 569d666c128a45b7a526c13be5090ae3 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:45.666880 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "569d666c128a45b7a526c13be5090ae3" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196"
I20250113 02:51:45.667073 771 raft_consensus.cc:3054] T 569d666c128a45b7a526c13be5090ae3 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:45.667035 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "569d666c128a45b7a526c13be5090ae3" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f"
W20250113 02:51:45.667680 998 leader_election.cc:343] T 569d666c128a45b7a526c13be5090ae3 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:45.668228 1198 ts_tablet_manager.cc:1905] T b5c1882125b5491ea79a29a6dac6771e P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:45.670219 771 raft_consensus.cc:2463] T 569d666c128a45b7a526c13be5090ae3 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 1.
I20250113 02:51:45.668399 2368 leader_election.cc:290] T 569d666c128a45b7a526c13be5090ae3 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Requested vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:45.670652 995 leader_election.cc:304] T 569d666c128a45b7a526c13be5090ae3 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters: df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:45.671036 2368 raft_consensus.cc:2798] T 569d666c128a45b7a526c13be5090ae3 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Leader election won for term 1
I20250113 02:51:45.671324 2368 raft_consensus.cc:695] T 569d666c128a45b7a526c13be5090ae3 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: Becoming Leader. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Running, Role: LEADER
I20250113 02:51:45.671628 2368 consensus_queue.cc:237] T 569d666c128a45b7a526c13be5090ae3 P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:45.676117 882 tablet_service.cc:1514] Processing DeleteTablet for tablet 33ccb3f7d2024f03b23d1e1c1046072a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:45 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:45.676333 2772 raft_consensus.cc:491] T 542df00229f645788f1b821c0ce00c46 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:45.676486 2772 raft_consensus.cc:513] T 542df00229f645788f1b821c0ce00c46 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:45.677155 2772 leader_election.cc:290] T 542df00229f645788f1b821c0ce00c46 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:45.677625 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "542df00229f645788f1b821c0ce00c46" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:45.678288 998 leader_election.cc:343] T 542df00229f645788f1b821c0ce00c46 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:45.679024 449 catalog_manager.cc:5618] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587) does not have the latest schema for tablet f2c6c8c17d6e499d84ad7c0c23889cc4 (table default.Table_92ea541283614df981d96c5b74362f1d [id=9e58548d1a184cc0b362f148c44005ed]). Expected version 2 got 0
I20250113 02:51:45.679406 449 catalog_manager.cc:5618] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587) does not have the latest schema for tablet b815003a27f64809baaa2d43d6293cef (table default.Table_c2e8fd7d2cc14fa39c0abfece783c6c9 [id=42e19138c08f422a92ba9621e9609957]). Expected version 1 got 0
I20250113 02:51:45.679805 449 catalog_manager.cc:5618] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587) does not have the latest schema for tablet 68c4a7e8c0fb4feb85498709b00d3258 (table default.Table_c90f3a5c35c842f6ad2cb74cf29dac13 [id=8d2f84cd670a4e8d9fa17b8313d3fab4]). Expected version 2 got 0
I20250113 02:51:45.680262 1198 ts_tablet_manager.cc:1918] T b5c1882125b5491ea79a29a6dac6771e P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250113 02:51:45.680394 1198 log.cc:1198] T b5c1882125b5491ea79a29a6dac6771e P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/b5c1882125b5491ea79a29a6dac6771e
I20250113 02:51:45.680366 2723 raft_consensus.cc:491] T c73c649a4ca44471af2cf6a77500b065 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:45.680511 449 catalog_manager.cc:5618] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587) does not have the latest schema for tablet f1237f7b04af4ebbb39b03aa3c78a499 (table default.Table_4c5e2e1008b54019a243ffe1ec72c020 [id=0817677719d045fd83bb85890f11df25]). Expected version 2 got 0
I20250113 02:51:45.680492 2723 raft_consensus.cc:513] T c73c649a4ca44471af2cf6a77500b065 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:45.680936 449 catalog_manager.cc:5618] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587) does not have the latest schema for tablet c13e9a51c1e74b44b352bade78390ba5 (table default.Table_0468611ec68e47109f5357935e50874d [id=caf098e2b57849a997bd7c8eaf1f9326]). Expected version 1 got 0
I20250113 02:51:45.681176 2723 leader_election.cc:290] T c73c649a4ca44471af2cf6a77500b065 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:45.681375 449 catalog_manager.cc:5618] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587) does not have the latest schema for tablet 019eb21c661547cfa0b619d119eb024e (table default.Table_022a7c61a6864393acd739f562893b7c [id=97bfbd46eecd4e57b7f9757c413f6795]). Expected version 1 got 0
I20250113 02:51:45.681701 449 catalog_manager.cc:5618] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587) does not have the latest schema for tablet e01e7f8db83d461bbfbb2e901419ce0b (table default.Table_e27f8b6f8232415388a5e53ab15494b7 [id=efc4ebba97f74c9ba12ab1dcc6577da0]). Expected version 1 got 0
I20250113 02:51:45.681815 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c73c649a4ca44471af2cf6a77500b065" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:45.682328 711 leader_election.cc:343] T c73c649a4ca44471af2cf6a77500b065 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:45.682859 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c73c649a4ca44471af2cf6a77500b065" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
W20250113 02:51:45.683454 709 leader_election.cc:343] T c73c649a4ca44471af2cf6a77500b065 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:45.683545 709 leader_election.cc:304] T c73c649a4ca44471af2cf6a77500b065 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:45.683895 2723 raft_consensus.cc:2743] T c73c649a4ca44471af2cf6a77500b065 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:45.684139 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "542df00229f645788f1b821c0ce00c46" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
I20250113 02:51:45.684409 771 raft_consensus.cc:2463] T 542df00229f645788f1b821c0ce00c46 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 0.
I20250113 02:51:45.684872 995 leader_election.cc:304] T 542df00229f645788f1b821c0ce00c46 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters: df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:45.685333 2772 raft_consensus.cc:2798] T 542df00229f645788f1b821c0ce00c46 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 02:51:45.685449 2772 raft_consensus.cc:491] T 542df00229f645788f1b821c0ce00c46 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 02:51:45.685544 2772 raft_consensus.cc:3054] T 542df00229f645788f1b821c0ce00c46 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:45.688338 2772 raft_consensus.cc:513] T 542df00229f645788f1b821c0ce00c46 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:45.688499 1195 ts_tablet_manager.cc:1905] T cf7b8bd6801743f49e9b9473747447c5 P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:45.689316 2772 leader_election.cc:290] T 542df00229f645788f1b821c0ce00c46 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Requested vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:45.689399 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "542df00229f645788f1b821c0ce00c46" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f"
W20250113 02:51:45.689922 998 leader_election.cc:343] T 542df00229f645788f1b821c0ce00c46 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:45.689900 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "542df00229f645788f1b821c0ce00c46" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196"
I20250113 02:51:45.690078 769 raft_consensus.cc:3054] T 542df00229f645788f1b821c0ce00c46 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:45.692142 2723 raft_consensus.cc:491] T 936b8ceaedd34a019a4fb89a80dd2708 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:45.692281 2723 raft_consensus.cc:513] T 936b8ceaedd34a019a4fb89a80dd2708 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:45.692570 769 raft_consensus.cc:2463] T 542df00229f645788f1b821c0ce00c46 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 1.
I20250113 02:51:45.692940 2782 raft_consensus.cc:491] T fe3d35d9aaf64188aaa3c1ac38d279d8 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:45.693069 2782 raft_consensus.cc:513] T fe3d35d9aaf64188aaa3c1ac38d279d8 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:45.693351 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "936b8ceaedd34a019a4fb89a80dd2708" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:45.693596 2782 leader_election.cc:290] T fe3d35d9aaf64188aaa3c1ac38d279d8 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:45.692942 2723 leader_election.cc:290] T 936b8ceaedd34a019a4fb89a80dd2708 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:45.694274 995 leader_election.cc:304] T 542df00229f645788f1b821c0ce00c46 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters: df69f48656b44bb890f76587ee69cc0f
W20250113 02:51:45.694293 711 leader_election.cc:343] T 936b8ceaedd34a019a4fb89a80dd2708 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:45.694729 2772 raft_consensus.cc:2798] T 542df00229f645788f1b821c0ce00c46 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Leader election won for term 1
I20250113 02:51:45.694886 2772 raft_consensus.cc:695] T 542df00229f645788f1b821c0ce00c46 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: Becoming Leader. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Running, Role: LEADER
I20250113 02:51:45.695194 2772 consensus_queue.cc:237] T 542df00229f645788f1b821c0ce00c46 P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:45.695465 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe3d35d9aaf64188aaa3c1ac38d279d8" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:45.695940 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe3d35d9aaf64188aaa3c1ac38d279d8" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:45.696398 711 leader_election.cc:343] T fe3d35d9aaf64188aaa3c1ac38d279d8 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:45.696465 1196 ts_tablet_manager.cc:1918] T 0644f5600c5b47e29f3fa73060ed96e3 P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:45.696573 1196 log.cc:1198] T 0644f5600c5b47e29f3fa73060ed96e3 P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/0644f5600c5b47e29f3fa73060ed96e3
W20250113 02:51:45.696826 709 leader_election.cc:343] T fe3d35d9aaf64188aaa3c1ac38d279d8 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:45.696887 1196 ts_tablet_manager.cc:1939] T 0644f5600c5b47e29f3fa73060ed96e3 P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:45.696915 709 leader_election.cc:304] T fe3d35d9aaf64188aaa3c1ac38d279d8 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:45.697194 1191 tablet_bootstrap.cc:492] T 36726b40ac5145bfb371871b3fff079a P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:45.697198 1196 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37366 (request call id 1351) took 1298 ms. Trace:
I20250113 02:51:45.697331 1191 ts_tablet_manager.cc:1397] T 36726b40ac5145bfb371871b3fff079a P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.262s user 0.050s sys 0.045s
I20250113 02:51:45.697273 1196 rpcz_store.cc:276] 0113 02:51:44.398870 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:44.400860 (+ 1990us) service_pool.cc:225] Handling call
0113 02:51:45.628848 (+1227988us) ts_tablet_manager.cc:1145] Deleting tablet 0644f5600c5b47e29f3fa73060ed96e3
0113 02:51:45.628858 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:45.663327 (+ 34469us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:45.696345 (+ 33018us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:45.697182 (+ 837us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1227745}
I20250113 02:51:45.697640 1196 tablet_replica.cc:331] T b5c1882125b5491ea79a29a6dac6771e P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:45.697669 2782 raft_consensus.cc:2743] T fe3d35d9aaf64188aaa3c1ac38d279d8 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:45.697813 1196 raft_consensus.cc:2238] T b5c1882125b5491ea79a29a6dac6771e P df69f48656b44bb890f76587ee69cc0f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:45.698163 1191 raft_consensus.cc:357] T 36726b40ac5145bfb371871b3fff079a P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:45.698333 1191 raft_consensus.cc:383] T 36726b40ac5145bfb371871b3fff079a P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:45.698388 1191 raft_consensus.cc:738] T 36726b40ac5145bfb371871b3fff079a P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:45.698431 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet 0644f5600c5b47e29f3fa73060ed96e3 (table default.Table_cd499f4a8487400690a9ab957c606b7f [id=be5f659ed070438bb6d0e61a3820eb58]) successfully deleted
I20250113 02:51:45.703536 1198 ts_tablet_manager.cc:1939] T b5c1882125b5491ea79a29a6dac6771e P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:45.704891 1195 ts_tablet_manager.cc:1918] T cf7b8bd6801743f49e9b9473747447c5 P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250113 02:51:45.705008 1195 log.cc:1198] T cf7b8bd6801743f49e9b9473747447c5 P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/cf7b8bd6801743f49e9b9473747447c5
I20250113 02:51:45.705359 1195 ts_tablet_manager.cc:1939] T cf7b8bd6801743f49e9b9473747447c5 P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:45.706104 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1262) took 5151 ms. Trace:
I20250113 02:51:45.706183 1195 rpcz_store.cc:276] 0113 02:51:40.554153 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:40.554632 (+ 479us) service_pool.cc:225] Handling call
0113 02:51:45.660871 (+5106239us) ts_tablet_manager.cc:1145] Deleting tablet cf7b8bd6801743f49e9b9473747447c5
0113 02:51:45.660883 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:45.691499 (+ 30616us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:45.704805 (+ 13306us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:45.706082 (+ 1277us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5106058}
I20250113 02:51:45.706660 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet cf7b8bd6801743f49e9b9473747447c5 (table default.Table_ed883bfc96094be9bb6e07bc510b6042 [id=3e7c139aef3847df965d0400bd2ffa45]) successfully deleted
I20250113 02:51:45.708112 739 tablet_service.cc:1514] Processing DeleteTablet for tablet 593c54696ea7473d8450744a7c4a394d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:45 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:45.708541 1191 consensus_queue.cc:260] T 36726b40ac5145bfb371871b3fff079a P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:45.708825 1196 raft_consensus.cc:2267] T b5c1882125b5491ea79a29a6dac6771e P df69f48656b44bb890f76587ee69cc0f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:45.708778 1060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "936b8ceaedd34a019a4fb89a80dd2708" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
W20250113 02:51:45.709527 709 leader_election.cc:343] T 936b8ceaedd34a019a4fb89a80dd2708 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:45.709638 709 leader_election.cc:304] T 936b8ceaedd34a019a4fb89a80dd2708 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:45.709889 1196 ts_tablet_manager.cc:1905] T b5c1882125b5491ea79a29a6dac6771e P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:45.710413 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1442) took 1154 ms. Trace:
I20250113 02:51:45.710541 1198 rpcz_store.cc:276] 0113 02:51:44.556204 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:44.556259 (+ 55us) service_pool.cc:225] Handling call
0113 02:51:45.664088 (+1107829us) ts_tablet_manager.cc:1145] Deleting tablet b5c1882125b5491ea79a29a6dac6771e
0113 02:51:45.664111 (+ 23us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:45.674830 (+ 10719us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:45.680143 (+ 5313us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:45.710377 (+ 30234us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1136384,"tablet-delete.queue_time_us":1102019}
I20250113 02:51:45.710716 879 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:37356 (request call id 279) took 1078 ms. Trace:
I20250113 02:51:45.710809 879 rpcz_store.cc:276] 0113 02:51:44.632402 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:44.632460 (+ 58us) service_pool.cc:225] Handling call
0113 02:51:44.704894 (+ 72434us) spinlock_profiling.cc:243] Waited 47.7 ms on lock 0x616000273454. stack: 00007f0b1ea7cfd6 00007f0b1ea7cd90 00007f0b1da5c48d 000000000096d544 000000000096d50d 000000000096ceda 00007f0b2a2effd0 00007f0b2e02cb3f 00007f0b289878e6 00007f0b28987615 00007f0b257e430e 00007f0b257e31ee 00007f0b257e9425 00007f0b257e9d4f 00007f0b257e9b78 0000000000a71ecc
0113 02:51:45.710691 (+1005797us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":109725184}
I20250113 02:51:45.710994 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet b5c1882125b5491ea79a29a6dac6771e (table default.Table_99031e0341fd4ad8a3bb04a6e222a1d0 [id=3237c1be7e1c48918ff095162d191dc1]) successfully deleted
I20250113 02:51:45.713168 2723 raft_consensus.cc:2743] T 936b8ceaedd34a019a4fb89a80dd2708 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250113 02:51:45.714282 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 08180ab0274a40fd955dda5b08197a08 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 08180ab0274a40fd955dda5b08197a08 already in progress: creating tablet
I20250113 02:51:45.714394 1191 ts_tablet_manager.cc:1428] T 36726b40ac5145bfb371871b3fff079a P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.017s user 0.001s sys 0.001s
I20250113 02:51:45.715045 1198 tablet_replica.cc:331] T 79aaab73797c474bae06831fa308b066 P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
W20250113 02:51:45.715250 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 3a89e991ebee4b4f8c639ea95e1d1ea3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3a89e991ebee4b4f8c639ea95e1d1ea3 already in progress: creating tablet
I20250113 02:51:45.715282 1198 raft_consensus.cc:2238] T 79aaab73797c474bae06831fa308b066 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: Raft consensus shutting down.
I20250113 02:51:45.715613 1035 tablet_service.cc:1467] Processing CreateTablet for tablet 2c7ce64e3f8f45018d6b4f552a1392e1 (DEFAULT_TABLE table=default.Table_906514aed3bf4e5e87a1748d6fffc261 [id=96cb0686ce4c4f1da81c955d99cd9438]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:45.716143 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c7ce64e3f8f45018d6b4f552a1392e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:45.715621 1198 raft_consensus.cc:2267] T 79aaab73797c474bae06831fa308b066 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Raft consensus is shut down!
W20250113 02:51:45.719918 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 59a447c6d0264ec38390fe688bc9555a consensus state has no leader
I20250113 02:51:45.722821 1198 ts_tablet_manager.cc:1905] T 79aaab73797c474bae06831fa308b066 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:45.724311 453 catalog_manager.cc:5618] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467) does not have the latest schema for tablet b815003a27f64809baaa2d43d6293cef (table default.Table_c2e8fd7d2cc14fa39c0abfece783c6c9 [id=42e19138c08f422a92ba9621e9609957]). Expected version 1 got 0
I20250113 02:51:45.726012 882 tablet_service.cc:1514] Processing DeleteTablet for tablet 64aa8390b1b1475c8743b6491d40674c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:29 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:45.726148 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_f0d0db84fe564a8ab3368977ba13d3f8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:45.726720 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f0d0db84fe564a8ab3368977ba13d3f8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:45.730068 1198 ts_tablet_manager.cc:1918] T 79aaab73797c474bae06831fa308b066 P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250113 02:51:45.730202 1192 tablet_bootstrap.cc:492] T 247af1a17df04b27898f86642336d51e P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:45.730227 1198 log.cc:1198] T 79aaab73797c474bae06831fa308b066 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/79aaab73797c474bae06831fa308b066
I20250113 02:51:45.730355 1192 ts_tablet_manager.cc:1397] T 247af1a17df04b27898f86642336d51e P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.081s user 0.001s sys 0.009s
I20250113 02:51:45.730515 882 tablet_service.cc:1467] Processing CreateTablet for tablet 76dc8fdb782744589332f31ce23f7f9b (DEFAULT_TABLE table=default.Table_55663eac113f43f4ad701ef384b143cb [id=4faa6410be2340ff9fd62a9dc6bc22be]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:45.730656 1198 ts_tablet_manager.cc:1939] T 79aaab73797c474bae06831fa308b066 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:45.730907 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76dc8fdb782744589332f31ce23f7f9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:45.731223 1192 raft_consensus.cc:357] T 247af1a17df04b27898f86642336d51e P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:45.731416 1192 raft_consensus.cc:383] T 247af1a17df04b27898f86642336d51e P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:45.731515 1192 raft_consensus.cc:738] T 247af1a17df04b27898f86642336d51e P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:45.731590 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet 79aaab73797c474bae06831fa308b066 (table default.Table_f7097c6cf1fd4839872c42a30fa5f64f [id=89b7f6f853ca4e3c87cbccd41698009a]) successfully deleted
I20250113 02:51:45.731729 1192 consensus_queue.cc:260] T 247af1a17df04b27898f86642336d51e P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
W20250113 02:51:45.732378 440 catalog_manager.cc:4839] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet aa1ebdb92a5f4c92a5195b614689c100 because the tablet was not found. No further retry: Not found: Tablet not found: aa1ebdb92a5f4c92a5195b614689c100
I20250113 02:51:45.732419 1192 ts_tablet_manager.cc:1428] T 247af1a17df04b27898f86642336d51e P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250113 02:51:45.732688 1192 tablet_bootstrap.cc:492] T 59fadc693f554854bc2874f8db7c64c8 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:45.734282 1198 tablet_replica.cc:331] T 8cc78bf7d6324b7ba10a1b1045923f2b P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:45.734475 1198 raft_consensus.cc:2238] T 8cc78bf7d6324b7ba10a1b1045923f2b P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: Raft consensus shutting down.
I20250113 02:51:45.734787 1198 raft_consensus.cc:2267] T 8cc78bf7d6324b7ba10a1b1045923f2b P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:45.735325 2588 raft_consensus.cc:491] T d59134b029a143038837366f51f00d28 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:45.735457 2588 raft_consensus.cc:513] T d59134b029a143038837366f51f00d28 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:45.736083 2588 leader_election.cc:290] T d59134b029a143038837366f51f00d28 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:45.736603 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d59134b029a143038837366f51f00d28" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:45.736951 1192 tablet_bootstrap.cc:654] T 59fadc693f554854bc2874f8db7c64c8 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
W20250113 02:51:45.737154 711 leader_election.cc:343] T d59134b029a143038837366f51f00d28 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:45.737500 1195 tablet_replica.cc:331] T eceed0f52c9f46bbaa78ca773a7afde5 P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:45.737480 1060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d59134b029a143038837366f51f00d28" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:45.737731 1195 raft_consensus.cc:2238] T eceed0f52c9f46bbaa78ca773a7afde5 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Raft consensus shutting down.
W20250113 02:51:45.737922 709 leader_election.cc:343] T d59134b029a143038837366f51f00d28 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:45.737987 1195 raft_consensus.cc:2267] T eceed0f52c9f46bbaa78ca773a7afde5 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:45.738035 709 leader_election.cc:304] T d59134b029a143038837366f51f00d28 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:45.738271 2588 raft_consensus.cc:2743] T d59134b029a143038837366f51f00d28 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:45.738550 1195 ts_tablet_manager.cc:1905] T eceed0f52c9f46bbaa78ca773a7afde5 P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:45.738894 1198 ts_tablet_manager.cc:1905] T 8cc78bf7d6324b7ba10a1b1045923f2b P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:45.743731 1191 tablet_bootstrap.cc:492] T b9ec434644c14acd95c72efbef8ded59 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:45.745229 1198 ts_tablet_manager.cc:1918] T 8cc78bf7d6324b7ba10a1b1045923f2b P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250113 02:51:45.745421 1198 log.cc:1198] T 8cc78bf7d6324b7ba10a1b1045923f2b P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/8cc78bf7d6324b7ba10a1b1045923f2b
I20250113 02:51:45.745834 1198 ts_tablet_manager.cc:1939] T 8cc78bf7d6324b7ba10a1b1045923f2b P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:45.746692 739 tablet_service.cc:1467] Processing CreateTablet for tablet 2c7ce64e3f8f45018d6b4f552a1392e1 (DEFAULT_TABLE table=default.Table_906514aed3bf4e5e87a1748d6fffc261 [id=96cb0686ce4c4f1da81c955d99cd9438]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:45.746805 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet 8cc78bf7d6324b7ba10a1b1045923f2b (table default.Table_a55eab4604df4ca5b767708b6533e6d3 [id=925246ca69314b9490c9f968f3597229]) successfully deleted
I20250113 02:51:45.747071 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c7ce64e3f8f45018d6b4f552a1392e1. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:45.747509 440 catalog_manager.cc:4839] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet fd0b8592997d4a61b8ca854ecbb67c55 because the tablet was not found. No further retry: Not found: Tablet not found: fd0b8592997d4a61b8ca854ecbb67c55
W20250113 02:51:45.748067 440 catalog_manager.cc:4839] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 67baaab1e36e415381579d7385d4ab5a because the tablet was not found. No further retry: Not found: Tablet not found: 67baaab1e36e415381579d7385d4ab5a
I20250113 02:51:45.749008 1035 tablet_service.cc:1467] Processing CreateTablet for tablet 76dc8fdb782744589332f31ce23f7f9b (DEFAULT_TABLE table=default.Table_55663eac113f43f4ad701ef384b143cb [id=4faa6410be2340ff9fd62a9dc6bc22be]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:45.749372 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76dc8fdb782744589332f31ce23f7f9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:45.749890 882 tablet_service.cc:1467] Processing CreateTablet for tablet 6d67e6ed65fc440b88db6f07a1cba32c (DEFAULT_TABLE table=default.Table_5be43673265c48b1905509b7852f0c11 [id=6ddded7f3bf9460fa06d66e5a25c6faa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:45.750375 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d67e6ed65fc440b88db6f07a1cba32c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:45.752283 740 tablet_service.cc:1467] Processing CreateTablet for tablet 6d67e6ed65fc440b88db6f07a1cba32c (DEFAULT_TABLE table=default.Table_5be43673265c48b1905509b7852f0c11 [id=6ddded7f3bf9460fa06d66e5a25c6faa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:45.752892 885 tablet_service.cc:1467] Processing CreateTablet for tablet a6603e63f3d643b2a2525adf9c7180c8 (DEFAULT_TABLE table=default.Table_f02ddc5b50b94468a206c5bb80525f28 [id=39085e6258b148c8b9ac8cd1a1c87309]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:45.753240 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6603e63f3d643b2a2525adf9c7180c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:45.753427 2588 raft_consensus.cc:491] T d47f544f723542bea1d396f586d8d658 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:45.753551 2588 raft_consensus.cc:513] T d47f544f723542bea1d396f586d8d658 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:45.754145 2588 leader_election.cc:290] T d47f544f723542bea1d396f586d8d658 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:45.754643 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d47f544f723542bea1d396f586d8d658" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:45.755044 1060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d47f544f723542bea1d396f586d8d658" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
W20250113 02:51:45.755573 709 leader_election.cc:343] T d47f544f723542bea1d396f586d8d658 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
W20250113 02:51:45.755908 711 leader_election.cc:343] T d47f544f723542bea1d396f586d8d658 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:45.756022 711 leader_election.cc:304] T d47f544f723542bea1d396f586d8d658 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:45.757669 1196 ts_tablet_manager.cc:1918] T b5c1882125b5491ea79a29a6dac6771e P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250113 02:51:45.757840 1196 log.cc:1198] T b5c1882125b5491ea79a29a6dac6771e P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/b5c1882125b5491ea79a29a6dac6771e
I20250113 02:51:45.757884 1032 tablet_service.cc:1467] Processing CreateTablet for tablet 6d67e6ed65fc440b88db6f07a1cba32c (DEFAULT_TABLE table=default.Table_5be43673265c48b1905509b7852f0c11 [id=6ddded7f3bf9460fa06d66e5a25c6faa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:45.758335 1196 ts_tablet_manager.cc:1939] T b5c1882125b5491ea79a29a6dac6771e P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:45.758708 1196 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37366 (request call id 1352) took 1204 ms. Trace:
I20250113 02:51:45.758833 1196 rpcz_store.cc:276] 0113 02:51:44.554032 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:44.554096 (+ 64us) service_pool.cc:225] Handling call
0113 02:51:45.697595 (+1143499us) ts_tablet_manager.cc:1145] Deleting tablet b5c1882125b5491ea79a29a6dac6771e
0113 02:51:45.697605 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:45.727413 (+ 29808us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:45.757583 (+ 30170us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:45.758692 (+ 1109us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1143259}
I20250113 02:51:45.759384 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet b5c1882125b5491ea79a29a6dac6771e (table default.Table_99031e0341fd4ad8a3bb04a6e222a1d0 [id=3237c1be7e1c48918ff095162d191dc1]) successfully deleted
I20250113 02:51:45.759478 1196 tablet_replica.cc:331] T 788ab82626424b6e833e31761a15b9b2 P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:45.759676 1196 raft_consensus.cc:2238] T 788ab82626424b6e833e31761a15b9b2 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:45.759764 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d67e6ed65fc440b88db6f07a1cba32c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:45.759835 1196 raft_consensus.cc:2267] T 788ab82626424b6e833e31761a15b9b2 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:45.760527 1191 tablet_bootstrap.cc:654] T b9ec434644c14acd95c72efbef8ded59 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:45.764230 2476 raft_consensus.cc:491] T b33025a462e74f2aa26cc3779f36615b P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:45.764365 2476 raft_consensus.cc:513] T b33025a462e74f2aa26cc3779f36615b P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:45.764937 2476 leader_election.cc:290] T b33025a462e74f2aa26cc3779f36615b P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:45.765972 2588 raft_consensus.cc:2743] T d47f544f723542bea1d396f586d8d658 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:45.766471 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d67e6ed65fc440b88db6f07a1cba32c. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:45.771260 851 leader_election.cc:343] T b33025a462e74f2aa26cc3779f36615b P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Not found: Tablet not found: b33025a462e74f2aa26cc3779f36615b
I20250113 02:51:45.775866 2368 consensus_queue.cc:1035] T 88a776aa820c44b3a3854ca5022bd353 P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 02:51:45.776337 736 tablet_service.cc:1467] Processing CreateTablet for tablet 76dc8fdb782744589332f31ce23f7f9b (DEFAULT_TABLE table=default.Table_55663eac113f43f4ad701ef384b143cb [id=4faa6410be2340ff9fd62a9dc6bc22be]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:45.776468 1194 tablet_bootstrap.cc:654] T 1d2fdc75d00f4e749d322ac6cd5fc410 P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:45.776952 736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76dc8fdb782744589332f31ce23f7f9b. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:45.777012 852 leader_election.cc:343] T b33025a462e74f2aa26cc3779f36615b P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Not found: Tablet not found: b33025a462e74f2aa26cc3779f36615b
I20250113 02:51:45.777143 852 leader_election.cc:304] T b33025a462e74f2aa26cc3779f36615b P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df69f48656b44bb890f76587ee69cc0f; no voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca
I20250113 02:51:45.777593 2476 raft_consensus.cc:2743] T b33025a462e74f2aa26cc3779f36615b P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:45.780867 1196 ts_tablet_manager.cc:1905] T 788ab82626424b6e833e31761a15b9b2 P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:45.784571 1024 tablet_service.cc:1467] Processing CreateTablet for tablet b90bf22dc4cd4c6089c6109c3b088417 (DEFAULT_TABLE table=default.Table_9eb9691792fb4e5da6a1299fbc96e962 [id=49098b820b3c4d5faa6b94c94efc3796]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:45.787410 883 tablet_service.cc:1467] Processing CreateTablet for tablet 234b1f0e98bc4b1c99e3bc5ae361b285 (DEFAULT_TABLE table=default.Table_2a70bebef78f48f09c6c3982df867cdc [id=39326dea264c4059be04ed0e7d9b5138]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:45.787698 882 tablet_service.cc:1467] Processing CreateTablet for tablet b90bf22dc4cd4c6089c6109c3b088417 (DEFAULT_TABLE table=default.Table_9eb9691792fb4e5da6a1299fbc96e962 [id=49098b820b3c4d5faa6b94c94efc3796]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:45.787777 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 234b1f0e98bc4b1c99e3bc5ae361b285. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:45.791175 891 tablet_service.cc:1467] Processing CreateTablet for tablet 6a998cc32ff34ba190f4951851f46403 (DEFAULT_TABLE table=default.Table_1ec47bba53df4a508134dbdc0e95b15b [id=c17fd9f6e31544f29458a71de7402b41]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:45.791604 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a998cc32ff34ba190f4951851f46403. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:45.792047 1039 tablet_service.cc:1467] Processing CreateTablet for tablet 234b1f0e98bc4b1c99e3bc5ae361b285 (DEFAULT_TABLE table=default.Table_2a70bebef78f48f09c6c3982df867cdc [id=39326dea264c4059be04ed0e7d9b5138]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:45.792658 1034 tablet_service.cc:1467] Processing CreateTablet for tablet 6a998cc32ff34ba190f4951851f46403 (DEFAULT_TABLE table=default.Table_1ec47bba53df4a508134dbdc0e95b15b [id=c17fd9f6e31544f29458a71de7402b41]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:45.794317 896 tablet_service.cc:1467] Processing CreateTablet for tablet 0a94f75386b248dc9b8e03b384834779 (DEFAULT_TABLE table=default.Table_ca9b419c256442dfb375e4043a0e0fa9 [id=7b79338aab3744a3a7627ac07b9bef43]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:45.794646 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0a94f75386b248dc9b8e03b384834779. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:45.795248 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b90bf22dc4cd4c6089c6109c3b088417. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:45.796998 2588 raft_consensus.cc:491] T 8f2fb58fc24d4692a3fe9a652e076aed P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:45.802336 1195 ts_tablet_manager.cc:1918] T eceed0f52c9f46bbaa78ca773a7afde5 P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250113 02:51:45.802470 1195 log.cc:1198] T eceed0f52c9f46bbaa78ca773a7afde5 P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/eceed0f52c9f46bbaa78ca773a7afde5
I20250113 02:51:45.802860 1195 ts_tablet_manager.cc:1939] T eceed0f52c9f46bbaa78ca773a7afde5 P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:45.803133 2772 consensus_queue.cc:1035] T 019eb21c661547cfa0b619d119eb024e P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 02:51:45.803699 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1275) took 5178 ms. Trace:
I20250113 02:51:45.806370 1198 tablet_replica.cc:331] T cf7b8bd6801743f49e9b9473747447c5 P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:45.806802 1198 raft_consensus.cc:2238] T cf7b8bd6801743f49e9b9473747447c5 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: Raft consensus shutting down.
I20250113 02:51:45.807080 1198 raft_consensus.cc:2267] T cf7b8bd6801743f49e9b9473747447c5 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:45.811331 1198 ts_tablet_manager.cc:1905] T cf7b8bd6801743f49e9b9473747447c5 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:45.803795 1195 rpcz_store.cc:276] 0113 02:51:40.625435 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:40.769039 (+143604us) service_pool.cc:225] Handling call
0113 02:51:45.727642 (+4958603us) ts_tablet_manager.cc:1145] Deleting tablet eceed0f52c9f46bbaa78ca773a7afde5
0113 02:51:45.737444 (+ 9802us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:45.741565 (+ 4121us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:45.802217 (+ 60652us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:45.803677 (+ 1460us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4924887}
I20250113 02:51:45.814410 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet eceed0f52c9f46bbaa78ca773a7afde5 (table default.Table_6a0195b054a043269e133f6ed8dd5979 [id=4b2bd999368c4b628c70cb3ee8c2b1f3]) successfully deleted
I20250113 02:51:45.814711 740 tablet_service.cc:1467] Processing CreateTablet for tablet 6a998cc32ff34ba190f4951851f46403 (DEFAULT_TABLE table=default.Table_1ec47bba53df4a508134dbdc0e95b15b [id=c17fd9f6e31544f29458a71de7402b41]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:45.814813 751 tablet_service.cc:1467] Processing CreateTablet for tablet b90bf22dc4cd4c6089c6109c3b088417 (DEFAULT_TABLE table=default.Table_9eb9691792fb4e5da6a1299fbc96e962 [id=49098b820b3c4d5faa6b94c94efc3796]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:45.815260 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b90bf22dc4cd4c6089c6109c3b088417. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:45.815537 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a998cc32ff34ba190f4951851f46403. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:45.817011 1196 ts_tablet_manager.cc:1918] T 788ab82626424b6e833e31761a15b9b2 P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:45.817695 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b90bf22dc4cd4c6089c6109c3b088417. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:45.817082 1194 tablet_bootstrap.cc:492] T 1d2fdc75d00f4e749d322ac6cd5fc410 P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:45.818010 1194 ts_tablet_manager.cc:1397] T 1d2fdc75d00f4e749d322ac6cd5fc410 P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.205s user 0.011s sys 0.000s
I20250113 02:51:45.818681 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 234b1f0e98bc4b1c99e3bc5ae361b285. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:45.819257 1194 raft_consensus.cc:357] T 1d2fdc75d00f4e749d322ac6cd5fc410 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:45.819445 1194 raft_consensus.cc:383] T 1d2fdc75d00f4e749d322ac6cd5fc410 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:45.819496 1194 raft_consensus.cc:738] T 1d2fdc75d00f4e749d322ac6cd5fc410 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:45.819689 1194 consensus_queue.cc:260] T 1d2fdc75d00f4e749d322ac6cd5fc410 P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:45.820214 1194 ts_tablet_manager.cc:1428] T 1d2fdc75d00f4e749d322ac6cd5fc410 P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250113 02:51:45.822280 1194 tablet_bootstrap.cc:492] T 3d85539cf0074a03b167d3279e940079 P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:45.823846 1194 tablet_bootstrap.cc:654] T 3d85539cf0074a03b167d3279e940079 P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:45.833177 1194 tablet_bootstrap.cc:492] T 3d85539cf0074a03b167d3279e940079 P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:45.833793 1194 ts_tablet_manager.cc:1397] T 3d85539cf0074a03b167d3279e940079 P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.001s
I20250113 02:51:45.834722 1194 raft_consensus.cc:357] T 3d85539cf0074a03b167d3279e940079 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:45.837077 1192 tablet_bootstrap.cc:492] T 59fadc693f554854bc2874f8db7c64c8 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:45.837234 1192 ts_tablet_manager.cc:1397] T 59fadc693f554854bc2874f8db7c64c8 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.105s user 0.011s sys 0.007s
I20250113 02:51:45.837242 1194 raft_consensus.cc:383] T 3d85539cf0074a03b167d3279e940079 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:45.837749 1194 raft_consensus.cc:738] T 3d85539cf0074a03b167d3279e940079 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:45.837981 1192 raft_consensus.cc:357] T 59fadc693f554854bc2874f8db7c64c8 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:45.838172 1192 raft_consensus.cc:383] T 59fadc693f554854bc2874f8db7c64c8 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:45.838233 1192 raft_consensus.cc:738] T 59fadc693f554854bc2874f8db7c64c8 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:45.838423 1194 consensus_queue.cc:260] T 3d85539cf0074a03b167d3279e940079 P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:45.838466 1192 consensus_queue.cc:260] T 59fadc693f554854bc2874f8db7c64c8 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:45.839747 1194 ts_tablet_manager.cc:1428] T 3d85539cf0074a03b167d3279e940079 P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20250113 02:51:45.840617 1194 tablet_bootstrap.cc:492] T 153b8f90b5a847b88944f683d306f579 P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:45.841511 1192 ts_tablet_manager.cc:1428] T 59fadc693f554854bc2874f8db7c64c8 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250113 02:51:45.841667 1039 tablet_service.cc:1467] Processing CreateTablet for tablet 0a94f75386b248dc9b8e03b384834779 (DEFAULT_TABLE table=default.Table_ca9b419c256442dfb375e4043a0e0fa9 [id=7b79338aab3744a3a7627ac07b9bef43]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:45.842131 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0a94f75386b248dc9b8e03b384834779. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:45.841782 1192 tablet_bootstrap.cc:492] T 2470b010b571444eba3f132323038f33 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:45.843379 1194 tablet_bootstrap.cc:654] T 153b8f90b5a847b88944f683d306f579 P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:45.844453 1192 tablet_bootstrap.cc:654] T 2470b010b571444eba3f132323038f33 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:45.846449 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a998cc32ff34ba190f4951851f46403. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:45.852469 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e57ac440265e4646bfc5d5d9ca4c35d2 consensus state has no leader
I20250113 02:51:45.852973 447 catalog_manager.cc:6071] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: default.Table_e5b4d614e4b743d7982b025ca6d297e5 [id=5824acd778ed47d09f087bbf8e7822d6] alter complete (version 1)
W20250113 02:51:45.855780 454 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7ff9b30d502a4aa1a553035bbc992ad2 consensus state has no leader
I20250113 02:51:45.855996 454 master_service.cc:946] ReplaceTablet: received request to replace tablet 93c0ce9f457e431193a2ceb0ad4d4af6 from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:45.858181 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet b5c1882125b5491ea79a29a6dac6771e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:44 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:45.858472 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet 79aaab73797c474bae06831fa308b066 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:30 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:45.858361 743 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c759831773849f2bc52e484ffa0cf49 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:30 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:45.859233 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet 8cc78bf7d6324b7ba10a1b1045923f2b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:32 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:45.859908 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet 58053845187f466599750f2a62f4c3e1 with delete_type TABLET_DATA_DELETED (replaced by tablet a04cec5ed9f34a16a80e112e5c9671e6) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:45.860642 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet cfa963a6df094ce0b9c54408f75ca6b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:37 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:45.864969 1194 tablet_bootstrap.cc:492] T 153b8f90b5a847b88944f683d306f579 P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:45.865140 1194 ts_tablet_manager.cc:1397] T 153b8f90b5a847b88944f683d306f579 P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.025s user 0.002s sys 0.002s
I20250113 02:51:45.865155 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_1c44e4571363423c9348ca3bb8c0e19a" } modify_external_catalogs: true
I20250113 02:51:45.865329 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_1c44e4571363423c9348ca3bb8c0e19a" } modify_external_catalogs: true
I20250113 02:51:45.865900 1194 raft_consensus.cc:357] T 153b8f90b5a847b88944f683d306f579 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:45.866104 1194 raft_consensus.cc:383] T 153b8f90b5a847b88944f683d306f579 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:45.866173 1194 raft_consensus.cc:738] T 153b8f90b5a847b88944f683d306f579 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:45.869156 1194 consensus_queue.cc:260] T 153b8f90b5a847b88944f683d306f579 P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:45.869447 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_a272ebee897d42c99c9c5012355291de"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:45.869728 1192 tablet_bootstrap.cc:492] T 2470b010b571444eba3f132323038f33 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:45.869853 1192 ts_tablet_manager.cc:1397] T 2470b010b571444eba3f132323038f33 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.028s user 0.007s sys 0.007s
W20250113 02:51:45.870025 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a272ebee897d42c99c9c5012355291de in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:45.870563 454 catalog_manager.cc:5618] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587) does not have the latest schema for tablet afcb5b1977c447d3a51de9cb72d7196a (table default.Table_1eb2867afbff468cbbca66486afd7075 [id=d0ffa26e65894ab89facfccd75985d51]). Expected version 1 got 0
I20250113 02:51:45.872365 1198 ts_tablet_manager.cc:1918] T cf7b8bd6801743f49e9b9473747447c5 P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250113 02:51:45.872473 1198 log.cc:1198] T cf7b8bd6801743f49e9b9473747447c5 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/cf7b8bd6801743f49e9b9473747447c5
I20250113 02:51:45.872825 1198 ts_tablet_manager.cc:1939] T cf7b8bd6801743f49e9b9473747447c5 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:45.871009 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_d20226843534409a8cabaa764bcd9ce5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:45.873355 1192 raft_consensus.cc:357] T 2470b010b571444eba3f132323038f33 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:45.873538 1192 raft_consensus.cc:383] T 2470b010b571444eba3f132323038f33 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250113 02:51:45.873546 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d20226843534409a8cabaa764bcd9ce5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:45.873590 1192 raft_consensus.cc:738] T 2470b010b571444eba3f132323038f33 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:45.873823 1192 consensus_queue.cc:260] T 2470b010b571444eba3f132323038f33 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:45.874315 1192 ts_tablet_manager.cc:1428] T 2470b010b571444eba3f132323038f33 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250113 02:51:45.871440 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_4b6d8da7a53240dab5cd6d788c17e913"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:45.874927 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4b6d8da7a53240dab5cd6d788c17e913 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:45.875089 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1447) took 1002 ms. Trace:
I20250113 02:51:45.875173 1198 rpcz_store.cc:276] 0113 02:51:44.872647 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:44.873685 (+ 1038us) service_pool.cc:225] Handling call
0113 02:51:45.748327 (+874642us) ts_tablet_manager.cc:1145] Deleting tablet cf7b8bd6801743f49e9b9473747447c5
0113 02:51:45.748336 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:45.866552 (+118216us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:45.872251 (+ 5699us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:45.875070 (+ 2819us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":874509}
I20250113 02:51:45.875492 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet cf7b8bd6801743f49e9b9473747447c5 (table default.Table_ed883bfc96094be9bb6e07bc510b6042 [id=3e7c139aef3847df965d0400bd2ffa45]) successfully deleted
I20250113 02:51:45.876530 739 tablet_service.cc:1467] Processing CreateTablet for tablet 234b1f0e98bc4b1c99e3bc5ae361b285 (DEFAULT_TABLE table=default.Table_2a70bebef78f48f09c6c3982df867cdc [id=39326dea264c4059be04ed0e7d9b5138]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:45.876960 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 234b1f0e98bc4b1c99e3bc5ae361b285. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:45.878051 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1291) took 5195 ms. Trace:
I20250113 02:51:45.878142 1195 rpcz_store.cc:276] 0113 02:51:40.682146 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:41.112398 (+430252us) service_pool.cc:225] Handling call
0113 02:51:45.814756 (+4702358us) ts_tablet_manager.cc:1145] Deleting tablet 42a8e01bb71f49b09c83a8f5c905f38d
0113 02:51:45.814767 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:45.878017 (+ 63250us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4702121}
I20250113 02:51:45.878538 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1292) took 5170 ms. Trace:
I20250113 02:51:45.878608 1195 rpcz_store.cc:276] 0113 02:51:40.708313 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:41.112655 (+404342us) service_pool.cc:225] Handling call
0113 02:51:45.878416 (+4765761us) ts_tablet_manager.cc:1145] Deleting tablet 94307d22b7f842ca89844768e4a2a2ab
0113 02:51:45.878427 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:45.878521 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4765568}
I20250113 02:51:45.878609 2603 fs_manager.cc:714] Time spent opening directory manager: real 0.213s user 0.002s sys 0.004s
I20250113 02:51:45.878902 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1293) took 5170 ms. Trace:
I20250113 02:51:45.878969 1195 rpcz_store.cc:276] 0113 02:51:40.708420 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:41.112858 (+404438us) service_pool.cc:225] Handling call
0113 02:51:45.878807 (+4765949us) ts_tablet_manager.cc:1145] Deleting tablet a0c14501ad0449aca28909bc14428dad
0113 02:51:45.878816 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:45.878889 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4765766}
I20250113 02:51:45.879091 1194 ts_tablet_manager.cc:1428] T 153b8f90b5a847b88944f683d306f579 P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.014s user 0.001s sys 0.001s
I20250113 02:51:45.879241 1195 tablet_replica.cc:331] T 6001369444894b48b51a48fb60ddc52a P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:45.879498 1195 raft_consensus.cc:2238] T 6001369444894b48b51a48fb60ddc52a P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:45.879683 1195 raft_consensus.cc:2267] T 6001369444894b48b51a48fb60ddc52a P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:45.880247 1195 ts_tablet_manager.cc:1905] T 6001369444894b48b51a48fb60ddc52a P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:45.881085 2588 raft_consensus.cc:513] T 8f2fb58fc24d4692a3fe9a652e076aed P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:45.872411 446 catalog_manager.cc:5618] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467) does not have the latest schema for tablet b815003a27f64809baaa2d43d6293cef (table default.Table_c2e8fd7d2cc14fa39c0abfece783c6c9 [id=42e19138c08f422a92ba9621e9609957]). Expected version 1 got 0
W20250113 02:51:45.881672 998 consensus_peers.cc:487] T 0839a35fcab04f3c9d03e256553e01c8 P 4216a7f853d642b7bc3e5cff458f3aca -> Peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Couldn't send request to peer df69f48656b44bb890f76587ee69cc0f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250113 02:51:45.881923 998 consensus_peers.cc:487] T 019eb21c661547cfa0b619d119eb024e P 4216a7f853d642b7bc3e5cff458f3aca -> Peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Couldn't send request to peer df69f48656b44bb890f76587ee69cc0f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250113 02:51:45.882952 882 tablet_service.cc:1514] Processing DeleteTablet for tablet 438604e8859f422f9caccb61c5b5b125 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:24 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:45.882975 446 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_292625e9d891410287fe0647a7c6b109" } modify_external_catalogs: true
I20250113 02:51:45.883128 1192 tablet_bootstrap.cc:492] T f4379f7fc6b54e6785f27975d9030cce P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:45.883145 446 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_292625e9d891410287fe0647a7c6b109" } modify_external_catalogs: true
I20250113 02:51:45.872424 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 2cd3b62ecc774d2eb658ed34e05864e1
I20250113 02:51:45.884768 448 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_bfd9035770884f2eb446fae15640ab22" } modify_external_catalogs: true
I20250113 02:51:45.884910 448 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_bfd9035770884f2eb446fae15640ab22" } modify_external_catalogs: true
I20250113 02:51:45.884774 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet 2cd3b62ecc774d2eb658ed34e05864e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:45 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:45.884932 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1451) took 1011 ms. Trace:
I20250113 02:51:45.885630 1028 tablet_service.cc:1467] Processing CreateTablet for tablet 5da22bf39b4b43d080165893cbf75a97 (DEFAULT_TABLE table=default.Table_d4a9d5f02f1243249011f1f7fa787487 [id=baecfdfb6bae4d12bfbb0e589bd114ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:45.885967 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5da22bf39b4b43d080165893cbf75a97. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:45.887346 1194 tablet_bootstrap.cc:492] T dc680b5680b84abbab219296bc7ee707 P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:45.889252 1024 tablet_service.cc:1467] Processing CreateTablet for tablet 4b1dd1347dfb4bd7ad836d2902776f65 (DEFAULT_TABLE table=default.Table_21dd5e7f8388499ea952554e9d7df194 [id=370f724eefcf4fa480cd56b63b9bfe60]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:45.889845 454 catalog_manager.cc:5618] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587) does not have the latest schema for tablet f1237f7b04af4ebbb39b03aa3c78a499 (table default.Table_4c5e2e1008b54019a243ffe1ec72c020 [id=0817677719d045fd83bb85890f11df25]). Expected version 2 got 0
I20250113 02:51:45.889986 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b1dd1347dfb4bd7ad836d2902776f65. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:45.890169 454 catalog_manager.cc:5618] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587) does not have the latest schema for tablet 247af1a17df04b27898f86642336d51e (table default.Table_b3926f9a1b6a4f249cf1e20b1129438e [id=43940b76df2c4fb5bc58e5e2975cac8b]). Expected version 2 got 0
I20250113 02:51:45.891083 1034 tablet_service.cc:1467] Processing CreateTablet for tablet 9f6e77970dbc43d1bfd8b28283f472ac (DEFAULT_TABLE table=default.Table_0b86c10add464ae3a38d1ec5305117b4 [id=4886c046def74a6e971904c54792d0ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:45.888029 891 tablet_service.cc:1467] Processing CreateTablet for tablet 4b1dd1347dfb4bd7ad836d2902776f65 (DEFAULT_TABLE table=default.Table_21dd5e7f8388499ea952554e9d7df194 [id=370f724eefcf4fa480cd56b63b9bfe60]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:45.892262 1032 tablet_service.cc:1514] Processing DeleteTablet for tablet 59fadc693f554854bc2874f8db7c64c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:40 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:45.892350 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b1dd1347dfb4bd7ad836d2902776f65. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:45.888157 882 tablet_service.cc:1514] Processing DeleteTablet for tablet 2cd3b62ecc774d2eb658ed34e05864e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:45 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:45.893307 1032 tablet_service.cc:1514] Processing DeleteTablet for tablet cf7b8bd6801743f49e9b9473747447c5 with delete_type TABLET_DATA_DELETED (replaced by tablet a8aedbc8d7bb4183bb1ab9902f07bea7) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:45.893951 1032 tablet_service.cc:1514] Processing DeleteTablet for tablet 542df00229f645788f1b821c0ce00c46 with delete_type TABLET_DATA_DELETED (replaced by tablet 2460e7cddcd84788a54dab342e7bb27b) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:45.885170 1198 rpcz_store.cc:276] 0113 02:51:44.872957 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:44.875855 (+ 2898us) service_pool.cc:225] Handling call
0113 02:51:45.884849 (+1008994us) ts_tablet_manager.cc:1145] Deleting tablet 6a4c6dc08df744e6856bdf99ab65c575
0113 02:51:45.884859 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:45.884917 (+ 58us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1008802}
I20250113 02:51:45.884526 1192 tablet_bootstrap.cc:654] T f4379f7fc6b54e6785f27975d9030cce P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
W20250113 02:51:45.895766 440 catalog_manager.cc:4839] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 6a4c6dc08df744e6856bdf99ab65c575 because the tablet was not found. No further retry: Not found: Tablet not found: 6a4c6dc08df744e6856bdf99ab65c575
I20250113 02:51:45.898157 882 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d2fdc75d00f4e749d322ac6cd5fc410 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:27 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:45.898240 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f6e77970dbc43d1bfd8b28283f472ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:45.899328 882 tablet_service.cc:1514] Processing DeleteTablet for tablet 153b8f90b5a847b88944f683d306f579 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:24 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:45.899832 896 tablet_service.cc:1467] Processing CreateTablet for tablet 9f6e77970dbc43d1bfd8b28283f472ac (DEFAULT_TABLE table=default.Table_0b86c10add464ae3a38d1ec5305117b4 [id=4886c046def74a6e971904c54792d0ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:45.900286 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f6e77970dbc43d1bfd8b28283f472ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:45.900370 1194 tablet_bootstrap.cc:654] T dc680b5680b84abbab219296bc7ee707 P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:45.872721 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_d86534ff16f04a45977372ccee26a336" } new_table_name: "default.Table_df638be048b742d18c261cbfc36b778e" modify_external_catalogs: true
W20250113 02:51:45.904266 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 7d0d6f8669dd41e39d402ad1d237418d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7d0d6f8669dd41e39d402ad1d237418d already in progress: creating tablet
W20250113 02:51:45.904698 440 catalog_manager.cc:4839] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 83762997f3bd47e6966898ccad38abbb because the tablet was not found. No further retry: Not found: Tablet not found: 83762997f3bd47e6966898ccad38abbb
W20250113 02:51:45.904910 440 catalog_manager.cc:4839] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 83762997f3bd47e6966898ccad38abbb because the tablet was not found. No further retry: Not found: Tablet not found: 83762997f3bd47e6966898ccad38abbb
W20250113 02:51:45.905112 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet b6293a3b355b4c409b1158b6ba51f727 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b6293a3b355b4c409b1158b6ba51f727 already in progress: creating tablet
W20250113 02:51:45.905287 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 99843ce7f1bf41da9b7cb5bd7ac2e875 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 99843ce7f1bf41da9b7cb5bd7ac2e875 already in progress: creating tablet
W20250113 02:51:45.905589 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet e9767ab120c04e2694c913b951020a13 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e9767ab120c04e2694c913b951020a13 already in progress: creating tablet
W20250113 02:51:45.906077 440 catalog_manager.cc:4839] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet fb2458e2044d49029ed2d6851db9de25 because the tablet was not found. No further retry: Not found: Tablet not found: fb2458e2044d49029ed2d6851db9de25
I20250113 02:51:45.906468 1198 tablet_replica.cc:331] T f105f8b32a654e4ca154e8fc0cb96091 P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:45.906524 1196 log.cc:1198] T 788ab82626424b6e833e31761a15b9b2 P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/788ab82626424b6e833e31761a15b9b2
I20250113 02:51:45.906666 1198 raft_consensus.cc:2238] T f105f8b32a654e4ca154e8fc0cb96091 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: Raft consensus shutting down.
I20250113 02:51:45.906873 1196 ts_tablet_manager.cc:1939] T 788ab82626424b6e833e31761a15b9b2 P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:45.906977 1198 raft_consensus.cc:2267] T f105f8b32a654e4ca154e8fc0cb96091 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:45.907665 1196 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37366 (request call id 1353) took 1344 ms. Trace:
I20250113 02:51:45.907768 1198 ts_tablet_manager.cc:1905] T f105f8b32a654e4ca154e8fc0cb96091 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:45.907747 1196 rpcz_store.cc:276] 0113 02:51:44.563171 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:44.563238 (+ 67us) service_pool.cc:225] Handling call
0113 02:51:45.759427 (+1196189us) ts_tablet_manager.cc:1145] Deleting tablet 788ab82626424b6e833e31761a15b9b2
0113 02:51:45.759439 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:45.810197 (+ 50758us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:45.816930 (+ 6733us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:45.907643 (+ 90713us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1195975}
I20250113 02:51:45.913558 1028 tablet_service.cc:1467] Processing CreateTablet for tablet 692a20cd59224ec39578c7f2a2c31eed (DEFAULT_TABLE table=default.Table_6fa6ea162f4c46b19013a23599df0f49 [id=710591fe6f764ce69cea7a6f94350979]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:45.913930 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 692a20cd59224ec39578c7f2a2c31eed. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:45.913938 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet 788ab82626424b6e833e31761a15b9b2 (table default.Table_9494f6e5f96141dbaec6769ed51d39b2 [id=0f07147c73f24decb09399a1dd188e1f]) successfully deleted
I20250113 02:51:45.914614 1196 tablet_replica.cc:331] T bbc7832c50114d6f890fec45a3d994e9 P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:45.914825 1196 raft_consensus.cc:2238] T bbc7832c50114d6f890fec45a3d994e9 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:45.914995 1196 raft_consensus.cc:2267] T bbc7832c50114d6f890fec45a3d994e9 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:45.915555 1196 ts_tablet_manager.cc:1905] T bbc7832c50114d6f890fec45a3d994e9 P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:45.916415 882 tablet_service.cc:1467] Processing CreateTablet for tablet 692a20cd59224ec39578c7f2a2c31eed (DEFAULT_TABLE table=default.Table_6fa6ea162f4c46b19013a23599df0f49 [id=710591fe6f764ce69cea7a6f94350979]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:45.916901 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 692a20cd59224ec39578c7f2a2c31eed. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:45.917284 891 tablet_service.cc:1467] Processing CreateTablet for tablet 7c7f54f1b0754423bcac4686d5f3a420 (DEFAULT_TABLE table=default.Table_d1c2e202f1bc4708ac8ff97a63254ea8 [id=4c47c713b81e411a9c9a838e2588071c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:45.917622 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c7f54f1b0754423bcac4686d5f3a420. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:45.919030 1198 ts_tablet_manager.cc:1918] T f105f8b32a654e4ca154e8fc0cb96091 P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250113 02:51:45.919118 1198 log.cc:1198] T f105f8b32a654e4ca154e8fc0cb96091 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/f105f8b32a654e4ca154e8fc0cb96091
I20250113 02:51:45.919481 1198 ts_tablet_manager.cc:1939] T f105f8b32a654e4ca154e8fc0cb96091 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:45.920054 2476 raft_consensus.cc:491] T 438604e8859f422f9caccb61c5b5b125 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:45.920181 2476 raft_consensus.cc:513] T 438604e8859f422f9caccb61c5b5b125 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:45.920521 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet f105f8b32a654e4ca154e8fc0cb96091 (table default.Table_3463038bce4545df8ec6c3834403a95d [id=505134ec091a46c9a1dc599ae2514a3c]) successfully deleted
I20250113 02:51:45.920858 2476 leader_election.cc:290] T 438604e8859f422f9caccb61c5b5b125 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:45.921080 883 tablet_service.cc:1467] Processing CreateTablet for tablet c821e1ffe83c4134837d4830895ec9d8 (DEFAULT_TABLE table=default.Table_fe0c2876b17f4e318de4048a7fba8c7a [id=229de359c465436582c6858aa856d42c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:45.921411 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c821e1ffe83c4134837d4830895ec9d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:45.922258 887 tablet_service.cc:1467] Processing CreateTablet for tablet baff26c23b51499298de2a8d9a9dc222 (DEFAULT_TABLE table=default.Table_7cdec29878244c9a9b06a312482a2ed6 [id=7429aaaa72244158bc6e8932831bb501]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:45.922564 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet baff26c23b51499298de2a8d9a9dc222. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:45.926685 1034 tablet_service.cc:1467] Processing CreateTablet for tablet 7c7f54f1b0754423bcac4686d5f3a420 (DEFAULT_TABLE table=default.Table_d1c2e202f1bc4708ac8ff97a63254ea8 [id=4c47c713b81e411a9c9a838e2588071c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:45.927016 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c7f54f1b0754423bcac4686d5f3a420. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:45.928254 899 tablet_service.cc:1467] Processing CreateTablet for tablet e263ea58dee54833957651d41d798991 (DEFAULT_TABLE table=default.Table_9054f47a8da54f029277b3cbaf88e62c [id=673df17ac67a4fbca74c829b63689b77]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:45.928567 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e263ea58dee54833957651d41d798991. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:45.932266 881 tablet_service.cc:1467] Processing CreateTablet for tablet ff1bd8bc7d9948779ab5a1cf8fb2a1ef (DEFAULT_TABLE table=default.Table_2142dbe5da9c4c2c824c7dc752e2f15b [id=9fa889d17f07420aa0812cb32c506959]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:45.932636 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff1bd8bc7d9948779ab5a1cf8fb2a1ef. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:45.936893 852 leader_election.cc:343] T 438604e8859f422f9caccb61c5b5b125 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Not found: Tablet not found: 438604e8859f422f9caccb61c5b5b125
I20250113 02:51:45.941334 1034 tablet_service.cc:1467] Processing CreateTablet for tablet c821e1ffe83c4134837d4830895ec9d8 (DEFAULT_TABLE table=default.Table_fe0c2876b17f4e318de4048a7fba8c7a [id=229de359c465436582c6858aa856d42c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:45.942005 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c821e1ffe83c4134837d4830895ec9d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:45.943248 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet 2470b010b571444eba3f132323038f33 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:24 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:45.943367 1196 ts_tablet_manager.cc:1918] T bbc7832c50114d6f890fec45a3d994e9 P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:45.943477 1196 log.cc:1198] T bbc7832c50114d6f890fec45a3d994e9 P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/bbc7832c50114d6f890fec45a3d994e9
I20250113 02:51:45.943768 1196 ts_tablet_manager.cc:1939] T bbc7832c50114d6f890fec45a3d994e9 P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:45.944563 1196 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37366 (request call id 1354) took 1180 ms. Trace:
I20250113 02:51:45.944660 1196 rpcz_store.cc:276] 0113 02:51:44.764540 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:44.764591 (+ 51us) service_pool.cc:225] Handling call
0113 02:51:45.914556 (+1149965us) ts_tablet_manager.cc:1145] Deleting tablet bbc7832c50114d6f890fec45a3d994e9
0113 02:51:45.914568 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:45.930112 (+ 15544us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:45.943267 (+ 13155us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:45.944542 (+ 1275us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1148833}
I20250113 02:51:45.945068 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet bbc7832c50114d6f890fec45a3d994e9 (table default.Table_3c637eb181574498b44ecb4fc41f0117 [id=10c0207c68d8463e91f09972031ab8d6]) successfully deleted
I20250113 02:51:45.948248 1039 tablet_service.cc:1467] Processing CreateTablet for tablet e263ea58dee54833957651d41d798991 (DEFAULT_TABLE table=default.Table_9054f47a8da54f029277b3cbaf88e62c [id=673df17ac67a4fbca74c829b63689b77]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:45.948729 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e263ea58dee54833957651d41d798991. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:45.949208 1198 tablet_replica.cc:331] T 883027967e0c442a91ac820210b9c6c7 P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:45.949407 1198 raft_consensus.cc:2238] T 883027967e0c442a91ac820210b9c6c7 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:45.949569 1198 raft_consensus.cc:2267] T 883027967e0c442a91ac820210b9c6c7 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:45.950363 1196 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37366 (request call id 1355) took 1128 ms. Trace:
I20250113 02:51:45.950456 1196 rpcz_store.cc:276] 0113 02:51:44.821850 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:44.821912 (+ 62us) service_pool.cc:225] Handling call
0113 02:51:45.950231 (+1128319us) ts_tablet_manager.cc:1145] Deleting tablet 08180ab0274a40fd955dda5b08197a08
0113 02:51:45.950240 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:45.950346 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1128069}
W20250113 02:51:45.951709 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 08180ab0274a40fd955dda5b08197a08 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 08180ab0274a40fd955dda5b08197a08 already in progress: creating tablet
I20250113 02:51:45.951772 1198 ts_tablet_manager.cc:1905] T 883027967e0c442a91ac820210b9c6c7 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:45.952504 1196 tablet_replica.cc:331] T 42dbd37e20fa47318bc6497423dbe565 P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:45.953274 1196 raft_consensus.cc:2238] T 42dbd37e20fa47318bc6497423dbe565 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:45.953975 1196 raft_consensus.cc:2267] T 42dbd37e20fa47318bc6497423dbe565 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:45.954751 1191 tablet_bootstrap.cc:492] T b9ec434644c14acd95c72efbef8ded59 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:45.954929 1191 ts_tablet_manager.cc:1397] T b9ec434644c14acd95c72efbef8ded59 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.211s user 0.003s sys 0.000s
I20250113 02:51:45.955793 1196 ts_tablet_manager.cc:1905] T 42dbd37e20fa47318bc6497423dbe565 P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:45.957171 2588 leader_election.cc:290] T 8f2fb58fc24d4692a3fe9a652e076aed P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:45.957515 882 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d85539cf0074a03b167d3279e940079 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:29 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:45.957608 743 tablet_service.cc:1514] Processing DeleteTablet for tablet 9174df876fe64adc805fba0a36fb4d38 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:30 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:45.957813 738 tablet_service.cc:1514] Processing DeleteTablet for tablet 936b8ceaedd34a019a4fb89a80dd2708 with delete_type TABLET_DATA_DELETED (replaced by tablet eba80548f52f4c32a098d1540c9cded0) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:45.957813 748 tablet_service.cc:1514] Processing DeleteTablet for tablet d565b77f90784d87b64996772c938085 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:34 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:45.958078 738 tablet_service.cc:1514] Processing DeleteTablet for tablet 7fb282707196444690352dea3eb1387f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:35 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:45.958207 743 tablet_service.cc:1514] Processing DeleteTablet for tablet 79aaab73797c474bae06831fa308b066 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:30 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:45.960403 885 tablet_service.cc:1467] Processing CreateTablet for tablet 5da22bf39b4b43d080165893cbf75a97 (DEFAULT_TABLE table=default.Table_d4a9d5f02f1243249011f1f7fa787487 [id=baecfdfb6bae4d12bfbb0e589bd114ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:45.960834 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f2fb58fc24d4692a3fe9a652e076aed" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:45.961344 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5da22bf39b4b43d080165893cbf75a97. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:45.962257 709 leader_election.cc:343] T 8f2fb58fc24d4692a3fe9a652e076aed P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:45.961558 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f2fb58fc24d4692a3fe9a652e076aed" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:45.963102 711 leader_election.cc:343] T 8f2fb58fc24d4692a3fe9a652e076aed P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:45.963250 711 leader_election.cc:304] T 8f2fb58fc24d4692a3fe9a652e076aed P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:45.963837 2588 raft_consensus.cc:2743] T 8f2fb58fc24d4692a3fe9a652e076aed P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:45.964426 1196 ts_tablet_manager.cc:1918] T 42dbd37e20fa47318bc6497423dbe565 P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:45.964537 1196 log.cc:1198] T 42dbd37e20fa47318bc6497423dbe565 P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/42dbd37e20fa47318bc6497423dbe565
I20250113 02:51:45.964843 1196 ts_tablet_manager.cc:1939] T 42dbd37e20fa47318bc6497423dbe565 P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:45.965642 1196 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37366 (request call id 1356) took 1139 ms. Trace:
I20250113 02:51:45.965740 1196 rpcz_store.cc:276] 0113 02:51:44.825823 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:44.825876 (+ 53us) service_pool.cc:225] Handling call
0113 02:51:45.952461 (+1126585us) ts_tablet_manager.cc:1145] Deleting tablet 42dbd37e20fa47318bc6497423dbe565
0113 02:51:45.952468 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:45.959600 (+ 7132us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:45.964347 (+ 4747us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:45.965622 (+ 1275us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1126404}
I20250113 02:51:45.966199 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet 42dbd37e20fa47318bc6497423dbe565 (table default.Table_37b52d3cfe0f4b0fb1ef1468117512dd [id=45aa63632c914a4e99eb69759b56c842]) successfully deleted
I20250113 02:51:45.966768 1196 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37366 (request call id 1357) took 1134 ms. Trace:
I20250113 02:51:45.966856 1196 rpcz_store.cc:276] 0113 02:51:44.832774 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:44.833232 (+ 458us) service_pool.cc:225] Handling call
0113 02:51:45.966630 (+1133398us) ts_tablet_manager.cc:1145] Deleting tablet 3a89e991ebee4b4f8c639ea95e1d1ea3
0113 02:51:45.966639 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:45.966749 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1133020}
W20250113 02:51:45.967307 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 3a89e991ebee4b4f8c639ea95e1d1ea3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3a89e991ebee4b4f8c639ea95e1d1ea3 already in progress: creating tablet
I20250113 02:51:45.967778 1196 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37366 (request call id 1358) took 1055 ms. Trace:
I20250113 02:51:45.967898 1196 rpcz_store.cc:276] 0113 02:51:44.912192 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:44.912278 (+ 86us) service_pool.cc:225] Handling call
0113 02:51:45.967618 (+1055340us) ts_tablet_manager.cc:1145] Deleting tablet 7d0d6f8669dd41e39d402ad1d237418d
0113 02:51:45.967629 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:45.967760 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1055032}
W20250113 02:51:45.968376 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 7d0d6f8669dd41e39d402ad1d237418d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7d0d6f8669dd41e39d402ad1d237418d already in progress: creating tablet
I20250113 02:51:45.968632 1194 tablet_bootstrap.cc:492] T dc680b5680b84abbab219296bc7ee707 P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:45.968845 1194 ts_tablet_manager.cc:1397] T dc680b5680b84abbab219296bc7ee707 P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.082s user 0.010s sys 0.010s
I20250113 02:51:45.969691 1194 raft_consensus.cc:357] T dc680b5680b84abbab219296bc7ee707 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:45.969967 1194 raft_consensus.cc:383] T dc680b5680b84abbab219296bc7ee707 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:45.970078 1194 raft_consensus.cc:738] T dc680b5680b84abbab219296bc7ee707 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:45.971024 752 tablet_service.cc:1467] Processing CreateTablet for tablet 5da22bf39b4b43d080165893cbf75a97 (DEFAULT_TABLE table=default.Table_d4a9d5f02f1243249011f1f7fa787487 [id=baecfdfb6bae4d12bfbb0e589bd114ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:45.971146 745 tablet_service.cc:1467] Processing CreateTablet for tablet 4b1dd1347dfb4bd7ad836d2902776f65 (DEFAULT_TABLE table=default.Table_21dd5e7f8388499ea952554e9d7df194 [id=370f724eefcf4fa480cd56b63b9bfe60]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:45.971439 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5da22bf39b4b43d080165893cbf75a97. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:45.971621 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b1dd1347dfb4bd7ad836d2902776f65. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:45.973731 748 tablet_service.cc:1467] Processing CreateTablet for tablet 0a94f75386b248dc9b8e03b384834779 (DEFAULT_TABLE table=default.Table_ca9b419c256442dfb375e4043a0e0fa9 [id=7b79338aab3744a3a7627ac07b9bef43]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:45.974068 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0a94f75386b248dc9b8e03b384834779. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:45.974573 1028 tablet_service.cc:1467] Processing CreateTablet for tablet baff26c23b51499298de2a8d9a9dc222 (DEFAULT_TABLE table=default.Table_7cdec29878244c9a9b06a312482a2ed6 [id=7429aaaa72244158bc6e8932831bb501]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:45.974960 738 tablet_service.cc:1514] Processing DeleteTablet for tablet 2cd3b62ecc774d2eb658ed34e05864e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:45 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:45.975020 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet baff26c23b51499298de2a8d9a9dc222. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:45.975617 1194 consensus_queue.cc:260] T dc680b5680b84abbab219296bc7ee707 P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:45.975708 743 tablet_service.cc:1467] Processing CreateTablet for tablet 692a20cd59224ec39578c7f2a2c31eed (DEFAULT_TABLE table=default.Table_6fa6ea162f4c46b19013a23599df0f49 [id=710591fe6f764ce69cea7a6f94350979]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:45.976066 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 692a20cd59224ec39578c7f2a2c31eed. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:45.976222 1194 ts_tablet_manager.cc:1428] T dc680b5680b84abbab219296bc7ee707 P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20250113 02:51:45.976393 754 tablet_service.cc:1467] Processing CreateTablet for tablet 9f6e77970dbc43d1bfd8b28283f472ac (DEFAULT_TABLE table=default.Table_0b86c10add464ae3a38d1ec5305117b4 [id=4886c046def74a6e971904c54792d0ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:45.976540 1194 tablet_bootstrap.cc:492] T 82505f1389184d0ca12ec0e2d6084d22 P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:45.976696 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f6e77970dbc43d1bfd8b28283f472ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:45.977221 1198 ts_tablet_manager.cc:1918] T 883027967e0c442a91ac820210b9c6c7 P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250113 02:51:45.977324 1198 log.cc:1198] T 883027967e0c442a91ac820210b9c6c7 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/883027967e0c442a91ac820210b9c6c7
I20250113 02:51:45.977651 1198 ts_tablet_manager.cc:1939] T 883027967e0c442a91ac820210b9c6c7 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:45.978608 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet 883027967e0c442a91ac820210b9c6c7 (table default.Table_a7a5a9569d064b488facac137ec35a85 [id=af62d77b2fb74001b86f0d760325eaa8]) successfully deleted
I20250113 02:51:45.978796 1194 tablet_bootstrap.cc:654] T 82505f1389184d0ca12ec0e2d6084d22 P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:45.980320 738 tablet_service.cc:1467] Processing CreateTablet for tablet 7c7f54f1b0754423bcac4686d5f3a420 (DEFAULT_TABLE table=default.Table_d1c2e202f1bc4708ac8ff97a63254ea8 [id=4c47c713b81e411a9c9a838e2588071c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:45.980782 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c7f54f1b0754423bcac4686d5f3a420. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:45.981730 1198 tablet_replica.cc:331] T 16a132384b0e4264a73d09a3ea133893 P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
W20250113 02:51:45.981679 851 leader_election.cc:343] T 438604e8859f422f9caccb61c5b5b125 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Not found: Tablet not found: 438604e8859f422f9caccb61c5b5b125
I20250113 02:51:45.981855 851 leader_election.cc:304] T 438604e8859f422f9caccb61c5b5b125 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df69f48656b44bb890f76587ee69cc0f; no voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca
I20250113 02:51:45.981932 1198 raft_consensus.cc:2238] T 16a132384b0e4264a73d09a3ea133893 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: Raft consensus shutting down.
I20250113 02:51:45.982170 1198 pending_rounds.cc:62] T 16a132384b0e4264a73d09a3ea133893 P 4216a7f853d642b7bc3e5cff458f3aca: Trying to abort 1 pending ops.
I20250113 02:51:45.982219 1198 pending_rounds.cc:69] T 16a132384b0e4264a73d09a3ea133893 P 4216a7f853d642b7bc3e5cff458f3aca: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113673545175977984 op_type: NO_OP noop_request { }
I20250113 02:51:45.982316 1198 raft_consensus.cc:2883] T 16a132384b0e4264a73d09a3ea133893 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250113 02:51:45.982393 1198 raft_consensus.cc:2267] T 16a132384b0e4264a73d09a3ea133893 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:45.982528 2476 raft_consensus.cc:2743] T 438604e8859f422f9caccb61c5b5b125 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:45.968827 1196 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37366 (request call id 1359) took 1044 ms. Trace:
I20250113 02:51:45.983640 1196 rpcz_store.cc:276] 0113 02:51:44.924663 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:44.924736 (+ 73us) service_pool.cc:225] Handling call
0113 02:51:45.968650 (+1043914us) ts_tablet_manager.cc:1145] Deleting tablet 83762997f3bd47e6966898ccad38abbb
0113 02:51:45.968661 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:45.968812 (+ 151us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1043709}
W20250113 02:51:45.984341 442 catalog_manager.cc:4839] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 83762997f3bd47e6966898ccad38abbb because the tablet was not found. No further retry: Not found: Tablet not found: 83762997f3bd47e6966898ccad38abbb
I20250113 02:51:45.985061 1196 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37366 (request call id 1360) took 1039 ms. Trace:
I20250113 02:51:45.985234 1196 rpcz_store.cc:276] 0113 02:51:44.945236 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:44.945335 (+ 99us) service_pool.cc:225] Handling call
0113 02:51:45.984924 (+1039589us) ts_tablet_manager.cc:1145] Deleting tablet 83762997f3bd47e6966898ccad38abbb
0113 02:51:45.984942 (+ 18us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:45.985040 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1039335}
I20250113 02:51:45.985780 1198 ts_tablet_manager.cc:1905] T 16a132384b0e4264a73d09a3ea133893 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250113 02:51:45.985873 442 catalog_manager.cc:4839] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 83762997f3bd47e6966898ccad38abbb because the tablet was not found. No further retry: Not found: Tablet not found: 83762997f3bd47e6966898ccad38abbb
I20250113 02:51:45.986235 1196 tablet_replica.cc:331] T ba8980ad3a364c888a825affbcbc9b1f P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:45.986476 1196 raft_consensus.cc:2238] T ba8980ad3a364c888a825affbcbc9b1f P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:45.986686 1196 raft_consensus.cc:2267] T ba8980ad3a364c888a825affbcbc9b1f P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:45.987285 1196 ts_tablet_manager.cc:1905] T ba8980ad3a364c888a825affbcbc9b1f P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:45.994740 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_f193626bef644c699ec47d1e88580317"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:45.995481 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f193626bef644c699ec47d1e88580317 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:45.997260 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_c89db70b70a64f88a0203c8c66867911" } new_table_name: "default.Table_956c8f70cb6a49b3bbed735003a9c0a6" modify_external_catalogs: true
I20250113 02:51:45.998988 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_f421dead901f4537860423130ed500fd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:45.999467 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f421dead901f4537860423130ed500fd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:46.000097 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 65139dcfe89249ec95f5425b294f04d8
I20250113 02:51:46.000242 1196 ts_tablet_manager.cc:1918] T ba8980ad3a364c888a825affbcbc9b1f P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:46.000398 1196 log.cc:1198] T ba8980ad3a364c888a825affbcbc9b1f P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/ba8980ad3a364c888a825affbcbc9b1f
I20250113 02:51:46.000838 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_f7a044df83094686b983b6a11fd518cd" } modify_external_catalogs: true
I20250113 02:51:46.001032 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_f7a044df83094686b983b6a11fd518cd" } modify_external_catalogs: true
I20250113 02:51:46.001081 1196 ts_tablet_manager.cc:1939] T ba8980ad3a364c888a825affbcbc9b1f P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:46.002892 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet ba8980ad3a364c888a825affbcbc9b1f (table default.Table_459ff6639996470cb94e898c9ef4880e [id=a1d8c84954c747adb2aa43726d7b5334]) successfully deleted
I20250113 02:51:46.003139 1196 tablet_replica.cc:331] T 0c408921a09a463eb464e08edcf042b3 P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:46.003417 1196 raft_consensus.cc:2238] T 0c408921a09a463eb464e08edcf042b3 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:46.003603 1196 raft_consensus.cc:2267] T 0c408921a09a463eb464e08edcf042b3 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:46.004551 1196 ts_tablet_manager.cc:1905] T 0c408921a09a463eb464e08edcf042b3 P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:46.005884 885 tablet_service.cc:1514] Processing DeleteTablet for tablet dc680b5680b84abbab219296bc7ee707 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:23 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:46.006093 896 tablet_service.cc:1514] Processing DeleteTablet for tablet 65139dcfe89249ec95f5425b294f04d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:45 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:46.006752 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet 65139dcfe89249ec95f5425b294f04d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:45 UTC) from {username='slave'} at 127.0.0.1:60230
W20250113 02:51:46.007130 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 99843ce7f1bf41da9b7cb5bd7ac2e875 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 99843ce7f1bf41da9b7cb5bd7ac2e875 already in progress: creating tablet
W20250113 02:51:46.007403 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet b6293a3b355b4c409b1158b6ba51f727 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b6293a3b355b4c409b1158b6ba51f727 already in progress: creating tablet
I20250113 02:51:46.008615 448 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet ef8f32459ea4405a8eef1e7bce8b2036
I20250113 02:51:46.008898 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_588573aeb9464e8088f6759b1ca47c54" } new_table_name: "default.Table_3c91d80eb0ae4bc2a4a5eb3b36a680b4" modify_external_catalogs: true
I20250113 02:51:46.013312 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet f29d7a9504d740528914fe6a66fd4028
I20250113 02:51:46.013330 1196 ts_tablet_manager.cc:1918] T 0c408921a09a463eb464e08edcf042b3 P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:46.013428 1196 log.cc:1198] T 0c408921a09a463eb464e08edcf042b3 P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/0c408921a09a463eb464e08edcf042b3
I20250113 02:51:46.013725 1196 ts_tablet_manager.cc:1939] T 0c408921a09a463eb464e08edcf042b3 P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:46.014912 896 tablet_service.cc:1514] Processing DeleteTablet for tablet f29d7a9504d740528914fe6a66fd4028 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:45 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:46.015399 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet f29d7a9504d740528914fe6a66fd4028 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:45 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:46.017689 1198 ts_tablet_manager.cc:1918] T 16a132384b0e4264a73d09a3ea133893 P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250113 02:51:46.019155 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet 0c408921a09a463eb464e08edcf042b3 (table default.Table_08def01561d44ec5a88f5b5a82d47e3b [id=c24af26d1a734152af598a5aa258cb06]) successfully deleted
I20250113 02:51:46.019527 1198 log.cc:1198] T 16a132384b0e4264a73d09a3ea133893 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/16a132384b0e4264a73d09a3ea133893
W20250113 02:51:46.020057 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet e9767ab120c04e2694c913b951020a13 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e9767ab120c04e2694c913b951020a13 already in progress: creating tablet
I20250113 02:51:46.020277 1196 tablet_replica.cc:331] T b33025a462e74f2aa26cc3779f36615b P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:46.020501 1196 raft_consensus.cc:2238] T b33025a462e74f2aa26cc3779f36615b P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
W20250113 02:51:46.020570 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet be2ca8d2ac6547189a0edeff12413330 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet be2ca8d2ac6547189a0edeff12413330 already in progress: creating tablet
I20250113 02:51:46.020675 1196 raft_consensus.cc:2267] T b33025a462e74f2aa26cc3779f36615b P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
W20250113 02:51:46.020792 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 28000f78e5dd408c8de58c674ad95f6c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 28000f78e5dd408c8de58c674ad95f6c already in progress: creating tablet
W20250113 02:51:46.021018 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet c7da2c406e64426faf123c4bddfaf213 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c7da2c406e64426faf123c4bddfaf213 already in progress: creating tablet
I20250113 02:51:46.021109 1034 tablet_service.cc:1467] Processing CreateTablet for tablet ff1bd8bc7d9948779ab5a1cf8fb2a1ef (DEFAULT_TABLE table=default.Table_2142dbe5da9c4c2c824c7dc752e2f15b [id=9fa889d17f07420aa0812cb32c506959]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:46.021214 1196 ts_tablet_manager.cc:1905] T b33025a462e74f2aa26cc3779f36615b P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:46.021522 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff1bd8bc7d9948779ab5a1cf8fb2a1ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:46.021525 1198 ts_tablet_manager.cc:1939] T 16a132384b0e4264a73d09a3ea133893 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:46.022782 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet 16a132384b0e4264a73d09a3ea133893 (table default.Table_eef7fe6c891b4fe984f8fb4a1c667a8e [id=ada38f49d3074fd49727e4cbe8b576d6]) successfully deleted
I20250113 02:51:46.024684 1198 tablet_replica.cc:331] T 65d9bbd8c15f40a4a50c1695fb5e648e P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:46.025290 1198 raft_consensus.cc:2238] T 65d9bbd8c15f40a4a50c1695fb5e648e P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: Raft consensus shutting down.
I20250113 02:51:46.026044 1198 pending_rounds.cc:62] T 65d9bbd8c15f40a4a50c1695fb5e648e P 4216a7f853d642b7bc3e5cff458f3aca: Trying to abort 1 pending ops.
I20250113 02:51:46.026580 1198 pending_rounds.cc:69] T 65d9bbd8c15f40a4a50c1695fb5e648e P 4216a7f853d642b7bc3e5cff458f3aca: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113673543688564736 op_type: NO_OP noop_request { }
I20250113 02:51:46.027000 1198 raft_consensus.cc:2883] T 65d9bbd8c15f40a4a50c1695fb5e648e P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250113 02:51:46.027262 1198 raft_consensus.cc:2267] T 65d9bbd8c15f40a4a50c1695fb5e648e P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:46.027628 1196 ts_tablet_manager.cc:1918] T b33025a462e74f2aa26cc3779f36615b P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:46.027745 1196 log.cc:1198] T b33025a462e74f2aa26cc3779f36615b P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/b33025a462e74f2aa26cc3779f36615b
I20250113 02:51:46.029323 1196 ts_tablet_manager.cc:1939] T b33025a462e74f2aa26cc3779f36615b P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:46.029549 2823 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 02:51:46.030369 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet b33025a462e74f2aa26cc3779f36615b (table default.Table_23c1655ab4cf4a30b67185ccf4fbb76d [id=a8e428598ca94e15b04f1df8112a4abc]) successfully deleted
I20250113 02:51:46.030711 1196 tablet_replica.cc:331] T 64aa8390b1b1475c8743b6491d40674c P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
W20250113 02:51:46.030876 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 1376cf72589d43048cb829e03240c8da because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1376cf72589d43048cb829e03240c8da already in progress: creating tablet
I20250113 02:51:46.030896 1196 raft_consensus.cc:2238] T 64aa8390b1b1475c8743b6491d40674c P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:46.031039 1196 raft_consensus.cc:2267] T 64aa8390b1b1475c8743b6491d40674c P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
W20250113 02:51:46.031046 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 593c54696ea7473d8450744a7c4a394d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 593c54696ea7473d8450744a7c4a394d already in progress: creating tablet
W20250113 02:51:46.031203 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 33ccb3f7d2024f03b23d1e1c1046072a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 33ccb3f7d2024f03b23d1e1c1046072a already in progress: creating tablet
I20250113 02:51:46.031790 896 tablet_service.cc:1467] Processing CreateTablet for tablet cb99b991d59746f6aef9248d73dc5692 (DEFAULT_TABLE table=default.Table_4c6fa7b378f94acea4082e8a5ad20579 [id=3e7a9af752aa44cab4e7dca0a9aba79d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:46.032192 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb99b991d59746f6aef9248d73dc5692. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:46.033429 882 tablet_service.cc:1467] Processing CreateTablet for tablet 83762997f3bd47e6966898ccad38abbb (DEFAULT_TABLE table=default.Table_82bda2c55fed4358ba3769c07352f440 [id=1c65b1cdf06d4a9791a87600c8089a4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:46.033769 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 83762997f3bd47e6966898ccad38abbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:46.035543 742 tablet_service.cc:1467] Processing CreateTablet for tablet c821e1ffe83c4134837d4830895ec9d8 (DEFAULT_TABLE table=default.Table_fe0c2876b17f4e318de4048a7fba8c7a [id=229de359c465436582c6858aa856d42c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:46.035921 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c821e1ffe83c4134837d4830895ec9d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:46.036273 749 tablet_service.cc:1467] Processing CreateTablet for tablet baff26c23b51499298de2a8d9a9dc222 (DEFAULT_TABLE table=default.Table_7cdec29878244c9a9b06a312482a2ed6 [id=7429aaaa72244158bc6e8932831bb501]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:46.036257 885 tablet_service.cc:1467] Processing CreateTablet for tablet 6d859229bd084f35975a615777be80a4 (DEFAULT_TABLE table=default.Table_27220e70104c4e69aab7262aef046bdb [id=ede56dd0d041496bbbad872b3213fe4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:46.036558 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet baff26c23b51499298de2a8d9a9dc222. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:46.036593 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d859229bd084f35975a615777be80a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:46.036770 1024 tablet_service.cc:1467] Processing CreateTablet for tablet 6d859229bd084f35975a615777be80a4 (DEFAULT_TABLE table=default.Table_27220e70104c4e69aab7262aef046bdb [id=ede56dd0d041496bbbad872b3213fe4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:46.036903 750 tablet_service.cc:1467] Processing CreateTablet for tablet e263ea58dee54833957651d41d798991 (DEFAULT_TABLE table=default.Table_9054f47a8da54f029277b3cbaf88e62c [id=673df17ac67a4fbca74c829b63689b77]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:46.037197 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d859229bd084f35975a615777be80a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:46.037202 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e263ea58dee54833957651d41d798991. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:46.037345 755 tablet_service.cc:1514] Processing DeleteTablet for tablet 65139dcfe89249ec95f5425b294f04d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:45 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:46.038512 1034 tablet_service.cc:1467] Processing CreateTablet for tablet cb99b991d59746f6aef9248d73dc5692 (DEFAULT_TABLE table=default.Table_4c6fa7b378f94acea4082e8a5ad20579 [id=3e7a9af752aa44cab4e7dca0a9aba79d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:46.038913 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb99b991d59746f6aef9248d73dc5692. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:46.040771 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 70554c1314ef4778bb60dd2f09d08b5d consensus state has no leader
I20250113 02:51:46.046509 1196 ts_tablet_manager.cc:1905] T 64aa8390b1b1475c8743b6491d40674c P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250113 02:51:46.052198 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 42a8e01bb71f49b09c83a8f5c905f38d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 42a8e01bb71f49b09c83a8f5c905f38d already in progress: creating tablet
W20250113 02:51:46.052488 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 94307d22b7f842ca89844768e4a2a2ab because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 94307d22b7f842ca89844768e4a2a2ab already in progress: creating tablet
W20250113 02:51:46.052708 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet a0c14501ad0449aca28909bc14428dad because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a0c14501ad0449aca28909bc14428dad already in progress: creating tablet
I20250113 02:51:46.052670 744 tablet_service.cc:1467] Processing CreateTablet for tablet ff1bd8bc7d9948779ab5a1cf8fb2a1ef (DEFAULT_TABLE table=default.Table_2142dbe5da9c4c2c824c7dc752e2f15b [id=9fa889d17f07420aa0812cb32c506959]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:46.053007 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff1bd8bc7d9948779ab5a1cf8fb2a1ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:46.055429 1196 ts_tablet_manager.cc:1918] T 64aa8390b1b1475c8743b6491d40674c P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:46.055563 1196 log.cc:1198] T 64aa8390b1b1475c8743b6491d40674c P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/64aa8390b1b1475c8743b6491d40674c
I20250113 02:51:46.055869 1196 ts_tablet_manager.cc:1939] T 64aa8390b1b1475c8743b6491d40674c P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:46.056294 741 tablet_service.cc:1467] Processing CreateTablet for tablet cb99b991d59746f6aef9248d73dc5692 (DEFAULT_TABLE table=default.Table_4c6fa7b378f94acea4082e8a5ad20579 [id=3e7a9af752aa44cab4e7dca0a9aba79d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:46.056696 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb99b991d59746f6aef9248d73dc5692. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:46.058442 755 tablet_service.cc:1514] Processing DeleteTablet for tablet f29d7a9504d740528914fe6a66fd4028 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:45 UTC) from {username='slave'} at 127.0.0.1:55398
W20250113 02:51:46.061278 998 consensus_peers.cc:487] T 88a776aa820c44b3a3854ca5022bd353 P 4216a7f853d642b7bc3e5cff458f3aca -> Peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Couldn't send request to peer df69f48656b44bb890f76587ee69cc0f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250113 02:51:46.061511 1196 tablet_replica.cc:331] T 438604e8859f422f9caccb61c5b5b125 P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:46.061717 1196 raft_consensus.cc:2238] T 438604e8859f422f9caccb61c5b5b125 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:46.061880 1196 raft_consensus.cc:2267] T 438604e8859f422f9caccb61c5b5b125 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
W20250113 02:51:46.062755 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a86210c2f74c41458621f0683bed3ce3 consensus state has no leader
I20250113 02:51:46.062834 746 tablet_service.cc:1467] Processing CreateTablet for tablet 6d859229bd084f35975a615777be80a4 (DEFAULT_TABLE table=default.Table_27220e70104c4e69aab7262aef046bdb [id=ede56dd0d041496bbbad872b3213fe4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:46.063163 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d859229bd084f35975a615777be80a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:46.063834 1196 ts_tablet_manager.cc:1905] T 438604e8859f422f9caccb61c5b5b125 P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:46.069852 1196 ts_tablet_manager.cc:1918] T 438604e8859f422f9caccb61c5b5b125 P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:46.069967 1196 log.cc:1198] T 438604e8859f422f9caccb61c5b5b125 P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/438604e8859f422f9caccb61c5b5b125
I20250113 02:51:46.070256 1196 ts_tablet_manager.cc:1939] T 438604e8859f422f9caccb61c5b5b125 P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
W20250113 02:51:46.074018 998 consensus_peers.cc:487] T b217cb8c09fb41cab4f53e605c37efe8 P 4216a7f853d642b7bc3e5cff458f3aca -> Peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Couldn't send request to peer df69f48656b44bb890f76587ee69cc0f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250113 02:51:46.077564 1024 tablet_service.cc:1467] Processing CreateTablet for tablet 83762997f3bd47e6966898ccad38abbb (DEFAULT_TABLE table=default.Table_82bda2c55fed4358ba3769c07352f440 [id=1c65b1cdf06d4a9791a87600c8089a4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:46.077981 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 83762997f3bd47e6966898ccad38abbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:46.078469 1196 tablet_replica.cc:331] T 1d2fdc75d00f4e749d322ac6cd5fc410 P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:46.078557 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet 64aa8390b1b1475c8743b6491d40674c (table default.Table_b89481a4a7be4e49b13f3260d8a31d5d [id=6c751a49967d448ba1432934782b08d3]) successfully deleted
I20250113 02:51:46.078646 1196 raft_consensus.cc:2238] T 1d2fdc75d00f4e749d322ac6cd5fc410 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:46.078787 1196 raft_consensus.cc:2267] T 1d2fdc75d00f4e749d322ac6cd5fc410 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:46.081121 1196 ts_tablet_manager.cc:1905] T 1d2fdc75d00f4e749d322ac6cd5fc410 P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:46.087093 1196 ts_tablet_manager.cc:1918] T 1d2fdc75d00f4e749d322ac6cd5fc410 P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:46.087222 1196 log.cc:1198] T 1d2fdc75d00f4e749d322ac6cd5fc410 P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/1d2fdc75d00f4e749d322ac6cd5fc410
I20250113 02:51:46.087538 1196 ts_tablet_manager.cc:1939] T 1d2fdc75d00f4e749d322ac6cd5fc410 P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:46.095466 2768 consensus_queue.cc:1035] T 0839a35fcab04f3c9d03e256553e01c8 P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 02:51:46.095719 1196 tablet_replica.cc:331] T 153b8f90b5a847b88944f683d306f579 P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:46.095940 1196 raft_consensus.cc:2238] T 153b8f90b5a847b88944f683d306f579 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:46.096114 1196 raft_consensus.cc:2267] T 153b8f90b5a847b88944f683d306f579 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:46.098546 1196 ts_tablet_manager.cc:1905] T 153b8f90b5a847b88944f683d306f579 P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:46.098875 736 tablet_service.cc:1467] Processing CreateTablet for tablet 83762997f3bd47e6966898ccad38abbb (DEFAULT_TABLE table=default.Table_82bda2c55fed4358ba3769c07352f440 [id=1c65b1cdf06d4a9791a87600c8089a4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:46.099308 736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 83762997f3bd47e6966898ccad38abbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:46.107517 2827 consensus_queue.cc:1035] T 9b2d0aa6f0e9456d95f4e48c360ab6e8 P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 02:51:46.112988 1196 ts_tablet_manager.cc:1918] T 153b8f90b5a847b88944f683d306f579 P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:46.113119 1196 log.cc:1198] T 153b8f90b5a847b88944f683d306f579 P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/153b8f90b5a847b88944f683d306f579
I20250113 02:51:46.113372 1196 ts_tablet_manager.cc:1939] T 153b8f90b5a847b88944f683d306f579 P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:46.113588 1191 raft_consensus.cc:357] T b9ec434644c14acd95c72efbef8ded59 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:46.113787 1191 raft_consensus.cc:383] T b9ec434644c14acd95c72efbef8ded59 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:46.113842 1191 raft_consensus.cc:738] T b9ec434644c14acd95c72efbef8ded59 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:46.114037 1191 consensus_queue.cc:260] T b9ec434644c14acd95c72efbef8ded59 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:46.114550 1191 ts_tablet_manager.cc:1428] T b9ec434644c14acd95c72efbef8ded59 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.160s user 0.002s sys 0.000s
I20250113 02:51:46.116513 1196 tablet_replica.cc:331] T 3d85539cf0074a03b167d3279e940079 P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:46.116729 1196 raft_consensus.cc:2238] T 3d85539cf0074a03b167d3279e940079 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:46.116878 1196 raft_consensus.cc:2267] T 3d85539cf0074a03b167d3279e940079 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:46.118330 1196 ts_tablet_manager.cc:1905] T 3d85539cf0074a03b167d3279e940079 P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:46.119745 1194 tablet_bootstrap.cc:492] T 82505f1389184d0ca12ec0e2d6084d22 P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:46.119874 1194 ts_tablet_manager.cc:1397] T 82505f1389184d0ca12ec0e2d6084d22 P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.143s user 0.030s sys 0.040s
I20250113 02:51:46.120594 1194 raft_consensus.cc:357] T 82505f1389184d0ca12ec0e2d6084d22 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:46.120810 1194 raft_consensus.cc:383] T 82505f1389184d0ca12ec0e2d6084d22 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:46.120864 1194 raft_consensus.cc:738] T 82505f1389184d0ca12ec0e2d6084d22 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:46.121102 1194 consensus_queue.cc:260] T 82505f1389184d0ca12ec0e2d6084d22 P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:46.124435 2588 raft_consensus.cc:491] T 36726b40ac5145bfb371871b3fff079a P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:46.124579 2588 raft_consensus.cc:513] T 36726b40ac5145bfb371871b3fff079a P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:46.125241 2588 leader_election.cc:290] T 36726b40ac5145bfb371871b3fff079a P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:46.125526 1194 ts_tablet_manager.cc:1428] T 82505f1389184d0ca12ec0e2d6084d22 P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250113 02:51:46.125558 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36726b40ac5145bfb371871b3fff079a" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:46.125778 1194 tablet_bootstrap.cc:492] T 1968c7e9357c4c6e901d334ace0467fb P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
W20250113 02:51:46.126029 711 leader_election.cc:343] T 36726b40ac5145bfb371871b3fff079a P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:46.127104 1194 tablet_bootstrap.cc:654] T 1968c7e9357c4c6e901d334ace0467fb P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:46.128774 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36726b40ac5145bfb371871b3fff079a" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:46.129334 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 476c89cc585b4e00947f5d1e8d0229f6
I20250113 02:51:46.130100 1039 tablet_service.cc:1514] Processing DeleteTablet for tablet 476c89cc585b4e00947f5d1e8d0229f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:45 UTC) from {username='slave'} at 127.0.0.1:60230
W20250113 02:51:46.130499 709 leader_election.cc:343] T 36726b40ac5145bfb371871b3fff079a P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:46.130615 709 leader_election.cc:304] T 36726b40ac5145bfb371871b3fff079a P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:46.131007 2588 raft_consensus.cc:2743] T 36726b40ac5145bfb371871b3fff079a P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:46.139372 1196 ts_tablet_manager.cc:1918] T 3d85539cf0074a03b167d3279e940079 P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:46.139496 1196 log.cc:1198] T 3d85539cf0074a03b167d3279e940079 P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/3d85539cf0074a03b167d3279e940079
I20250113 02:51:46.139786 1196 ts_tablet_manager.cc:1939] T 3d85539cf0074a03b167d3279e940079 P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:46.142694 449 catalog_manager.cc:5618] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095) does not have the latest schema for tablet e01e7f8db83d461bbfbb2e901419ce0b (table default.Table_e27f8b6f8232415388a5e53ab15494b7 [id=efc4ebba97f74c9ba12ab1dcc6577da0]). Expected version 1 got 0
I20250113 02:51:46.143065 449 catalog_manager.cc:5618] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095) does not have the latest schema for tablet 019eb21c661547cfa0b619d119eb024e (table default.Table_022a7c61a6864393acd739f562893b7c [id=97bfbd46eecd4e57b7f9757c413f6795]). Expected version 1 got 0
I20250113 02:51:46.143352 449 catalog_manager.cc:5618] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095) does not have the latest schema for tablet d59134b029a143038837366f51f00d28 (table default.Table_8641ddb6d21949528d69f2be6ed9011f [id=6ae8663361aa4ba989265452dfa21f89]). Expected version 2 got 0
W20250113 02:51:46.144129 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 568fae43734b4d3f81802e01251c6a7a consensus state has no leader
I20250113 02:51:46.144162 1194 tablet_bootstrap.cc:492] T 1968c7e9357c4c6e901d334ace0467fb P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:46.144292 1194 ts_tablet_manager.cc:1397] T 1968c7e9357c4c6e901d334ace0467fb P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.004s
I20250113 02:51:46.144487 1195 ts_tablet_manager.cc:1918] T 6001369444894b48b51a48fb60ddc52a P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:46.144603 1195 log.cc:1198] T 6001369444894b48b51a48fb60ddc52a P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/6001369444894b48b51a48fb60ddc52a
I20250113 02:51:46.144753 2769 raft_consensus.cc:491] T 68c4a7e8c0fb4feb85498709b00d3258 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:46.144905 1195 ts_tablet_manager.cc:1939] T 6001369444894b48b51a48fb60ddc52a P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:46.144887 2769 raft_consensus.cc:513] T 68c4a7e8c0fb4feb85498709b00d3258 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:46.144938 1194 raft_consensus.cc:357] T 1968c7e9357c4c6e901d334ace0467fb P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:46.145103 1194 raft_consensus.cc:383] T 1968c7e9357c4c6e901d334ace0467fb P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:46.145148 1194 raft_consensus.cc:738] T 1968c7e9357c4c6e901d334ace0467fb P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:46.145371 1194 consensus_queue.cc:260] T 1968c7e9357c4c6e901d334ace0467fb P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:46.145529 2769 leader_election.cc:290] T 68c4a7e8c0fb4feb85498709b00d3258 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:46.145993 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68c4a7e8c0fb4feb85498709b00d3258" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:46.146601 998 leader_election.cc:343] T 68c4a7e8c0fb4feb85498709b00d3258 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:46.149693 1194 ts_tablet_manager.cc:1428] T 1968c7e9357c4c6e901d334ace0467fb P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250113 02:51:46.145473 449 catalog_manager.cc:5618] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095) does not have the latest schema for tablet f1237f7b04af4ebbb39b03aa3c78a499 (table default.Table_4c5e2e1008b54019a243ffe1ec72c020 [id=0817677719d045fd83bb85890f11df25]). Expected version 2 got 0
W20250113 02:51:46.151451 995 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.1:40095, user_credentials={real_user=slave}} blocked reactor thread for 44167.5us
I20250113 02:51:46.155685 1194 tablet_bootstrap.cc:492] T 938cc1ccd8dd449996836188ba9e64ff P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:46.157213 1196 tablet_replica.cc:331] T dc680b5680b84abbab219296bc7ee707 P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:46.157392 1196 raft_consensus.cc:2238] T dc680b5680b84abbab219296bc7ee707 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:46.157523 1196 raft_consensus.cc:2267] T dc680b5680b84abbab219296bc7ee707 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
W20250113 02:51:46.157539 998 consensus_peers.cc:487] T 9b2d0aa6f0e9456d95f4e48c360ab6e8 P 4216a7f853d642b7bc3e5cff458f3aca -> Peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Couldn't send request to peer df69f48656b44bb890f76587ee69cc0f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250113 02:51:46.158233 1194 tablet_bootstrap.cc:654] T 938cc1ccd8dd449996836188ba9e64ff P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:46.159564 1196 ts_tablet_manager.cc:1905] T dc680b5680b84abbab219296bc7ee707 P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:46.162717 447 catalog_manager.cc:5618] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587) does not have the latest schema for tablet 019eb21c661547cfa0b619d119eb024e (table default.Table_022a7c61a6864393acd739f562893b7c [id=97bfbd46eecd4e57b7f9757c413f6795]). Expected version 1 got 0
I20250113 02:51:46.164999 740 tablet_service.cc:1514] Processing DeleteTablet for tablet 36726b40ac5145bfb371871b3fff079a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:32 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:46.165832 1196 ts_tablet_manager.cc:1918] T dc680b5680b84abbab219296bc7ee707 P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:46.165932 1196 log.cc:1198] T dc680b5680b84abbab219296bc7ee707 P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/dc680b5680b84abbab219296bc7ee707
I20250113 02:51:46.166210 1196 ts_tablet_manager.cc:1939] T dc680b5680b84abbab219296bc7ee707 P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:46.167202 739 tablet_service.cc:1514] Processing DeleteTablet for tablet 476c89cc585b4e00947f5d1e8d0229f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:45 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:46.167636 2829 raft_consensus.cc:491] T 58053845187f466599750f2a62f4c3e1 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:46.167825 2829 raft_consensus.cc:513] T 58053845187f466599750f2a62f4c3e1 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:46.169800 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58053845187f466599750f2a62f4c3e1" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:46.170032 1194 tablet_bootstrap.cc:492] T 938cc1ccd8dd449996836188ba9e64ff P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:46.170163 1194 ts_tablet_manager.cc:1397] T 938cc1ccd8dd449996836188ba9e64ff P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.020s user 0.005s sys 0.000s
W20250113 02:51:46.170358 998 leader_election.cc:343] T 58053845187f466599750f2a62f4c3e1 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:46.170760 2829 leader_election.cc:290] T 58053845187f466599750f2a62f4c3e1 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:46.170774 1194 raft_consensus.cc:357] T 938cc1ccd8dd449996836188ba9e64ff P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:46.170969 1194 raft_consensus.cc:383] T 938cc1ccd8dd449996836188ba9e64ff P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:46.171078 1194 raft_consensus.cc:738] T 938cc1ccd8dd449996836188ba9e64ff P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:46.171350 1194 consensus_queue.cc:260] T 938cc1ccd8dd449996836188ba9e64ff P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:46.178128 1194 ts_tablet_manager.cc:1428] T 938cc1ccd8dd449996836188ba9e64ff P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20250113 02:51:46.178119 449 catalog_manager.cc:5618] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467) does not have the latest schema for tablet 938cc1ccd8dd449996836188ba9e64ff (table default.Table_e5b4d614e4b743d7982b025ca6d297e5 [id=5824acd778ed47d09f087bbf8e7822d6]). Expected version 1 got 0
I20250113 02:51:46.182337 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58053845187f466599750f2a62f4c3e1" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
I20250113 02:51:46.182533 770 raft_consensus.cc:2463] T 58053845187f466599750f2a62f4c3e1 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 0.
I20250113 02:51:46.183813 1191 tablet_bootstrap.cc:492] T 95e47db8a4d74615800ba6c049ec7131 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:46.184973 1194 tablet_bootstrap.cc:492] T b38cd34d67834a3f9bf5b5636bb79217 P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:46.185823 1191 tablet_bootstrap.cc:654] T 95e47db8a4d74615800ba6c049ec7131 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:46.186435 1194 tablet_bootstrap.cc:654] T b38cd34d67834a3f9bf5b5636bb79217 P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:46.190524 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68c4a7e8c0fb4feb85498709b00d3258" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
I20250113 02:51:46.190740 774 raft_consensus.cc:2463] T 68c4a7e8c0fb4feb85498709b00d3258 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 0.
I20250113 02:51:46.193428 995 leader_election.cc:304] T 58053845187f466599750f2a62f4c3e1 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters: df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:46.193830 740 tablet_service.cc:1514] Processing DeleteTablet for tablet b9ec434644c14acd95c72efbef8ded59 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:28 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:46.193863 995 leader_election.cc:304] T 68c4a7e8c0fb4feb85498709b00d3258 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters: df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:46.194053 2772 raft_consensus.cc:2798] T 58053845187f466599750f2a62f4c3e1 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 02:51:46.194118 2772 raft_consensus.cc:491] T 58053845187f466599750f2a62f4c3e1 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 02:51:46.194190 2772 raft_consensus.cc:3054] T 58053845187f466599750f2a62f4c3e1 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:46.194448 2827 raft_consensus.cc:2798] T 68c4a7e8c0fb4feb85498709b00d3258 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 02:51:46.194516 2827 raft_consensus.cc:491] T 68c4a7e8c0fb4feb85498709b00d3258 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 02:51:46.194577 2827 raft_consensus.cc:3054] T 68c4a7e8c0fb4feb85498709b00d3258 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:46.196424 2368 consensus_queue.cc:1035] T f2c6c8c17d6e499d84ad7c0c23889cc4 P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 02:51:46.196797 2772 raft_consensus.cc:513] T 58053845187f466599750f2a62f4c3e1 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:46.197482 2772 leader_election.cc:290] T 58053845187f466599750f2a62f4c3e1 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Requested vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:46.198212 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58053845187f466599750f2a62f4c3e1" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f"
W20250113 02:51:46.198850 998 leader_election.cc:343] T 58053845187f466599750f2a62f4c3e1 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:46.199699 2827 raft_consensus.cc:513] T 68c4a7e8c0fb4feb85498709b00d3258 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:46.200248 2827 leader_election.cc:290] T 68c4a7e8c0fb4feb85498709b00d3258 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Requested vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:46.200745 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68c4a7e8c0fb4feb85498709b00d3258" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f"
W20250113 02:51:46.201216 998 leader_election.cc:343] T 68c4a7e8c0fb4feb85498709b00d3258 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:46.203840 449 catalog_manager.cc:5618] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467) does not have the latest schema for tablet 938cc1ccd8dd449996836188ba9e64ff (table default.Table_e5b4d614e4b743d7982b025ca6d297e5 [id=5824acd778ed47d09f087bbf8e7822d6]). Expected version 1 got 0
I20250113 02:51:46.204739 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58053845187f466599750f2a62f4c3e1" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196"
I20250113 02:51:46.204931 774 raft_consensus.cc:3054] T 58053845187f466599750f2a62f4c3e1 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:46.207396 774 raft_consensus.cc:2463] T 58053845187f466599750f2a62f4c3e1 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 1.
I20250113 02:51:46.207696 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68c4a7e8c0fb4feb85498709b00d3258" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196"
I20250113 02:51:46.207850 770 raft_consensus.cc:3054] T 68c4a7e8c0fb4feb85498709b00d3258 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:46.210427 770 raft_consensus.cc:2463] T 68c4a7e8c0fb4feb85498709b00d3258 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 1.
I20250113 02:51:46.211145 1194 tablet_bootstrap.cc:492] T b38cd34d67834a3f9bf5b5636bb79217 P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:46.211517 2835 tablet.cc:1721] T 019eb21c661547cfa0b619d119eb024e P 4216a7f853d642b7bc3e5cff458f3aca: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250113 02:51:46.211845 1194 ts_tablet_manager.cc:1397] T b38cd34d67834a3f9bf5b5636bb79217 P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.027s user 0.005s sys 0.018s
I20250113 02:51:46.213361 1194 raft_consensus.cc:357] T b38cd34d67834a3f9bf5b5636bb79217 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:46.213563 1194 raft_consensus.cc:383] T b38cd34d67834a3f9bf5b5636bb79217 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:46.213620 1194 raft_consensus.cc:738] T b38cd34d67834a3f9bf5b5636bb79217 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:46.213860 1194 consensus_queue.cc:260] T b38cd34d67834a3f9bf5b5636bb79217 P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:46.214537 995 leader_election.cc:304] T 58053845187f466599750f2a62f4c3e1 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters: df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:46.215124 1194 ts_tablet_manager.cc:1428] T b38cd34d67834a3f9bf5b5636bb79217 P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250113 02:51:46.215095 2829 raft_consensus.cc:2798] T 58053845187f466599750f2a62f4c3e1 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Leader election won for term 1
I20250113 02:51:46.215915 2829 raft_consensus.cc:695] T 58053845187f466599750f2a62f4c3e1 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: Becoming Leader. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Running, Role: LEADER
I20250113 02:51:46.216625 2829 consensus_queue.cc:237] T 58053845187f466599750f2a62f4c3e1 P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:46.217125 1194 tablet_bootstrap.cc:492] T c13e9a51c1e74b44b352bade78390ba5 P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:46.215198 995 leader_election.cc:304] T 68c4a7e8c0fb4feb85498709b00d3258 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters: df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:46.220465 2588 raft_consensus.cc:491] T 2470b010b571444eba3f132323038f33 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:46.220569 1194 tablet_bootstrap.cc:654] T c13e9a51c1e74b44b352bade78390ba5 P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:46.220628 2588 raft_consensus.cc:513] T 2470b010b571444eba3f132323038f33 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:46.221424 2588 leader_election.cc:290] T 2470b010b571444eba3f132323038f33 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:46.220307 2827 raft_consensus.cc:2798] T 68c4a7e8c0fb4feb85498709b00d3258 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Leader election won for term 1
I20250113 02:51:46.221889 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2470b010b571444eba3f132323038f33" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:46.221989 2827 raft_consensus.cc:695] T 68c4a7e8c0fb4feb85498709b00d3258 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: Becoming Leader. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Running, Role: LEADER
I20250113 02:51:46.222062 1059 raft_consensus.cc:2463] T 2470b010b571444eba3f132323038f33 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1186fa869eb14812832fa6562531f196 in term 0.
I20250113 02:51:46.222213 2827 consensus_queue.cc:237] T 68c4a7e8c0fb4feb85498709b00d3258 P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:46.222481 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2470b010b571444eba3f132323038f33" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:46.222843 709 leader_election.cc:304] T 2470b010b571444eba3f132323038f33 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters:
W20250113 02:51:46.223001 711 leader_election.cc:343] T 2470b010b571444eba3f132323038f33 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:46.223196 2588 raft_consensus.cc:2798] T 2470b010b571444eba3f132323038f33 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 02:51:46.223268 2588 raft_consensus.cc:491] T 2470b010b571444eba3f132323038f33 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 02:51:46.223325 2588 raft_consensus.cc:3054] T 2470b010b571444eba3f132323038f33 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:46.226485 2588 raft_consensus.cc:513] T 2470b010b571444eba3f132323038f33 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:46.229079 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2470b010b571444eba3f132323038f33" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f"
W20250113 02:51:46.229914 711 leader_election.cc:343] T 2470b010b571444eba3f132323038f33 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:46.230993 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2470b010b571444eba3f132323038f33" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca"
I20250113 02:51:46.231227 1059 raft_consensus.cc:3054] T 2470b010b571444eba3f132323038f33 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:46.234669 1059 raft_consensus.cc:2463] T 2470b010b571444eba3f132323038f33 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1186fa869eb14812832fa6562531f196 in term 1.
I20250113 02:51:46.235033 1191 tablet_bootstrap.cc:492] T 95e47db8a4d74615800ba6c049ec7131 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:46.235198 1191 ts_tablet_manager.cc:1397] T 95e47db8a4d74615800ba6c049ec7131 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.051s user 0.003s sys 0.000s
I20250113 02:51:46.235334 709 leader_election.cc:304] T 2470b010b571444eba3f132323038f33 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters: df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:46.235635 2723 raft_consensus.cc:2798] T 2470b010b571444eba3f132323038f33 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Leader election won for term 1
I20250113 02:51:46.236665 1191 raft_consensus.cc:357] T 95e47db8a4d74615800ba6c049ec7131 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:46.236900 1191 raft_consensus.cc:383] T 95e47db8a4d74615800ba6c049ec7131 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:46.236956 1191 raft_consensus.cc:738] T 95e47db8a4d74615800ba6c049ec7131 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:46.238898 2588 leader_election.cc:290] T 2470b010b571444eba3f132323038f33 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 election: Requested vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:46.248643 914 raft_consensus.cc:3054] T 938cc1ccd8dd449996836188ba9e64ff P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:46.264986 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1294) took 5530 ms. Trace:
I20250113 02:51:46.265128 1195 rpcz_store.cc:276] 0113 02:51:40.734108 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:41.113067 (+378959us) service_pool.cc:225] Handling call
0113 02:51:45.879194 (+4766127us) ts_tablet_manager.cc:1145] Deleting tablet 6001369444894b48b51a48fb60ddc52a
0113 02:51:45.879203 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:46.091824 (+212621us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:46.144378 (+ 52554us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:46.181386 (+ 37008us) spinlock_profiling.cc:243] Waited 36.1 ms on lock 0x61800000fd18. stack: 00007f2b88d31fd6 00007f2b88d31d90 00007f2b87d1148d 000000000096d544 000000000096d50d 000000000096ceda 00007f2b88d1a0fd 00007f2b88d1b16e 00007f2b9c14ec7d 00007f2b9c0cad43 00007f2b9836e323 00007f2b983a0584 00007f2b98381690 00007f2b9838141d 0000000000a71ecc 00007f2b88d78b64
0113 02:51:46.264963 (+ 83577us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":82995072,"tablet-delete.queue_time_us":4765953}
I20250113 02:51:46.265580 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1295) took 5480 ms. Trace:
I20250113 02:51:46.265655 1195 rpcz_store.cc:276] 0113 02:51:40.785619 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:41.113258 (+327639us) service_pool.cc:225] Handling call
0113 02:51:46.265459 (+5152201us) ts_tablet_manager.cc:1145] Deleting tablet a894199f21aa4dba8a03583ea14a9092
0113 02:51:46.265468 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:46.265566 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5152029}
I20250113 02:51:46.265962 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1299) took 5304 ms. Trace:
I20250113 02:51:46.266033 1195 rpcz_store.cc:276] 0113 02:51:40.961530 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:41.121395 (+159865us) service_pool.cc:225] Handling call
0113 02:51:46.265873 (+5144478us) ts_tablet_manager.cc:1145] Deleting tablet 8f2b4df7c6a74896ade95a490ea9b54c
0113 02:51:46.265881 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:46.265949 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5144274}
I20250113 02:51:46.266305 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1301) took 5301 ms. Trace:
I20250113 02:51:46.266374 1195 rpcz_store.cc:276] 0113 02:51:40.964846 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:41.122331 (+157485us) service_pool.cc:225] Handling call
0113 02:51:46.266222 (+5143891us) ts_tablet_manager.cc:1145] Deleting tablet 3c623bd0fbbf4922aedda031a88b79eb
0113 02:51:46.266229 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:46.266293 (+ 64us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5143738}
I20250113 02:51:46.266985 1195 tablet_replica.cc:331] T 438d1d2300e044f39a15586f335935ba P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:46.267168 1195 raft_consensus.cc:2238] T 438d1d2300e044f39a15586f335935ba P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:46.267349 1195 raft_consensus.cc:2267] T 438d1d2300e044f39a15586f335935ba P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:46.267911 1195 ts_tablet_manager.cc:1905] T 438d1d2300e044f39a15586f335935ba P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:46.270576 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet 6001369444894b48b51a48fb60ddc52a (table default.Table_fda8e0268b174c6f81a15f80e24825b0 [id=b8c683d257794905b44a447d158917ef]) successfully deleted
W20250113 02:51:46.270799 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet a894199f21aa4dba8a03583ea14a9092 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a894199f21aa4dba8a03583ea14a9092 already in progress: creating tablet
W20250113 02:51:46.270938 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 8f2b4df7c6a74896ade95a490ea9b54c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8f2b4df7c6a74896ade95a490ea9b54c already in progress: creating tablet
I20250113 02:51:46.270886 1191 consensus_queue.cc:260] T 95e47db8a4d74615800ba6c049ec7131 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
W20250113 02:51:46.271179 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 3c623bd0fbbf4922aedda031a88b79eb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3c623bd0fbbf4922aedda031a88b79eb already in progress: creating tablet
I20250113 02:51:46.273186 1191 ts_tablet_manager.cc:1428] T 95e47db8a4d74615800ba6c049ec7131 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.038s user 0.015s sys 0.005s
I20250113 02:51:46.274679 1191 tablet_bootstrap.cc:492] T 80c647a3b5a1411e9130461115c53550 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:46.276121 1191 tablet_bootstrap.cc:654] T 80c647a3b5a1411e9130461115c53550 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:46.282713 1192 tablet_bootstrap.cc:492] T f4379f7fc6b54e6785f27975d9030cce P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:46.282909 1192 ts_tablet_manager.cc:1397] T f4379f7fc6b54e6785f27975d9030cce P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.400s user 0.017s sys 0.012s
I20250113 02:51:46.283654 1192 raft_consensus.cc:357] T f4379f7fc6b54e6785f27975d9030cce P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:46.283912 1192 raft_consensus.cc:383] T f4379f7fc6b54e6785f27975d9030cce P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:46.284022 1192 raft_consensus.cc:738] T f4379f7fc6b54e6785f27975d9030cce P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:46.284324 1192 consensus_queue.cc:260] T f4379f7fc6b54e6785f27975d9030cce P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:46.285437 1192 ts_tablet_manager.cc:1428] T f4379f7fc6b54e6785f27975d9030cce P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250113 02:51:46.286160 1192 tablet_bootstrap.cc:492] T b3ce6d851f564b849796f174860aacd5 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:46.301578 1198 ts_tablet_manager.cc:1905] T 65d9bbd8c15f40a4a50c1695fb5e648e P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:46.307091 2837 tablet.cc:1721] T 019eb21c661547cfa0b619d119eb024e P 1186fa869eb14812832fa6562531f196: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20250113 02:51:46.310138 653 catalog_manager.cc:5989] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Tablet ee09d2b74c05432c9d7917b036bf8898 (table default.Table_6a88830e770641c4bbe405c0ef700d67 [id=ba1edf2b734e48cb89a77ca59fa717c6]) was not created within the allowed timeout. Replacing with a new tablet a11377e635b044e58a730fda3c25ae27
I20250113 02:51:46.310254 2603 fs_manager.cc:730] Time spent opening block manager: real 0.428s user 0.012s sys 0.017s
I20250113 02:51:46.313158 2723 raft_consensus.cc:695] T 2470b010b571444eba3f132323038f33 P 1186fa869eb14812832fa6562531f196 [term 1 LEADER]: Becoming Leader. State: Replica: 1186fa869eb14812832fa6562531f196, State: Running, Role: LEADER
I20250113 02:51:46.313565 2723 consensus_queue.cc:237] T 2470b010b571444eba3f132323038f33 P 1186fa869eb14812832fa6562531f196 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
W20250113 02:51:46.315622 653 catalog_manager.cc:5989] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Tablet eb4a985261f4452db4de74891b7bf65a (table default.Table_e38f2c9d348948238d245b93b82b0dce [id=7caed6f6020f49a5898a2261f653070d]) was not created within the allowed timeout. Replacing with a new tablet ec0fb729e3a14d50a23faf81c8082f1e
W20250113 02:51:46.339223 442 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.104.2:38467, user_credentials={real_user=slave}} blocked reactor thread for 260650us
W20250113 02:51:46.341822 998 consensus_peers.cc:487] T f2c6c8c17d6e499d84ad7c0c23889cc4 P 4216a7f853d642b7bc3e5cff458f3aca -> Peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Couldn't send request to peer df69f48656b44bb890f76587ee69cc0f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250113 02:51:46.358634 2827 consensus_queue.cc:1035] T b217cb8c09fb41cab4f53e605c37efe8 P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 02:51:46.362983 1192 tablet_bootstrap.cc:654] T b3ce6d851f564b849796f174860aacd5 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:46.375434 2849 tablet.cc:1721] T 938cc1ccd8dd449996836188ba9e64ff P df69f48656b44bb890f76587ee69cc0f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250113 02:51:46.376080 1191 tablet_bootstrap.cc:492] T 80c647a3b5a1411e9130461115c53550 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:46.376242 1191 ts_tablet_manager.cc:1397] T 80c647a3b5a1411e9130461115c53550 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.103s user 0.000s sys 0.040s
I20250113 02:51:46.310678 2603 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/master-2/data,/tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/master-2/wal
uuid: "45e9d43ecac0401fb2c18fe5eaf089b5"
format_stamp: "Formatted at 2025-01-13 02:51:13 on dist-test-slave-v0t5"
I20250113 02:51:46.377076 2603 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 02:51:46.377561 1191 raft_consensus.cc:357] T 80c647a3b5a1411e9130461115c53550 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:46.377811 1191 raft_consensus.cc:383] T 80c647a3b5a1411e9130461115c53550 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:46.377876 1191 raft_consensus.cc:738] T 80c647a3b5a1411e9130461115c53550 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:46.380440 1191 consensus_queue.cc:260] T 80c647a3b5a1411e9130461115c53550 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:46.381790 1191 ts_tablet_manager.cc:1428] T 80c647a3b5a1411e9130461115c53550 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
W20250113 02:51:46.354249 440 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 59a447c6d0264ec38390fe688bc9555a (table default.Table_d467386709824eb68cc352a120812d3e [id=a59f6a19ab9541d7bf386d352f1962ac], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 59a447c6d0264ec38390fe688bc9555a consensus state has no leader
I20250113 02:51:46.380849 454 catalog_manager.cc:5618] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467) does not have the latest schema for tablet 938cc1ccd8dd449996836188ba9e64ff (table default.Table_e5b4d614e4b743d7982b025ca6d297e5 [id=5824acd778ed47d09f087bbf8e7822d6]). Expected version 1 got 0
I20250113 02:51:46.399994 1191 tablet_bootstrap.cc:492] T 29c17fd90d774c029a35e77888358067 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:46.400969 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet 438604e8859f422f9caccb61c5b5b125 (table default.Table_2cb862a84f3f44c5bcfb646461a5af1c [id=c6e65e4bab5040cca1f0f55721639d2e]) successfully deleted
I20250113 02:51:46.406713 1191 tablet_bootstrap.cc:654] T 29c17fd90d774c029a35e77888358067 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
W20250113 02:51:46.410547 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 2cd3b62ecc774d2eb658ed34e05864e1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2cd3b62ecc774d2eb658ed34e05864e1 already in progress: creating tablet
I20250113 02:51:46.411365 1195 ts_tablet_manager.cc:1918] T 438d1d2300e044f39a15586f335935ba P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250113 02:51:46.411634 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet 1d2fdc75d00f4e749d322ac6cd5fc410 (table default.Table_3fa4f9f57be84226a01935b139545f6e [id=0ab8e0c80dda42fcb07f2fd6fbf01609]) successfully deleted
I20250113 02:51:46.411885 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet 153b8f90b5a847b88944f683d306f579 (table default.Table_a7c3697e25a94815ac03a3cb1a57bc0c [id=39d8412c30d24c44a01c24b3c9645a8d]) successfully deleted
I20250113 02:51:46.412161 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet 3d85539cf0074a03b167d3279e940079 (table default.Table_3a73b4672b154b99af1a24e962007ea0 [id=f742014cd0d44db99c4c1a031630d827]) successfully deleted
I20250113 02:51:46.412361 1195 log.cc:1198] T 438d1d2300e044f39a15586f335935ba P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/438d1d2300e044f39a15586f335935ba
I20250113 02:51:46.413138 1195 ts_tablet_manager.cc:1939] T 438d1d2300e044f39a15586f335935ba P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:46.413372 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet dc680b5680b84abbab219296bc7ee707 (table default.Table_25a2c280183748df92b064c40756f5ac [id=a1ac1dc79bce47258f1dbf33837b782c]) successfully deleted
W20250113 02:51:46.413671 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 65139dcfe89249ec95f5425b294f04d8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 65139dcfe89249ec95f5425b294f04d8 already in progress: creating tablet
I20250113 02:51:46.414705 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1320) took 5365 ms. Trace:
W20250113 02:51:46.415009 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet f29d7a9504d740528914fe6a66fd4028 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f29d7a9504d740528914fe6a66fd4028 already in progress: creating tablet
I20250113 02:51:46.417666 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_8a69c12869c44953bde2307180dc1a88"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:46.418381 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8a69c12869c44953bde2307180dc1a88 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:46.423638 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_ea862ea580c043c5bd2f56522b97897d" } new_table_name: "default.Table_a3eb3dedc42f4da8b4e5e7489d21cfa9" modify_external_catalogs: true
I20250113 02:51:46.425272 1198 ts_tablet_manager.cc:1918] T 65d9bbd8c15f40a4a50c1695fb5e648e P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250113 02:51:46.426219 1198 log.cc:1198] T 65d9bbd8c15f40a4a50c1695fb5e648e P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/65d9bbd8c15f40a4a50c1695fb5e648e
I20250113 02:51:46.426682 1198 ts_tablet_manager.cc:1939] T 65d9bbd8c15f40a4a50c1695fb5e648e P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:46.426962 1192 tablet_bootstrap.cc:492] T b3ce6d851f564b849796f174860aacd5 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:46.427112 1192 ts_tablet_manager.cc:1397] T b3ce6d851f564b849796f174860aacd5 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.141s user 0.014s sys 0.010s
I20250113 02:51:46.427892 1192 raft_consensus.cc:357] T b3ce6d851f564b849796f174860aacd5 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:46.432097 1192 raft_consensus.cc:383] T b3ce6d851f564b849796f174860aacd5 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:46.432582 1192 raft_consensus.cc:738] T b3ce6d851f564b849796f174860aacd5 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:46.432871 1192 consensus_queue.cc:260] T b3ce6d851f564b849796f174860aacd5 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
W20250113 02:51:46.433902 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f748397a164d42e3a7fda3585673e633 consensus state has no leader
I20250113 02:51:46.434396 1192 ts_tablet_manager.cc:1428] T b3ce6d851f564b849796f174860aacd5 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20250113 02:51:46.434759 1192 tablet_bootstrap.cc:492] T c04be975c8c049b892b16129ed47a142 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:46.433032 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1469) took 1246 ms. Trace:
I20250113 02:51:46.436546 1198 rpcz_store.cc:276] 0113 02:51:45.186963 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:45.187006 (+ 43us) service_pool.cc:225] Handling call
0113 02:51:46.024638 (+837632us) ts_tablet_manager.cc:1145] Deleting tablet 65d9bbd8c15f40a4a50c1695fb5e648e
0113 02:51:46.024648 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:46.345737 (+321089us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:46.425154 (+ 79417us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:46.433012 (+ 7858us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":837491}
I20250113 02:51:46.437072 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet 65d9bbd8c15f40a4a50c1695fb5e648e (table default.Table_1df1ad89ec084bd4928ccfa8376b0bc6 [id=32ea1ffb81004a9db8e67189bea55237]) successfully deleted
I20250113 02:51:46.437402 1198 tablet_replica.cc:331] T b217cb8c09fb41cab4f53e605c37efe8 P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:46.437687 1198 raft_consensus.cc:2238] T b217cb8c09fb41cab4f53e605c37efe8 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: Raft consensus shutting down.
I20250113 02:51:46.438086 1198 raft_consensus.cc:2267] T b217cb8c09fb41cab4f53e605c37efe8 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:46.438329 1192 tablet_bootstrap.cc:654] T c04be975c8c049b892b16129ed47a142 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:46.438465 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_1e0ca8b977914d1aaf79792357ea51a7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:46.439431 1195 rpcz_store.cc:276] 0113 02:51:41.049642 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:41.187577 (+137935us) service_pool.cc:225] Handling call
0113 02:51:46.266943 (+5079366us) ts_tablet_manager.cc:1145] Deleting tablet 438d1d2300e044f39a15586f335935ba
0113 02:51:46.266951 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:46.313780 (+ 46829us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:46.411266 (+ 97486us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:46.414680 (+ 3414us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5075078}
I20250113 02:51:46.440584 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1313) took 5409 ms. Trace:
I20250113 02:51:46.440786 1195 rpcz_store.cc:276] 0113 02:51:41.030978 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:41.168893 (+137915us) service_pool.cc:225] Handling call
0113 02:51:46.440344 (+5271451us) ts_tablet_manager.cc:1145] Deleting tablet aef2ffd67def48eea501a98fbc9dc43b
0113 02:51:46.440355 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:46.440542 (+ 187us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5244157}
I20250113 02:51:46.441303 1195 tablet_replica.cc:331] T 01a455eda0ee488fb3d45a6a09c43833 P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:46.441371 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet 438d1d2300e044f39a15586f335935ba (table default.Table_d7d1f5a34e5f4c05b474b0b61f912cd9 [id=2c7925ec0d8d4cd7b8761268d3962f15]) successfully deleted
W20250113 02:51:46.441596 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet aef2ffd67def48eea501a98fbc9dc43b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet aef2ffd67def48eea501a98fbc9dc43b already in progress: creating tablet
I20250113 02:51:46.441671 1195 raft_consensus.cc:2238] T 01a455eda0ee488fb3d45a6a09c43833 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:46.441918 1195 raft_consensus.cc:2267] T 01a455eda0ee488fb3d45a6a09c43833 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:46.443354 1194 tablet_bootstrap.cc:492] T c13e9a51c1e74b44b352bade78390ba5 P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:46.443496 1194 ts_tablet_manager.cc:1397] T c13e9a51c1e74b44b352bade78390ba5 P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.227s user 0.020s sys 0.062s
I20250113 02:51:46.443439 1195 ts_tablet_manager.cc:1905] T 01a455eda0ee488fb3d45a6a09c43833 P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:46.444193 1194 raft_consensus.cc:357] T c13e9a51c1e74b44b352bade78390ba5 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:46.444367 1194 raft_consensus.cc:383] T c13e9a51c1e74b44b352bade78390ba5 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:46.444416 1194 raft_consensus.cc:738] T c13e9a51c1e74b44b352bade78390ba5 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:46.444608 1194 consensus_queue.cc:260] T c13e9a51c1e74b44b352bade78390ba5 P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:46.444778 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_2cdb0197edfc4223a764898c3f226cd8" } new_table_name: "default.Table_8e5d26dfdb7e4f2289e792e637df2407" modify_external_catalogs: true
I20250113 02:51:46.445773 1194 ts_tablet_manager.cc:1428] T c13e9a51c1e74b44b352bade78390ba5 P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250113 02:51:46.446080 1194 tablet_bootstrap.cc:492] T acd83024592e431898e9675c17f50d4f P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:46.447747 1194 tablet_bootstrap.cc:654] T acd83024592e431898e9675c17f50d4f P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
W20250113 02:51:46.451764 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7944d77262ae4b99b6735cf9ca0593f6 has no consensus state
I20250113 02:51:46.452921 1198 ts_tablet_manager.cc:1905] T b217cb8c09fb41cab4f53e605c37efe8 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:46.458066 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_a01645da1d1b4ba7852c3986e29de0c1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:46.458948 1198 ts_tablet_manager.cc:1918] T b217cb8c09fb41cab4f53e605c37efe8 P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250113 02:51:46.459100 1198 log.cc:1198] T b217cb8c09fb41cab4f53e605c37efe8 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/b217cb8c09fb41cab4f53e605c37efe8
I20250113 02:51:46.459524 1198 ts_tablet_manager.cc:1939] T b217cb8c09fb41cab4f53e605c37efe8 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:46.461266 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1470) took 1273 ms. Trace:
I20250113 02:51:46.461429 1198 rpcz_store.cc:276] 0113 02:51:45.187469 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:45.187516 (+ 47us) service_pool.cc:225] Handling call
0113 02:51:46.437356 (+1249840us) ts_tablet_manager.cc:1145] Deleting tablet b217cb8c09fb41cab4f53e605c37efe8
0113 02:51:46.437366 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:46.455927 (+ 18561us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:46.458865 (+ 2938us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:46.461244 (+ 2379us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1249683}
I20250113 02:51:46.462023 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet b217cb8c09fb41cab4f53e605c37efe8 (table default.Table_3ed64b1d864b4b79bdaef850c30c486b [id=f8180eb6217846b6be7819aab2edaf99]) successfully deleted
W20250113 02:51:46.464108 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a01645da1d1b4ba7852c3986e29de0c1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250113 02:51:46.466676 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1e0ca8b977914d1aaf79792357ea51a7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:46.468860 1194 tablet_bootstrap.cc:492] T acd83024592e431898e9675c17f50d4f P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:46.469039 1194 ts_tablet_manager.cc:1397] T acd83024592e431898e9675c17f50d4f P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.023s user 0.002s sys 0.003s
I20250113 02:51:46.469218 455 catalog_manager.cc:5618] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467) does not have the latest schema for tablet c13e9a51c1e74b44b352bade78390ba5 (table default.Table_0468611ec68e47109f5357935e50874d [id=caf098e2b57849a997bd7c8eaf1f9326]). Expected version 1 got 0
I20250113 02:51:46.469810 1194 raft_consensus.cc:357] T acd83024592e431898e9675c17f50d4f P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:46.469990 1194 raft_consensus.cc:383] T acd83024592e431898e9675c17f50d4f P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:46.470041 1194 raft_consensus.cc:738] T acd83024592e431898e9675c17f50d4f P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:46.470266 1194 consensus_queue.cc:260] T acd83024592e431898e9675c17f50d4f P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:46.470968 1194 ts_tablet_manager.cc:1428] T acd83024592e431898e9675c17f50d4f P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20250113 02:51:46.473803 1192 tablet_bootstrap.cc:492] T c04be975c8c049b892b16129ed47a142 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:46.473986 1192 ts_tablet_manager.cc:1397] T c04be975c8c049b892b16129ed47a142 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.039s user 0.005s sys 0.011s
I20250113 02:51:46.474750 1192 raft_consensus.cc:357] T c04be975c8c049b892b16129ed47a142 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:46.475000 1192 raft_consensus.cc:383] T c04be975c8c049b892b16129ed47a142 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:46.475100 1192 raft_consensus.cc:738] T c04be975c8c049b892b16129ed47a142 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:46.475437 1192 consensus_queue.cc:260] T c04be975c8c049b892b16129ed47a142 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:46.476907 1198 tablet_replica.cc:331] T 88a776aa820c44b3a3854ca5022bd353 P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:46.477089 1198 raft_consensus.cc:2238] T 88a776aa820c44b3a3854ca5022bd353 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: Raft consensus shutting down.
I20250113 02:51:46.477394 1198 raft_consensus.cc:2267] T 88a776aa820c44b3a3854ca5022bd353 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:46.433929 1191 tablet_bootstrap.cc:492] T 29c17fd90d774c029a35e77888358067 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:46.480567 1191 ts_tablet_manager.cc:1397] T 29c17fd90d774c029a35e77888358067 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.081s user 0.003s sys 0.007s
I20250113 02:51:46.481393 1191 raft_consensus.cc:357] T 29c17fd90d774c029a35e77888358067 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:46.481653 1191 raft_consensus.cc:383] T 29c17fd90d774c029a35e77888358067 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:46.481756 1191 raft_consensus.cc:738] T 29c17fd90d774c029a35e77888358067 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:46.482692 2827 raft_consensus.cc:491] T c04be975c8c049b892b16129ed47a142 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:46.482836 2827 raft_consensus.cc:513] T c04be975c8c049b892b16129ed47a142 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:46.483088 453 master_service.cc:946] ReplaceTablet: received request to replace tablet 4fe54131aafb40ee9d8bc5ea299af193 from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:46.483525 2827 leader_election.cc:290] T c04be975c8c049b892b16129ed47a142 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:46.484551 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_d5096557ae6445e6bcd3b6a9b3299e8b" } modify_external_catalogs: true
I20250113 02:51:46.484716 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_d5096557ae6445e6bcd3b6a9b3299e8b" } modify_external_catalogs: true
I20250113 02:51:46.484826 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c04be975c8c049b892b16129ed47a142" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
I20250113 02:51:46.484992 769 raft_consensus.cc:2463] T c04be975c8c049b892b16129ed47a142 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 0.
I20250113 02:51:46.485311 995 leader_election.cc:304] T c04be975c8c049b892b16129ed47a142 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters:
I20250113 02:51:46.485682 2768 raft_consensus.cc:2798] T c04be975c8c049b892b16129ed47a142 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 02:51:46.485754 2768 raft_consensus.cc:491] T c04be975c8c049b892b16129ed47a142 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 02:51:46.485809 2768 raft_consensus.cc:3054] T c04be975c8c049b892b16129ed47a142 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:46.486028 455 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_8cf41dedb3ca4e848b36d1e6e19e5126" } modify_external_catalogs: true
I20250113 02:51:46.486173 455 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_8cf41dedb3ca4e848b36d1e6e19e5126" } modify_external_catalogs: true
I20250113 02:51:46.486488 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_eaf6851fc514494a95392ba107626e41"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:46.486663 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_8a46b6886be646d48357f268651520c6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:46.486989 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_eaf6851fc514494a95392ba107626e41 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250113 02:51:46.487102 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8a46b6886be646d48357f268651520c6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:46.488273 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_1b00bd87905943a480ae8c6344417112" } modify_external_catalogs: true
I20250113 02:51:46.488435 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_1b00bd87905943a480ae8c6344417112" } modify_external_catalogs: true
I20250113 02:51:46.488582 896 tablet_service.cc:1514] Processing DeleteTablet for tablet 82505f1389184d0ca12ec0e2d6084d22 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:27 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:46.488582 882 tablet_service.cc:1514] Processing DeleteTablet for tablet 476c89cc585b4e00947f5d1e8d0229f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:45 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:46.488757 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_0532c8897e3a478ca8602439e3752e72"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:46.489176 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0532c8897e3a478ca8602439e3752e72 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:46.490469 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_643c47db97214c82861277c1935d5032"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:46.490953 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_643c47db97214c82861277c1935d5032 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:46.490998 1198 ts_tablet_manager.cc:1905] T 88a776aa820c44b3a3854ca5022bd353 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:46.494598 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c04be975c8c049b892b16129ed47a142" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:46.495440 998 leader_election.cc:343] T c04be975c8c049b892b16129ed47a142 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:46.497144 1196 tablet_replica.cc:331] T 82505f1389184d0ca12ec0e2d6084d22 P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:46.497336 1196 raft_consensus.cc:2238] T 82505f1389184d0ca12ec0e2d6084d22 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:46.497478 1196 raft_consensus.cc:2267] T 82505f1389184d0ca12ec0e2d6084d22 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:46.498176 1196 ts_tablet_manager.cc:1905] T 82505f1389184d0ca12ec0e2d6084d22 P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:46.498528 2768 raft_consensus.cc:513] T c04be975c8c049b892b16129ed47a142 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:46.499516 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c04be975c8c049b892b16129ed47a142" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196"
I20250113 02:51:46.499645 653 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet ee09d2b74c05432c9d7917b036bf8898
I20250113 02:51:46.499693 769 raft_consensus.cc:3054] T c04be975c8c049b892b16129ed47a142 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:46.502240 769 raft_consensus.cc:2463] T c04be975c8c049b892b16129ed47a142 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 1.
I20250113 02:51:46.502650 995 leader_election.cc:304] T c04be975c8c049b892b16129ed47a142 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters:
I20250113 02:51:46.502861 2772 raft_consensus.cc:491] T f4379f7fc6b54e6785f27975d9030cce P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:46.502956 2772 raft_consensus.cc:513] T f4379f7fc6b54e6785f27975d9030cce P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:46.503485 2772 leader_election.cc:290] T f4379f7fc6b54e6785f27975d9030cce P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:46.503604 2772 raft_consensus.cc:2798] T c04be975c8c049b892b16129ed47a142 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Leader election won for term 1
I20250113 02:51:46.503692 2772 raft_consensus.cc:695] T c04be975c8c049b892b16129ed47a142 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: Becoming Leader. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Running, Role: LEADER
I20250113 02:51:46.503799 751 tablet_service.cc:1514] Processing DeleteTablet for tablet ee09d2b74c05432c9d7917b036bf8898 with delete_type TABLET_DATA_DELETED (Replaced by a11377e635b044e58a730fda3c25ae27 at 2025-01-13 02:51:46 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:46.503916 2772 consensus_queue.cc:237] T c04be975c8c049b892b16129ed47a142 P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:46.504366 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet ee09d2b74c05432c9d7917b036bf8898 with delete_type TABLET_DATA_DELETED (Replaced by a11377e635b044e58a730fda3c25ae27 at 2025-01-13 02:51:46 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:46.504796 1196 ts_tablet_manager.cc:1918] T 82505f1389184d0ca12ec0e2d6084d22 P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:46.504962 1196 log.cc:1198] T 82505f1389184d0ca12ec0e2d6084d22 P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/82505f1389184d0ca12ec0e2d6084d22
I20250113 02:51:46.504609 449 catalog_manager.cc:5526] T 68c4a7e8c0fb4feb85498709b00d3258 P 4216a7f853d642b7bc3e5cff458f3aca reported cstate change: term changed from 0 to 1, leader changed from <none> to 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3). New cstate: current_term: 1 leader_uuid: "4216a7f853d642b7bc3e5cff458f3aca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } health_report { overall_health: UNKNOWN } } }
I20250113 02:51:46.505354 449 catalog_manager.cc:5618] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587) does not have the latest schema for tablet 68c4a7e8c0fb4feb85498709b00d3258 (table default.Table_c90f3a5c35c842f6ad2cb74cf29dac13 [id=8d2f84cd670a4e8d9fa17b8313d3fab4]). Expected version 2 got 0
I20250113 02:51:46.505429 1196 ts_tablet_manager.cc:1939] T 82505f1389184d0ca12ec0e2d6084d22 P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:46.505951 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4379f7fc6b54e6785f27975d9030cce" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:46.506284 447 catalog_manager.cc:6071] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: default.Table_022a7c61a6864393acd739f562893b7c [id=97bfbd46eecd4e57b7f9757c413f6795] alter complete (version 1)
I20250113 02:51:46.506470 1194 tablet_bootstrap.cc:492] T 3f62f344c5a648328ca2de8ca1069c19 P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:46.506520 653 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet eb4a985261f4452db4de74891b7bf65a
I20250113 02:51:46.507190 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet eb4a985261f4452db4de74891b7bf65a with delete_type TABLET_DATA_DELETED (Replaced by ec0fb729e3a14d50a23faf81c8082f1e at 2025-01-13 02:51:46 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:46.507190 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4379f7fc6b54e6785f27975d9030cce" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
W20250113 02:51:46.508419 998 leader_election.cc:343] T f4379f7fc6b54e6785f27975d9030cce P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:46.509205 1034 tablet_service.cc:1467] Processing CreateTablet for tablet 873988747b7549fe89f1229e1303d3cf (DEFAULT_TABLE table=default.Table_f193626bef644c699ec47d1e88580317 [id=85d29efa591a4ca9bc369fcf5f04f660]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:46.509624 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 873988747b7549fe89f1229e1303d3cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:46.507413 775 raft_consensus.cc:2463] T f4379f7fc6b54e6785f27975d9030cce P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 0.
I20250113 02:51:46.511682 751 tablet_service.cc:1514] Processing DeleteTablet for tablet eb4a985261f4452db4de74891b7bf65a with delete_type TABLET_DATA_DELETED (Replaced by ec0fb729e3a14d50a23faf81c8082f1e at 2025-01-13 02:51:46 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:46.511817 995 leader_election.cc:304] T f4379f7fc6b54e6785f27975d9030cce P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters: df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:46.512235 2827 raft_consensus.cc:2798] T f4379f7fc6b54e6785f27975d9030cce P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 02:51:46.512308 2827 raft_consensus.cc:491] T f4379f7fc6b54e6785f27975d9030cce P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 02:51:46.512382 2827 raft_consensus.cc:3054] T f4379f7fc6b54e6785f27975d9030cce P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:46.513936 1194 tablet_bootstrap.cc:654] T 3f62f344c5a648328ca2de8ca1069c19 P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:46.515297 2827 raft_consensus.cc:513] T f4379f7fc6b54e6785f27975d9030cce P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:46.516794 2827 leader_election.cc:290] T f4379f7fc6b54e6785f27975d9030cce P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Requested vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:46.517042 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4379f7fc6b54e6785f27975d9030cce" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f"
W20250113 02:51:46.517532 998 leader_election.cc:343] T f4379f7fc6b54e6785f27975d9030cce P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:46.518232 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4379f7fc6b54e6785f27975d9030cce" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196"
I20250113 02:51:46.518422 772 raft_consensus.cc:3054] T f4379f7fc6b54e6785f27975d9030cce P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:46.518878 751 tablet_service.cc:1467] Processing CreateTablet for tablet 873988747b7549fe89f1229e1303d3cf (DEFAULT_TABLE table=default.Table_f193626bef644c699ec47d1e88580317 [id=85d29efa591a4ca9bc369fcf5f04f660]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:46.519223 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 873988747b7549fe89f1229e1303d3cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:46.519445 1024 tablet_service.cc:1467] Processing CreateTablet for tablet f46a1fddc0d94ddfa41946d5252d70f3 (DEFAULT_TABLE table=default.Table_4b6d8da7a53240dab5cd6d788c17e913 [id=14efb26c5802415487699f2be72a2e25]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:46.519573 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c04be975c8c049b892b16129ed47a142" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f"
I20250113 02:51:46.519773 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f46a1fddc0d94ddfa41946d5252d70f3. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:46.520021 998 leader_election.cc:343] T c04be975c8c049b892b16129ed47a142 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:46.520643 2768 leader_election.cc:290] T c04be975c8c049b892b16129ed47a142 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Requested vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:46.520924 772 raft_consensus.cc:2463] T f4379f7fc6b54e6785f27975d9030cce P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 1.
I20250113 02:51:46.521463 995 leader_election.cc:304] T f4379f7fc6b54e6785f27975d9030cce P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters: df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:46.522670 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 4fe54131aafb40ee9d8bc5ea299af193
I20250113 02:51:46.523119 453 catalog_manager.cc:6596] ReplaceTablet: tablet 4fe54131aafb40ee9d8bc5ea299af193 deleted and replaced by tablet a35558e3f1fa4a36a174adde70d7cd44
I20250113 02:51:46.521797 2768 raft_consensus.cc:2798] T f4379f7fc6b54e6785f27975d9030cce P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Leader election won for term 1
I20250113 02:51:46.525524 2768 raft_consensus.cc:695] T f4379f7fc6b54e6785f27975d9030cce P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: Becoming Leader. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Running, Role: LEADER
I20250113 02:51:46.526064 2768 consensus_queue.cc:237] T f4379f7fc6b54e6785f27975d9030cce P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:46.526590 748 tablet_service.cc:1514] Processing DeleteTablet for tablet 4fe54131aafb40ee9d8bc5ea299af193 with delete_type TABLET_DATA_DELETED (replaced by tablet a35558e3f1fa4a36a174adde70d7cd44) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:46.526489 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet 4fe54131aafb40ee9d8bc5ea299af193 with delete_type TABLET_DATA_DELETED (replaced by tablet a35558e3f1fa4a36a174adde70d7cd44) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:46.527431 748 tablet_service.cc:1467] Processing CreateTablet for tablet f46a1fddc0d94ddfa41946d5252d70f3 (DEFAULT_TABLE table=default.Table_4b6d8da7a53240dab5cd6d788c17e913 [id=14efb26c5802415487699f2be72a2e25]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:46.527786 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f46a1fddc0d94ddfa41946d5252d70f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:46.528007 1192 ts_tablet_manager.cc:1428] T c04be975c8c049b892b16129ed47a142 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.054s user 0.001s sys 0.001s
I20250113 02:51:46.528306 1034 tablet_service.cc:1467] Processing CreateTablet for tablet baf54f3ef2264e3391699a52aac2f4d6 (DEFAULT_TABLE table=default.Table_a272ebee897d42c99c9c5012355291de [id=86c2f92dbd1643dd9cedaa6e94ac137b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:46.528494 1192 tablet_bootstrap.cc:492] T a2781ba5061a4225b5636366857a229e P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:46.528684 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet baf54f3ef2264e3391699a52aac2f4d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:46.533739 1194 tablet_bootstrap.cc:492] T 3f62f344c5a648328ca2de8ca1069c19 P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:46.533939 1194 ts_tablet_manager.cc:1397] T 3f62f344c5a648328ca2de8ca1069c19 P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.028s user 0.004s sys 0.000s
I20250113 02:51:46.534718 1194 raft_consensus.cc:357] T 3f62f344c5a648328ca2de8ca1069c19 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:46.534976 1194 raft_consensus.cc:383] T 3f62f344c5a648328ca2de8ca1069c19 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:46.535086 1194 raft_consensus.cc:738] T 3f62f344c5a648328ca2de8ca1069c19 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:46.535383 1198 ts_tablet_manager.cc:1918] T 88a776aa820c44b3a3854ca5022bd353 P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250113 02:51:46.535496 1198 log.cc:1198] T 88a776aa820c44b3a3854ca5022bd353 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/88a776aa820c44b3a3854ca5022bd353
I20250113 02:51:46.535522 1194 consensus_queue.cc:260] T 3f62f344c5a648328ca2de8ca1069c19 P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:46.535852 1198 ts_tablet_manager.cc:1939] T 88a776aa820c44b3a3854ca5022bd353 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:46.536836 1194 ts_tablet_manager.cc:1428] T 3f62f344c5a648328ca2de8ca1069c19 P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250113 02:51:46.536931 748 tablet_service.cc:1467] Processing CreateTablet for tablet baf54f3ef2264e3391699a52aac2f4d6 (DEFAULT_TABLE table=default.Table_a272ebee897d42c99c9c5012355291de [id=86c2f92dbd1643dd9cedaa6e94ac137b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:46.537163 1194 tablet_bootstrap.cc:492] T 9f8eb43a39774d2c8e9cb4e88a405b0f P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:46.537318 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet baf54f3ef2264e3391699a52aac2f4d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:46.537534 1024 tablet_service.cc:1467] Processing CreateTablet for tablet 9ac00c087be84f758af6af2a895c86e1 (DEFAULT_TABLE table=default.Table_f0d0db84fe564a8ab3368977ba13d3f8 [id=5bf22f304b3142f08894c54789804673]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:46.538008 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ac00c087be84f758af6af2a895c86e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:46.539644 1194 tablet_bootstrap.cc:654] T 9f8eb43a39774d2c8e9cb4e88a405b0f P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:46.541510 451 master_service.cc:946] ReplaceTablet: received request to replace tablet 348c21d783ef4103a37409ef8634eb35 from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:46.543071 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_31180c034b5f487e9f6880d11e0239ee"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:46.544384 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_8530e7723ac8458d9990e3c5c6286447"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:46.544795 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8530e7723ac8458d9990e3c5c6286447 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:46.544831 453 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_5893047163024e76bd537c27b9b10458" } modify_external_catalogs: true
I20250113 02:51:46.545032 1192 tablet_bootstrap.cc:654] T a2781ba5061a4225b5636366857a229e P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:46.544991 453 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_5893047163024e76bd537c27b9b10458" } modify_external_catalogs: true
I20250113 02:51:46.545969 454 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_720678f42d2849259edbacc2f1a9c24b" } new_table_name: "default.Table_47d4611ba2bc4d9a9f169e377bc16ddb" modify_external_catalogs: true
I20250113 02:51:46.547113 742 tablet_service.cc:1467] Processing CreateTablet for tablet 9ac00c087be84f758af6af2a895c86e1 (DEFAULT_TABLE table=default.Table_f0d0db84fe564a8ab3368977ba13d3f8 [id=5bf22f304b3142f08894c54789804673]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:46.547456 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ac00c087be84f758af6af2a895c86e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:46.548449 736 tablet_service.cc:1467] Processing CreateTablet for tablet c4883512d8974e3d9b3bbecd80479b76 (DEFAULT_TABLE table=default.Table_b88ada2171ff424baba4ef155ad02b02 [id=f7235431acf14aa8942bb7277cea67b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:46.548764 736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c4883512d8974e3d9b3bbecd80479b76. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:46.548753 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_777e191d02f54883afa306010890a105" } new_table_name: "default.Table_daf614ed470045c4a6c29cf45b5ef178" modify_external_catalogs: true
I20250113 02:51:46.549904 1039 tablet_service.cc:1467] Processing CreateTablet for tablet c4883512d8974e3d9b3bbecd80479b76 (DEFAULT_TABLE table=default.Table_b88ada2171ff424baba4ef155ad02b02 [id=f7235431acf14aa8942bb7277cea67b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:46.550230 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c4883512d8974e3d9b3bbecd80479b76. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:46.554522 2769 raft_consensus.cc:491] T afcb5b1977c447d3a51de9cb72d7196a P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:46.554639 2769 raft_consensus.cc:513] T afcb5b1977c447d3a51de9cb72d7196a P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:46.555210 2769 leader_election.cc:290] T afcb5b1977c447d3a51de9cb72d7196a P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:46.555794 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afcb5b1977c447d3a51de9cb72d7196a" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:46.556345 998 leader_election.cc:343] T afcb5b1977c447d3a51de9cb72d7196a P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
W20250113 02:51:46.556511 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_31180c034b5f487e9f6880d11e0239ee in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:46.556893 2476 raft_consensus.cc:491] T b38cd34d67834a3f9bf5b5636bb79217 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:46.557017 2476 raft_consensus.cc:513] T b38cd34d67834a3f9bf5b5636bb79217 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:46.557756 2476 leader_election.cc:290] T b38cd34d67834a3f9bf5b5636bb79217 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:46.558305 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b38cd34d67834a3f9bf5b5636bb79217" candidate_uuid: "df69f48656b44bb890f76587ee69cc0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
I20250113 02:51:46.559881 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afcb5b1977c447d3a51de9cb72d7196a" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
I20250113 02:51:46.481999 1191 consensus_queue.cc:260] T 29c17fd90d774c029a35e77888358067 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:46.560099 772 raft_consensus.cc:2463] T afcb5b1977c447d3a51de9cb72d7196a P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 0.
I20250113 02:51:46.560578 1028 tablet_service.cc:1467] Processing CreateTablet for tablet c7f540efce9a4e84b13c4d41b25722f0 (DEFAULT_TABLE table=default.Table_469a767a55b84a8b930b57666c06575b [id=088397900049451e9a3681e151dbc0ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:46.560899 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c7f540efce9a4e84b13c4d41b25722f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:46.561484 995 leader_election.cc:304] T afcb5b1977c447d3a51de9cb72d7196a P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters: df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:46.561800 2769 raft_consensus.cc:2798] T afcb5b1977c447d3a51de9cb72d7196a P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 02:51:46.561859 2769 raft_consensus.cc:491] T afcb5b1977c447d3a51de9cb72d7196a P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 02:51:46.561910 2769 raft_consensus.cc:3054] T afcb5b1977c447d3a51de9cb72d7196a P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:46.564159 1192 tablet_bootstrap.cc:492] T a2781ba5061a4225b5636366857a229e P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:46.564280 1192 ts_tablet_manager.cc:1397] T a2781ba5061a4225b5636366857a229e P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.036s user 0.001s sys 0.004s
I20250113 02:51:46.564306 2769 raft_consensus.cc:513] T afcb5b1977c447d3a51de9cb72d7196a P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:46.564834 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b38cd34d67834a3f9bf5b5636bb79217" candidate_uuid: "df69f48656b44bb890f76587ee69cc0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:46.565008 1039 tablet_service.cc:1467] Processing CreateTablet for tablet 71b38c7d3ae7419a9a20f68c43517ecf (DEFAULT_TABLE table=default.Table_c2fe3752f4a1482fb35693231429d1df [id=30719be151404bf4a53902e9d5c04538]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:46.565330 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 71b38c7d3ae7419a9a20f68c43517ecf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:46.565429 2769 leader_election.cc:290] T afcb5b1977c447d3a51de9cb72d7196a P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Requested vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
W20250113 02:51:46.565553 998 consensus_peers.cc:487] T 569d666c128a45b7a526c13be5090ae3 P 4216a7f853d642b7bc3e5cff458f3aca -> Peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Couldn't send request to peer df69f48656b44bb890f76587ee69cc0f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250113 02:51:46.565722 1191 ts_tablet_manager.cc:1428] T 29c17fd90d774c029a35e77888358067 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.085s user 0.001s sys 0.001s
I20250113 02:51:46.565987 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afcb5b1977c447d3a51de9cb72d7196a" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f"
I20250113 02:51:46.566161 1191 tablet_bootstrap.cc:492] T ed68a2b7ebcf4841bef54798dd3ed80d P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
W20250113 02:51:46.566710 998 leader_election.cc:343] T afcb5b1977c447d3a51de9cb72d7196a P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:46.570533 852 leader_election.cc:304] T b38cd34d67834a3f9bf5b5636bb79217 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df69f48656b44bb890f76587ee69cc0f; no voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca
I20250113 02:51:46.570899 2476 raft_consensus.cc:2743] T b38cd34d67834a3f9bf5b5636bb79217 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:46.571096 882 tablet_service.cc:1514] Processing DeleteTablet for tablet 1968c7e9357c4c6e901d334ace0467fb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:26 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:46.571285 1196 tablet_replica.cc:331] T 1968c7e9357c4c6e901d334ace0467fb P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:46.571468 1196 raft_consensus.cc:2238] T 1968c7e9357c4c6e901d334ace0467fb P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:46.571604 1196 raft_consensus.cc:2267] T 1968c7e9357c4c6e901d334ace0467fb P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:46.572486 1196 ts_tablet_manager.cc:1905] T 1968c7e9357c4c6e901d334ace0467fb P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:46.575562 1194 tablet_bootstrap.cc:492] T 9f8eb43a39774d2c8e9cb4e88a405b0f P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:46.575680 1194 ts_tablet_manager.cc:1397] T 9f8eb43a39774d2c8e9cb4e88a405b0f P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.039s user 0.005s sys 0.000s
I20250113 02:51:46.575807 1028 tablet_service.cc:1467] Processing CreateTablet for tablet 7944d77262ae4b99b6735cf9ca0593f6 (DEFAULT_TABLE table=default.Table_8e5d26dfdb7e4f2289e792e637df2407 [id=35cb9d91202e42e0b4ad1beb4d995d0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:46.576187 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7944d77262ae4b99b6735cf9ca0593f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:46.576478 1194 raft_consensus.cc:357] T 9f8eb43a39774d2c8e9cb4e88a405b0f P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:46.576596 751 tablet_service.cc:1467] Processing CreateTablet for tablet 71b38c7d3ae7419a9a20f68c43517ecf (DEFAULT_TABLE table=default.Table_c2fe3752f4a1482fb35693231429d1df [id=30719be151404bf4a53902e9d5c04538]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:46.576779 1194 raft_consensus.cc:383] T 9f8eb43a39774d2c8e9cb4e88a405b0f P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:46.576884 1194 raft_consensus.cc:738] T 9f8eb43a39774d2c8e9cb4e88a405b0f P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:46.577152 1194 consensus_queue.cc:260] T 9f8eb43a39774d2c8e9cb4e88a405b0f P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:46.577546 736 tablet_service.cc:1467] Processing CreateTablet for tablet c7f540efce9a4e84b13c4d41b25722f0 (DEFAULT_TABLE table=default.Table_469a767a55b84a8b930b57666c06575b [id=088397900049451e9a3681e151dbc0ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:46.578377 1194 ts_tablet_manager.cc:1428] T 9f8eb43a39774d2c8e9cb4e88a405b0f P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250113 02:51:46.578981 1194 tablet_bootstrap.cc:492] T 67baaab1e36e415381579d7385d4ab5a P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:46.580641 1194 tablet_bootstrap.cc:654] T 67baaab1e36e415381579d7385d4ab5a P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:46.579947 1196 ts_tablet_manager.cc:1918] T 1968c7e9357c4c6e901d334ace0467fb P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:46.584671 1192 raft_consensus.cc:357] T a2781ba5061a4225b5636366857a229e P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:46.584867 1196 log.cc:1198] T 1968c7e9357c4c6e901d334ace0467fb P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/1968c7e9357c4c6e901d334ace0467fb
I20250113 02:51:46.585343 1196 ts_tablet_manager.cc:1939] T 1968c7e9357c4c6e901d334ace0467fb P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:46.588744 882 tablet_service.cc:1514] Processing DeleteTablet for tablet ee09d2b74c05432c9d7917b036bf8898 with delete_type TABLET_DATA_DELETED (Replaced by a11377e635b044e58a730fda3c25ae27 at 2025-01-13 02:51:46 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:46.589241 1192 raft_consensus.cc:383] T a2781ba5061a4225b5636366857a229e P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:46.589216 742 tablet_service.cc:1467] Processing CreateTablet for tablet 7944d77262ae4b99b6735cf9ca0593f6 (DEFAULT_TABLE table=default.Table_8e5d26dfdb7e4f2289e792e637df2407 [id=35cb9d91202e42e0b4ad1beb4d995d0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:46.589437 1192 raft_consensus.cc:738] T a2781ba5061a4225b5636366857a229e P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:46.589622 749 tablet_service.cc:1467] Processing CreateTablet for tablet f46c28be32b24a0ba697629ea17b39c8 (DEFAULT_TABLE table=default.Table_880629d9242b4654a306a4b3843a636a [id=a0fc70e132f441db878c2dc0421158de]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:46.589672 1024 tablet_service.cc:1467] Processing CreateTablet for tablet f46c28be32b24a0ba697629ea17b39c8 (DEFAULT_TABLE table=default.Table_880629d9242b4654a306a4b3843a636a [id=a0fc70e132f441db878c2dc0421158de]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:46.589682 1192 consensus_queue.cc:260] T a2781ba5061a4225b5636366857a229e P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:46.590029 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f46c28be32b24a0ba697629ea17b39c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:46.591542 1032 tablet_service.cc:1467] Processing CreateTablet for tablet dbbcb086a31a49d3b32367da67de0760 (DEFAULT_TABLE table=default.Table_da4eeeb904054a9aa6807387f76b1d8f [id=5763436c8e6140aa99af2a28f45bb17d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:46.591636 1192 ts_tablet_manager.cc:1428] T a2781ba5061a4225b5636366857a229e P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.026s user 0.002s sys 0.000s
I20250113 02:51:46.597800 1194 tablet_bootstrap.cc:492] T 67baaab1e36e415381579d7385d4ab5a P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:46.586982 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1466) took 1401 ms. Trace:
I20250113 02:51:46.598023 1194 ts_tablet_manager.cc:1397] T 67baaab1e36e415381579d7385d4ab5a P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.000s
I20250113 02:51:46.598040 1198 rpcz_store.cc:276] 0113 02:51:45.185753 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:45.188789 (+ 3036us) service_pool.cc:225] Handling call
0113 02:51:46.462412 (+1273623us) ts_tablet_manager.cc:1145] Deleting tablet 88a776aa820c44b3a3854ca5022bd353
0113 02:51:46.476862 (+ 14450us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:46.510914 (+ 34052us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:46.535283 (+ 24369us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:46.586962 (+ 51679us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1273373}
I20250113 02:51:46.600260 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet 88a776aa820c44b3a3854ca5022bd353 (table default.Table_6d2c74a667ab4954a8380355f54d648f [id=5f6159d685c54a29b39e171bb1b2b24a]) successfully deleted
I20250113 02:51:46.600520 1194 raft_consensus.cc:357] T 67baaab1e36e415381579d7385d4ab5a P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:46.600808 1194 raft_consensus.cc:383] T 67baaab1e36e415381579d7385d4ab5a P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:46.600916 1194 raft_consensus.cc:738] T 67baaab1e36e415381579d7385d4ab5a P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:46.601245 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afcb5b1977c447d3a51de9cb72d7196a" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196"
I20250113 02:51:46.601449 772 raft_consensus.cc:3054] T afcb5b1977c447d3a51de9cb72d7196a P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:46.601715 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 71b38c7d3ae7419a9a20f68c43517ecf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:46.601625 1194 consensus_queue.cc:260] T 67baaab1e36e415381579d7385d4ab5a P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:46.602718 1194 ts_tablet_manager.cc:1428] T 67baaab1e36e415381579d7385d4ab5a P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250113 02:51:46.603317 1194 tablet_bootstrap.cc:492] T e570465a3d4046b1ac19a3a034fe91f1 P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:46.604241 772 raft_consensus.cc:2463] T afcb5b1977c447d3a51de9cb72d7196a P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 1.
I20250113 02:51:46.605086 736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c7f540efce9a4e84b13c4d41b25722f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:46.606010 1194 tablet_bootstrap.cc:654] T e570465a3d4046b1ac19a3a034fe91f1 P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:46.608176 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f46c28be32b24a0ba697629ea17b39c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:46.611181 1194 tablet_bootstrap.cc:492] T e570465a3d4046b1ac19a3a034fe91f1 P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:46.611295 1194 ts_tablet_manager.cc:1397] T e570465a3d4046b1ac19a3a034fe91f1 P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.002s
I20250113 02:51:46.611963 1194 raft_consensus.cc:357] T e570465a3d4046b1ac19a3a034fe91f1 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:46.612180 1194 raft_consensus.cc:383] T e570465a3d4046b1ac19a3a034fe91f1 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:46.612231 1194 raft_consensus.cc:738] T e570465a3d4046b1ac19a3a034fe91f1 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:46.612247 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dbbcb086a31a49d3b32367da67de0760. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:46.612411 1194 consensus_queue.cc:260] T e570465a3d4046b1ac19a3a034fe91f1 P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:46.613435 1194 ts_tablet_manager.cc:1428] T e570465a3d4046b1ac19a3a034fe91f1 P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250113 02:51:46.613611 744 tablet_service.cc:1467] Processing CreateTablet for tablet dbbcb086a31a49d3b32367da67de0760 (DEFAULT_TABLE table=default.Table_da4eeeb904054a9aa6807387f76b1d8f [id=5763436c8e6140aa99af2a28f45bb17d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:46.614063 1194 tablet_bootstrap.cc:492] T 283fc7f1c74e480c8a63c74d3ae9eb7d P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:46.615367 750 tablet_service.cc:1467] Processing CreateTablet for tablet 48fdb58340a742f680ed6e62a7eca532 (DEFAULT_TABLE table=default.Table_9a6c1e2713d74f7186906076d1f1212e [id=49984a2b4b434405a43eafed5cd441cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:46.616044 746 tablet_service.cc:1467] Processing CreateTablet for tablet af8d36e1b0584aeeb7fb651454a3efc8 (DEFAULT_TABLE table=default.Table_22efb860611f468c98e2b9df40e31e83 [id=0829928cf7e9474789fc2fdac9a49f90]), partition=RANGE (key) PARTITION UNBOUNDED
W20250113 02:51:46.616504 709 consensus_peers.cc:487] T 883027967e0c442a91ac820210b9c6c7 P 1186fa869eb14812832fa6562531f196 -> Peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Couldn't send request to peer 4216a7f853d642b7bc3e5cff458f3aca. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 883027967e0c442a91ac820210b9c6c7. This is attempt 1: this message will repeat every 5th retry.
I20250113 02:51:46.618530 745 tablet_service.cc:1467] Processing CreateTablet for tablet 1041048c522742dbbca710038b92ec15 (DEFAULT_TABLE table=default.Table_6441bd25946b4757af31d88201d6e3f2 [id=5b2f13bde21d4c87a609c980a55ab13c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:46.618850 741 tablet_service.cc:1467] Processing CreateTablet for tablet 0a378ce2d64e44d1a1088e705d439b9c (DEFAULT_TABLE table=default.Table_36b33b166f834edbb7a465fc17ac3720 [id=1e8097bb261649a7b8af4917a9a3ca16]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:46.612268 1192 tablet_bootstrap.cc:492] T 8f2fb58fc24d4692a3fe9a652e076aed P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:46.619403 754 tablet_service.cc:1467] Processing CreateTablet for tablet f33f59dee0f7489eb6e555c3d062a0f6 (DEFAULT_TABLE table=default.Table_41ea2489c272425292a6a6562964690f [id=33fe2300ba2c49559c1bc4b0e543e4f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:46.622754 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dbbcb086a31a49d3b32367da67de0760. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:46.628186 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 48fdb58340a742f680ed6e62a7eca532. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:46.629120 1028 tablet_service.cc:1467] Processing CreateTablet for tablet 1041048c522742dbbca710038b92ec15 (DEFAULT_TABLE table=default.Table_6441bd25946b4757af31d88201d6e3f2 [id=5b2f13bde21d4c87a609c980a55ab13c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:46.629249 995 leader_election.cc:304] T afcb5b1977c447d3a51de9cb72d7196a P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters: df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:46.629297 1039 tablet_service.cc:1467] Processing CreateTablet for tablet 0a378ce2d64e44d1a1088e705d439b9c (DEFAULT_TABLE table=default.Table_36b33b166f834edbb7a465fc17ac3720 [id=1e8097bb261649a7b8af4917a9a3ca16]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:46.629451 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1041048c522742dbbca710038b92ec15. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:46.629706 2772 raft_consensus.cc:2798] T afcb5b1977c447d3a51de9cb72d7196a P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Leader election won for term 1
I20250113 02:51:46.629810 2772 raft_consensus.cc:695] T afcb5b1977c447d3a51de9cb72d7196a P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: Becoming Leader. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Running, Role: LEADER
I20250113 02:51:46.630028 2772 consensus_queue.cc:237] T afcb5b1977c447d3a51de9cb72d7196a P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:46.630699 1034 tablet_service.cc:1467] Processing CreateTablet for tablet af8d36e1b0584aeeb7fb651454a3efc8 (DEFAULT_TABLE table=default.Table_22efb860611f468c98e2b9df40e31e83 [id=0829928cf7e9474789fc2fdac9a49f90]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:46.631008 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af8d36e1b0584aeeb7fb651454a3efc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:46.631528 1024 tablet_service.cc:1467] Processing CreateTablet for tablet 48fdb58340a742f680ed6e62a7eca532 (DEFAULT_TABLE table=default.Table_9a6c1e2713d74f7186906076d1f1212e [id=49984a2b4b434405a43eafed5cd441cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:46.631796 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 48fdb58340a742f680ed6e62a7eca532. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:46.628060 1194 tablet_bootstrap.cc:654] T 283fc7f1c74e480c8a63c74d3ae9eb7d P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:46.632118 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0a378ce2d64e44d1a1088e705d439b9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:46.632861 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af8d36e1b0584aeeb7fb651454a3efc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:46.634106 1035 tablet_service.cc:1467] Processing CreateTablet for tablet f33f59dee0f7489eb6e555c3d062a0f6 (DEFAULT_TABLE table=default.Table_41ea2489c272425292a6a6562964690f [id=33fe2300ba2c49559c1bc4b0e543e4f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:46.634397 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f33f59dee0f7489eb6e555c3d062a0f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:46.634583 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1472) took 1386 ms. Trace:
I20250113 02:51:46.634654 1198 rpcz_store.cc:276] 0113 02:51:45.248626 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:45.248696 (+ 70us) service_pool.cc:225] Handling call
0113 02:51:46.598580 (+1349884us) ts_tablet_manager.cc:1145] Deleting tablet be2ca8d2ac6547189a0edeff12413330
0113 02:51:46.623679 (+ 25099us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:46.634569 (+ 10890us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1349547}
W20250113 02:51:46.635046 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet be2ca8d2ac6547189a0edeff12413330 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet be2ca8d2ac6547189a0edeff12413330 already in progress: creating tablet
I20250113 02:51:46.640216 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7944d77262ae4b99b6735cf9ca0593f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:46.644224 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1041048c522742dbbca710038b92ec15. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:46.648200 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0a378ce2d64e44d1a1088e705d439b9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:46.648592 914 raft_consensus.cc:3054] T 3f62f344c5a648328ca2de8ca1069c19 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:46.652242 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f33f59dee0f7489eb6e555c3d062a0f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:46.654989 2476 raft_consensus.cc:491] T acd83024592e431898e9675c17f50d4f P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:46.655128 2476 raft_consensus.cc:513] T acd83024592e431898e9675c17f50d4f P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:46.655889 2476 leader_election.cc:290] T acd83024592e431898e9675c17f50d4f P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587), 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095)
W20250113 02:51:46.657213 851 leader_election.cc:343] T acd83024592e431898e9675c17f50d4f P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Not found: Tablet not found: acd83024592e431898e9675c17f50d4f
W20250113 02:51:46.657563 852 leader_election.cc:343] T acd83024592e431898e9675c17f50d4f P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Not found: Tablet not found: acd83024592e431898e9675c17f50d4f
I20250113 02:51:46.657668 852 leader_election.cc:304] T acd83024592e431898e9675c17f50d4f P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df69f48656b44bb890f76587ee69cc0f; no voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca
I20250113 02:51:46.657999 2476 raft_consensus.cc:2743] T acd83024592e431898e9675c17f50d4f P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:46.658926 882 tablet_service.cc:1514] Processing DeleteTablet for tablet eb4a985261f4452db4de74891b7bf65a with delete_type TABLET_DATA_DELETED (Replaced by ec0fb729e3a14d50a23faf81c8082f1e at 2025-01-13 02:51:46 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:46.659732 1194 tablet_bootstrap.cc:492] T 283fc7f1c74e480c8a63c74d3ae9eb7d P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:46.659868 1194 ts_tablet_manager.cc:1397] T 283fc7f1c74e480c8a63c74d3ae9eb7d P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.046s user 0.012s sys 0.002s
I20250113 02:51:46.660727 1194 raft_consensus.cc:357] T 283fc7f1c74e480c8a63c74d3ae9eb7d P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:46.660893 1194 raft_consensus.cc:383] T 283fc7f1c74e480c8a63c74d3ae9eb7d P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:46.660944 1194 raft_consensus.cc:738] T 283fc7f1c74e480c8a63c74d3ae9eb7d P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:46.661175 1194 consensus_queue.cc:260] T 283fc7f1c74e480c8a63c74d3ae9eb7d P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:46.662202 1194 ts_tablet_manager.cc:1428] T 283fc7f1c74e480c8a63c74d3ae9eb7d P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250113 02:51:46.662444 1194 tablet_bootstrap.cc:492] T 6a4c6dc08df744e6856bdf99ab65c575 P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:46.662448 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 9572215fd2eb45faa99cc236fcc0d5ea
I20250113 02:51:46.663722 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 905542fb0dce4bbb8852b74d94c7b2a6
I20250113 02:51:46.664012 1194 tablet_bootstrap.cc:654] T 6a4c6dc08df744e6856bdf99ab65c575 P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:46.664373 452 master_service.cc:946] ReplaceTablet: received request to replace tablet cb897004cbfb4a0283b42d06509a9ee5 from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:46.664772 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet b946dd2d428a43d399bb75a43d843a28
I20250113 02:51:46.665180 450 master_service.cc:946] ReplaceTablet: received request to replace tablet e51b750440a24570b734bb51e7a6f923 from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:46.665937 736 tablet_service.cc:1514] Processing DeleteTablet for tablet 905542fb0dce4bbb8852b74d94c7b2a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:46 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:46.667436 2768 consensus_queue.cc:1035] T 569d666c128a45b7a526c13be5090ae3 P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 02:51:46.671881 446 catalog_manager.cc:5618] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095) does not have the latest schema for tablet f2c6c8c17d6e499d84ad7c0c23889cc4 (table default.Table_92ea541283614df981d96c5b74362f1d [id=9e58548d1a184cc0b362f148c44005ed]). Expected version 2 got 0
I20250113 02:51:46.672098 742 tablet_service.cc:1514] Processing DeleteTablet for tablet b946dd2d428a43d399bb75a43d843a28 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:46 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:46.673463 1192 tablet_bootstrap.cc:654] T 8f2fb58fc24d4692a3fe9a652e076aed P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:46.674301 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1473) took 1425 ms. Trace:
I20250113 02:51:46.674396 1198 rpcz_store.cc:276] 0113 02:51:45.248793 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:45.248867 (+ 74us) service_pool.cc:225] Handling call
0113 02:51:46.635317 (+1386450us) ts_tablet_manager.cc:1145] Deleting tablet c7da2c406e64426faf123c4bddfaf213
0113 02:51:46.635327 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:46.674281 (+ 38954us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1386188}
I20250113 02:51:46.676533 742 tablet_service.cc:1514] Processing DeleteTablet for tablet 95e47db8a4d74615800ba6c049ec7131 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:27 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:46.677073 744 tablet_service.cc:1467] Processing CreateTablet for tablet e9f7aa2f369f4aad9561deb4f1ca06f3 (DEFAULT_TABLE table=default.Table_28f9de083c724932a0a03761e6148593 [id=baeeedacf6e94931b1201a333b5fdf5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:46.677628 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9f7aa2f369f4aad9561deb4f1ca06f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:46.680095 745 tablet_service.cc:1514] Processing DeleteTablet for tablet 80c647a3b5a1411e9130461115c53550 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:29 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:46.681977 741 tablet_service.cc:1514] Processing DeleteTablet for tablet 9572215fd2eb45faa99cc236fcc0d5ea with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:46 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:46.682037 1032 tablet_service.cc:1514] Processing DeleteTablet for tablet 9572215fd2eb45faa99cc236fcc0d5ea with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:46 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:46.684672 2368 raft_consensus.cc:491] T 247af1a17df04b27898f86642336d51e P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:46.684857 2368 raft_consensus.cc:513] T 247af1a17df04b27898f86642336d51e P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:46.685746 2368 leader_election.cc:290] T 247af1a17df04b27898f86642336d51e P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:46.686298 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "247af1a17df04b27898f86642336d51e" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:46.686882 998 leader_election.cc:343] T 247af1a17df04b27898f86642336d51e P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:46.689641 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1478) took 1295 ms. Trace:
I20250113 02:51:46.689739 1198 rpcz_store.cc:276] 0113 02:51:45.393713 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:45.393769 (+ 56us) service_pool.cc:225] Handling call
0113 02:51:46.674634 (+1280865us) ts_tablet_manager.cc:1145] Deleting tablet 28000f78e5dd408c8de58c674ad95f6c
0113 02:51:46.674645 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:46.689622 (+ 14977us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1280605}
I20250113 02:51:46.690061 1198 tablet_replica.cc:331] T 569d666c128a45b7a526c13be5090ae3 P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:46.690255 1198 raft_consensus.cc:2238] T 569d666c128a45b7a526c13be5090ae3 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: Raft consensus shutting down.
I20250113 02:51:46.690522 1198 raft_consensus.cc:2267] T 569d666c128a45b7a526c13be5090ae3 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:46.691367 1198 ts_tablet_manager.cc:1905] T 569d666c128a45b7a526c13be5090ae3 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:46.692827 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "247af1a17df04b27898f86642336d51e" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
I20250113 02:51:46.693024 772 raft_consensus.cc:2463] T 247af1a17df04b27898f86642336d51e P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 0.
I20250113 02:51:46.693295 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet 905542fb0dce4bbb8852b74d94c7b2a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:46 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:46.693611 995 leader_election.cc:304] T 247af1a17df04b27898f86642336d51e P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters: df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:46.694203 2829 raft_consensus.cc:2798] T 247af1a17df04b27898f86642336d51e P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 02:51:46.694293 2829 raft_consensus.cc:491] T 247af1a17df04b27898f86642336d51e P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 02:51:46.694387 2829 raft_consensus.cc:3054] T 247af1a17df04b27898f86642336d51e P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:46.701370 1035 tablet_service.cc:1467] Processing CreateTablet for tablet e9f7aa2f369f4aad9561deb4f1ca06f3 (DEFAULT_TABLE table=default.Table_28f9de083c724932a0a03761e6148593 [id=baeeedacf6e94931b1201a333b5fdf5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:46.701762 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9f7aa2f369f4aad9561deb4f1ca06f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:46.701831 882 tablet_service.cc:1467] Processing CreateTablet for tablet 873988747b7549fe89f1229e1303d3cf (DEFAULT_TABLE table=default.Table_f193626bef644c699ec47d1e88580317 [id=85d29efa591a4ca9bc369fcf5f04f660]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:46.702303 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 873988747b7549fe89f1229e1303d3cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:46.707731 1036 tablet_service.cc:1467] Processing CreateTablet for tablet 132eda23315c4af1820a0c5433cdc7c1 (DEFAULT_TABLE table=default.Table_4fcb7bb3569b4eddafedff2f3dec10f8 [id=4a3910c2627a44d98e927b08a65599e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:46.707901 1030 tablet_service.cc:1514] Processing DeleteTablet for tablet f4379f7fc6b54e6785f27975d9030cce with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:31 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:46.708109 1036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 132eda23315c4af1820a0c5433cdc7c1. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:46.708328 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet c7da2c406e64426faf123c4bddfaf213 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c7da2c406e64426faf123c4bddfaf213 already in progress: creating tablet
I20250113 02:51:46.708360 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet b946dd2d428a43d399bb75a43d843a28 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:46 UTC) from {username='slave'} at 127.0.0.1:60230
W20250113 02:51:46.708729 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 28000f78e5dd408c8de58c674ad95f6c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 28000f78e5dd408c8de58c674ad95f6c already in progress: creating tablet
I20250113 02:51:46.709406 1032 tablet_service.cc:1467] Processing CreateTablet for tablet ea7b44be353f4ab1b2ac8d1b9916382f (DEFAULT_TABLE table=default.Table_7f54426882784252bf8e26b7028d253a [id=616a5b5350d1455e8d928d8e95694c21]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:46.709807 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea7b44be353f4ab1b2ac8d1b9916382f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:46.717880 2829 raft_consensus.cc:513] T 247af1a17df04b27898f86642336d51e P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:46.718729 2829 leader_election.cc:290] T 247af1a17df04b27898f86642336d51e P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Requested vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:46.718989 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "247af1a17df04b27898f86642336d51e" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196"
I20250113 02:51:46.719205 769 raft_consensus.cc:3054] T 247af1a17df04b27898f86642336d51e P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:46.719110 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "247af1a17df04b27898f86642336d51e" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f"
W20250113 02:51:46.719801 998 leader_election.cc:343] T 247af1a17df04b27898f86642336d51e P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:46.721741 769 raft_consensus.cc:2463] T 247af1a17df04b27898f86642336d51e P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 1.
I20250113 02:51:46.722209 995 leader_election.cc:304] T 247af1a17df04b27898f86642336d51e P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters: df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:46.722630 2829 raft_consensus.cc:2798] T 247af1a17df04b27898f86642336d51e P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Leader election won for term 1
I20250113 02:51:46.722837 2829 raft_consensus.cc:695] T 247af1a17df04b27898f86642336d51e P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: Becoming Leader. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Running, Role: LEADER
I20250113 02:51:46.723090 2829 consensus_queue.cc:237] T 247af1a17df04b27898f86642336d51e P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:46.726594 1198 ts_tablet_manager.cc:1918] T 569d666c128a45b7a526c13be5090ae3 P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250113 02:51:46.730980 1198 log.cc:1198] T 569d666c128a45b7a526c13be5090ae3 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/569d666c128a45b7a526c13be5090ae3
I20250113 02:51:46.731690 1198 ts_tablet_manager.cc:1939] T 569d666c128a45b7a526c13be5090ae3 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:46.732064 1191 tablet_bootstrap.cc:700] T ed68a2b7ebcf4841bef54798dd3ed80d P 1186fa869eb14812832fa6562531f196: Time spent opening tablet: real 0.154s user 0.001s sys 0.000s
I20250113 02:51:46.732939 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1483) took 1178 ms. Trace:
I20250113 02:51:46.733424 882 tablet_service.cc:1467] Processing CreateTablet for tablet f46a1fddc0d94ddfa41946d5252d70f3 (DEFAULT_TABLE table=default.Table_4b6d8da7a53240dab5cd6d788c17e913 [id=14efb26c5802415487699f2be72a2e25]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:46.734309 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f46a1fddc0d94ddfa41946d5252d70f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:46.735486 1191 tablet_bootstrap.cc:654] T ed68a2b7ebcf4841bef54798dd3ed80d P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:46.735374 1198 rpcz_store.cc:276] 0113 02:51:45.554044 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:45.554107 (+ 63us) service_pool.cc:225] Handling call
0113 02:51:46.690011 (+1135904us) ts_tablet_manager.cc:1145] Deleting tablet 569d666c128a45b7a526c13be5090ae3
0113 02:51:46.690024 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:46.711777 (+ 21753us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:46.726509 (+ 14732us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:46.732925 (+ 6416us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1101574}
I20250113 02:51:46.735960 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1485) took 1181 ms. Trace:
I20250113 02:51:46.736061 1198 rpcz_store.cc:276] 0113 02:51:45.554308 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:45.554366 (+ 58us) service_pool.cc:225] Handling call
0113 02:51:46.735856 (+1181490us) ts_tablet_manager.cc:1145] Deleting tablet 4c759831773849f2bc52e484ffa0cf49
0113 02:51:46.735867 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:46.735944 (+ 77us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1147374}
I20250113 02:51:46.736356 1198 tablet_replica.cc:331] T 9b2d0aa6f0e9456d95f4e48c360ab6e8 P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:46.736553 1198 raft_consensus.cc:2238] T 9b2d0aa6f0e9456d95f4e48c360ab6e8 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: Raft consensus shutting down.
I20250113 02:51:46.736608 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet 569d666c128a45b7a526c13be5090ae3 (table default.Table_349bd0d1105d42e4889232adbacc4d3a [id=f10700c3143d4a6c9f4f45508db7fa93]) successfully deleted
W20250113 02:51:46.736843 440 catalog_manager.cc:4839] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 4c759831773849f2bc52e484ffa0cf49 because the tablet was not found. No further retry: Not found: Tablet not found: 4c759831773849f2bc52e484ffa0cf49
I20250113 02:51:46.736876 1198 raft_consensus.cc:2267] T 9b2d0aa6f0e9456d95f4e48c360ab6e8 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:46.742451 1198 ts_tablet_manager.cc:1905] T 9b2d0aa6f0e9456d95f4e48c360ab6e8 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:46.735260 896 tablet_service.cc:1514] Processing DeleteTablet for tablet 4fe54131aafb40ee9d8bc5ea299af193 with delete_type TABLET_DATA_DELETED (replaced by tablet a35558e3f1fa4a36a174adde70d7cd44) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:46.743366 1196 tablet_replica.cc:331] T 4fe54131aafb40ee9d8bc5ea299af193 P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:46.743568 1196 raft_consensus.cc:2238] T 4fe54131aafb40ee9d8bc5ea299af193 P df69f48656b44bb890f76587ee69cc0f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:46.743770 1196 raft_consensus.cc:2267] T 4fe54131aafb40ee9d8bc5ea299af193 P df69f48656b44bb890f76587ee69cc0f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:46.744288 1196 ts_tablet_manager.cc:1905] T 4fe54131aafb40ee9d8bc5ea299af193 P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:46.742578 885 tablet_service.cc:1467] Processing CreateTablet for tablet baf54f3ef2264e3391699a52aac2f4d6 (DEFAULT_TABLE table=default.Table_a272ebee897d42c99c9c5012355291de [id=86c2f92dbd1643dd9cedaa6e94ac137b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:46.743235 745 tablet_service.cc:1467] Processing CreateTablet for tablet 0d8cdaa36b424a53b0501a65f4e2ce7d (DEFAULT_TABLE table=default.Table_3fda809831ba43319e2ee62fb49dde92 [id=d3e184260dd8493d9ba55ad9729ace32]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:46.744253 741 tablet_service.cc:1467] Processing CreateTablet for tablet 132eda23315c4af1820a0c5433cdc7c1 (DEFAULT_TABLE table=default.Table_4fcb7bb3569b4eddafedff2f3dec10f8 [id=4a3910c2627a44d98e927b08a65599e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:46.745437 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet baf54f3ef2264e3391699a52aac2f4d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:46.747303 1032 tablet_service.cc:1467] Processing CreateTablet for tablet e5eb5247d3704e7aae0d4e08dc19ae5d (DEFAULT_TABLE table=default.Table_6ea04b79ee2c446c986176fbd2cad259 [id=3bb015e2f6df4fa9b4f048debad038af]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:46.747684 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5eb5247d3704e7aae0d4e08dc19ae5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:46.750994 742 tablet_service.cc:1467] Processing CreateTablet for tablet e5eb5247d3704e7aae0d4e08dc19ae5d (DEFAULT_TABLE table=default.Table_6ea04b79ee2c446c986176fbd2cad259 [id=3bb015e2f6df4fa9b4f048debad038af]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:46.752023 1024 tablet_service.cc:1467] Processing CreateTablet for tablet b4e03b52495d46f8be3f1544e5fda5e9 (DEFAULT_TABLE table=default.Table_d7c36054e0c84d26a64741fa4e91cd8e [id=a7082e6637444da1ab32067db8bcd4f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:46.752342 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b4e03b52495d46f8be3f1544e5fda5e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:46.754568 1039 tablet_service.cc:1467] Processing CreateTablet for tablet 0d8cdaa36b424a53b0501a65f4e2ce7d (DEFAULT_TABLE table=default.Table_3fda809831ba43319e2ee62fb49dde92 [id=d3e184260dd8493d9ba55ad9729ace32]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:46.754840 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d8cdaa36b424a53b0501a65f4e2ce7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:46.755214 1196 ts_tablet_manager.cc:1918] T 4fe54131aafb40ee9d8bc5ea299af193 P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20250113 02:51:46.760497 1196 log.cc:1198] T 4fe54131aafb40ee9d8bc5ea299af193 P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/4fe54131aafb40ee9d8bc5ea299af193
I20250113 02:51:46.760905 1196 ts_tablet_manager.cc:1939] T 4fe54131aafb40ee9d8bc5ea299af193 P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:46.761410 744 tablet_service.cc:1467] Processing CreateTablet for tablet ea7b44be353f4ab1b2ac8d1b9916382f (DEFAULT_TABLE table=default.Table_7f54426882784252bf8e26b7028d253a [id=616a5b5350d1455e8d928d8e95694c21]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:46.764770 885 tablet_service.cc:1467] Processing CreateTablet for tablet 9ac00c087be84f758af6af2a895c86e1 (DEFAULT_TABLE table=default.Table_f0d0db84fe564a8ab3368977ba13d3f8 [id=5bf22f304b3142f08894c54789804673]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:46.765168 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ac00c087be84f758af6af2a895c86e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:46.765419 754 tablet_service.cc:1467] Processing CreateTablet for tablet b4e03b52495d46f8be3f1544e5fda5e9 (DEFAULT_TABLE table=default.Table_d7c36054e0c84d26a64741fa4e91cd8e [id=a7082e6637444da1ab32067db8bcd4f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:46.766971 1024 tablet_service.cc:1467] Processing CreateTablet for tablet 69e200d941164e21848c5cfc5fbcd8f6 (DEFAULT_TABLE table=default.Table_d2fb23238827478fbe0d1afa409ba711 [id=2f4f6b0d225641d38ae54e47eadc12c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:46.767424 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69e200d941164e21848c5cfc5fbcd8f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:46.769495 1192 tablet_bootstrap.cc:492] T 8f2fb58fc24d4692a3fe9a652e076aed P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:46.769626 1192 ts_tablet_manager.cc:1397] T 8f2fb58fc24d4692a3fe9a652e076aed P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.157s user 0.003s sys 0.011s
I20250113 02:51:46.770404 1192 raft_consensus.cc:357] T 8f2fb58fc24d4692a3fe9a652e076aed P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:46.770517 736 tablet_service.cc:1467] Processing CreateTablet for tablet 69e200d941164e21848c5cfc5fbcd8f6 (DEFAULT_TABLE table=default.Table_d2fb23238827478fbe0d1afa409ba711 [id=2f4f6b0d225641d38ae54e47eadc12c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:46.770586 1192 raft_consensus.cc:383] T 8f2fb58fc24d4692a3fe9a652e076aed P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:46.770633 1192 raft_consensus.cc:738] T 8f2fb58fc24d4692a3fe9a652e076aed P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:46.770846 1192 consensus_queue.cc:260] T 8f2fb58fc24d4692a3fe9a652e076aed P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:46.771358 1192 ts_tablet_manager.cc:1428] T 8f2fb58fc24d4692a3fe9a652e076aed P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250113 02:51:46.771622 1192 tablet_bootstrap.cc:492] T 6001369444894b48b51a48fb60ddc52a P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:46.773164 1192 tablet_bootstrap.cc:654] T 6001369444894b48b51a48fb60ddc52a P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:46.776055 750 tablet_service.cc:1467] Processing CreateTablet for tablet 9bfecc0884a64b4583506f6a050820f0 (DEFAULT_TABLE table=default.Table_28e6f68bf4304b12b107c8e71e66e1e3 [id=e9f4feaeaee04ae3a223b3762b87da66]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:46.776496 1198 ts_tablet_manager.cc:1918] T 9b2d0aa6f0e9456d95f4e48c360ab6e8 P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250113 02:51:46.776456 1028 tablet_service.cc:1467] Processing CreateTablet for tablet 52e2b698832844baa0edfe732909e075 (DEFAULT_TABLE table=default.Table_f166593d31ee4c56965f2c6eda65615b [id=7e5d042f6f81474093c3128db8d6cdf1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:46.776777 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 52e2b698832844baa0edfe732909e075. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:46.776898 1036 tablet_service.cc:1467] Processing CreateTablet for tablet 9bfecc0884a64b4583506f6a050820f0 (DEFAULT_TABLE table=default.Table_28e6f68bf4304b12b107c8e71e66e1e3 [id=e9f4feaeaee04ae3a223b3762b87da66]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:46.777313 1036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9bfecc0884a64b4583506f6a050820f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:46.777945 746 tablet_service.cc:1467] Processing CreateTablet for tablet 52e2b698832844baa0edfe732909e075 (DEFAULT_TABLE table=default.Table_f166593d31ee4c56965f2c6eda65615b [id=7e5d042f6f81474093c3128db8d6cdf1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:46.779541 749 tablet_service.cc:1467] Processing CreateTablet for tablet 766d0c959e5a4c969c75828e2523cfd1 (DEFAULT_TABLE table=default.Table_9f1f5d4e13b44704a2e7edf981461d71 [id=46af1be1bcee41e4baabe573cf885c56]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:46.780409 1035 tablet_service.cc:1467] Processing CreateTablet for tablet 766d0c959e5a4c969c75828e2523cfd1 (DEFAULT_TABLE table=default.Table_9f1f5d4e13b44704a2e7edf981461d71 [id=46af1be1bcee41e4baabe573cf885c56]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:46.780601 1034 tablet_service.cc:1467] Processing CreateTablet for tablet 9e02c3d5511e47f59c38c8e6e3cb51d4 (DEFAULT_TABLE table=default.Table_3e1f7bdb9ecf47d0baa280c94babfc41 [id=b359b53f229d4214976fb303d0228ba2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:46.780714 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 766d0c959e5a4c969c75828e2523cfd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:46.780848 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e02c3d5511e47f59c38c8e6e3cb51d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:46.785828 896 tablet_service.cc:1467] Processing CreateTablet for tablet c4883512d8974e3d9b3bbecd80479b76 (DEFAULT_TABLE table=default.Table_b88ada2171ff424baba4ef155ad02b02 [id=f7235431acf14aa8942bb7277cea67b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:46.786289 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c4883512d8974e3d9b3bbecd80479b76. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:46.793355 1198 log.cc:1198] T 9b2d0aa6f0e9456d95f4e48c360ab6e8 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/9b2d0aa6f0e9456d95f4e48c360ab6e8
I20250113 02:51:46.793745 1198 ts_tablet_manager.cc:1939] T 9b2d0aa6f0e9456d95f4e48c360ab6e8 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:46.794341 2854 raft_consensus.cc:491] T 29c17fd90d774c029a35e77888358067 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:46.794493 738 tablet_service.cc:1467] Processing CreateTablet for tablet 9e02c3d5511e47f59c38c8e6e3cb51d4 (DEFAULT_TABLE table=default.Table_3e1f7bdb9ecf47d0baa280c94babfc41 [id=b359b53f229d4214976fb303d0228ba2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:46.794530 2854 raft_consensus.cc:513] T 29c17fd90d774c029a35e77888358067 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:46.794737 2853 raft_consensus.cc:491] T 7527e6e0b65e4a43b7f48899a34cdf7f P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4216a7f853d642b7bc3e5cff458f3aca)
I20250113 02:51:46.794868 2853 raft_consensus.cc:513] T 7527e6e0b65e4a43b7f48899a34cdf7f P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:46.795454 2853 leader_election.cc:290] T 7527e6e0b65e4a43b7f48899a34cdf7f P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
W20250113 02:51:46.796311 709 leader_election.cc:343] T 7527e6e0b65e4a43b7f48899a34cdf7f P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Not found: Tablet not found: 7527e6e0b65e4a43b7f48899a34cdf7f
I20250113 02:51:46.797098 743 tablet_service.cc:1467] Processing CreateTablet for tablet 391615b7658d44dea2a72a7bd1656856 (DEFAULT_TABLE table=default.Table_a9e430e1146e40d7945c1680ae7ef2ba [id=6d7b4b3c78aa445cb4ab74fc6a40c8a5]), partition=RANGE (key) PARTITION UNBOUNDED
W20250113 02:51:46.797294 711 leader_election.cc:343] T 7527e6e0b65e4a43b7f48899a34cdf7f P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Not found: Tablet not found: 7527e6e0b65e4a43b7f48899a34cdf7f
I20250113 02:51:46.797382 711 leader_election.cc:304] T 7527e6e0b65e4a43b7f48899a34cdf7f P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:46.797685 2853 raft_consensus.cc:2743] T 7527e6e0b65e4a43b7f48899a34cdf7f P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250113 02:51:46.798390 913 raft_consensus.cc:3054] T c13e9a51c1e74b44b352bade78390ba5 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:46.800197 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1484) took 1246 ms. Trace:
I20250113 02:51:46.800278 1198 rpcz_store.cc:276] 0113 02:51:45.554167 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:45.554934 (+ 767us) service_pool.cc:225] Handling call
0113 02:51:46.736309 (+1181375us) ts_tablet_manager.cc:1145] Deleting tablet 9b2d0aa6f0e9456d95f4e48c360ab6e8
0113 02:51:46.736319 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:46.763366 (+ 27047us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:46.776418 (+ 13052us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:46.800180 (+ 23762us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1147791}
I20250113 02:51:46.800624 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet 9b2d0aa6f0e9456d95f4e48c360ab6e8 (table default.Table_7bb55657ad1b451c86260845c932b23e [id=8b48305e2d1b4a0fa1c1fab70d570cf6]) successfully deleted
I20250113 02:51:46.801151 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1486) took 1206 ms. Trace:
I20250113 02:51:46.801224 1198 rpcz_store.cc:276] 0113 02:51:45.594221 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:45.594385 (+ 164us) service_pool.cc:225] Handling call
0113 02:51:46.800850 (+1206465us) ts_tablet_manager.cc:1145] Deleting tablet 593c54696ea7473d8450744a7c4a394d
0113 02:51:46.800861 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:46.801133 (+ 272us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1206288}
W20250113 02:51:46.801576 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 593c54696ea7473d8450744a7c4a394d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 593c54696ea7473d8450744a7c4a394d already in progress: creating tablet
I20250113 02:51:46.802585 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29c17fd90d774c029a35e77888358067" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:46.802963 1035 tablet_service.cc:1467] Processing CreateTablet for tablet 657c15d348ec43df8074de4c62c213a4 (DEFAULT_TABLE table=default.Table_63be23fc2ed64476b299996a4175c5d5 [id=455f66a951ff4c70ae9d7c192fcb2de5]), partition=RANGE (key) PARTITION UNBOUNDED
W20250113 02:51:46.803071 711 leader_election.cc:343] T 29c17fd90d774c029a35e77888358067 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:46.803397 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 657c15d348ec43df8074de4c62c213a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:46.803496 752 tablet_service.cc:1467] Processing CreateTablet for tablet 657c15d348ec43df8074de4c62c213a4 (DEFAULT_TABLE table=default.Table_63be23fc2ed64476b299996a4175c5d5 [id=455f66a951ff4c70ae9d7c192fcb2de5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:46.803768 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29c17fd90d774c029a35e77888358067" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
W20250113 02:51:46.804250 709 leader_election.cc:343] T 29c17fd90d774c029a35e77888358067 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:46.804335 709 leader_election.cc:304] T 29c17fd90d774c029a35e77888358067 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:46.804546 2853 raft_consensus.cc:2743] T 29c17fd90d774c029a35e77888358067 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:46.804737 2854 leader_election.cc:290] T 29c17fd90d774c029a35e77888358067 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:46.804802 1028 tablet_service.cc:1467] Processing CreateTablet for tablet e64d7e847dcd4253b6b4c7464fc5a798 (DEFAULT_TABLE table=default.Table_3d4ca369c9434e42b3455bd2f325eff9 [id=c536a0f3507f43379482aec6e651e8b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:46.804843 739 tablet_service.cc:1467] Processing CreateTablet for tablet e64d7e847dcd4253b6b4c7464fc5a798 (DEFAULT_TABLE table=default.Table_3d4ca369c9434e42b3455bd2f325eff9 [id=c536a0f3507f43379482aec6e651e8b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:46.805225 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e64d7e847dcd4253b6b4c7464fc5a798. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:46.805868 1030 tablet_service.cc:1467] Processing CreateTablet for tablet 0d560b06a2f64a02b81d8d1e4bb75f66 (DEFAULT_TABLE table=default.Table_51d4d166147f48b9b374db8c8a89e368 [id=d2ec4eb5d98148c38e4ce4a27cb92704]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:46.806169 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d560b06a2f64a02b81d8d1e4bb75f66. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:46.808228 881 tablet_service.cc:1467] Processing CreateTablet for tablet 71b38c7d3ae7419a9a20f68c43517ecf (DEFAULT_TABLE table=default.Table_c2fe3752f4a1482fb35693231429d1df [id=30719be151404bf4a53902e9d5c04538]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:46.808564 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 71b38c7d3ae7419a9a20f68c43517ecf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:46.813186 887 tablet_service.cc:1467] Processing CreateTablet for tablet c7f540efce9a4e84b13c4d41b25722f0 (DEFAULT_TABLE table=default.Table_469a767a55b84a8b930b57666c06575b [id=088397900049451e9a3681e151dbc0ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:46.813637 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c7f540efce9a4e84b13c4d41b25722f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:46.813735 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet 82505f1389184d0ca12ec0e2d6084d22 (table default.Table_855f1069eae9472998eb347f20b003b8 [id=210a731382514eb098fb9442e6c290ba]) successfully deleted
W20250113 02:51:46.813993 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 476c89cc585b4e00947f5d1e8d0229f6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 476c89cc585b4e00947f5d1e8d0229f6 already in progress: creating tablet
I20250113 02:51:46.814157 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet 1968c7e9357c4c6e901d334ace0467fb (table default.Table_f2da084a4aee4d19b45e784e56f926a6 [id=fa9bcd8c66c34d65b43561572abb8890]) successfully deleted
W20250113 02:51:46.814327 442 catalog_manager.cc:4839] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet ee09d2b74c05432c9d7917b036bf8898 because the tablet was not found. No further retry: Not found: Tablet not found: ee09d2b74c05432c9d7917b036bf8898
W20250113 02:51:46.814505 442 catalog_manager.cc:4839] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet eb4a985261f4452db4de74891b7bf65a because the tablet was not found. No further retry: Not found: Tablet not found: eb4a985261f4452db4de74891b7bf65a
I20250113 02:51:46.814893 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet 4fe54131aafb40ee9d8bc5ea299af193 (table default.Table_567d446e6ae54abf9ae7586692b5780a [id=70089f757a1c45f6b983f216fb167478]) successfully deleted
I20250113 02:51:46.815675 883 tablet_service.cc:1467] Processing CreateTablet for tablet f46c28be32b24a0ba697629ea17b39c8 (DEFAULT_TABLE table=default.Table_880629d9242b4654a306a4b3843a636a [id=a0fc70e132f441db878c2dc0421158de]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:46.815965 2772 raft_consensus.cc:491] T 59fadc693f554854bc2874f8db7c64c8 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:46.816037 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f46c28be32b24a0ba697629ea17b39c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:46.816098 2772 raft_consensus.cc:513] T 59fadc693f554854bc2874f8db7c64c8 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:46.816170 899 tablet_service.cc:1467] Processing CreateTablet for tablet 7944d77262ae4b99b6735cf9ca0593f6 (DEFAULT_TABLE table=default.Table_8e5d26dfdb7e4f2289e792e637df2407 [id=35cb9d91202e42e0b4ad1beb4d995d0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:46.816459 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7944d77262ae4b99b6735cf9ca0593f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:46.817137 2772 leader_election.cc:290] T 59fadc693f554854bc2874f8db7c64c8 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:46.819384 2854 raft_consensus.cc:491] T b3ce6d851f564b849796f174860aacd5 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:46.819514 2854 raft_consensus.cc:513] T b3ce6d851f564b849796f174860aacd5 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
W20250113 02:51:46.820086 967 tablet.cc:2367] T 283fc7f1c74e480c8a63c74d3ae9eb7d P df69f48656b44bb890f76587ee69cc0f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250113 02:51:46.820922 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3ce6d851f564b849796f174860aacd5" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:46.821398 711 leader_election.cc:343] T b3ce6d851f564b849796f174860aacd5 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:46.821529 2854 leader_election.cc:290] T b3ce6d851f564b849796f174860aacd5 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:46.825030 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59fadc693f554854bc2874f8db7c64c8" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:46.825563 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea7b44be353f4ab1b2ac8d1b9916382f. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:46.825548 995 leader_election.cc:343] T 59fadc693f554854bc2874f8db7c64c8 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Not found: Tablet not found: 59fadc693f554854bc2874f8db7c64c8
W20250113 02:51:46.825682 998 leader_election.cc:343] T 59fadc693f554854bc2874f8db7c64c8 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:46.825800 998 leader_election.cc:304] T 59fadc693f554854bc2874f8db7c64c8 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4216a7f853d642b7bc3e5cff458f3aca; no voters: 1186fa869eb14812832fa6562531f196, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:46.826480 2769 raft_consensus.cc:2743] T 59fadc693f554854bc2874f8db7c64c8 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:46.826818 2849 tablet.cc:1721] T 3f62f344c5a648328ca2de8ca1069c19 P df69f48656b44bb890f76587ee69cc0f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250113 02:51:46.828459 1191 tablet_bootstrap.cc:492] T ed68a2b7ebcf4841bef54798dd3ed80d P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:46.828588 1191 ts_tablet_manager.cc:1397] T ed68a2b7ebcf4841bef54798dd3ed80d P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.262s user 0.002s sys 0.001s
I20250113 02:51:46.829087 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d8cdaa36b424a53b0501a65f4e2ce7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:46.831157 2368 raft_consensus.cc:491] T a2781ba5061a4225b5636366857a229e P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:46.831300 2368 raft_consensus.cc:513] T a2781ba5061a4225b5636366857a229e P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:46.825567 2772 consensus_queue.cc:1035] T 542df00229f645788f1b821c0ce00c46 P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.006s
I20250113 02:51:46.831933 2368 leader_election.cc:290] T a2781ba5061a4225b5636366857a229e P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:46.832248 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5eb5247d3704e7aae0d4e08dc19ae5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:46.832688 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2781ba5061a4225b5636366857a229e" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
I20250113 02:51:46.832917 773 raft_consensus.cc:2463] T a2781ba5061a4225b5636366857a229e P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 0.
I20250113 02:51:46.832912 446 catalog_manager.cc:5618] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587) does not have the latest schema for tablet f2c6c8c17d6e499d84ad7c0c23889cc4 (table default.Table_92ea541283614df981d96c5b74362f1d [id=9e58548d1a184cc0b362f148c44005ed]). Expected version 2 got 0
I20250113 02:51:46.833112 446 catalog_manager.cc:5618] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587) does not have the latest schema for tablet b815003a27f64809baaa2d43d6293cef (table default.Table_c2e8fd7d2cc14fa39c0abfece783c6c9 [id=42e19138c08f422a92ba9621e9609957]). Expected version 1 got 0
I20250113 02:51:46.833307 446 catalog_manager.cc:5526] T afcb5b1977c447d3a51de9cb72d7196a P 4216a7f853d642b7bc3e5cff458f3aca reported cstate change: term changed from 0 to 1, leader changed from <none> to 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3). New cstate: current_term: 1 leader_uuid: "4216a7f853d642b7bc3e5cff458f3aca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } health_report { overall_health: UNKNOWN } } }
I20250113 02:51:46.833675 446 catalog_manager.cc:5618] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587) does not have the latest schema for tablet afcb5b1977c447d3a51de9cb72d7196a (table default.Table_1eb2867afbff468cbbca66486afd7075 [id=d0ffa26e65894ab89facfccd75985d51]). Expected version 1 got 0
I20250113 02:51:46.833937 995 leader_election.cc:304] T a2781ba5061a4225b5636366857a229e P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters:
I20250113 02:51:46.834172 2368 raft_consensus.cc:2798] T a2781ba5061a4225b5636366857a229e P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 02:51:46.834028 446 catalog_manager.cc:5526] T c04be975c8c049b892b16129ed47a142 P 4216a7f853d642b7bc3e5cff458f3aca reported cstate change: term changed from 0 to 1, leader changed from <none> to 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3). New cstate: current_term: 1 leader_uuid: "4216a7f853d642b7bc3e5cff458f3aca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } health_report { overall_health: HEALTHY } } }
I20250113 02:51:46.834265 2368 raft_consensus.cc:491] T a2781ba5061a4225b5636366857a229e P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 02:51:46.834347 2368 raft_consensus.cc:3054] T a2781ba5061a4225b5636366857a229e P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:46.834406 446 catalog_manager.cc:5618] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587) does not have the latest schema for tablet c04be975c8c049b892b16129ed47a142 (table default.Table_f98f15e1cb6a43348c4438096a7f9ea4 [id=36184fc3b73745c487d1d7a9a69e85ef]). Expected version 1 got 0
I20250113 02:51:46.836218 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b4e03b52495d46f8be3f1544e5fda5e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:46.837363 2368 raft_consensus.cc:513] T a2781ba5061a4225b5636366857a229e P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:46.837951 2368 leader_election.cc:290] T a2781ba5061a4225b5636366857a229e P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Requested vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:46.838973 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2781ba5061a4225b5636366857a229e" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196"
I20250113 02:51:46.839188 773 raft_consensus.cc:3054] T a2781ba5061a4225b5636366857a229e P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:46.839327 455 catalog_manager.cc:5618] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467) does not have the latest schema for tablet 3f62f344c5a648328ca2de8ca1069c19 (table default.Table_9ca0c47692a840c39dbc1c73c7a6bc7c [id=7783df86a6ee404f87055036508adb20]). Expected version 1 got 0
I20250113 02:51:46.840632 736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69e200d941164e21848c5cfc5fbcd8f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:46.841531 773 raft_consensus.cc:2463] T a2781ba5061a4225b5636366857a229e P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 1.
I20250113 02:51:46.842156 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2781ba5061a4225b5636366857a229e" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f"
W20250113 02:51:46.842686 998 leader_election.cc:343] T a2781ba5061a4225b5636366857a229e P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:46.844179 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 657c15d348ec43df8074de4c62c213a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:46.846372 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2781ba5061a4225b5636366857a229e" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:46.846786 998 leader_election.cc:343] T a2781ba5061a4225b5636366857a229e P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:46.848176 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 391615b7658d44dea2a72a7bd1656856. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:46.849794 1039 tablet_service.cc:1467] Processing CreateTablet for tablet 391615b7658d44dea2a72a7bd1656856 (DEFAULT_TABLE table=default.Table_a9e430e1146e40d7945c1680ae7ef2ba [id=6d7b4b3c78aa445cb4ab74fc6a40c8a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:46.851680 1032 tablet_service.cc:1467] Processing CreateTablet for tablet 752c1291da404f3a81548cfa70eee063 (DEFAULT_TABLE table=default.Table_50f8312d0d834e25ab560f9c59fa3ce0 [id=bd87a87c82c9467f96c3c7638457ae0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:46.852218 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e02c3d5511e47f59c38c8e6e3cb51d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:46.852892 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 132eda23315c4af1820a0c5433cdc7c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:46.853793 740 tablet_service.cc:1467] Processing CreateTablet for tablet 752c1291da404f3a81548cfa70eee063 (DEFAULT_TABLE table=default.Table_50f8312d0d834e25ab560f9c59fa3ce0 [id=bd87a87c82c9467f96c3c7638457ae0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:46.854106 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 752c1291da404f3a81548cfa70eee063. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:46.856801 995 leader_election.cc:304] T a2781ba5061a4225b5636366857a229e P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters: df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:46.857772 1025 tablet_service.cc:1467] Processing CreateTablet for tablet a11377e635b044e58a730fda3c25ae27 (DEFAULT_TABLE table=default.Table_6a88830e770641c4bbe405c0ef700d67 [id=ba1edf2b734e48cb89a77ca59fa717c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:46.858130 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 766d0c959e5a4c969c75828e2523cfd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:46.859413 2368 raft_consensus.cc:2798] T a2781ba5061a4225b5636366857a229e P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Leader election won for term 1
I20250113 02:51:46.859532 2368 raft_consensus.cc:695] T a2781ba5061a4225b5636366857a229e P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: Becoming Leader. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Running, Role: LEADER
I20250113 02:51:46.859747 2368 consensus_queue.cc:237] T a2781ba5061a4225b5636366857a229e P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:46.860133 755 tablet_service.cc:1467] Processing CreateTablet for tablet a11377e635b044e58a730fda3c25ae27 (DEFAULT_TABLE table=default.Table_6a88830e770641c4bbe405c0ef700d67 [id=ba1edf2b734e48cb89a77ca59fa717c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:46.860435 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a11377e635b044e58a730fda3c25ae27. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:46.866178 1031 tablet_service.cc:1467] Processing CreateTablet for tablet 3389e6747fb145c5a78c90a5ab7d3f2a (DEFAULT_TABLE table=default.Table_e0d35592e5c842afbe63a4fb305cae3a [id=a76443d62ea04c03ad5942a8c1fb0c5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:46.867491 1033 tablet_service.cc:1467] Processing CreateTablet for tablet 04078d87cb8043fbbd51a7c1cbcd398d (DEFAULT_TABLE table=default.Table_6f8ebd124a0e4bfb83502fc9f0f97130 [id=07fc217c213342aaa6c47f0ef8325725]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:46.872292 745 tablet_service.cc:1467] Processing CreateTablet for tablet 3389e6747fb145c5a78c90a5ab7d3f2a (DEFAULT_TABLE table=default.Table_e0d35592e5c842afbe63a4fb305cae3a [id=a76443d62ea04c03ad5942a8c1fb0c5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:46.872473 1027 tablet_service.cc:1467] Processing CreateTablet for tablet 9670fe8bf3a3403b8c8cb99d03cc3a78 (DEFAULT_TABLE table=default.Table_f6c8709d5ec649f782b85cbbc6517f5b [id=ac06bedbd2bb46e688a236a61e8f75fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:46.872604 740 tablet_service.cc:1467] Processing CreateTablet for tablet 04078d87cb8043fbbd51a7c1cbcd398d (DEFAULT_TABLE table=default.Table_6f8ebd124a0e4bfb83502fc9f0f97130 [id=07fc217c213342aaa6c47f0ef8325725]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:46.872759 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3389e6747fb145c5a78c90a5ab7d3f2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:46.872917 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 04078d87cb8043fbbd51a7c1cbcd398d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:46.873564 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9bfecc0884a64b4583506f6a050820f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:46.873750 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e64d7e847dcd4253b6b4c7464fc5a798. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:46.873934 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 52e2b698832844baa0edfe732909e075. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:46.877969 998 consensus_peers.cc:487] T 542df00229f645788f1b821c0ce00c46 P 4216a7f853d642b7bc3e5cff458f3aca -> Peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Couldn't send request to peer df69f48656b44bb890f76587ee69cc0f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250113 02:51:46.880559 744 tablet_service.cc:1467] Processing CreateTablet for tablet 9670fe8bf3a3403b8c8cb99d03cc3a78 (DEFAULT_TABLE table=default.Table_f6c8709d5ec649f782b85cbbc6517f5b [id=ac06bedbd2bb46e688a236a61e8f75fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:46.881243 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9670fe8bf3a3403b8c8cb99d03cc3a78. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:46.889760 1042 tablet_service.cc:1467] Processing CreateTablet for tablet 423d5d1a57484806b76565aa2292c2a1 (DEFAULT_TABLE table=default.Table_cf16887c855544b1987f7539a58537fb [id=7eb31eded10f498b918c00ec5e3f860a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:46.890558 748 tablet_service.cc:1467] Processing CreateTablet for tablet 0d560b06a2f64a02b81d8d1e4bb75f66 (DEFAULT_TABLE table=default.Table_51d4d166147f48b9b374db8c8a89e368 [id=d2ec4eb5d98148c38e4ce4a27cb92704]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:46.891038 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d560b06a2f64a02b81d8d1e4bb75f66. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:46.892309 1026 tablet_service.cc:1467] Processing CreateTablet for tablet 9842ee1a8e3c4011add4b117a874f620 (DEFAULT_TABLE table=default.Table_22ace32627b84d33b3fe01731f8d2c07 [id=2661227c8a684d4eb08d2c46738f7aed]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:46.893573 1037 tablet_service.cc:1467] Processing CreateTablet for tablet 5f1e613a8b934e2c89a6eb08ff18c913 (DEFAULT_TABLE table=default.Table_1416b17502f140db98544370a86bdcb6 [id=d7c4c089b0454fada09f7d063d41b876]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:46.895268 744 tablet_service.cc:1467] Processing CreateTablet for tablet 5f1e613a8b934e2c89a6eb08ff18c913 (DEFAULT_TABLE table=default.Table_1416b17502f140db98544370a86bdcb6 [id=d7c4c089b0454fada09f7d063d41b876]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:46.896597 1038 tablet_service.cc:1467] Processing CreateTablet for tablet b786b007e4e04e34a70f9f585acee9fa (DEFAULT_TABLE table=default.Table_83eaa3dec71e42e2921fc5957777f78e [id=f554a27db5a4443185c24bd70fec2461]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:46.897199 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f1e613a8b934e2c89a6eb08ff18c913. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:46.897766 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3ce6d851f564b849796f174860aacd5" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:46.897975 1059 raft_consensus.cc:2463] T b3ce6d851f564b849796f174860aacd5 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1186fa869eb14812832fa6562531f196 in term 0.
I20250113 02:51:46.898705 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 04078d87cb8043fbbd51a7c1cbcd398d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:46.899786 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b786b007e4e04e34a70f9f585acee9fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:46.899951 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9670fe8bf3a3403b8c8cb99d03cc3a78. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:46.904228 1042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 423d5d1a57484806b76565aa2292c2a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:46.905054 1037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f1e613a8b934e2c89a6eb08ff18c913. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:46.908187 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9842ee1a8e3c4011add4b117a874f620. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:46.908439 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a11377e635b044e58a730fda3c25ae27. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:46.898407 709 leader_election.cc:304] T b3ce6d851f564b849796f174860aacd5 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters: df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:46.908649 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3389e6747fb145c5a78c90a5ab7d3f2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:46.908931 2854 raft_consensus.cc:2798] T b3ce6d851f564b849796f174860aacd5 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 02:51:46.909058 2854 raft_consensus.cc:491] T b3ce6d851f564b849796f174860aacd5 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 02:51:46.909134 2854 raft_consensus.cc:3054] T b3ce6d851f564b849796f174860aacd5 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:46.909440 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 752c1291da404f3a81548cfa70eee063. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:46.911662 2854 raft_consensus.cc:513] T b3ce6d851f564b849796f174860aacd5 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:46.912653 2853 raft_consensus.cc:491] T b9ec434644c14acd95c72efbef8ded59 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:46.912719 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 391615b7658d44dea2a72a7bd1656856. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:46.912776 2853 raft_consensus.cc:513] T b9ec434644c14acd95c72efbef8ded59 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:46.913007 2854 leader_election.cc:290] T b3ce6d851f564b849796f174860aacd5 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 election: Requested vote from peers 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:46.913331 2853 leader_election.cc:290] T b9ec434644c14acd95c72efbef8ded59 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:46.914163 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9ec434644c14acd95c72efbef8ded59" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:46.914705 711 leader_election.cc:343] T b9ec434644c14acd95c72efbef8ded59 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:46.915143 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3ce6d851f564b849796f174860aacd5" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f"
W20250113 02:51:46.915714 711 leader_election.cc:343] T b3ce6d851f564b849796f174860aacd5 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:46.916579 749 tablet_service.cc:1467] Processing CreateTablet for tablet b786b007e4e04e34a70f9f585acee9fa (DEFAULT_TABLE table=default.Table_83eaa3dec71e42e2921fc5957777f78e [id=f554a27db5a4443185c24bd70fec2461]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:46.916998 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b786b007e4e04e34a70f9f585acee9fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:46.920224 741 tablet_service.cc:1467] Processing CreateTablet for tablet d664e48a65ae4e53aedbff0d55bfc17b (DEFAULT_TABLE table=default.Table_ed460a21a2184db893d2e76e67cdb974 [id=fb3d90554f7943099cd541f610084cbf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:46.920646 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d664e48a65ae4e53aedbff0d55bfc17b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:46.922945 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3ce6d851f564b849796f174860aacd5" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca"
I20250113 02:51:46.923120 1059 raft_consensus.cc:3054] T b3ce6d851f564b849796f174860aacd5 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:46.925571 1059 raft_consensus.cc:2463] T b3ce6d851f564b849796f174860aacd5 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1186fa869eb14812832fa6562531f196 in term 1.
I20250113 02:51:46.926223 709 leader_election.cc:304] T b3ce6d851f564b849796f174860aacd5 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters: df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:46.926709 2853 raft_consensus.cc:2798] T b3ce6d851f564b849796f174860aacd5 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Leader election won for term 1
I20250113 02:51:46.926996 2853 raft_consensus.cc:695] T b3ce6d851f564b849796f174860aacd5 P 1186fa869eb14812832fa6562531f196 [term 1 LEADER]: Becoming Leader. State: Replica: 1186fa869eb14812832fa6562531f196, State: Running, Role: LEADER
I20250113 02:51:46.927178 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1487) took 1330 ms. Trace:
I20250113 02:51:46.927234 2853 consensus_queue.cc:237] T b3ce6d851f564b849796f174860aacd5 P 1186fa869eb14812832fa6562531f196 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:46.927260 1198 rpcz_store.cc:276] 0113 02:51:45.596329 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:45.597535 (+ 1206us) service_pool.cc:225] Handling call
0113 02:51:46.813116 (+1215581us) ts_tablet_manager.cc:1145] Deleting tablet 1376cf72589d43048cb829e03240c8da
0113 02:51:46.904857 (+ 91741us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:46.927159 (+ 22302us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1215408}
I20250113 02:51:46.928175 1038 tablet_service.cc:1467] Processing CreateTablet for tablet 03daa471e30240688d711b493cff1e6d (DEFAULT_TABLE table=default.Table_9bdc99d09f2d405b88dcb4a48144ca4b [id=e3f0e6356ea6410b91977cb88255ca20]), partition=RANGE (key) PARTITION UNBOUNDED
W20250113 02:51:46.928303 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 1376cf72589d43048cb829e03240c8da because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1376cf72589d43048cb829e03240c8da already in progress: creating tablet
I20250113 02:51:46.928691 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03daa471e30240688d711b493cff1e6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:46.930637 743 tablet_service.cc:1467] Processing CreateTablet for tablet 03daa471e30240688d711b493cff1e6d (DEFAULT_TABLE table=default.Table_9bdc99d09f2d405b88dcb4a48144ca4b [id=e3f0e6356ea6410b91977cb88255ca20]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:46.930994 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03daa471e30240688d711b493cff1e6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:46.931177 1042 tablet_service.cc:1467] Processing CreateTablet for tablet d75822a1e2184e3288e692e564045a6a (DEFAULT_TABLE table=default.Table_a4837fc75b6f4d2c8756d0967cce0b31 [id=edf26613cff84dbb95fb2800dad48e06]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:46.931483 1042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d75822a1e2184e3288e692e564045a6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:46.932013 1060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9ec434644c14acd95c72efbef8ded59" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
W20250113 02:51:46.932633 709 leader_election.cc:343] T b9ec434644c14acd95c72efbef8ded59 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:46.932725 709 leader_election.cc:304] T b9ec434644c14acd95c72efbef8ded59 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:46.933055 2854 raft_consensus.cc:2743] T b9ec434644c14acd95c72efbef8ded59 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:46.936281 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1490) took 1274 ms. Trace:
I20250113 02:51:46.936372 1198 rpcz_store.cc:276] 0113 02:51:45.662308 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:45.662714 (+ 406us) service_pool.cc:225] Handling call
0113 02:51:46.927497 (+1264783us) ts_tablet_manager.cc:1145] Deleting tablet 33ccb3f7d2024f03b23d1e1c1046072a
0113 02:51:46.927505 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:46.936264 (+ 8759us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1264622}
W20250113 02:51:46.936782 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 33ccb3f7d2024f03b23d1e1c1046072a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 33ccb3f7d2024f03b23d1e1c1046072a already in progress: creating tablet
I20250113 02:51:46.937153 754 tablet_service.cc:1467] Processing CreateTablet for tablet d75822a1e2184e3288e692e564045a6a (DEFAULT_TABLE table=default.Table_a4837fc75b6f4d2c8756d0967cce0b31 [id=edf26613cff84dbb95fb2800dad48e06]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:46.937518 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d75822a1e2184e3288e692e564045a6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:46.938735 1026 tablet_service.cc:1467] Processing CreateTablet for tablet ac5e8cb88591488089575fe679815124 (DEFAULT_TABLE table=default.Table_f88977415dbf4f43abaa5fdfa7c46f4d [id=1791821062614708b2787fecac4fc837]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:46.939023 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac5e8cb88591488089575fe679815124. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:46.940258 752 tablet_service.cc:1467] Processing CreateTablet for tablet ac5e8cb88591488089575fe679815124 (DEFAULT_TABLE table=default.Table_f88977415dbf4f43abaa5fdfa7c46f4d [id=1791821062614708b2787fecac4fc837]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:46.940577 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac5e8cb88591488089575fe679815124. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:46.941558 742 tablet_service.cc:1467] Processing CreateTablet for tablet 32e99ae9bbe743ff8e3e71b0043d5a1e (DEFAULT_TABLE table=default.Table_e01a09e962074d5c87c15ad97b576d09 [id=5b4c62389b5c4487982fec74f9c15280]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:46.941849 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 32e99ae9bbe743ff8e3e71b0043d5a1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:46.943682 1033 tablet_service.cc:1467] Processing CreateTablet for tablet 32e99ae9bbe743ff8e3e71b0043d5a1e (DEFAULT_TABLE table=default.Table_e01a09e962074d5c87c15ad97b576d09 [id=5b4c62389b5c4487982fec74f9c15280]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:46.944016 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 32e99ae9bbe743ff8e3e71b0043d5a1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:46.945778 1031 tablet_service.cc:1467] Processing CreateTablet for tablet d664e48a65ae4e53aedbff0d55bfc17b (DEFAULT_TABLE table=default.Table_ed460a21a2184db893d2e76e67cdb974 [id=fb3d90554f7943099cd541f610084cbf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:46.946401 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d664e48a65ae4e53aedbff0d55bfc17b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:46.946520 1034 tablet_service.cc:1467] Processing CreateTablet for tablet 4200521dc098474fb5f498676d70f023 (DEFAULT_TABLE table=default.Table_186f88df8da44d0c87ac23e9406895d6 [id=67ae377d727840489d2edccd7579e037]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:46.946915 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4200521dc098474fb5f498676d70f023. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:46.948233 743 tablet_service.cc:1467] Processing CreateTablet for tablet 4200521dc098474fb5f498676d70f023 (DEFAULT_TABLE table=default.Table_186f88df8da44d0c87ac23e9406895d6 [id=67ae377d727840489d2edccd7579e037]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:46.948578 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4200521dc098474fb5f498676d70f023. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:46.950001 1042 tablet_service.cc:1467] Processing CreateTablet for tablet bc88415f52a0424db18574c886dd6f1a (DEFAULT_TABLE table=default.Table_27eb308f2dc84858b3d1a2fa4b42c3f6 [id=1cfda70b239543dbbd14d5857a6bbf3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:46.950364 1042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc88415f52a0424db18574c886dd6f1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:46.951495 738 tablet_service.cc:1467] Processing CreateTablet for tablet bc88415f52a0424db18574c886dd6f1a (DEFAULT_TABLE table=default.Table_27eb308f2dc84858b3d1a2fa4b42c3f6 [id=1cfda70b239543dbbd14d5857a6bbf3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:46.951805 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc88415f52a0424db18574c886dd6f1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:46.953006 744 tablet_service.cc:1467] Processing CreateTablet for tablet 802998e270094c258883579852ff4b4e (DEFAULT_TABLE table=default.Table_484890185d6b4dea9fe5ed3102b5b625 [id=6a7f6dedc485453b86d7f885bb4ad20e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:46.954816 1037 tablet_service.cc:1467] Processing CreateTablet for tablet ae75154319bb4c3991cd3c5b08ce4a1e (DEFAULT_TABLE table=default.Table_d20226843534409a8cabaa764bcd9ce5 [id=4309d6902f5c4d3b8ee14a7f4e63eda9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:46.955118 1037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae75154319bb4c3991cd3c5b08ce4a1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:46.957743 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 802998e270094c258883579852ff4b4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:46.963042 452 catalog_manager.cc:5618] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467) does not have the latest schema for tablet c13e9a51c1e74b44b352bade78390ba5 (table default.Table_0468611ec68e47109f5357935e50874d [id=caf098e2b57849a997bd7c8eaf1f9326]). Expected version 1 got 0
W20250113 02:51:46.963142 454 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f7b219de824c4bfdb097cb81ae5f16f2 consensus state has no leader
I20250113 02:51:46.963495 752 tablet_service.cc:1467] Processing CreateTablet for tablet ae75154319bb4c3991cd3c5b08ce4a1e (DEFAULT_TABLE table=default.Table_d20226843534409a8cabaa764bcd9ce5 [id=4309d6902f5c4d3b8ee14a7f4e63eda9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:46.963630 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet e8a0482683c7493eb19e7da5c78c095c
I20250113 02:51:46.963816 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae75154319bb4c3991cd3c5b08ce4a1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:46.964058 1035 tablet_service.cc:1467] Processing CreateTablet for tablet ec0fb729e3a14d50a23faf81c8082f1e (DEFAULT_TABLE table=default.Table_e38f2c9d348948238d245b93b82b0dce [id=7caed6f6020f49a5898a2261f653070d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:46.964272 740 tablet_service.cc:1467] Processing CreateTablet for tablet ec0fb729e3a14d50a23faf81c8082f1e (DEFAULT_TABLE table=default.Table_e38f2c9d348948238d245b93b82b0dce [id=7caed6f6020f49a5898a2261f653070d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:46.964417 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ec0fb729e3a14d50a23faf81c8082f1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:46.964437 1030 tablet_service.cc:1514] Processing DeleteTablet for tablet e8a0482683c7493eb19e7da5c78c095c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:46 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:46.964581 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ec0fb729e3a14d50a23faf81c8082f1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:46.965126 739 tablet_service.cc:1514] Processing DeleteTablet for tablet e8a0482683c7493eb19e7da5c78c095c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:46 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:46.965354 736 tablet_service.cc:1467] Processing CreateTablet for tablet 1fc811dda69f413e92833ec302694468 (DEFAULT_TABLE table=default.Table_5b9ead21b07d41b2ad99e51e4c522ac2 [id=fc41dd4bce6048b0ac0953ae0ea4c382]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:46.965567 1030 tablet_service.cc:1467] Processing CreateTablet for tablet fd2151a6bf884603907340f73d709975 (DEFAULT_TABLE table=default.Table_a0736ea5f5ac4ab1a819a6c08d410086 [id=dd6eee7466da4e5499f8193de2c56e02]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:46.965674 736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1fc811dda69f413e92833ec302694468. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:46.965863 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd2151a6bf884603907340f73d709975. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:46.970474 739 tablet_service.cc:1467] Processing CreateTablet for tablet fd2151a6bf884603907340f73d709975 (DEFAULT_TABLE table=default.Table_a0736ea5f5ac4ab1a819a6c08d410086 [id=dd6eee7466da4e5499f8193de2c56e02]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:46.970791 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd2151a6bf884603907340f73d709975. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:46.999523 746 tablet_service.cc:1467] Processing CreateTablet for tablet 9842ee1a8e3c4011add4b117a874f620 (DEFAULT_TABLE table=default.Table_22ace32627b84d33b3fe01731f8d2c07 [id=2661227c8a684d4eb08d2c46738f7aed]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:47.001813 2854 raft_consensus.cc:491] T 0b119704bc774ca9b4155c9619d4765b P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4216a7f853d642b7bc3e5cff458f3aca)
I20250113 02:51:47.001952 2854 raft_consensus.cc:513] T 0b119704bc774ca9b4155c9619d4765b P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } }
I20250113 02:51:47.002614 2854 leader_election.cc:290] T 0b119704bc774ca9b4155c9619d4765b P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
W20250113 02:51:47.003608 711 leader_election.cc:343] T 0b119704bc774ca9b4155c9619d4765b P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Not found: Tablet not found: 0b119704bc774ca9b4155c9619d4765b
I20250113 02:51:46.829308 1191 raft_consensus.cc:357] T ed68a2b7ebcf4841bef54798dd3ed80d P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:47.003906 1191 raft_consensus.cc:383] T ed68a2b7ebcf4841bef54798dd3ed80d P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:47.003988 1191 raft_consensus.cc:738] T ed68a2b7ebcf4841bef54798dd3ed80d P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:47.004206 1191 consensus_queue.cc:260] T ed68a2b7ebcf4841bef54798dd3ed80d P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:47.004868 1191 ts_tablet_manager.cc:1428] T ed68a2b7ebcf4841bef54798dd3ed80d P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.176s user 0.001s sys 0.001s
I20250113 02:51:47.005151 1191 tablet_bootstrap.cc:492] T e13267498917432ba92b6badb76f5d73 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:47.006762 1191 tablet_bootstrap.cc:654] T e13267498917432ba92b6badb76f5d73 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:47.009239 2854 raft_consensus.cc:491] T 95e47db8a4d74615800ba6c049ec7131 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:47.009349 2854 raft_consensus.cc:513] T 95e47db8a4d74615800ba6c049ec7131 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:47.009881 2854 leader_election.cc:290] T 95e47db8a4d74615800ba6c049ec7131 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:47.011390 452 catalog_manager.cc:5618] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467) does not have the latest schema for tablet c13e9a51c1e74b44b352bade78390ba5 (table default.Table_0468611ec68e47109f5357935e50874d [id=caf098e2b57849a997bd7c8eaf1f9326]). Expected version 1 got 0
I20250113 02:51:47.011559 750 tablet_service.cc:1467] Processing CreateTablet for tablet 423d5d1a57484806b76565aa2292c2a1 (DEFAULT_TABLE table=default.Table_cf16887c855544b1987f7539a58537fb [id=7eb31eded10f498b918c00ec5e3f860a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:47.011966 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 423d5d1a57484806b76565aa2292c2a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:47.013741 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95e47db8a4d74615800ba6c049ec7131" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:47.014317 711 leader_election.cc:343] T 95e47db8a4d74615800ba6c049ec7131 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:47.028898 2476 raft_consensus.cc:491] T e570465a3d4046b1ac19a3a034fe91f1 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:47.029089 2476 raft_consensus.cc:513] T e570465a3d4046b1ac19a3a034fe91f1 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:47.029744 2476 leader_election.cc:290] T e570465a3d4046b1ac19a3a034fe91f1 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
W20250113 02:51:47.031272 852 leader_election.cc:343] T e570465a3d4046b1ac19a3a034fe91f1 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Not found: Tablet not found: e570465a3d4046b1ac19a3a034fe91f1
W20250113 02:51:47.031701 851 leader_election.cc:343] T e570465a3d4046b1ac19a3a034fe91f1 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Not found: Tablet not found: e570465a3d4046b1ac19a3a034fe91f1
I20250113 02:51:47.031795 851 leader_election.cc:304] T e570465a3d4046b1ac19a3a034fe91f1 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df69f48656b44bb890f76587ee69cc0f; no voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca
W20250113 02:51:47.032053 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6265e9a6cfde4161b0cd5dd1b76be8fc consensus state has no leader
I20250113 02:51:47.032228 2476 raft_consensus.cc:2743] T e570465a3d4046b1ac19a3a034fe91f1 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:47.036952 881 tablet_service.cc:1467] Processing CreateTablet for tablet dbbcb086a31a49d3b32367da67de0760 (DEFAULT_TABLE table=default.Table_da4eeeb904054a9aa6807387f76b1d8f [id=5763436c8e6140aa99af2a28f45bb17d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:47.037613 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dbbcb086a31a49d3b32367da67de0760. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:47.037998 883 tablet_service.cc:1467] Processing CreateTablet for tablet 0a378ce2d64e44d1a1088e705d439b9c (DEFAULT_TABLE table=default.Table_36b33b166f834edbb7a465fc17ac3720 [id=1e8097bb261649a7b8af4917a9a3ca16]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:47.038390 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0a378ce2d64e44d1a1088e705d439b9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:47.040434 887 tablet_service.cc:1467] Processing CreateTablet for tablet af8d36e1b0584aeeb7fb651454a3efc8 (DEFAULT_TABLE table=default.Table_22efb860611f468c98e2b9df40e31e83 [id=0829928cf7e9474789fc2fdac9a49f90]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:47.040892 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af8d36e1b0584aeeb7fb651454a3efc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:47.037057 899 tablet_service.cc:1467] Processing CreateTablet for tablet 48fdb58340a742f680ed6e62a7eca532 (DEFAULT_TABLE table=default.Table_9a6c1e2713d74f7186906076d1f1212e [id=49984a2b4b434405a43eafed5cd441cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:47.044518 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 48fdb58340a742f680ed6e62a7eca532. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:47.054996 1195 ts_tablet_manager.cc:1918] T 01a455eda0ee488fb3d45a6a09c43833 P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:47.055155 1195 log.cc:1198] T 01a455eda0ee488fb3d45a6a09c43833 P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/01a455eda0ee488fb3d45a6a09c43833
I20250113 02:51:47.055472 1195 ts_tablet_manager.cc:1939] T 01a455eda0ee488fb3d45a6a09c43833 P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:47.056504 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1321) took 5957 ms. Trace:
I20250113 02:51:47.056602 1195 rpcz_store.cc:276] 0113 02:51:41.098957 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:41.191578 (+ 92621us) service_pool.cc:225] Handling call
0113 02:51:46.441228 (+5249650us) ts_tablet_manager.cc:1145] Deleting tablet 01a455eda0ee488fb3d45a6a09c43833
0113 02:51:46.441242 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:46.594140 (+152898us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:47.054880 (+460740us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:47.056481 (+ 1601us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5143317}
I20250113 02:51:47.057145 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet 01a455eda0ee488fb3d45a6a09c43833 (table default.Table_be2b756d1ad24981880b0debe778986a [id=694cd0f0a6624241947e7be0c71dac12]) successfully deleted
I20250113 02:51:47.057586 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1336) took 5587 ms. Trace:
I20250113 02:51:47.057672 1195 rpcz_store.cc:276] 0113 02:51:41.515334 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:41.540543 (+ 25209us) service_pool.cc:225] Handling call
0113 02:51:47.057430 (+5516887us) ts_tablet_manager.cc:1145] Deleting tablet c8966b3b93144a30813bfdaf04f9c43a
0113 02:51:47.057444 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:47.057566 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5486585}
W20250113 02:51:47.058208 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet c8966b3b93144a30813bfdaf04f9c43a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c8966b3b93144a30813bfdaf04f9c43a already in progress: creating tablet
I20250113 02:51:47.058596 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9842ee1a8e3c4011add4b117a874f620. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:47.064456 891 tablet_service.cc:1514] Processing DeleteTablet for tablet 905542fb0dce4bbb8852b74d94c7b2a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:46 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:47.065316 883 tablet_service.cc:1514] Processing DeleteTablet for tablet 9572215fd2eb45faa99cc236fcc0d5ea with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:46 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:47.068890 885 tablet_service.cc:1514] Processing DeleteTablet for tablet acd83024592e431898e9675c17f50d4f with delete_type TABLET_DATA_DELETED (replaced by tablet 18716752335043e89038f01b4d65d711) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:47.068908 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1498) took 1210 ms. Trace:
I20250113 02:51:47.069016 1198 rpcz_store.cc:276] 0113 02:51:45.858080 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:45.858136 (+ 56us) service_pool.cc:225] Handling call
0113 02:51:46.936620 (+1078484us) ts_tablet_manager.cc:1145] Deleting tablet b5c1882125b5491ea79a29a6dac6771e
0113 02:51:46.936629 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:47.068880 (+132251us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1078226}
I20250113 02:51:47.069149 1196 tablet_replica.cc:331] T acd83024592e431898e9675c17f50d4f P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:47.069319 1196 raft_consensus.cc:2238] T acd83024592e431898e9675c17f50d4f P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:47.069499 1196 raft_consensus.cc:2267] T acd83024592e431898e9675c17f50d4f P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:47.069819 1024 tablet_service.cc:1467] Processing CreateTablet for tablet 1fc811dda69f413e92833ec302694468 (DEFAULT_TABLE table=default.Table_5b9ead21b07d41b2ad99e51e4c522ac2 [id=fc41dd4bce6048b0ac0953ae0ea4c382]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:47.069984 1196 ts_tablet_manager.cc:1905] T acd83024592e431898e9675c17f50d4f P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:47.070317 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1fc811dda69f413e92833ec302694468. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:47.078559 440 catalog_manager.cc:4839] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet b5c1882125b5491ea79a29a6dac6771e because the tablet was not found. No further retry: Not found: Tablet not found: b5c1882125b5491ea79a29a6dac6771e
I20250113 02:51:47.079295 891 tablet_service.cc:1467] Processing CreateTablet for tablet ea7b44be353f4ab1b2ac8d1b9916382f (DEFAULT_TABLE table=default.Table_7f54426882784252bf8e26b7028d253a [id=616a5b5350d1455e8d928d8e95694c21]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:47.079493 889 tablet_service.cc:1467] Processing CreateTablet for tablet 132eda23315c4af1820a0c5433cdc7c1 (DEFAULT_TABLE table=default.Table_4fcb7bb3569b4eddafedff2f3dec10f8 [id=4a3910c2627a44d98e927b08a65599e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:47.079689 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea7b44be353f4ab1b2ac8d1b9916382f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:47.079869 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 132eda23315c4af1820a0c5433cdc7c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:47.085813 883 tablet_service.cc:1467] Processing CreateTablet for tablet e9f7aa2f369f4aad9561deb4f1ca06f3 (DEFAULT_TABLE table=default.Table_28f9de083c724932a0a03761e6148593 [id=baeeedacf6e94931b1201a333b5fdf5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:47.086234 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9f7aa2f369f4aad9561deb4f1ca06f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:47.089435 913 raft_consensus.cc:3054] T b38cd34d67834a3f9bf5b5636bb79217 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:47.091686 885 tablet_service.cc:1514] Processing DeleteTablet for tablet b946dd2d428a43d399bb75a43d843a28 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:46 UTC) from {username='slave'} at 127.0.0.1:37366
W20250113 02:51:47.102937 711 consensus_peers.cc:487] T 2470b010b571444eba3f132323038f33 P 1186fa869eb14812832fa6562531f196 -> Peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Couldn't send request to peer df69f48656b44bb890f76587ee69cc0f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250113 02:51:47.105221 899 tablet_service.cc:1467] Processing CreateTablet for tablet 1041048c522742dbbca710038b92ec15 (DEFAULT_TABLE table=default.Table_6441bd25946b4757af31d88201d6e3f2 [id=5b2f13bde21d4c87a609c980a55ab13c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:47.105852 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1041048c522742dbbca710038b92ec15. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:47.109054 1038 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b2d0aa6f0e9456d95f4e48c360ab6e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:33 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:47.109434 1042 tablet_service.cc:1514] Processing DeleteTablet for tablet 569d666c128a45b7a526c13be5090ae3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:26 UTC) from {username='slave'} at 127.0.0.1:60230
W20250113 02:51:47.110527 709 leader_election.cc:343] T 0b119704bc774ca9b4155c9619d4765b P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Not found: Tablet not found: 0b119704bc774ca9b4155c9619d4765b
I20250113 02:51:47.110648 709 leader_election.cc:304] T 0b119704bc774ca9b4155c9619d4765b P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:47.112567 2853 raft_consensus.cc:2743] T 0b119704bc774ca9b4155c9619d4765b P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250113 02:51:47.112831 1036 tablet_service.cc:1514] Processing DeleteTablet for tablet a2781ba5061a4225b5636366857a229e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:24 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:47.113281 1026 tablet_service.cc:1467] Processing CreateTablet for tablet 802998e270094c258883579852ff4b4e (DEFAULT_TABLE table=default.Table_484890185d6b4dea9fe5ed3102b5b625 [id=6a7f6dedc485453b86d7f885bb4ad20e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:47.113857 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 802998e270094c258883579852ff4b4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:47.116976 2476 raft_consensus.cc:491] T 67baaab1e36e415381579d7385d4ab5a P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:47.117111 2476 raft_consensus.cc:513] T 67baaab1e36e415381579d7385d4ab5a P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:47.117727 2476 leader_election.cc:290] T 67baaab1e36e415381579d7385d4ab5a P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
W20250113 02:51:47.119561 851 leader_election.cc:343] T 67baaab1e36e415381579d7385d4ab5a P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Not found: Tablet not found: 67baaab1e36e415381579d7385d4ab5a
W20250113 02:51:47.119942 852 leader_election.cc:343] T 67baaab1e36e415381579d7385d4ab5a P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Not found: Tablet not found: 67baaab1e36e415381579d7385d4ab5a
I20250113 02:51:47.120069 852 leader_election.cc:304] T 67baaab1e36e415381579d7385d4ab5a P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df69f48656b44bb890f76587ee69cc0f; no voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca
I20250113 02:51:47.120039 893 tablet_service.cc:1467] Processing CreateTablet for tablet 9bfecc0884a64b4583506f6a050820f0 (DEFAULT_TABLE table=default.Table_28e6f68bf4304b12b107c8e71e66e1e3 [id=e9f4feaeaee04ae3a223b3762b87da66]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:47.120512 2476 raft_consensus.cc:2743] T 67baaab1e36e415381579d7385d4ab5a P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:47.120540 897 tablet_service.cc:1467] Processing CreateTablet for tablet 766d0c959e5a4c969c75828e2523cfd1 (DEFAULT_TABLE table=default.Table_9f1f5d4e13b44704a2e7edf981461d71 [id=46af1be1bcee41e4baabe573cf885c56]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:47.120559 880 tablet_service.cc:1467] Processing CreateTablet for tablet 9e02c3d5511e47f59c38c8e6e3cb51d4 (DEFAULT_TABLE table=default.Table_3e1f7bdb9ecf47d0baa280c94babfc41 [id=b359b53f229d4214976fb303d0228ba2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:47.120043 895 tablet_service.cc:1467] Processing CreateTablet for tablet 52e2b698832844baa0edfe732909e075 (DEFAULT_TABLE table=default.Table_f166593d31ee4c56965f2c6eda65615b [id=7e5d042f6f81474093c3128db8d6cdf1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:47.120043 891 tablet_service.cc:1467] Processing CreateTablet for tablet 69e200d941164e21848c5cfc5fbcd8f6 (DEFAULT_TABLE table=default.Table_d2fb23238827478fbe0d1afa409ba711 [id=2f4f6b0d225641d38ae54e47eadc12c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:47.121383 888 tablet_service.cc:1467] Processing CreateTablet for tablet 391615b7658d44dea2a72a7bd1656856 (DEFAULT_TABLE table=default.Table_a9e430e1146e40d7945c1680ae7ef2ba [id=6d7b4b3c78aa445cb4ab74fc6a40c8a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:47.122296 898 tablet_service.cc:1467] Processing CreateTablet for tablet 657c15d348ec43df8074de4c62c213a4 (DEFAULT_TABLE table=default.Table_63be23fc2ed64476b299996a4175c5d5 [id=455f66a951ff4c70ae9d7c192fcb2de5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:47.122135 889 tablet_service.cc:1467] Processing CreateTablet for tablet b4e03b52495d46f8be3f1544e5fda5e9 (DEFAULT_TABLE table=default.Table_d7c36054e0c84d26a64741fa4e91cd8e [id=a7082e6637444da1ab32067db8bcd4f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:47.123266 885 tablet_service.cc:1467] Processing CreateTablet for tablet e5eb5247d3704e7aae0d4e08dc19ae5d (DEFAULT_TABLE table=default.Table_6ea04b79ee2c446c986176fbd2cad259 [id=3bb015e2f6df4fa9b4f048debad038af]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:47.123471 884 tablet_service.cc:1467] Processing CreateTablet for tablet e64d7e847dcd4253b6b4c7464fc5a798 (DEFAULT_TABLE table=default.Table_3d4ca369c9434e42b3455bd2f325eff9 [id=c536a0f3507f43379482aec6e651e8b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:47.124949 890 tablet_service.cc:1467] Processing CreateTablet for tablet 0d560b06a2f64a02b81d8d1e4bb75f66 (DEFAULT_TABLE table=default.Table_51d4d166147f48b9b374db8c8a89e368 [id=d2ec4eb5d98148c38e4ce4a27cb92704]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:47.126838 1194 tablet_bootstrap.cc:492] T 6a4c6dc08df744e6856bdf99ab65c575 P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:47.127143 1194 ts_tablet_manager.cc:1397] T 6a4c6dc08df744e6856bdf99ab65c575 P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.465s user 0.057s sys 0.051s
I20250113 02:51:47.129086 1196 ts_tablet_manager.cc:1918] T acd83024592e431898e9675c17f50d4f P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:47.109674 1060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95e47db8a4d74615800ba6c049ec7131" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:47.096004 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1499) took 1237 ms. Trace:
W20250113 02:51:47.135149 709 leader_election.cc:343] T 95e47db8a4d74615800ba6c049ec7131 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:47.135272 709 leader_election.cc:304] T 95e47db8a4d74615800ba6c049ec7131 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:47.136360 2853 raft_consensus.cc:2743] T 95e47db8a4d74615800ba6c049ec7131 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:47.136433 449 catalog_manager.cc:5526] T b3ce6d851f564b849796f174860aacd5 P 1186fa869eb14812832fa6562531f196 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1186fa869eb14812832fa6562531f196 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "1186fa869eb14812832fa6562531f196" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } health_report { overall_health: UNKNOWN } } }
I20250113 02:51:47.138152 1191 tablet_bootstrap.cc:492] T e13267498917432ba92b6badb76f5d73 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:47.138286 1191 ts_tablet_manager.cc:1397] T e13267498917432ba92b6badb76f5d73 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.133s user 0.001s sys 0.001s
I20250113 02:51:47.138973 1191 raft_consensus.cc:357] T e13267498917432ba92b6badb76f5d73 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:47.139182 1191 raft_consensus.cc:383] T e13267498917432ba92b6badb76f5d73 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:47.139241 1191 raft_consensus.cc:738] T e13267498917432ba92b6badb76f5d73 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:47.140023 446 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_5dea789d6bb64579bde4b03309ab08a7" } modify_external_catalogs: true
I20250113 02:51:47.140336 446 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_5dea789d6bb64579bde4b03309ab08a7" } modify_external_catalogs: true
I20250113 02:51:47.141901 1196 log.cc:1198] T acd83024592e431898e9675c17f50d4f P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/acd83024592e431898e9675c17f50d4f
I20250113 02:51:47.145918 741 tablet_service.cc:1514] Processing DeleteTablet for tablet 29c17fd90d774c029a35e77888358067 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:35 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:47.147910 1196 ts_tablet_manager.cc:1939] T acd83024592e431898e9675c17f50d4f P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:47.148526 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_204b526f8a5448f48f761b4cc7101fef"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:47.149448 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_204b526f8a5448f48f761b4cc7101fef in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:47.149987 1191 consensus_queue.cc:260] T e13267498917432ba92b6badb76f5d73 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:47.154682 447 master_service.cc:946] ReplaceTablet: received request to replace tablet 344fa45adf1749f1b5b299d9a6d893c9 from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:47.154950 887 tablet_service.cc:1467] Processing CreateTablet for tablet f33f59dee0f7489eb6e555c3d062a0f6 (DEFAULT_TABLE table=default.Table_41ea2489c272425292a6a6562964690f [id=33fe2300ba2c49559c1bc4b0e543e4f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:47.155846 1191 ts_tablet_manager.cc:1428] T e13267498917432ba92b6badb76f5d73 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.017s user 0.001s sys 0.001s
W20250113 02:51:47.158069 998 consensus_peers.cc:487] T 4fe54131aafb40ee9d8bc5ea299af193 P 4216a7f853d642b7bc3e5cff458f3aca -> Peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Couldn't send request to peer df69f48656b44bb890f76587ee69cc0f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4fe54131aafb40ee9d8bc5ea299af193. This is attempt 1: this message will repeat every 5th retry.
I20250113 02:51:47.161034 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1346) took 5375 ms. Trace:
I20250113 02:51:47.162670 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9bfecc0884a64b4583506f6a050820f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:47.162941 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 391615b7658d44dea2a72a7bd1656856. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:47.158291 1191 tablet_bootstrap.cc:492] T d8419fec408c40f6bd46c39de7b3dce9 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:47.165596 1191 tablet_bootstrap.cc:654] T d8419fec408c40f6bd46c39de7b3dce9 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:47.168102 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e64d7e847dcd4253b6b4c7464fc5a798. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:47.168361 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69e200d941164e21848c5cfc5fbcd8f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:47.168619 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d560b06a2f64a02b81d8d1e4bb75f66. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:47.172071 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 52e2b698832844baa0edfe732909e075. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:47.172304 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e02c3d5511e47f59c38c8e6e3cb51d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:47.172605 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 766d0c959e5a4c969c75828e2523cfd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:47.176084 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 657c15d348ec43df8074de4c62c213a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:47.181219 1191 tablet_bootstrap.cc:492] T d8419fec408c40f6bd46c39de7b3dce9 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:47.181406 1191 ts_tablet_manager.cc:1397] T d8419fec408c40f6bd46c39de7b3dce9 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.023s user 0.001s sys 0.001s
I20250113 02:51:47.182111 1191 raft_consensus.cc:357] T d8419fec408c40f6bd46c39de7b3dce9 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:47.186419 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b4e03b52495d46f8be3f1544e5fda5e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:47.183167 2853 raft_consensus.cc:491] T 8337f3c82d654ecdac2cbc1ffa88ef05 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4216a7f853d642b7bc3e5cff458f3aca)
I20250113 02:51:47.188613 2853 raft_consensus.cc:513] T 8337f3c82d654ecdac2cbc1ffa88ef05 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } }
I20250113 02:51:47.189311 2853 leader_election.cc:290] T 8337f3c82d654ecdac2cbc1ffa88ef05 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:47.189613 1191 raft_consensus.cc:383] T d8419fec408c40f6bd46c39de7b3dce9 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:47.189882 1191 raft_consensus.cc:738] T d8419fec408c40f6bd46c39de7b3dce9 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
W20250113 02:51:47.190162 711 leader_election.cc:343] T 8337f3c82d654ecdac2cbc1ffa88ef05 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Not found: Tablet not found: 8337f3c82d654ecdac2cbc1ffa88ef05
I20250113 02:51:47.190274 1191 consensus_queue.cc:260] T d8419fec408c40f6bd46c39de7b3dce9 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:47.192312 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5eb5247d3704e7aae0d4e08dc19ae5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:47.192636 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f33f59dee0f7489eb6e555c3d062a0f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:47.161334 1195 rpcz_store.cc:276] 0113 02:51:41.785829 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:41.785948 (+ 119us) service_pool.cc:225] Handling call
0113 02:51:47.155822 (+5369874us) ts_tablet_manager.cc:1145] Deleting tablet 4cc15487be474354b995c4e415276a18
0113 02:51:47.155890 (+ 68us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:47.160963 (+ 5073us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5369633}
I20250113 02:51:47.200533 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_026ff82c780e4fb5803728166c8952ce" } new_table_name: "default.Table_b7a11e4fd6464f4f9c70e8ab146858fb" modify_external_catalogs: true
I20250113 02:51:47.200891 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_80b52e8cafc549b1b530660a5fd2bb1d" } new_table_name: "default.Table_6047c5df892b46db84ecbfff30a81eec" modify_external_catalogs: true
I20250113 02:51:47.202443 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_5c113fa1072647119025f2ea5c159722"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:47.202963 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5c113fa1072647119025f2ea5c159722 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:47.201601 1191 ts_tablet_manager.cc:1428] T d8419fec408c40f6bd46c39de7b3dce9 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.020s user 0.001s sys 0.001s
I20250113 02:51:47.203620 1191 tablet_bootstrap.cc:492] T 303d911dfa2f4abbb053ead33db6d71c P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:47.204294 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1348) took 5368 ms. Trace:
I20250113 02:51:47.204384 1195 rpcz_store.cc:276] 0113 02:51:41.836162 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:41.836747 (+ 585us) service_pool.cc:225] Handling call
0113 02:51:47.204158 (+5367411us) ts_tablet_manager.cc:1145] Deleting tablet 8ee0b2e06dc74faab48912dd16fe1537
0113 02:51:47.204169 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:47.204276 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5336396}
I20250113 02:51:47.204999 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_0468611ec68e47109f5357935e50874d" } new_table_name: "default.Table_eadba9f9ab724461b018a9423dcac195" modify_external_catalogs: true
I20250113 02:51:47.205657 2853 raft_consensus.cc:491] T e13267498917432ba92b6badb76f5d73 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:47.205773 2853 raft_consensus.cc:513] T e13267498917432ba92b6badb76f5d73 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:47.206300 2853 leader_election.cc:290] T e13267498917432ba92b6badb76f5d73 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:47.206867 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e13267498917432ba92b6badb76f5d73" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:47.207525 711 leader_election.cc:343] T e13267498917432ba92b6badb76f5d73 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:47.208470 1060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e13267498917432ba92b6badb76f5d73" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
W20250113 02:51:47.208488 709 leader_election.cc:343] T 8337f3c82d654ecdac2cbc1ffa88ef05 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Not found: Tablet not found: 8337f3c82d654ecdac2cbc1ffa88ef05
I20250113 02:51:47.208626 709 leader_election.cc:304] T 8337f3c82d654ecdac2cbc1ffa88ef05 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:47.208914 1194 raft_consensus.cc:357] T 6a4c6dc08df744e6856bdf99ab65c575 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:47.209125 1194 raft_consensus.cc:383] T 6a4c6dc08df744e6856bdf99ab65c575 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:47.209200 1194 raft_consensus.cc:738] T 6a4c6dc08df744e6856bdf99ab65c575 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:47.209228 2853 raft_consensus.cc:2743] T 8337f3c82d654ecdac2cbc1ffa88ef05 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250113 02:51:47.209430 1194 consensus_queue.cc:260] T 6a4c6dc08df744e6856bdf99ab65c575 P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:47.209961 1191 tablet_bootstrap.cc:654] T 303d911dfa2f4abbb053ead33db6d71c P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:47.209918 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_c483e974da4240c59118749d623aa221"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:47.210475 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c483e974da4240c59118749d623aa221 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:47.210091 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1350) took 5362 ms. Trace:
I20250113 02:51:47.210754 1195 rpcz_store.cc:276] 0113 02:51:41.847573 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:41.847731 (+ 158us) service_pool.cc:225] Handling call
0113 02:51:47.209976 (+5362245us) ts_tablet_manager.cc:1145] Deleting tablet 9b9e7deda7334ff4a3429f70d9b3a7fc
0113 02:51:47.209986 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:47.210074 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5341302}
I20250113 02:51:47.211174 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1345) took 5436 ms. Trace:
I20250113 02:51:47.211297 1195 rpcz_store.cc:276] 0113 02:51:41.774275 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:41.774366 (+ 91us) service_pool.cc:225] Handling call
0113 02:51:47.211074 (+5436708us) ts_tablet_manager.cc:1145] Deleting tablet 316cb2863ed645158a8f72e3cb0a94fd
0113 02:51:47.211083 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:47.211159 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5334796}
I20250113 02:51:47.211647 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_7fd1c363a0154f30b11fbb0b6a091d39"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:47.212018 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1347) took 5387 ms. Trace:
I20250113 02:51:47.212126 1195 rpcz_store.cc:276] 0113 02:51:41.824856 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:41.836055 (+ 11199us) service_pool.cc:225] Handling call
0113 02:51:47.211784 (+5375729us) ts_tablet_manager.cc:1145] Deleting tablet 85d13f8abbec484d82c4d0b86480531b
0113 02:51:47.211792 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:47.211999 (+ 207us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5330082}
W20250113 02:51:47.212314 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7fd1c363a0154f30b11fbb0b6a091d39 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:47.212792 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1361) took 4967 ms. Trace:
I20250113 02:51:47.212926 1195 rpcz_store.cc:276] 0113 02:51:42.245738 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:42.245792 (+ 54us) service_pool.cc:225] Handling call
0113 02:51:47.212692 (+4966900us) ts_tablet_manager.cc:1145] Deleting tablet 1c153e47a7aa4bf5ae61dd19a91930b9
0113 02:51:47.212701 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:47.212776 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4966712}
I20250113 02:51:47.213425 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1362) took 4669 ms. Trace:
I20250113 02:51:47.213547 1195 rpcz_store.cc:276] 0113 02:51:42.544354 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:42.544441 (+ 87us) service_pool.cc:225] Handling call
0113 02:51:47.213306 (+4668865us) ts_tablet_manager.cc:1145] Deleting tablet 86c8c07f90d2470ebb2b3f1d16daf4cd
0113 02:51:47.213314 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:47.213410 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4668603}
W20250113 02:51:47.213819 709 leader_election.cc:343] T e13267498917432ba92b6badb76f5d73 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:47.213927 709 leader_election.cc:304] T e13267498917432ba92b6badb76f5d73 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:47.214056 1195 tablet_replica.cc:331] T f105f8b32a654e4ca154e8fc0cb96091 P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:47.214278 1195 raft_consensus.cc:2238] T f105f8b32a654e4ca154e8fc0cb96091 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:47.214320 2853 raft_consensus.cc:2743] T e13267498917432ba92b6badb76f5d73 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:47.214776 1195 raft_consensus.cc:2267] T f105f8b32a654e4ca154e8fc0cb96091 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:47.132076 1198 rpcz_store.cc:276] 0113 02:51:45.858386 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:45.858437 (+ 51us) service_pool.cc:225] Handling call
0113 02:51:47.069297 (+1210860us) ts_tablet_manager.cc:1145] Deleting tablet 79aaab73797c474bae06831fa308b066
0113 02:51:47.069307 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:47.095949 (+ 26642us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1210729}
I20250113 02:51:47.216334 1195 ts_tablet_manager.cc:1905] T f105f8b32a654e4ca154e8fc0cb96091 P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:47.217536 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1500) took 1358 ms. Trace:
I20250113 02:51:47.217634 1198 rpcz_store.cc:276] 0113 02:51:45.859141 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:45.859197 (+ 56us) service_pool.cc:225] Handling call
0113 02:51:47.215754 (+1356557us) ts_tablet_manager.cc:1145] Deleting tablet 8cc78bf7d6324b7ba10a1b1045923f2b
0113 02:51:47.215764 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:47.217513 (+ 1749us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1356419}
I20250113 02:51:47.217949 1198 tablet_replica.cc:331] T 58053845187f466599750f2a62f4c3e1 P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:47.218174 1198 raft_consensus.cc:2238] T 58053845187f466599750f2a62f4c3e1 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: Raft consensus shutting down.
I20250113 02:51:47.218410 1198 pending_rounds.cc:62] T 58053845187f466599750f2a62f4c3e1 P 4216a7f853d642b7bc3e5cff458f3aca: Trying to abort 1 pending ops.
I20250113 02:51:47.218452 1198 pending_rounds.cc:69] T 58053845187f466599750f2a62f4c3e1 P 4216a7f853d642b7bc3e5cff458f3aca: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113673548677251072 op_type: NO_OP noop_request { }
I20250113 02:51:47.218564 1198 raft_consensus.cc:2883] T 58053845187f466599750f2a62f4c3e1 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250113 02:51:47.218644 1198 raft_consensus.cc:2267] T 58053845187f466599750f2a62f4c3e1 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:47.219698 1198 ts_tablet_manager.cc:1905] T 58053845187f466599750f2a62f4c3e1 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250113 02:51:47.220270 440 catalog_manager.cc:4839] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 79aaab73797c474bae06831fa308b066 because the tablet was not found. No further retry: Not found: Tablet not found: 79aaab73797c474bae06831fa308b066
W20250113 02:51:47.220835 440 catalog_manager.cc:4839] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 8cc78bf7d6324b7ba10a1b1045923f2b because the tablet was not found. No further retry: Not found: Tablet not found: 8cc78bf7d6324b7ba10a1b1045923f2b
I20250113 02:51:47.221529 1191 tablet_bootstrap.cc:492] T 303d911dfa2f4abbb053ead33db6d71c P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:47.222985 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_929bc069a74042cf8ce8cfef9ab4028e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:47.223495 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_929bc069a74042cf8ce8cfef9ab4028e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:47.232595 1194 ts_tablet_manager.cc:1428] T 6a4c6dc08df744e6856bdf99ab65c575 P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.103s user 0.001s sys 0.001s
W20250113 02:51:47.229653 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 905542fb0dce4bbb8852b74d94c7b2a6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 905542fb0dce4bbb8852b74d94c7b2a6 already in progress: creating tablet
I20250113 02:51:47.234413 893 tablet_service.cc:1467] Processing CreateTablet for tablet a11377e635b044e58a730fda3c25ae27 (DEFAULT_TABLE table=default.Table_6a88830e770641c4bbe405c0ef700d67 [id=ba1edf2b734e48cb89a77ca59fa717c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:47.235044 896 tablet_service.cc:1467] Processing CreateTablet for tablet d664e48a65ae4e53aedbff0d55bfc17b (DEFAULT_TABLE table=default.Table_ed460a21a2184db893d2e76e67cdb974 [id=fb3d90554f7943099cd541f610084cbf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:47.235338 894 tablet_service.cc:1467] Processing CreateTablet for tablet ac5e8cb88591488089575fe679815124 (DEFAULT_TABLE table=default.Table_f88977415dbf4f43abaa5fdfa7c46f4d [id=1791821062614708b2787fecac4fc837]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:47.235417 897 tablet_service.cc:1467] Processing CreateTablet for tablet 03daa471e30240688d711b493cff1e6d (DEFAULT_TABLE table=default.Table_9bdc99d09f2d405b88dcb4a48144ca4b [id=e3f0e6356ea6410b91977cb88255ca20]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:47.235811 892 tablet_service.cc:1467] Processing CreateTablet for tablet 32e99ae9bbe743ff8e3e71b0043d5a1e (DEFAULT_TABLE table=default.Table_e01a09e962074d5c87c15ad97b576d09 [id=5b4c62389b5c4487982fec74f9c15280]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:47.235875 891 tablet_service.cc:1467] Processing CreateTablet for tablet 5f1e613a8b934e2c89a6eb08ff18c913 (DEFAULT_TABLE table=default.Table_1416b17502f140db98544370a86bdcb6 [id=d7c4c089b0454fada09f7d063d41b876]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:47.235811 886 tablet_service.cc:1467] Processing CreateTablet for tablet 4200521dc098474fb5f498676d70f023 (DEFAULT_TABLE table=default.Table_186f88df8da44d0c87ac23e9406895d6 [id=67ae377d727840489d2edccd7579e037]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:47.236277 895 tablet_service.cc:1467] Processing CreateTablet for tablet b786b007e4e04e34a70f9f585acee9fa (DEFAULT_TABLE table=default.Table_83eaa3dec71e42e2921fc5957777f78e [id=f554a27db5a4443185c24bd70fec2461]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:47.219681 1192 tablet_bootstrap.cc:492] T 6001369444894b48b51a48fb60ddc52a P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:47.237315 1192 ts_tablet_manager.cc:1397] T 6001369444894b48b51a48fb60ddc52a P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.466s user 0.019s sys 0.069s
I20250113 02:51:47.238407 888 tablet_service.cc:1467] Processing CreateTablet for tablet bc88415f52a0424db18574c886dd6f1a (DEFAULT_TABLE table=default.Table_27eb308f2dc84858b3d1a2fa4b42c3f6 [id=1cfda70b239543dbbd14d5857a6bbf3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:47.239367 898 tablet_service.cc:1467] Processing CreateTablet for tablet 9670fe8bf3a3403b8c8cb99d03cc3a78 (DEFAULT_TABLE table=default.Table_f6c8709d5ec649f782b85cbbc6517f5b [id=ac06bedbd2bb46e688a236a61e8f75fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:47.239464 884 tablet_service.cc:1467] Processing CreateTablet for tablet 3389e6747fb145c5a78c90a5ab7d3f2a (DEFAULT_TABLE table=default.Table_e0d35592e5c842afbe63a4fb305cae3a [id=a76443d62ea04c03ad5942a8c1fb0c5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:47.239862 880 tablet_service.cc:1467] Processing CreateTablet for tablet 423d5d1a57484806b76565aa2292c2a1 (DEFAULT_TABLE table=default.Table_cf16887c855544b1987f7539a58537fb [id=7eb31eded10f498b918c00ec5e3f860a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:47.240290 890 tablet_service.cc:1467] Processing CreateTablet for tablet 9842ee1a8e3c4011add4b117a874f620 (DEFAULT_TABLE table=default.Table_22ace32627b84d33b3fe01731f8d2c07 [id=2661227c8a684d4eb08d2c46738f7aed]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:47.240739 881 tablet_service.cc:1467] Processing CreateTablet for tablet d75822a1e2184e3288e692e564045a6a (DEFAULT_TABLE table=default.Table_a4837fc75b6f4d2c8756d0967cce0b31 [id=edf26613cff84dbb95fb2800dad48e06]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:47.247139 1194 tablet_bootstrap.cc:492] T fe8f075d4c9a4f9abe372bfdb81a829c P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
I20250113 02:51:47.246487 889 tablet_service.cc:1467] Processing CreateTablet for tablet 04078d87cb8043fbbd51a7c1cbcd398d (DEFAULT_TABLE table=default.Table_6f8ebd124a0e4bfb83502fc9f0f97130 [id=07fc217c213342aaa6c47f0ef8325725]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:47.248533 1198 ts_tablet_manager.cc:1918] T 58053845187f466599750f2a62f4c3e1 P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250113 02:51:47.248713 1198 log.cc:1198] T 58053845187f466599750f2a62f4c3e1 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/58053845187f466599750f2a62f4c3e1
I20250113 02:51:47.249184 1198 ts_tablet_manager.cc:1939] T 58053845187f466599750f2a62f4c3e1 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:47.229971 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 7aa4a8fbf5204ee89aa8bdc3047bc7e5
I20250113 02:51:47.256034 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1501) took 1391 ms. Trace:
I20250113 02:51:47.256300 1198 rpcz_store.cc:276] 0113 02:51:45.859836 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:45.859879 (+ 43us) service_pool.cc:225] Handling call
0113 02:51:47.217899 (+1358020us) ts_tablet_manager.cc:1145] Deleting tablet 58053845187f466599750f2a62f4c3e1
0113 02:51:47.217910 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:47.244762 (+ 26852us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:47.248428 (+ 3666us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:47.250803 (+ 2375us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1357827}
I20250113 02:51:47.258535 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_edfa039267c74167bfbbabc4eed8e0c1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:47.259079 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_edfa039267c74167bfbbabc4eed8e0c1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:47.256208 885 tablet_service.cc:1467] Processing CreateTablet for tablet 752c1291da404f3a81548cfa70eee063 (DEFAULT_TABLE table=default.Table_50f8312d0d834e25ab560f9c59fa3ce0 [id=bd87a87c82c9467f96c3c7638457ae0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:47.250429 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_547e8a172dd04141ba47718ef76d5234"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:47.261452 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_547e8a172dd04141ba47718ef76d5234 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:47.262423 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet 58053845187f466599750f2a62f4c3e1 (table default.Table_6a1167a25a9d452e9b7c477aadd97279 [id=2c51fbdb672d4b01a5932fade2beb3d4]) successfully deleted
I20250113 02:51:47.262684 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet 7aa4a8fbf5204ee89aa8bdc3047bc7e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:47 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:47.262851 899 tablet_service.cc:1467] Processing CreateTablet for tablet 802998e270094c258883579852ff4b4e (DEFAULT_TABLE table=default.Table_484890185d6b4dea9fe5ed3102b5b625 [id=6a7f6dedc485453b86d7f885bb4ad20e]), partition=RANGE (key) PARTITION UNBOUNDED
W20250113 02:51:47.238875 823 tablet.cc:2367] T d8419fec408c40f6bd46c39de7b3dce9 P 1186fa869eb14812832fa6562531f196: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250113 02:51:47.241951 2853 consensus_queue.cc:1035] T 2470b010b571444eba3f132323038f33 P 1186fa869eb14812832fa6562531f196 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 02:51:47.265309 1191 ts_tablet_manager.cc:1397] T 303d911dfa2f4abbb053ead33db6d71c P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.062s user 0.003s sys 0.000s
W20250113 02:51:47.261435 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet 9572215fd2eb45faa99cc236fcc0d5ea because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9572215fd2eb45faa99cc236fcc0d5ea already in progress: creating tablet
I20250113 02:51:47.267139 1191 raft_consensus.cc:357] T 303d911dfa2f4abbb053ead33db6d71c P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:47.267633 1191 raft_consensus.cc:383] T 303d911dfa2f4abbb053ead33db6d71c P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:47.267827 1191 raft_consensus.cc:738] T 303d911dfa2f4abbb053ead33db6d71c P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
W20250113 02:51:47.268685 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet becee3bc8d824a6d94e58730416aa076 consensus state has no leader
I20250113 02:51:47.272694 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_3487754fcaa24feaa2e5c3d3c267f9fb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:47.273202 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3487754fcaa24feaa2e5c3d3c267f9fb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:47.275677 746 tablet_service.cc:1514] Processing DeleteTablet for tablet ed68a2b7ebcf4841bef54798dd3ed80d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:27 UTC) from {username='slave'} at 127.0.0.1:55398
W20250113 02:51:47.276384 439 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.60:42067: connect: Connection refused (error 111) [suppressed 6 similar messages]
I20250113 02:51:47.276505 746 tablet_service.cc:1514] Processing DeleteTablet for tablet 7aa4a8fbf5204ee89aa8bdc3047bc7e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:47 UTC) from {username='slave'} at 127.0.0.1:55398
W20250113 02:51:47.279587 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet acb0ce94537f477a97fef095f7d16a09 consensus state has no leader
W20250113 02:51:47.284318 998 consensus_peers.cc:487] T f4379f7fc6b54e6785f27975d9030cce P 4216a7f853d642b7bc3e5cff458f3aca -> Peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Couldn't send request to peer df69f48656b44bb890f76587ee69cc0f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250113 02:51:47.268337 1191 consensus_queue.cc:260] T 303d911dfa2f4abbb053ead33db6d71c P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:47.288367 1191 ts_tablet_manager.cc:1428] T 303d911dfa2f4abbb053ead33db6d71c P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.023s user 0.003s sys 0.000s
W20250113 02:51:47.293406 998 consensus_peers.cc:487] T c04be975c8c049b892b16129ed47a142 P 4216a7f853d642b7bc3e5cff458f3aca -> Peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Couldn't send request to peer df69f48656b44bb890f76587ee69cc0f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250113 02:51:47.293668 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 4cc15487be474354b995c4e415276a18 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4cc15487be474354b995c4e415276a18 already in progress: creating tablet
W20250113 02:51:47.294121 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 8ee0b2e06dc74faab48912dd16fe1537 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8ee0b2e06dc74faab48912dd16fe1537 already in progress: creating tablet
W20250113 02:51:47.294564 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 9b9e7deda7334ff4a3429f70d9b3a7fc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9b9e7deda7334ff4a3429f70d9b3a7fc already in progress: creating tablet
W20250113 02:51:47.295073 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 316cb2863ed645158a8f72e3cb0a94fd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 316cb2863ed645158a8f72e3cb0a94fd already in progress: creating tablet
W20250113 02:51:47.295575 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 85d13f8abbec484d82c4d0b86480531b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 85d13f8abbec484d82c4d0b86480531b already in progress: creating tablet
W20250113 02:51:47.296025 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 1c153e47a7aa4bf5ae61dd19a91930b9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1c153e47a7aa4bf5ae61dd19a91930b9 already in progress: creating tablet
W20250113 02:51:47.296499 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 86c8c07f90d2470ebb2b3f1d16daf4cd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 86c8c07f90d2470ebb2b3f1d16daf4cd already in progress: creating tablet
I20250113 02:51:47.298277 1191 tablet_bootstrap.cc:492] T f9269967dea94fc3a337546ad0842e11 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:47.299454 1192 raft_consensus.cc:357] T 6001369444894b48b51a48fb60ddc52a P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:47.299669 1192 raft_consensus.cc:383] T 6001369444894b48b51a48fb60ddc52a P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:47.299728 1192 raft_consensus.cc:738] T 6001369444894b48b51a48fb60ddc52a P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:47.299937 1192 consensus_queue.cc:260] T 6001369444894b48b51a48fb60ddc52a P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:47.300501 1192 ts_tablet_manager.cc:1428] T 6001369444894b48b51a48fb60ddc52a P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.063s user 0.001s sys 0.000s
I20250113 02:51:47.301275 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1502) took 1440 ms. Trace:
I20250113 02:51:47.301486 1198 rpcz_store.cc:276] 0113 02:51:45.860535 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:45.860607 (+ 72us) service_pool.cc:225] Handling call
0113 02:51:47.257150 (+1396543us) ts_tablet_manager.cc:1145] Deleting tablet cfa963a6df094ce0b9c54408f75ca6b2
0113 02:51:47.258066 (+ 916us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:47.301253 (+ 43187us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1396393}
W20250113 02:51:47.303346 440 catalog_manager.cc:4839] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet cfa963a6df094ce0b9c54408f75ca6b2 because the tablet was not found. No further retry: Not found: Tablet not found: cfa963a6df094ce0b9c54408f75ca6b2
I20250113 02:51:47.305266 1192 tablet_bootstrap.cc:492] T fe3d35d9aaf64188aaa3c1ac38d279d8 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:47.307721 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1503) took 1423 ms. Trace:
I20250113 02:51:47.308041 1198 rpcz_store.cc:276] 0113 02:51:45.884677 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:45.884729 (+ 52us) service_pool.cc:225] Handling call
0113 02:51:47.302273 (+1417544us) ts_tablet_manager.cc:1145] Deleting tablet 2cd3b62ecc774d2eb658ed34e05864e1
0113 02:51:47.307417 (+ 5144us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:47.307702 (+ 285us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1417133}
I20250113 02:51:47.309067 1198 tablet_replica.cc:331] T 59fadc693f554854bc2874f8db7c64c8 P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:47.309624 1198 raft_consensus.cc:2238] T 59fadc693f554854bc2874f8db7c64c8 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus shutting down.
W20250113 02:51:47.309845 440 catalog_manager.cc:4852] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet 2cd3b62ecc774d2eb658ed34e05864e1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2cd3b62ecc774d2eb658ed34e05864e1 already in progress: creating tablet
I20250113 02:51:47.310168 1198 raft_consensus.cc:2267] T 59fadc693f554854bc2874f8db7c64c8 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:47.311777 1198 ts_tablet_manager.cc:1905] T 59fadc693f554854bc2874f8db7c64c8 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:47.316252 750 tablet_service.cc:1514] Processing DeleteTablet for tablet e13267498917432ba92b6badb76f5d73 with delete_type TABLET_DATA_DELETED (replaced by tablet 166f58d45b7b443ea54087e4ab48eaba) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:47.316509 746 tablet_service.cc:1514] Processing DeleteTablet for tablet d8419fec408c40f6bd46c39de7b3dce9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:25 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:47.321590 2842 raft_consensus.cc:491] T 80c647a3b5a1411e9130461115c53550 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:47.321717 2842 raft_consensus.cc:513] T 80c647a3b5a1411e9130461115c53550 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:47.322489 2842 leader_election.cc:290] T 80c647a3b5a1411e9130461115c53550 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:47.324426 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80c647a3b5a1411e9130461115c53550" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
W20250113 02:51:47.324990 709 leader_election.cc:343] T 80c647a3b5a1411e9130461115c53550 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:47.325505 746 tablet_service.cc:1514] Processing DeleteTablet for tablet 303d911dfa2f4abbb053ead33db6d71c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:25 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:47.333433 1191 tablet_bootstrap.cc:654] T f9269967dea94fc3a337546ad0842e11 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:47.263484 882 tablet_service.cc:1467] Processing CreateTablet for tablet 0d8cdaa36b424a53b0501a65f4e2ce7d (DEFAULT_TABLE table=default.Table_3fda809831ba43319e2ee62fb49dde92 [id=d3e184260dd8493d9ba55ad9729ace32]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:47.271762 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 32e99ae9bbe743ff8e3e71b0043d5a1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:47.344763 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac5e8cb88591488089575fe679815124. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:47.345176 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 752c1291da404f3a81548cfa70eee063. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:47.345551 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4200521dc098474fb5f498676d70f023. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:47.347025 1194 tablet_bootstrap.cc:654] T fe8f075d4c9a4f9abe372bfdb81a829c P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:47.348069 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d664e48a65ae4e53aedbff0d55bfc17b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:47.348304 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d8cdaa36b424a53b0501a65f4e2ce7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:47.350409 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f1e613a8b934e2c89a6eb08ff18c913. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:47.352596 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 04078d87cb8043fbbd51a7c1cbcd398d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:47.356122 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b786b007e4e04e34a70f9f585acee9fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:47.356163 894 tablet_service.cc:1467] Processing CreateTablet for tablet 1fc811dda69f413e92833ec302694468 (DEFAULT_TABLE table=default.Table_5b9ead21b07d41b2ad99e51e4c522ac2 [id=fc41dd4bce6048b0ac0953ae0ea4c382]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:47.356402 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc88415f52a0424db18574c886dd6f1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:47.356608 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a11377e635b044e58a730fda3c25ae27. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:47.356794 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d75822a1e2184e3288e692e564045a6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:47.356956 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03daa471e30240688d711b493cff1e6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:47.357203 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9670fe8bf3a3403b8c8cb99d03cc3a78. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:47.357805 1195 ts_tablet_manager.cc:1918] T f105f8b32a654e4ca154e8fc0cb96091 P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250113 02:51:47.357918 1195 log.cc:1198] T f105f8b32a654e4ca154e8fc0cb96091 P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/f105f8b32a654e4ca154e8fc0cb96091
I20250113 02:51:47.358021 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3389e6747fb145c5a78c90a5ab7d3f2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:47.358255 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 423d5d1a57484806b76565aa2292c2a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:47.358249 1195 ts_tablet_manager.cc:1939] T f105f8b32a654e4ca154e8fc0cb96091 P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:47.359019 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9842ee1a8e3c4011add4b117a874f620. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:47.303809 887 tablet_service.cc:1467] Processing CreateTablet for tablet ae75154319bb4c3991cd3c5b08ce4a1e (DEFAULT_TABLE table=default.Table_d20226843534409a8cabaa764bcd9ce5 [id=4309d6902f5c4d3b8ee14a7f4e63eda9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:47.323097 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80c647a3b5a1411e9130461115c53550" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:47.360205 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae75154319bb4c3991cd3c5b08ce4a1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:47.361191 1192 tablet_bootstrap.cc:654] T fe3d35d9aaf64188aaa3c1ac38d279d8 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:47.366765 881 tablet_service.cc:1467] Processing CreateTablet for tablet ec0fb729e3a14d50a23faf81c8082f1e (DEFAULT_TABLE table=default.Table_e38f2c9d348948238d245b93b82b0dce [id=7caed6f6020f49a5898a2261f653070d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:47.367089 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ec0fb729e3a14d50a23faf81c8082f1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:47.368645 896 tablet_service.cc:1514] Processing DeleteTablet for tablet e8a0482683c7493eb19e7da5c78c095c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:46 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:47.369377 888 tablet_service.cc:1514] Processing DeleteTablet for tablet 67baaab1e36e415381579d7385d4ab5a with delete_type TABLET_DATA_DELETED (replaced by tablet c368b5b0c3a940c4b4efb64c946375c3) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:47.369530 2368 raft_consensus.cc:491] T 8f2fb58fc24d4692a3fe9a652e076aed P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:47.369717 2368 raft_consensus.cc:513] T 8f2fb58fc24d4692a3fe9a652e076aed P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:47.369448 882 tablet_service.cc:1467] Processing CreateTablet for tablet fd2151a6bf884603907340f73d709975 (DEFAULT_TABLE table=default.Table_a0736ea5f5ac4ab1a819a6c08d410086 [id=dd6eee7466da4e5499f8193de2c56e02]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:47.370978 1196 tablet_replica.cc:331] T 67baaab1e36e415381579d7385d4ab5a P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:47.371142 1196 raft_consensus.cc:2238] T 67baaab1e36e415381579d7385d4ab5a P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:47.371265 1196 raft_consensus.cc:2267] T 67baaab1e36e415381579d7385d4ab5a P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:47.371717 1196 ts_tablet_manager.cc:1905] T 67baaab1e36e415381579d7385d4ab5a P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250113 02:51:47.372125 1110 tablet.cc:2367] T 6001369444894b48b51a48fb60ddc52a P 4216a7f853d642b7bc3e5cff458f3aca: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250113 02:51:47.375046 888 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f8eb43a39774d2c8e9cb4e88a405b0f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:23 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:47.375442 888 tablet_service.cc:1514] Processing DeleteTablet for tablet e570465a3d4046b1ac19a3a034fe91f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:27 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:47.375839 888 tablet_service.cc:1514] Processing DeleteTablet for tablet 283fc7f1c74e480c8a63c74d3ae9eb7d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:24 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:47.377156 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f2fb58fc24d4692a3fe9a652e076aed" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:47.377498 2368 leader_election.cc:290] T 8f2fb58fc24d4692a3fe9a652e076aed P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
W20250113 02:51:47.377604 998 leader_election.cc:343] T 8f2fb58fc24d4692a3fe9a652e076aed P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:47.369618 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1fc811dda69f413e92833ec302694468. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:47.380623 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd2151a6bf884603907340f73d709975. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:47.380884 442 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.0.104.2:38467, user_credentials={real_user=slave}} blocked reactor thread for 110128us
I20250113 02:51:47.381248 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet acd83024592e431898e9675c17f50d4f (table default.Table_25cb4fc43dc6485ca0e4ec0cf048105d [id=044304e1c4b44cc28b8990d6f28145a8]) successfully deleted
W20250113 02:51:47.381454 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet b946dd2d428a43d399bb75a43d843a28 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b946dd2d428a43d399bb75a43d843a28 already in progress: creating tablet
W20250113 02:51:47.381914 711 leader_election.cc:343] T 80c647a3b5a1411e9130461115c53550 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:47.382048 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 802998e270094c258883579852ff4b4e. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:47.383325 442 catalog_manager.cc:4852] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): delete failed for tablet e8a0482683c7493eb19e7da5c78c095c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e8a0482683c7493eb19e7da5c78c095c already in progress: creating tablet
I20250113 02:51:47.387249 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f2fb58fc24d4692a3fe9a652e076aed" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196" is_pre_election: true
W20250113 02:51:47.392365 440 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet becee3bc8d824a6d94e58730416aa076 (table default.Table_6047c5df892b46db84ecbfff30a81eec [id=413cdddd25cb4a80b5b9a32fba699a35], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet becee3bc8d824a6d94e58730416aa076 consensus state has no leader
I20250113 02:51:47.402719 448 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_dc40c4f395d846c7aa489903545cfc12" } modify_external_catalogs: true
I20250113 02:51:47.402930 448 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_dc40c4f395d846c7aa489903545cfc12" } modify_external_catalogs: true
I20250113 02:51:47.403751 446 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_f6e2816287cf46e1b05c86ba38199184" } modify_external_catalogs: true
I20250113 02:51:47.403900 446 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_f6e2816287cf46e1b05c86ba38199184" } modify_external_catalogs: true
I20250113 02:51:47.405261 449 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_7698d677addf448fa62c659dad28b1e4" } modify_external_catalogs: true
I20250113 02:51:47.405398 449 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_7698d677addf448fa62c659dad28b1e4" } modify_external_catalogs: true
I20250113 02:51:47.406059 451 master_service.cc:946] ReplaceTablet: received request to replace tablet cd6dcfecf4324542b241f909926756ea from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:47.411796 883 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a4c6dc08df744e6856bdf99ab65c575 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:26 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:47.413094 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_d0917724c0264ec3bc07ba9a1d519cdf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:47.413556 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d0917724c0264ec3bc07ba9a1d519cdf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:47.417163 455 catalog_manager.cc:5526] T 247af1a17df04b27898f86642336d51e P 4216a7f853d642b7bc3e5cff458f3aca reported cstate change: term changed from 0 to 1, leader changed from <none> to 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3). New cstate: current_term: 1 leader_uuid: "4216a7f853d642b7bc3e5cff458f3aca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } health_report { overall_health: UNKNOWN } } }
I20250113 02:51:47.417594 455 catalog_manager.cc:5618] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587) does not have the latest schema for tablet 247af1a17df04b27898f86642336d51e (table default.Table_b3926f9a1b6a4f249cf1e20b1129438e [id=43940b76df2c4fb5bc58e5e2975cac8b]). Expected version 2 got 0
I20250113 02:51:47.417846 455 catalog_manager.cc:5618] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587) does not have the latest schema for tablet 8f2fb58fc24d4692a3fe9a652e076aed (table default.Table_a752553e5fcc4ed1a56d9d5197d6b885 [id=8edee52f05e24c03b088ce58f4b3f4d9]). Expected version 1 got 0
I20250113 02:51:47.420233 1032 tablet_service.cc:1467] Processing CreateTablet for tablet a6a8e747ba2f4f749fa70694cf6fdd83 (DEFAULT_TABLE table=default.Table_0532c8897e3a478ca8602439e3752e72 [id=a84b79dfaadb4e92957fbe5f739b7a01]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:47.420598 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6a8e747ba2f4f749fa70694cf6fdd83. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:47.420866 1026 tablet_service.cc:1467] Processing CreateTablet for tablet fcd60b89f5bb46f6b682f231c84204ef (DEFAULT_TABLE table=default.Table_643c47db97214c82861277c1935d5032 [id=410ff6c1baa3428aa118233a87a8d001]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:47.421212 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fcd60b89f5bb46f6b682f231c84204ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:47.426602 1035 tablet_service.cc:1467] Processing CreateTablet for tablet ab366fbcf41e4969a71f3ba30ba24cf0 (DEFAULT_TABLE table=default.Table_8a46b6886be646d48357f268651520c6 [id=7dd719e3dbc045a7b5290b0d0d05b29b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:47.426903 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab366fbcf41e4969a71f3ba30ba24cf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:47.440009 1024 tablet_service.cc:1467] Processing CreateTablet for tablet 1bdb4f2ad28b4aec9d574816ab9d4f32 (DEFAULT_TABLE table=default.Table_8a69c12869c44953bde2307180dc1a88 [id=93deb7801dcb418e825d465529ceb4c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:47.440429 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1bdb4f2ad28b4aec9d574816ab9d4f32. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:47.442677 1030 tablet_service.cc:1467] Processing CreateTablet for tablet a35558e3f1fa4a36a174adde70d7cd44 (DEFAULT_TABLE table=default.Table_567d446e6ae54abf9ae7586692b5780a [id=70089f757a1c45f6b983f216fb167478]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:47.443017 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a35558e3f1fa4a36a174adde70d7cd44. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:47.451715 893 tablet_service.cc:1514] Processing DeleteTablet for tablet 7aa4a8fbf5204ee89aa8bdc3047bc7e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:47 UTC) from {username='slave'} at 127.0.0.1:37366
W20250113 02:51:47.451764 998 consensus_peers.cc:487] T 68c4a7e8c0fb4feb85498709b00d3258 P 4216a7f853d642b7bc3e5cff458f3aca -> Peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Couldn't send request to peer df69f48656b44bb890f76587ee69cc0f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250113 02:51:47.452878 2904 tablet.cc:1721] T c13e9a51c1e74b44b352bade78390ba5 P 4216a7f853d642b7bc3e5cff458f3aca: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250113 02:51:47.453073 1030 tablet_service.cc:1467] Processing CreateTablet for tablet d88f22c3f74a417c964e282d951f0c99 (DEFAULT_TABLE table=default.Table_8530e7723ac8458d9990e3c5c6286447 [id=1fd319f7088445388c4149bba313c531]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:47.453402 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d88f22c3f74a417c964e282d951f0c99. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:47.459451 1026 tablet_service.cc:1467] Processing CreateTablet for tablet 6720c7558e084472a1f67c08323dc98e (DEFAULT_TABLE table=default.Table_eaf6851fc514494a95392ba107626e41 [id=e05095ce043c43e4bb56ea6e2f3bca8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:47.459872 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6720c7558e084472a1f67c08323dc98e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:47.463207 1032 tablet_service.cc:1467] Processing CreateTablet for tablet 796b5f37479d460e8df1d00f1780ec7d (DEFAULT_TABLE table=default.Table_1e0ca8b977914d1aaf79792357ea51a7 [id=d5ed8b5ba94a43768b4ee73516124ca3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:47.463253 2476 raft_consensus.cc:491] T 9f8eb43a39774d2c8e9cb4e88a405b0f P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:47.463366 2476 raft_consensus.cc:513] T 9f8eb43a39774d2c8e9cb4e88a405b0f P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:47.463707 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 796b5f37479d460e8df1d00f1780ec7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:47.463076 899 tablet_service.cc:1467] Processing CreateTablet for tablet ab366fbcf41e4969a71f3ba30ba24cf0 (DEFAULT_TABLE table=default.Table_8a46b6886be646d48357f268651520c6 [id=7dd719e3dbc045a7b5290b0d0d05b29b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:47.463492 893 tablet_service.cc:1467] Processing CreateTablet for tablet a6a8e747ba2f4f749fa70694cf6fdd83 (DEFAULT_TABLE table=default.Table_0532c8897e3a478ca8602439e3752e72 [id=a84b79dfaadb4e92957fbe5f739b7a01]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:47.464473 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6a8e747ba2f4f749fa70694cf6fdd83. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:47.465219 2476 leader_election.cc:290] T 9f8eb43a39774d2c8e9cb4e88a405b0f P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
W20250113 02:51:47.465960 852 leader_election.cc:343] T 9f8eb43a39774d2c8e9cb4e88a405b0f P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Not found: Tablet not found: 9f8eb43a39774d2c8e9cb4e88a405b0f
I20250113 02:51:47.467962 882 tablet_service.cc:1467] Processing CreateTablet for tablet fcd60b89f5bb46f6b682f231c84204ef (DEFAULT_TABLE table=default.Table_643c47db97214c82861277c1935d5032 [id=410ff6c1baa3428aa118233a87a8d001]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:47.468658 889 tablet_service.cc:1467] Processing CreateTablet for tablet 6720c7558e084472a1f67c08323dc98e (DEFAULT_TABLE table=default.Table_eaf6851fc514494a95392ba107626e41 [id=e05095ce043c43e4bb56ea6e2f3bca8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:47.469594 897 tablet_service.cc:1467] Processing CreateTablet for tablet a35558e3f1fa4a36a174adde70d7cd44 (DEFAULT_TABLE table=default.Table_567d446e6ae54abf9ae7586692b5780a [id=70089f757a1c45f6b983f216fb167478]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:47.470036 884 tablet_service.cc:1467] Processing CreateTablet for tablet 1bdb4f2ad28b4aec9d574816ab9d4f32 (DEFAULT_TABLE table=default.Table_8a69c12869c44953bde2307180dc1a88 [id=93deb7801dcb418e825d465529ceb4c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:47.470248 1035 tablet_service.cc:1467] Processing CreateTablet for tablet 883db38457e54193b6e9cd2b8e76e6d2 (DEFAULT_TABLE table=default.Table_a01645da1d1b4ba7852c3986e29de0c1 [id=bb3cdeea760f489b89375a7ed3caba19]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:47.470636 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 883db38457e54193b6e9cd2b8e76e6d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:47.470983 1194 tablet_bootstrap.cc:492] T fe8f075d4c9a4f9abe372bfdb81a829c P df69f48656b44bb890f76587ee69cc0f: No bootstrap required, opened a new log
I20250113 02:51:47.471630 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 344fa45adf1749f1b5b299d9a6d893c9
I20250113 02:51:47.472081 447 catalog_manager.cc:6596] ReplaceTablet: tablet 344fa45adf1749f1b5b299d9a6d893c9 deleted and replaced by tablet 73ec021ab51d4f1181c182e37a9a92a3
I20250113 02:51:47.472584 1194 ts_tablet_manager.cc:1397] T fe8f075d4c9a4f9abe372bfdb81a829c P df69f48656b44bb890f76587ee69cc0f: Time spent bootstrapping tablet: real 0.225s user 0.007s sys 0.007s
I20250113 02:51:47.472998 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 546ee73435444c90ad286943807af984
I20250113 02:51:47.473847 1037 tablet_service.cc:1514] Processing DeleteTablet for tablet 546ee73435444c90ad286943807af984 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:47 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:47.473824 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet 344fa45adf1749f1b5b299d9a6d893c9 with delete_type TABLET_DATA_DELETED (replaced by tablet 73ec021ab51d4f1181c182e37a9a92a3) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:47.474375 895 tablet_service.cc:1467] Processing CreateTablet for tablet d88f22c3f74a417c964e282d951f0c99 (DEFAULT_TABLE table=default.Table_8530e7723ac8458d9990e3c5c6286447 [id=1fd319f7088445388c4149bba313c531]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:47.474493 898 tablet_service.cc:1467] Processing CreateTablet for tablet 796b5f37479d460e8df1d00f1780ec7d (DEFAULT_TABLE table=default.Table_1e0ca8b977914d1aaf79792357ea51a7 [id=d5ed8b5ba94a43768b4ee73516124ca3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:47.474572 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fcd60b89f5bb46f6b682f231c84204ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:47.474978 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 796b5f37479d460e8df1d00f1780ec7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:47.475226 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d88f22c3f74a417c964e282d951f0c99. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:47.480280 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a35558e3f1fa4a36a174adde70d7cd44. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:47.480823 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6720c7558e084472a1f67c08323dc98e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:47.483606 2476 raft_consensus.cc:491] T 283fc7f1c74e480c8a63c74d3ae9eb7d P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:47.484125 447 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_bb6e88b17f3d497386f8098edb3719b9" } modify_external_catalogs: true
I20250113 02:51:47.484295 447 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_bb6e88b17f3d497386f8098edb3719b9" } modify_external_catalogs: true
I20250113 02:51:47.485703 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_28e6f68bf4304b12b107c8e71e66e1e3" } new_table_name: "default.Table_4cf75878cd2243068f1b88b284ff6c5a" modify_external_catalogs: true
I20250113 02:51:47.486128 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_d5a79bb9160448269ae65f45fcd592a9" } new_table_name: "default.Table_782bf5e1073b467d89fabf889d90ff4c" modify_external_catalogs: true
I20250113 02:51:47.490877 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_0a219279fac1400295a14ee438d8c741"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:47.491037 892 tablet_service.cc:1514] Processing DeleteTablet for tablet 344fa45adf1749f1b5b299d9a6d893c9 with delete_type TABLET_DATA_DELETED (replaced by tablet 73ec021ab51d4f1181c182e37a9a92a3) from {username='slave'} at 127.0.0.1:37366
W20250113 02:51:47.491353 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0a219279fac1400295a14ee438d8c741 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:47.491854 892 tablet_service.cc:1514] Processing DeleteTablet for tablet 546ee73435444c90ad286943807af984 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:47 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:47.492879 892 tablet_service.cc:1467] Processing CreateTablet for tablet 02fe7f15df0f429595f842df2a0ee348 (DEFAULT_TABLE table=default.Table_f421dead901f4537860423130ed500fd [id=9b734237ea134bb68afba80aed07aa9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:47.495647 1032 tablet_service.cc:1467] Processing CreateTablet for tablet 02fe7f15df0f429595f842df2a0ee348 (DEFAULT_TABLE table=default.Table_f421dead901f4537860423130ed500fd [id=9b734237ea134bb68afba80aed07aa9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:47.496107 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 02fe7f15df0f429595f842df2a0ee348. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:47.499691 711 leader_election.cc:304] T 80c647a3b5a1411e9130461115c53550 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:47.490548 2476 raft_consensus.cc:513] T 283fc7f1c74e480c8a63c74d3ae9eb7d P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:47.501227 2853 raft_consensus.cc:2743] T 80c647a3b5a1411e9130461115c53550 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:47.501526 2476 leader_election.cc:290] T 283fc7f1c74e480c8a63c74d3ae9eb7d P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:47.501755 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab366fbcf41e4969a71f3ba30ba24cf0. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:47.503146 852 leader_election.cc:343] T 283fc7f1c74e480c8a63c74d3ae9eb7d P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Not found: Tablet not found: 283fc7f1c74e480c8a63c74d3ae9eb7d
I20250113 02:51:47.503775 775 raft_consensus.cc:2463] T 8f2fb58fc24d4692a3fe9a652e076aed P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 0.
I20250113 02:51:47.504966 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1363) took 4818 ms. Trace:
I20250113 02:51:47.490509 887 tablet_service.cc:1467] Processing CreateTablet for tablet 883db38457e54193b6e9cd2b8e76e6d2 (DEFAULT_TABLE table=default.Table_a01645da1d1b4ba7852c3986e29de0c1 [id=bb3cdeea760f489b89375a7ed3caba19]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:47.507408 746 tablet_service.cc:1467] Processing CreateTablet for tablet fcd60b89f5bb46f6b682f231c84204ef (DEFAULT_TABLE table=default.Table_643c47db97214c82861277c1935d5032 [id=410ff6c1baa3428aa118233a87a8d001]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:47.507616 743 tablet_service.cc:1467] Processing CreateTablet for tablet 883db38457e54193b6e9cd2b8e76e6d2 (DEFAULT_TABLE table=default.Table_a01645da1d1b4ba7852c3986e29de0c1 [id=bb3cdeea760f489b89375a7ed3caba19]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:47.507999 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 546ee73435444c90ad286943807af984 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:47 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:47.508179 750 tablet_service.cc:1467] Processing CreateTablet for tablet a6a8e747ba2f4f749fa70694cf6fdd83 (DEFAULT_TABLE table=default.Table_0532c8897e3a478ca8602439e3752e72 [id=a84b79dfaadb4e92957fbe5f739b7a01]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:47.508632 741 tablet_service.cc:1467] Processing CreateTablet for tablet ab366fbcf41e4969a71f3ba30ba24cf0 (DEFAULT_TABLE table=default.Table_8a46b6886be646d48357f268651520c6 [id=7dd719e3dbc045a7b5290b0d0d05b29b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:47.507428 751 tablet_service.cc:1467] Processing CreateTablet for tablet 796b5f37479d460e8df1d00f1780ec7d (DEFAULT_TABLE table=default.Table_1e0ca8b977914d1aaf79792357ea51a7 [id=d5ed8b5ba94a43768b4ee73516124ca3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:47.509130 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fcd60b89f5bb46f6b682f231c84204ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:47.509351 738 tablet_service.cc:1467] Processing CreateTablet for tablet 6720c7558e084472a1f67c08323dc98e (DEFAULT_TABLE table=default.Table_eaf6851fc514494a95392ba107626e41 [id=e05095ce043c43e4bb56ea6e2f3bca8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:47.509709 752 tablet_service.cc:1467] Processing CreateTablet for tablet a35558e3f1fa4a36a174adde70d7cd44 (DEFAULT_TABLE table=default.Table_567d446e6ae54abf9ae7586692b5780a [id=70089f757a1c45f6b983f216fb167478]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:47.510025 742 tablet_service.cc:1467] Processing CreateTablet for tablet d88f22c3f74a417c964e282d951f0c99 (DEFAULT_TABLE table=default.Table_8530e7723ac8458d9990e3c5c6286447 [id=1fd319f7088445388c4149bba313c531]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:47.510366 744 tablet_service.cc:1467] Processing CreateTablet for tablet 02fe7f15df0f429595f842df2a0ee348 (DEFAULT_TABLE table=default.Table_f421dead901f4537860423130ed500fd [id=9b734237ea134bb68afba80aed07aa9e]), partition=RANGE (key) PARTITION UNBOUNDED
W20250113 02:51:47.510812 851 leader_election.cc:343] T 283fc7f1c74e480c8a63c74d3ae9eb7d P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Not found: Tablet not found: 283fc7f1c74e480c8a63c74d3ae9eb7d
I20250113 02:51:47.509056 748 tablet_service.cc:1467] Processing CreateTablet for tablet 1bdb4f2ad28b4aec9d574816ab9d4f32 (DEFAULT_TABLE table=default.Table_8a69c12869c44953bde2307180dc1a88 [id=93deb7801dcb418e825d465529ceb4c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:47.511230 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab366fbcf41e4969a71f3ba30ba24cf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:47.512105 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 883db38457e54193b6e9cd2b8e76e6d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:47.512822 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d88f22c3f74a417c964e282d951f0c99. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:47.516253 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6720c7558e084472a1f67c08323dc98e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:47.516902 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1bdb4f2ad28b4aec9d574816ab9d4f32. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:47.518103 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 796b5f37479d460e8df1d00f1780ec7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:47.520046 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6a8e747ba2f4f749fa70694cf6fdd83. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:47.521536 995 leader_election.cc:304] T 8f2fb58fc24d4692a3fe9a652e076aed P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters: df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:47.522172 2827 raft_consensus.cc:2798] T 8f2fb58fc24d4692a3fe9a652e076aed P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 02:51:47.522243 2827 raft_consensus.cc:491] T 8f2fb58fc24d4692a3fe9a652e076aed P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 02:51:47.522315 2827 raft_consensus.cc:3054] T 8f2fb58fc24d4692a3fe9a652e076aed P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:47.525048 2827 raft_consensus.cc:513] T 8f2fb58fc24d4692a3fe9a652e076aed P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:47.525732 2827 leader_election.cc:290] T 8f2fb58fc24d4692a3fe9a652e076aed P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Requested vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
I20250113 02:51:47.527247 2772 consensus_queue.cc:1035] T 68c4a7e8c0fb4feb85498709b00d3258 P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 02:51:47.533844 851 leader_election.cc:304] T 283fc7f1c74e480c8a63c74d3ae9eb7d P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df69f48656b44bb890f76587ee69cc0f; no voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca
W20250113 02:51:47.534266 851 leader_election.cc:343] T 9f8eb43a39774d2c8e9cb4e88a405b0f P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Not found: Tablet not found: 9f8eb43a39774d2c8e9cb4e88a405b0f
I20250113 02:51:47.534389 851 leader_election.cc:304] T 9f8eb43a39774d2c8e9cb4e88a405b0f P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df69f48656b44bb890f76587ee69cc0f; no voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca
I20250113 02:51:47.534682 2476 raft_consensus.cc:2743] T 283fc7f1c74e480c8a63c74d3ae9eb7d P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:47.534829 2476 raft_consensus.cc:2743] T 9f8eb43a39774d2c8e9cb4e88a405b0f P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:47.535523 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1bdb4f2ad28b4aec9d574816ab9d4f32. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:47.536828 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 02fe7f15df0f429595f842df2a0ee348. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:47.537528 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 883db38457e54193b6e9cd2b8e76e6d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:47.543762 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 05e6e2dd6b364b1590469e400159a6dd
W20250113 02:51:47.543913 451 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b98da6f197764465a1e10aaadcddf3a4 consensus state has no leader
W20250113 02:51:47.544170 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9bfecc0884a64b4583506f6a050820f0 consensus state has no leader
I20250113 02:51:47.544490 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 6887e74b88aa458f857728460521a96a
I20250113 02:51:47.544665 883 tablet_service.cc:1514] Processing DeleteTablet for tablet 05e6e2dd6b364b1590469e400159a6dd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:47 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:47.545647 453 catalog_manager.cc:5618] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587) does not have the latest schema for tablet c13e9a51c1e74b44b352bade78390ba5 (table default.Table_eadba9f9ab724461b018a9423dcac195 [id=caf098e2b57849a997bd7c8eaf1f9326]). Expected version 2 got 1
I20250113 02:51:47.545694 883 tablet_service.cc:1514] Processing DeleteTablet for tablet 6887e74b88aa458f857728460521a96a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:47 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:47.545733 1030 tablet_service.cc:1514] Processing DeleteTablet for tablet 6887e74b88aa458f857728460521a96a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:47 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:47.545878 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet 05e6e2dd6b364b1590469e400159a6dd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:47 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:47.546948 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 1e135eb5a19d45c3bfea54b8cf1bb898
I20250113 02:51:47.547161 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet 6001369444894b48b51a48fb60ddc52a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:40 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:47.547463 883 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e135eb5a19d45c3bfea54b8cf1bb898 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:47 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:47.547921 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e135eb5a19d45c3bfea54b8cf1bb898 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:47 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:47.550856 2772 raft_consensus.cc:491] T 6001369444894b48b51a48fb60ddc52a P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:47.550981 2772 raft_consensus.cc:513] T 6001369444894b48b51a48fb60ddc52a P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:47.551486 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f2fb58fc24d4692a3fe9a652e076aed" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f"
I20250113 02:51:47.551599 2772 leader_election.cc:290] T 6001369444894b48b51a48fb60ddc52a P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
W20250113 02:51:47.551946 998 leader_election.cc:343] T 8f2fb58fc24d4692a3fe9a652e076aed P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:47.505107 1195 rpcz_store.cc:276] 0113 02:51:42.564158 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:42.564221 (+ 63us) service_pool.cc:225] Handling call
0113 02:51:47.213972 (+4649751us) ts_tablet_manager.cc:1145] Deleting tablet f105f8b32a654e4ca154e8fc0cb96091
0113 02:51:47.213980 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:47.279695 (+ 65715us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:47.357711 (+ 78016us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:47.382897 (+ 25186us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4649516}
I20250113 02:51:47.560283 754 tablet_service.cc:1514] Processing DeleteTablet for tablet 344fa45adf1749f1b5b299d9a6d893c9 with delete_type TABLET_DATA_DELETED (replaced by tablet 73ec021ab51d4f1181c182e37a9a92a3) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:47.561388 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f2fb58fc24d4692a3fe9a652e076aed" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1186fa869eb14812832fa6562531f196"
I20250113 02:51:47.561571 769 raft_consensus.cc:3054] T 8f2fb58fc24d4692a3fe9a652e076aed P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Advancing to term 1
I20250113 02:51:47.562108 751 tablet_service.cc:1514] Processing DeleteTablet for tablet 6887e74b88aa458f857728460521a96a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:47 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:47.562923 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1364) took 4998 ms. Trace:
I20250113 02:51:47.563033 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet f105f8b32a654e4ca154e8fc0cb96091 (table default.Table_3463038bce4545df8ec6c3834403a95d [id=505134ec091a46c9a1dc599ae2514a3c]) successfully deleted
I20250113 02:51:47.562992 1195 rpcz_store.cc:276] 0113 02:51:42.564749 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:42.564814 (+ 65us) service_pool.cc:225] Handling call
0113 02:51:47.560468 (+4995654us) ts_tablet_manager.cc:1145] Deleting tablet 0c408921a09a463eb464e08edcf042b3
0113 02:51:47.560481 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:47.562908 (+ 2427us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4995487}
I20250113 02:51:47.563253 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e135eb5a19d45c3bfea54b8cf1bb898 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:47 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:47.563261 1195 tablet_replica.cc:331] T 16a132384b0e4264a73d09a3ea133893 P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:47.565177 1195 raft_consensus.cc:2238] T 16a132384b0e4264a73d09a3ea133893 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:47.565349 1195 raft_consensus.cc:2267] T 16a132384b0e4264a73d09a3ea133893 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:47.565865 1195 ts_tablet_manager.cc:1905] T 16a132384b0e4264a73d09a3ea133893 P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:47.555580 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6001369444894b48b51a48fb60ddc52a" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
W20250113 02:51:47.569661 998 leader_election.cc:343] T 6001369444894b48b51a48fb60ddc52a P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:47.572808 754 tablet_service.cc:1514] Processing DeleteTablet for tablet 05e6e2dd6b364b1590469e400159a6dd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:47 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:47.575649 1191 tablet_bootstrap.cc:492] T f9269967dea94fc3a337546ad0842e11 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:47.575877 1191 ts_tablet_manager.cc:1397] T f9269967dea94fc3a337546ad0842e11 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.287s user 0.003s sys 0.005s
I20250113 02:51:47.577001 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a35558e3f1fa4a36a174adde70d7cd44. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:47.578444 1192 tablet_bootstrap.cc:492] T fe3d35d9aaf64188aaa3c1ac38d279d8 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:47.578800 1192 ts_tablet_manager.cc:1397] T fe3d35d9aaf64188aaa3c1ac38d279d8 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.274s user 0.005s sys 0.004s
I20250113 02:51:47.589471 2905 tablet.cc:1721] T c13e9a51c1e74b44b352bade78390ba5 P df69f48656b44bb890f76587ee69cc0f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250113 02:51:47.603276 446 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_5a88d7e91d464a8aa804fec7b174d051" } modify_external_catalogs: true
I20250113 02:51:47.603356 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_c4b56c0ef9c84e3494e2fc5f0436ce81"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:47.603479 446 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_5a88d7e91d464a8aa804fec7b174d051" } modify_external_catalogs: true
W20250113 02:51:47.603891 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c4b56c0ef9c84e3494e2fc5f0436ce81 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:47.605924 448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_da4eeeb904054a9aa6807387f76b1d8f" } new_table_name: "default.Table_53afceeb17dd4e71961773849182013d" modify_external_catalogs: true
I20250113 02:51:47.605983 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_32a3dae98e444e1f8f79fefacd999119"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:47.606487 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_32a3dae98e444e1f8f79fefacd999119 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:47.606676 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_ffe02411b95246779d563a7896030496"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:47.607082 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ffe02411b95246779d563a7896030496 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:47.607290 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_68db23b2977d4f0c9a7c2a96ee28fc80"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:47.607700 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_68db23b2977d4f0c9a7c2a96ee28fc80 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:47.607668 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_61e9db1815324f248afc5bd24612a9d9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:47.608114 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_61e9db1815324f248afc5bd24612a9d9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:47.608840 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_4b6d8da7a53240dab5cd6d788c17e913" } new_table_name: "default.Table_4c4def251d8c488098f63fcf7dfd87bb" modify_external_catalogs: true
I20250113 02:51:47.609228 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_d3095de5abfb4fb7ae50a24c7c9445ca"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:47.609643 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d3095de5abfb4fb7ae50a24c7c9445ca in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:47.606504 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_4d5b54216098465fb88c7b9d8b605344" } new_table_name: "default.Table_b6d92dda0de94aa0a96ddecc59c7fa48" modify_external_catalogs: true
W20250113 02:51:47.625597 448 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet dbbcb086a31a49d3b32367da67de0760 consensus state has no leader
I20250113 02:51:47.632337 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 02fe7f15df0f429595f842df2a0ee348. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:47.638803 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_433fdf8640d14c58aae8a24bcea4edf2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:47.638850 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_e78e59f833614fad8928e0598bc8e7fa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:47.639389 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_433fdf8640d14c58aae8a24bcea4edf2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250113 02:51:47.639392 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e78e59f833614fad8928e0598bc8e7fa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:47.649519 1191 raft_consensus.cc:357] T f9269967dea94fc3a337546ad0842e11 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:47.650120 1191 raft_consensus.cc:383] T f9269967dea94fc3a337546ad0842e11 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:47.650194 1191 raft_consensus.cc:738] T f9269967dea94fc3a337546ad0842e11 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:47.650460 1191 consensus_queue.cc:260] T f9269967dea94fc3a337546ad0842e11 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:47.656244 1196 ts_tablet_manager.cc:1918] T 67baaab1e36e415381579d7385d4ab5a P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:47.656384 1196 log.cc:1198] T 67baaab1e36e415381579d7385d4ab5a P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/67baaab1e36e415381579d7385d4ab5a
W20250113 02:51:47.657506 439 catalog_manager.cc:4839] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 0c408921a09a463eb464e08edcf042b3 because the tablet was not found. No further retry: Not found: Tablet not found: 0c408921a09a463eb464e08edcf042b3
W20250113 02:51:47.657939 998 consensus_peers.cc:487] T afcb5b1977c447d3a51de9cb72d7196a P 4216a7f853d642b7bc3e5cff458f3aca -> Peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Couldn't send request to peer df69f48656b44bb890f76587ee69cc0f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250113 02:51:47.658195 1196 ts_tablet_manager.cc:1939] T 67baaab1e36e415381579d7385d4ab5a P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
W20250113 02:51:47.659706 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f46a1fddc0d94ddfa41946d5252d70f3 consensus state has no leader
W20250113 02:51:47.660238 995 leader_election.cc:343] T 6001369444894b48b51a48fb60ddc52a P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): Not found: Tablet not found: 6001369444894b48b51a48fb60ddc52a
I20250113 02:51:47.660385 995 leader_election.cc:304] T 6001369444894b48b51a48fb60ddc52a P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4216a7f853d642b7bc3e5cff458f3aca; no voters: 1186fa869eb14812832fa6562531f196, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:47.661295 2368 raft_consensus.cc:2743] T 6001369444894b48b51a48fb60ddc52a P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:47.661757 2368 consensus_queue.cc:1035] T c04be975c8c049b892b16129ed47a142 P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 02:51:47.662348 453 catalog_manager.cc:5618] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467) does not have the latest schema for tablet c13e9a51c1e74b44b352bade78390ba5 (table default.Table_eadba9f9ab724461b018a9423dcac195 [id=caf098e2b57849a997bd7c8eaf1f9326]). Expected version 2 got 1
I20250113 02:51:47.662552 2904 tablet.cc:1721] T e01e7f8db83d461bbfbb2e901419ce0b P 4216a7f853d642b7bc3e5cff458f3aca: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250113 02:51:47.669418 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet 67baaab1e36e415381579d7385d4ab5a (table default.Table_c3ba3f6047274ab885a4c853b06e1417 [id=6666a614063942c99a78b95d35add037]) successfully deleted
I20250113 02:51:47.669795 1196 tablet_replica.cc:331] T 9f8eb43a39774d2c8e9cb4e88a405b0f P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:47.669901 1194 raft_consensus.cc:357] T fe8f075d4c9a4f9abe372bfdb81a829c P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:47.670194 1196 raft_consensus.cc:2238] T 9f8eb43a39774d2c8e9cb4e88a405b0f P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:47.670219 1194 raft_consensus.cc:383] T fe8f075d4c9a4f9abe372bfdb81a829c P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:47.670575 1194 raft_consensus.cc:738] T fe8f075d4c9a4f9abe372bfdb81a829c P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f48656b44bb890f76587ee69cc0f, State: Initialized, Role: FOLLOWER
I20250113 02:51:47.670897 1194 consensus_queue.cc:260] T fe8f075d4c9a4f9abe372bfdb81a829c P df69f48656b44bb890f76587ee69cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:47.671676 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_82f0839904a94b9db9014e2aa6ad9db2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:47.672168 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_82f0839904a94b9db9014e2aa6ad9db2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:47.672660 1194 ts_tablet_manager.cc:1428] T fe8f075d4c9a4f9abe372bfdb81a829c P df69f48656b44bb890f76587ee69cc0f: Time spent starting tablet: real 0.200s user 0.001s sys 0.001s
I20250113 02:51:47.672828 1196 raft_consensus.cc:2267] T 9f8eb43a39774d2c8e9cb4e88a405b0f P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:47.673624 1196 ts_tablet_manager.cc:1905] T 9f8eb43a39774d2c8e9cb4e88a405b0f P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:47.673085 1194 tablet_bootstrap.cc:492] T fd0b8592997d4a61b8ca854ecbb67c55 P df69f48656b44bb890f76587ee69cc0f: Bootstrap starting.
W20250113 02:51:47.676311 451 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 702e03e213b14ef5a1b64a02df1b3bf9 consensus state has no leader
I20250113 02:51:47.676246 887 tablet_service.cc:1467] Processing CreateTablet for tablet e292cf24779849c883b448ba0545df07 (DEFAULT_TABLE table=default.Table_c483e974da4240c59118749d623aa221 [id=8ad308c901684c8b9bc7dce80a134ce8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:47.678097 1191 ts_tablet_manager.cc:1428] T f9269967dea94fc3a337546ad0842e11 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.102s user 0.001s sys 0.001s
I20250113 02:51:47.678402 1191 tablet_bootstrap.cc:492] T 146dad5c977f464496f7d21611030dad P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:47.680657 1032 tablet_service.cc:1467] Processing CreateTablet for tablet e292cf24779849c883b448ba0545df07 (DEFAULT_TABLE table=default.Table_c483e974da4240c59118749d623aa221 [id=8ad308c901684c8b9bc7dce80a134ce8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:47.681661 1035 tablet_service.cc:1467] Processing CreateTablet for tablet c88632848f7b4d5db89ee6d1f8cb8264 (DEFAULT_TABLE table=default.Table_5c113fa1072647119025f2ea5c159722 [id=e080f5d9ddbe4207a11552cfe5bca7e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:47.686259 1026 tablet_service.cc:1467] Processing CreateTablet for tablet b012b9c27228408786adf499a8af3e3f (DEFAULT_TABLE table=default.Table_204b526f8a5448f48f761b4cc7101fef [id=3bbe5e79ed0f4092b0cbed630fcb9766]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:47.686637 1030 tablet_service.cc:1467] Processing CreateTablet for tablet 59477b97ac10415ea26e4c7d7372b139 (DEFAULT_TABLE table=default.Table_31180c034b5f487e9f6880d11e0239ee [id=e13334709a464a618efa771e41f83ee9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:47.686972 1024 tablet_service.cc:1467] Processing CreateTablet for tablet bf34a44ed6f84ac39e9d98f48b0f2207 (DEFAULT_TABLE table=default.Table_3487754fcaa24feaa2e5c3d3c267f9fb [id=64b27086ea4c4efd9afa927c6a871eb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:47.687289 1037 tablet_service.cc:1467] Processing CreateTablet for tablet 7ddce8fe30d94d8184d6eda2eeef3fea (DEFAULT_TABLE table=default.Table_547e8a172dd04141ba47718ef76d5234 [id=c4ef58546e2e4b99861a53b134ea44ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:47.687572 1034 tablet_service.cc:1467] Processing CreateTablet for tablet c743a26cbecf4d71ac768317245a417e (DEFAULT_TABLE table=default.Table_d0917724c0264ec3bc07ba9a1d519cdf [id=3d70e772179f4bd3b95e657c275dd3a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:47.688810 889 tablet_service.cc:1467] Processing CreateTablet for tablet 59477b97ac10415ea26e4c7d7372b139 (DEFAULT_TABLE table=default.Table_31180c034b5f487e9f6880d11e0239ee [id=e13334709a464a618efa771e41f83ee9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:47.689184 890 tablet_service.cc:1514] Processing DeleteTablet for tablet fe8f075d4c9a4f9abe372bfdb81a829c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:36 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:47.690073 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 9cf14ed03b3c418c941cd20ed371d7e2
I20250113 02:51:47.688827 880 tablet_service.cc:1467] Processing CreateTablet for tablet 7ddce8fe30d94d8184d6eda2eeef3fea (DEFAULT_TABLE table=default.Table_547e8a172dd04141ba47718ef76d5234 [id=c4ef58546e2e4b99861a53b134ea44ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:47.690845 1039 tablet_service.cc:1467] Processing CreateTablet for tablet c1d8da75e6bd41db8cfe8d069ebe98ba (DEFAULT_TABLE table=default.Table_edfa039267c74167bfbbabc4eed8e0c1 [id=60bbbe1a29314f399246c159cc3139b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:47.691059 1036 tablet_service.cc:1467] Processing CreateTablet for tablet 4f7d6dd9bd4f4f308f954c01867153b0 (DEFAULT_TABLE table=default.Table_7fd1c363a0154f30b11fbb0b6a091d39 [id=32848189910c4136a497c2b748ee6270]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:47.691165 1042 tablet_service.cc:1467] Processing CreateTablet for tablet e2ea5acdcdcc45a3a0865ca9ff050e57 (DEFAULT_TABLE table=default.Table_929bc069a74042cf8ce8cfef9ab4028e [id=d7b64dd49d3640b585a86898ca3153bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:47.691463 1038 tablet_service.cc:1467] Processing CreateTablet for tablet 73ec021ab51d4f1181c182e37a9a92a3 (DEFAULT_TABLE table=default.Table_0cfdfcf64e1e47a7951f9170072704c8 [id=d2ecded77d744b94913a3efbd4f477bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:47.692305 898 tablet_service.cc:1467] Processing CreateTablet for tablet c743a26cbecf4d71ac768317245a417e (DEFAULT_TABLE table=default.Table_d0917724c0264ec3bc07ba9a1d519cdf [id=3d70e772179f4bd3b95e657c275dd3a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:47.692786 895 tablet_service.cc:1467] Processing CreateTablet for tablet bf34a44ed6f84ac39e9d98f48b0f2207 (DEFAULT_TABLE table=default.Table_3487754fcaa24feaa2e5c3d3c267f9fb [id=64b27086ea4c4efd9afa927c6a871eb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:47.696077 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet 9cf14ed03b3c418c941cd20ed371d7e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:47 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:47.697059 448 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_593fc8a8736e4d4aaeda0d72bbb30337" } modify_external_catalogs: true
I20250113 02:51:47.697245 448 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_593fc8a8736e4d4aaeda0d72bbb30337" } modify_external_catalogs: true
I20250113 02:51:47.701916 892 tablet_service.cc:1467] Processing CreateTablet for tablet b012b9c27228408786adf499a8af3e3f (DEFAULT_TABLE table=default.Table_204b526f8a5448f48f761b4cc7101fef [id=3bbe5e79ed0f4092b0cbed630fcb9766]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:47.703857 897 tablet_service.cc:1467] Processing CreateTablet for tablet c88632848f7b4d5db89ee6d1f8cb8264 (DEFAULT_TABLE table=default.Table_5c113fa1072647119025f2ea5c159722 [id=e080f5d9ddbe4207a11552cfe5bca7e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:47.705957 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 298eb28ce7da4818acea3060a6ef427c
I20250113 02:51:47.706115 890 tablet_service.cc:1467] Processing CreateTablet for tablet 4f7d6dd9bd4f4f308f954c01867153b0 (DEFAULT_TABLE table=default.Table_7fd1c363a0154f30b11fbb0b6a091d39 [id=32848189910c4136a497c2b748ee6270]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:47.706890 885 tablet_service.cc:1467] Processing CreateTablet for tablet e2ea5acdcdcc45a3a0865ca9ff050e57 (DEFAULT_TABLE table=default.Table_929bc069a74042cf8ce8cfef9ab4028e [id=d7b64dd49d3640b585a86898ca3153bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:47.708223 883 tablet_service.cc:1467] Processing CreateTablet for tablet 73ec021ab51d4f1181c182e37a9a92a3 (DEFAULT_TABLE table=default.Table_0cfdfcf64e1e47a7951f9170072704c8 [id=d2ecded77d744b94913a3efbd4f477bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:47.708693 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet 298eb28ce7da4818acea3060a6ef427c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:47 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:47.709596 886 tablet_service.cc:1514] Processing DeleteTablet for tablet 9cf14ed03b3c418c941cd20ed371d7e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:47 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:47.709791 891 tablet_service.cc:1514] Processing DeleteTablet for tablet 298eb28ce7da4818acea3060a6ef427c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:47 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:47.710520 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59477b97ac10415ea26e4c7d7372b139. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:47.711027 1194 tablet_bootstrap.cc:654] T fd0b8592997d4a61b8ca854ecbb67c55 P df69f48656b44bb890f76587ee69cc0f: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:47.711333 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e2ea5acdcdcc45a3a0865ca9ff050e57. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:47.711838 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f7d6dd9bd4f4f308f954c01867153b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:47.712093 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 73ec021ab51d4f1181c182e37a9a92a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:47.713357 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ddce8fe30d94d8184d6eda2eeef3fea. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:47.714116 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b012b9c27228408786adf499a8af3e3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:47.714690 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e292cf24779849c883b448ba0545df07. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:47.718266 893 tablet_service.cc:1467] Processing CreateTablet for tablet c1d8da75e6bd41db8cfe8d069ebe98ba (DEFAULT_TABLE table=default.Table_edfa039267c74167bfbbabc4eed8e0c1 [id=60bbbe1a29314f399246c159cc3139b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:47.721119 1195 ts_tablet_manager.cc:1918] T 16a132384b0e4264a73d09a3ea133893 P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:47.721246 1195 log.cc:1198] T 16a132384b0e4264a73d09a3ea133893 P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/16a132384b0e4264a73d09a3ea133893
I20250113 02:51:47.721551 1195 ts_tablet_manager.cc:1939] T 16a132384b0e4264a73d09a3ea133893 P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:47.722278 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1365) took 5157 ms. Trace:
I20250113 02:51:47.722364 1195 rpcz_store.cc:276] 0113 02:51:42.565207 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:42.565376 (+ 169us) service_pool.cc:225] Handling call
0113 02:51:47.563216 (+4997840us) ts_tablet_manager.cc:1145] Deleting tablet 16a132384b0e4264a73d09a3ea133893
0113 02:51:47.563224 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:47.718035 (+154811us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:47.721020 (+ 2985us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:47.722255 (+ 1235us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4997524}
I20250113 02:51:47.722514 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e292cf24779849c883b448ba0545df07. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:47.722719 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1366) took 5119 ms. Trace:
I20250113 02:51:47.722781 1195 rpcz_store.cc:276] 0113 02:51:42.603317 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:42.603381 (+ 64us) service_pool.cc:225] Handling call
0113 02:51:47.722605 (+5119224us) ts_tablet_manager.cc:1145] Deleting tablet 125ebde7a9f14918ba328a75e88a7746
0113 02:51:47.722614 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:47.722706 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5119038}
I20250113 02:51:47.725318 1191 tablet_bootstrap.cc:654] T 146dad5c977f464496f7d21611030dad P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:47.725780 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c88632848f7b4d5db89ee6d1f8cb8264. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:47.727329 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1367) took 5122 ms. Trace:
I20250113 02:51:47.727476 1195 rpcz_store.cc:276] 0113 02:51:42.604997 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:42.605059 (+ 62us) service_pool.cc:225] Handling call
0113 02:51:47.722953 (+5117894us) ts_tablet_manager.cc:1145] Deleting tablet 166f58d45b7b443ea54087e4ab48eaba
0113 02:51:47.722960 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:47.727279 (+ 4319us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5117721}
I20250113 02:51:47.728277 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59477b97ac10415ea26e4c7d7372b139. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:47.732184 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf34a44ed6f84ac39e9d98f48b0f2207. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:47.734408 1037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ddce8fe30d94d8184d6eda2eeef3fea. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:47.734604 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b012b9c27228408786adf499a8af3e3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:47.737354 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c743a26cbecf4d71ac768317245a417e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:47.740186 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c1d8da75e6bd41db8cfe8d069ebe98ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:47.744874 1036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f7d6dd9bd4f4f308f954c01867153b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:47.748447 1042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e2ea5acdcdcc45a3a0865ca9ff050e57. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:47.728020 1195 tablet_replica.cc:331] T 8cc78bf7d6324b7ba10a1b1045923f2b P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:47.750459 1195 raft_consensus.cc:2238] T 8cc78bf7d6324b7ba10a1b1045923f2b P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:47.750991 448 catalog_manager.cc:6071] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: default.Table_e27f8b6f8232415388a5e53ab15494b7 [id=efc4ebba97f74c9ba12ab1dcc6577da0] alter complete (version 1)
I20250113 02:51:47.750815 1195 raft_consensus.cc:2267] T 8cc78bf7d6324b7ba10a1b1045923f2b P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:47.752291 1195 ts_tablet_manager.cc:1905] T 8cc78bf7d6324b7ba10a1b1045923f2b P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:47.752871 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 73ec021ab51d4f1181c182e37a9a92a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:47.745939 1192 raft_consensus.cc:357] T fe3d35d9aaf64188aaa3c1ac38d279d8 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:47.753892 1192 raft_consensus.cc:383] T fe3d35d9aaf64188aaa3c1ac38d279d8 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:47.753995 1192 raft_consensus.cc:738] T fe3d35d9aaf64188aaa3c1ac38d279d8 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:47.754386 1192 consensus_queue.cc:260] T fe3d35d9aaf64188aaa3c1ac38d279d8 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:47.755347 1192 ts_tablet_manager.cc:1428] T fe3d35d9aaf64188aaa3c1ac38d279d8 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.176s user 0.001s sys 0.001s
I20250113 02:51:47.756963 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_8530e7723ac8458d9990e3c5c6286447" } new_table_name: "default.Table_cfbbfdeacaab4c7198f9744cc16e185a" modify_external_catalogs: true
I20250113 02:51:47.757117 446 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_a44fcb8c7ef04ef68933abf5950c4d3a" } modify_external_catalogs: true
I20250113 02:51:47.757241 448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_220329606556483ba605d7e7cdf05e1b" } new_table_name: "default.Table_909535230dbb49e8b9a95b7923f67291" modify_external_catalogs: true
I20250113 02:51:47.757277 446 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_a44fcb8c7ef04ef68933abf5950c4d3a" } modify_external_catalogs: true
I20250113 02:51:47.758252 447 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_70b5a5b3cec84af19415f09e22c1718c" } modify_external_catalogs: true
I20250113 02:51:47.758374 447 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_70b5a5b3cec84af19415f09e22c1718c" } modify_external_catalogs: true
I20250113 02:51:47.740432 769 raft_consensus.cc:2463] T 8f2fb58fc24d4692a3fe9a652e076aed P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4216a7f853d642b7bc3e5cff458f3aca in term 1.
I20250113 02:51:47.760623 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_44cadaad4c454a31b14e50db0301c838" } modify_external_catalogs: true
I20250113 02:51:47.760749 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_44cadaad4c454a31b14e50db0301c838" } modify_external_catalogs: true
I20250113 02:51:47.763160 995 leader_election.cc:304] T 8f2fb58fc24d4692a3fe9a652e076aed P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1186fa869eb14812832fa6562531f196, 4216a7f853d642b7bc3e5cff458f3aca; no voters: df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:47.764227 2772 raft_consensus.cc:2798] T 8f2fb58fc24d4692a3fe9a652e076aed P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Leader election won for term 1
I20250113 02:51:47.764365 2772 raft_consensus.cc:695] T 8f2fb58fc24d4692a3fe9a652e076aed P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: Becoming Leader. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Running, Role: LEADER
I20250113 02:51:47.764576 2772 consensus_queue.cc:237] T 8f2fb58fc24d4692a3fe9a652e076aed P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:47.766700 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet 16a132384b0e4264a73d09a3ea133893 (table default.Table_eef7fe6c891b4fe984f8fb4a1c667a8e [id=ada38f49d3074fd49727e4cbe8b576d6]) successfully deleted
W20250113 02:51:47.766906 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 125ebde7a9f14918ba328a75e88a7746 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 125ebde7a9f14918ba328a75e88a7746 already in progress: creating tablet
W20250113 02:51:47.767047 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 166f58d45b7b443ea54087e4ab48eaba because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 166f58d45b7b443ea54087e4ab48eaba already in progress: creating tablet
I20250113 02:51:47.769232 743 tablet_service.cc:1467] Processing CreateTablet for tablet c88632848f7b4d5db89ee6d1f8cb8264 (DEFAULT_TABLE table=default.Table_5c113fa1072647119025f2ea5c159722 [id=e080f5d9ddbe4207a11552cfe5bca7e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:47.769604 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c88632848f7b4d5db89ee6d1f8cb8264. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:47.772205 1196 ts_tablet_manager.cc:1918] T 9f8eb43a39774d2c8e9cb4e88a405b0f P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:47.772316 1196 log.cc:1198] T 9f8eb43a39774d2c8e9cb4e88a405b0f P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/9f8eb43a39774d2c8e9cb4e88a405b0f
I20250113 02:51:47.772609 1196 ts_tablet_manager.cc:1939] T 9f8eb43a39774d2c8e9cb4e88a405b0f P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:47.773198 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_9cb075e6723c4e7bb1527ec917a3a2e0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:47.773695 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9cb075e6723c4e7bb1527ec917a3a2e0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:47.774072 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c743a26cbecf4d71ac768317245a417e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:47.775357 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c1d8da75e6bd41db8cfe8d069ebe98ba. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:47.777175 998 consensus_peers.cc:487] T a2781ba5061a4225b5636366857a229e P 4216a7f853d642b7bc3e5cff458f3aca -> Peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Couldn't send request to peer df69f48656b44bb890f76587ee69cc0f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250113 02:51:47.778983 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c88632848f7b4d5db89ee6d1f8cb8264. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:47.779937 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf34a44ed6f84ac39e9d98f48b0f2207. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:47.781179 742 tablet_service.cc:1467] Processing CreateTablet for tablet 7ddce8fe30d94d8184d6eda2eeef3fea (DEFAULT_TABLE table=default.Table_547e8a172dd04141ba47718ef76d5234 [id=c4ef58546e2e4b99861a53b134ea44ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:47.781296 744 tablet_service.cc:1467] Processing CreateTablet for tablet bf34a44ed6f84ac39e9d98f48b0f2207 (DEFAULT_TABLE table=default.Table_3487754fcaa24feaa2e5c3d3c267f9fb [id=64b27086ea4c4efd9afa927c6a871eb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:47.781539 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ddce8fe30d94d8184d6eda2eeef3fea. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:47.781756 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf34a44ed6f84ac39e9d98f48b0f2207. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:47.788239 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 5fead010e60544d394e0114ee4afc00c
W20250113 02:51:47.788591 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d88f22c3f74a417c964e282d951f0c99 consensus state has no leader
I20250113 02:51:47.789753 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet 5fead010e60544d394e0114ee4afc00c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:47 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:47.790043 748 tablet_service.cc:1467] Processing CreateTablet for tablet 59477b97ac10415ea26e4c7d7372b139 (DEFAULT_TABLE table=default.Table_31180c034b5f487e9f6880d11e0239ee [id=e13334709a464a618efa771e41f83ee9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:47.792595 751 tablet_service.cc:1467] Processing CreateTablet for tablet c1d8da75e6bd41db8cfe8d069ebe98ba (DEFAULT_TABLE table=default.Table_edfa039267c74167bfbbabc4eed8e0c1 [id=60bbbe1a29314f399246c159cc3139b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:47.792901 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c1d8da75e6bd41db8cfe8d069ebe98ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:47.793236 754 tablet_service.cc:1467] Processing CreateTablet for tablet c743a26cbecf4d71ac768317245a417e (DEFAULT_TABLE table=default.Table_d0917724c0264ec3bc07ba9a1d519cdf [id=3d70e772179f4bd3b95e657c275dd3a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:47.793541 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c743a26cbecf4d71ac768317245a417e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:47.795161 2854 consensus_queue.cc:1035] T b3ce6d851f564b849796f174860aacd5 P 1186fa869eb14812832fa6562531f196 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 02:51:47.798540 738 tablet_service.cc:1467] Processing CreateTablet for tablet e292cf24779849c883b448ba0545df07 (DEFAULT_TABLE table=default.Table_c483e974da4240c59118749d623aa221 [id=8ad308c901684c8b9bc7dce80a134ce8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:47.798871 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e292cf24779849c883b448ba0545df07. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:47.798910 1192 tablet_bootstrap.cc:492] T d565b77f90784d87b64996772c938085 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
W20250113 02:51:47.800454 448 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5f5315047bd346b99e51909438f9e60f consensus state has no leader
I20250113 02:51:47.801488 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet 045ff4572f3347fc890b439d90cb0bbe
I20250113 02:51:47.801771 1192 tablet_bootstrap.cc:654] T d565b77f90784d87b64996772c938085 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:47.802922 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet e90242bf273c42beb57ef160991a89ed
I20250113 02:51:47.803210 1198 ts_tablet_manager.cc:1918] T 59fadc693f554854bc2874f8db7c64c8 P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:47.803310 1198 log.cc:1198] T 59fadc693f554854bc2874f8db7c64c8 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/59fadc693f554854bc2874f8db7c64c8
I20250113 02:51:47.803615 1198 ts_tablet_manager.cc:1939] T 59fadc693f554854bc2874f8db7c64c8 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:47.805080 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet fe3d35d9aaf64188aaa3c1ac38d279d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:36 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:47.808172 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59477b97ac10415ea26e4c7d7372b139. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:47.817231 1039 tablet_service.cc:1514] Processing DeleteTablet for tablet 045ff4572f3347fc890b439d90cb0bbe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:47 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:47.817824 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet e90242bf273c42beb57ef160991a89ed with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:47 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:47.819496 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1507) took 1927 ms. Trace:
I20250113 02:51:47.819576 1198 rpcz_store.cc:276] 0113 02:51:45.892152 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:45.892214 (+ 62us) service_pool.cc:225] Handling call
0113 02:51:47.308901 (+1416687us) ts_tablet_manager.cc:1145] Deleting tablet 59fadc693f554854bc2874f8db7c64c8
0113 02:51:47.308912 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:47.566846 (+257934us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:47.803117 (+236271us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:47.819479 (+ 16362us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1416472}
I20250113 02:51:47.819950 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1508) took 1926 ms. Trace:
I20250113 02:51:47.819967 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet 59fadc693f554854bc2874f8db7c64c8 (table default.Table_17489ca6f295469eb97e9a3a5c7cb1aa [id=0e77d79275d44f37b11395f03d737c59]) successfully deleted
I20250113 02:51:47.820057 1198 rpcz_store.cc:276] 0113 02:51:45.893213 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:45.893268 (+ 55us) service_pool.cc:225] Handling call
0113 02:51:47.819857 (+1926589us) ts_tablet_manager.cc:1145] Deleting tablet cf7b8bd6801743f49e9b9473747447c5
0113 02:51:47.819867 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:47.819937 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1926440}
W20250113 02:51:47.820456 440 catalog_manager.cc:4839] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): delete failed for tablet cf7b8bd6801743f49e9b9473747447c5 because the tablet was not found. No further retry: Not found: Tablet not found: cf7b8bd6801743f49e9b9473747447c5
I20250113 02:51:47.829140 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_c506ab8e150647aaaa2baac66ae763b5" } new_table_name: "default.Table_ee43f17fb1394d9ea1ccbb4169886187" modify_external_catalogs: true
I20250113 02:51:47.829686 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_2dd967b1f0c34ec397869e598ff30a11"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:47.830145 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2dd967b1f0c34ec397869e598ff30a11 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:47.831498 746 tablet_service.cc:1467] Processing CreateTablet for tablet b012b9c27228408786adf499a8af3e3f (DEFAULT_TABLE table=default.Table_204b526f8a5448f48f761b4cc7101fef [id=3bbe5e79ed0f4092b0cbed630fcb9766]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:47.831945 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b012b9c27228408786adf499a8af3e3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:47.833765 447 catalog_manager.cc:5618] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095) does not have the latest schema for tablet 68c4a7e8c0fb4feb85498709b00d3258 (table default.Table_c90f3a5c35c842f6ad2cb74cf29dac13 [id=8d2f84cd670a4e8d9fa17b8313d3fab4]). Expected version 2 got 0
I20250113 02:51:47.834285 1198 tablet_replica.cc:331] T 542df00229f645788f1b821c0ce00c46 P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:47.834517 1198 raft_consensus.cc:2238] T 542df00229f645788f1b821c0ce00c46 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 LEADER]: Raft consensus shutting down.
I20250113 02:51:47.848423 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_e5a829394d5d42808382bbb11e7034ee"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:47.849123 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e5a829394d5d42808382bbb11e7034ee in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:47.851023 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_2ea5f56bb6c34801b1d89043155a8947"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:47.851810 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2ea5f56bb6c34801b1d89043155a8947 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:47.855123 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_5b13099e483d45e780e7500b2f81a4b0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:47.855582 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5b13099e483d45e780e7500b2f81a4b0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:47.860399 1195 ts_tablet_manager.cc:1918] T 8cc78bf7d6324b7ba10a1b1045923f2b P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250113 02:51:47.863145 1195 log.cc:1198] T 8cc78bf7d6324b7ba10a1b1045923f2b P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/8cc78bf7d6324b7ba10a1b1045923f2b
I20250113 02:51:47.866801 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_27eb308f2dc84858b3d1a2fa4b42c3f6" } new_table_name: "default.Table_eb8a90c415b04a15ac04f150bdec266b" modify_external_catalogs: true
I20250113 02:51:47.867950 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_44a921666dd64b9bb62acbb02339bd75"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:47.868435 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_44a921666dd64b9bb62acbb02339bd75 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:47.870409 1195 ts_tablet_manager.cc:1939] T 8cc78bf7d6324b7ba10a1b1045923f2b P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:47.872617 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1368) took 5264 ms. Trace:
W20250113 02:51:47.873204 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet bc88415f52a0424db18574c886dd6f1a consensus state has no leader
I20250113 02:51:47.872702 1195 rpcz_store.cc:276] 0113 02:51:42.608417 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:42.608478 (+ 61us) service_pool.cc:225] Handling call
0113 02:51:47.727940 (+5119462us) ts_tablet_manager.cc:1145] Deleting tablet 8cc78bf7d6324b7ba10a1b1045923f2b
0113 02:51:47.727949 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:47.813458 (+ 85509us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:47.860280 (+ 46822us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:47.872597 (+ 12317us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5119261}
I20250113 02:51:47.879405 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_b0050f363a17412aa670daf91361c7bb" } new_table_name: "default.Table_86189f7fa87c4a7b9b4e431ed6c647a6" modify_external_catalogs: true
I20250113 02:51:47.881608 742 tablet_service.cc:1467] Processing CreateTablet for tablet 4f7d6dd9bd4f4f308f954c01867153b0 (DEFAULT_TABLE table=default.Table_7fd1c363a0154f30b11fbb0b6a091d39 [id=32848189910c4136a497c2b748ee6270]), partition=RANGE (key) PARTITION UNBOUNDED
W20250113 02:51:47.886067 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 471cd6055e454540bbb479d44daf586a consensus state has no leader
I20250113 02:51:47.890491 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1370) took 5170 ms. Trace:
I20250113 02:51:47.890602 1195 rpcz_store.cc:276] 0113 02:51:42.720229 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:42.720291 (+ 62us) service_pool.cc:225] Handling call
0113 02:51:47.877596 (+5157305us) ts_tablet_manager.cc:1145] Deleting tablet 4791a82b5e13429184d0c7192dd7aef6
0113 02:51:47.877605 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:47.890464 (+ 12859us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5150913}
I20250113 02:51:47.891456 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_33e3d2b4659e4fd2a99a98885823a4b6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:47.891947 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_33e3d2b4659e4fd2a99a98885823a4b6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:47.892715 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1371) took 5165 ms. Trace:
I20250113 02:51:47.893199 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f7d6dd9bd4f4f308f954c01867153b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:47.892791 1195 rpcz_store.cc:276] 0113 02:51:42.727059 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:42.727123 (+ 64us) service_pool.cc:225] Handling call
0113 02:51:47.890889 (+5163766us) ts_tablet_manager.cc:1145] Deleting tablet 495064a93dfb41ff999fc73b579baca0
0113 02:51:47.892590 (+ 1701us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:47.892698 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5163391}
I20250113 02:51:47.894874 446 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_fe0c2876b17f4e318de4048a7fba8c7a" } modify_external_catalogs: true
I20250113 02:51:47.898712 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1372) took 5065 ms. Trace:
I20250113 02:51:47.898803 1195 rpcz_store.cc:276] 0113 02:51:42.833571 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:42.833971 (+ 400us) service_pool.cc:225] Handling call
0113 02:51:47.893998 (+5060027us) ts_tablet_manager.cc:1145] Deleting tablet e3a1e65ec9874cc7b83485683aafff77
0113 02:51:47.894007 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:47.898687 (+ 4680us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5059811}
I20250113 02:51:47.908463 741 tablet_service.cc:1514] Processing DeleteTablet for tablet 9cf14ed03b3c418c941cd20ed371d7e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:47 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:47.909087 751 tablet_service.cc:1467] Processing CreateTablet for tablet 73ec021ab51d4f1181c182e37a9a92a3 (DEFAULT_TABLE table=default.Table_0cfdfcf64e1e47a7951f9170072704c8 [id=d2ecded77d744b94913a3efbd4f477bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:47.909747 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 73ec021ab51d4f1181c182e37a9a92a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:47.916451 748 tablet_service.cc:1467] Processing CreateTablet for tablet e2ea5acdcdcc45a3a0865ca9ff050e57 (DEFAULT_TABLE table=default.Table_929bc069a74042cf8ce8cfef9ab4028e [id=d7b64dd49d3640b585a86898ca3153bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:47.917111 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e2ea5acdcdcc45a3a0865ca9ff050e57. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:47.920393 741 tablet_service.cc:1514] Processing DeleteTablet for tablet 298eb28ce7da4818acea3060a6ef427c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:47 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:47.919926 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1373) took 5041 ms. Trace:
I20250113 02:51:47.920892 1195 rpcz_store.cc:276] 0113 02:51:42.878698 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:42.878764 (+ 66us) service_pool.cc:225] Handling call
0113 02:51:47.899036 (+5020272us) ts_tablet_manager.cc:1145] Deleting tablet a446180f980640268170b4ab652d32cc
0113 02:51:47.899044 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:47.919901 (+ 20857us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5020094}
I20250113 02:51:47.921356 1195 tablet_replica.cc:331] T 0b119704bc774ca9b4155c9619d4765b P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:47.921669 1195 raft_consensus.cc:2238] T 0b119704bc774ca9b4155c9619d4765b P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:47.921929 1195 raft_consensus.cc:2267] T 0b119704bc774ca9b4155c9619d4765b P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:47.922636 1195 ts_tablet_manager.cc:1905] T 0b119704bc774ca9b4155c9619d4765b P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:47.938061 2603 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 02:51:47.939357 2603 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250113 02:51:47.939711 2603 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 02:51:47.900483 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_7d9a80e30e064eefaccf8e5972ef894d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:47.910166 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_cc67725cd94f450e966c32b58209681c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:47.951563 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cc67725cd94f450e966c32b58209681c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:47.910321 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_803441e825524c2aa95092f843cc84b2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:47.910709 448 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_1a6787745a7a47f9834090244c9f5654" } modify_external_catalogs: true
I20250113 02:51:47.952188 448 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_1a6787745a7a47f9834090244c9f5654" } modify_external_catalogs: true
W20250113 02:51:47.952425 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_803441e825524c2aa95092f843cc84b2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:47.931696 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet 8cc78bf7d6324b7ba10a1b1045923f2b (table default.Table_a55eab4604df4ca5b767708b6533e6d3 [id=925246ca69314b9490c9f968f3597229]) successfully deleted
W20250113 02:51:47.952819 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 4791a82b5e13429184d0c7192dd7aef6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4791a82b5e13429184d0c7192dd7aef6 already in progress: creating tablet
W20250113 02:51:47.953094 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7d9a80e30e064eefaccf8e5972ef894d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250113 02:51:47.953256 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet 495064a93dfb41ff999fc73b579baca0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 495064a93dfb41ff999fc73b579baca0 already in progress: creating tablet
W20250113 02:51:47.953433 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet e3a1e65ec9874cc7b83485683aafff77 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e3a1e65ec9874cc7b83485683aafff77 already in progress: creating tablet
W20250113 02:51:47.953601 439 catalog_manager.cc:4852] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): delete failed for tablet a446180f980640268170b4ab652d32cc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a446180f980640268170b4ab652d32cc already in progress: creating tablet
I20250113 02:51:47.923192 741 tablet_service.cc:1514] Processing DeleteTablet for tablet 5fead010e60544d394e0114ee4afc00c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:47 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:47.932090 744 tablet_service.cc:1514] Processing DeleteTablet for tablet e90242bf273c42beb57ef160991a89ed with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:47 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:47.932215 738 tablet_service.cc:1514] Processing DeleteTablet for tablet f9269967dea94fc3a337546ad0842e11 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:34 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:47.934656 1191 tablet_bootstrap.cc:492] T 146dad5c977f464496f7d21611030dad P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:47.949036 754 tablet_service.cc:1514] Processing DeleteTablet for tablet 045ff4572f3347fc890b439d90cb0bbe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:47 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:47.959264 1191 ts_tablet_manager.cc:1397] T 146dad5c977f464496f7d21611030dad P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.281s user 0.013s sys 0.010s
I20250113 02:51:47.956992 446 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_fe0c2876b17f4e318de4048a7fba8c7a" } modify_external_catalogs: true
I20250113 02:51:47.964712 1191 raft_consensus.cc:357] T 146dad5c977f464496f7d21611030dad P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:47.964937 1191 raft_consensus.cc:383] T 146dad5c977f464496f7d21611030dad P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:47.964995 1191 raft_consensus.cc:738] T 146dad5c977f464496f7d21611030dad P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:47.965221 1191 consensus_queue.cc:260] T 146dad5c977f464496f7d21611030dad P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:47.965792 1191 ts_tablet_manager.cc:1428] T 146dad5c977f464496f7d21611030dad P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20250113 02:51:47.968487 1191 tablet_bootstrap.cc:492] T 943d3a125ac5406a83f3c8d80e073b6e P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:47.970500 1191 tablet_bootstrap.cc:654] T 943d3a125ac5406a83f3c8d80e073b6e P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:47.972110 2842 raft_consensus.cc:491] T f9269967dea94fc3a337546ad0842e11 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:47.972227 2842 raft_consensus.cc:513] T f9269967dea94fc3a337546ad0842e11 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:47.972747 2842 leader_election.cc:290] T f9269967dea94fc3a337546ad0842e11 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:47.990025 883 tablet_service.cc:1514] Processing DeleteTablet for tablet 5fead010e60544d394e0114ee4afc00c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:47 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:47.992298 453 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_30951971e07445449dd042657deb91ec" } modify_external_catalogs: true
I20250113 02:51:47.992491 453 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_30951971e07445449dd042657deb91ec" } modify_external_catalogs: true
I20250113 02:51:47.995347 447 catalog_manager.cc:5618] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095) does not have the latest schema for tablet c04be975c8c049b892b16129ed47a142 (table default.Table_f98f15e1cb6a43348c4438096a7f9ea4 [id=36184fc3b73745c487d1d7a9a69e85ef]). Expected version 1 got 0
I20250113 02:51:48.001780 2917 tablet.cc:1721] T e01e7f8db83d461bbfbb2e901419ce0b P 1186fa869eb14812832fa6562531f196: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250113 02:51:48.002188 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_cc1ce461db3e461387263aacbd6af4b3" } new_table_name: "default.Table_685e3e58857540f0b317fe7b8a752a79" modify_external_catalogs: true
I20250113 02:51:48.003630 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_3fda809831ba43319e2ee62fb49dde92" } new_table_name: "default.Table_d46fb8340014430cb6986bdd9b1358bc" modify_external_catalogs: true
W20250113 02:51:48.010829 995 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.1:40095, user_credentials={real_user=slave}} blocked reactor thread for 247240us
I20250113 02:51:48.011165 2829 consensus_queue.cc:1035] T f4379f7fc6b54e6785f27975d9030cce P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 02:51:48.011325 1198 raft_consensus.cc:2267] T 542df00229f645788f1b821c0ce00c46 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:48.011560 2829 consensus_queue.cc:1035] T 247af1a17df04b27898f86642336d51e P 4216a7f853d642b7bc3e5cff458f3aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 02:51:48.011906 1198 ts_tablet_manager.cc:1905] T 542df00229f645788f1b821c0ce00c46 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:48.013782 1191 tablet_bootstrap.cc:492] T 943d3a125ac5406a83f3c8d80e073b6e P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:48.013926 1191 ts_tablet_manager.cc:1397] T 943d3a125ac5406a83f3c8d80e073b6e P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.046s user 0.003s sys 0.001s
I20250113 02:51:48.014562 1191 raft_consensus.cc:357] T 943d3a125ac5406a83f3c8d80e073b6e P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:48.014743 1191 raft_consensus.cc:383] T 943d3a125ac5406a83f3c8d80e073b6e P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:48.014796 1191 raft_consensus.cc:738] T 943d3a125ac5406a83f3c8d80e073b6e P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:48.014973 1191 consensus_queue.cc:260] T 943d3a125ac5406a83f3c8d80e073b6e P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:48.015902 1191 ts_tablet_manager.cc:1428] T 943d3a125ac5406a83f3c8d80e073b6e P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250113 02:51:48.016682 1191 tablet_bootstrap.cc:492] T 6d6e462c8bfa47eaa43d3695c69d9c0a P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:48.017161 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet c821e1ffe83c4134837d4830895ec9d8
W20250113 02:51:48.022078 711 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:38467, user_credentials={real_user=slave}} blocked reactor thread for 640111us
I20250113 02:51:48.032738 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_c2e8fd7d2cc14fa39c0abfece783c6c9" } modify_external_catalogs: true
I20250113 02:51:48.032919 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_c2e8fd7d2cc14fa39c0abfece783c6c9" } modify_external_catalogs: true
I20250113 02:51:48.034135 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet c821e1ffe83c4134837d4830895ec9d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:47 UTC) from {username='slave'} at 127.0.0.1:60230
W20250113 02:51:48.036036 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9083a6e1940e467cb73644c38c0bcc78 consensus state has no leader
I20250113 02:51:48.039397 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_41e74e05d3c64b3293212cff6b5fadde"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:48.039995 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_41e74e05d3c64b3293212cff6b5fadde in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:48.041206 454 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_9417f16fb6384fde8e191ca02808aa78" } modify_external_catalogs: true
I20250113 02:51:48.041344 454 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_9417f16fb6384fde8e191ca02808aa78" } modify_external_catalogs: true
I20250113 02:51:48.043720 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_f6f2b94540f64e45a00fa44fdab358eb" } new_table_name: "default.Table_66e09cfd64864de391467e30e7fa5e61" modify_external_catalogs: true
W20250113 02:51:48.048251 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0d8cdaa36b424a53b0501a65f4e2ce7d consensus state has no leader
I20250113 02:51:48.026214 1191 tablet_bootstrap.cc:654] T 6d6e462c8bfa47eaa43d3695c69d9c0a P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:48.053485 746 tablet_service.cc:1514] Processing DeleteTablet for tablet 146dad5c977f464496f7d21611030dad with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:26 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:48.058827 2842 raft_consensus.cc:491] T d8419fec408c40f6bd46c39de7b3dce9 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:48.058990 2853 raft_consensus.cc:491] T 303d911dfa2f4abbb053ead33db6d71c P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:48.059016 2842 raft_consensus.cc:513] T d8419fec408c40f6bd46c39de7b3dce9 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:48.059695 2842 leader_election.cc:290] T d8419fec408c40f6bd46c39de7b3dce9 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:48.059079 2853 raft_consensus.cc:513] T 303d911dfa2f4abbb053ead33db6d71c P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:48.060593 2853 leader_election.cc:290] T 303d911dfa2f4abbb053ead33db6d71c P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:48.062073 2853 raft_consensus.cc:491] T ed68a2b7ebcf4841bef54798dd3ed80d P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:48.062220 2853 raft_consensus.cc:513] T ed68a2b7ebcf4841bef54798dd3ed80d P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:48.062752 2853 leader_election.cc:290] T ed68a2b7ebcf4841bef54798dd3ed80d P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:48.062942 1198 ts_tablet_manager.cc:1918] T 542df00229f645788f1b821c0ce00c46 P 4216a7f853d642b7bc3e5cff458f3aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250113 02:51:48.063048 1198 log.cc:1198] T 542df00229f645788f1b821c0ce00c46 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-2/wal/wals/542df00229f645788f1b821c0ce00c46
I20250113 02:51:48.066238 1196 tablet_replica.cc:331] T e570465a3d4046b1ac19a3a034fe91f1 P df69f48656b44bb890f76587ee69cc0f: stopping tablet replica
I20250113 02:51:48.066421 1196 raft_consensus.cc:2238] T e570465a3d4046b1ac19a3a034fe91f1 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:48.066556 1196 raft_consensus.cc:2267] T e570465a3d4046b1ac19a3a034fe91f1 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:48.067028 1196 ts_tablet_manager.cc:1905] T e570465a3d4046b1ac19a3a034fe91f1 P df69f48656b44bb890f76587ee69cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:48.071854 1191 tablet_bootstrap.cc:492] T 6d6e462c8bfa47eaa43d3695c69d9c0a P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:48.072036 1191 ts_tablet_manager.cc:1397] T 6d6e462c8bfa47eaa43d3695c69d9c0a P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.056s user 0.006s sys 0.001s
I20250113 02:51:48.072778 1191 raft_consensus.cc:357] T 6d6e462c8bfa47eaa43d3695c69d9c0a P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:48.072966 1191 raft_consensus.cc:383] T 6d6e462c8bfa47eaa43d3695c69d9c0a P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:48.073019 1191 raft_consensus.cc:738] T 6d6e462c8bfa47eaa43d3695c69d9c0a P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:48.073251 1191 consensus_queue.cc:260] T 6d6e462c8bfa47eaa43d3695c69d9c0a P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:48.075033 1191 ts_tablet_manager.cc:1428] T 6d6e462c8bfa47eaa43d3695c69d9c0a P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250113 02:51:48.075398 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8419fec408c40f6bd46c39de7b3dce9" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:48.076040 1191 tablet_bootstrap.cc:492] T a78081b7efe34af8a053f5272cfaf346 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:48.077569 1191 tablet_bootstrap.cc:654] T a78081b7efe34af8a053f5272cfaf346 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:48.085832 1191 tablet_bootstrap.cc:492] T a78081b7efe34af8a053f5272cfaf346 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:48.085963 1191 ts_tablet_manager.cc:1397] T a78081b7efe34af8a053f5272cfaf346 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.000s
I20250113 02:51:48.090154 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9269967dea94fc3a337546ad0842e11" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:48.094333 1191 raft_consensus.cc:357] T a78081b7efe34af8a053f5272cfaf346 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:48.094522 1191 raft_consensus.cc:383] T a78081b7efe34af8a053f5272cfaf346 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:48.094579 1191 raft_consensus.cc:738] T a78081b7efe34af8a053f5272cfaf346 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
W20250113 02:51:48.094758 711 consensus_peers.cc:487] T b3ce6d851f564b849796f174860aacd5 P 1186fa869eb14812832fa6562531f196 -> Peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Couldn't send request to peer df69f48656b44bb890f76587ee69cc0f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250113 02:51:48.094790 1191 consensus_queue.cc:260] T a78081b7efe34af8a053f5272cfaf346 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
W20250113 02:51:48.099650 711 leader_election.cc:343] T d8419fec408c40f6bd46c39de7b3dce9 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:48.099699 1191 ts_tablet_manager.cc:1428] T a78081b7efe34af8a053f5272cfaf346 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20250113 02:51:48.100090 2854 raft_consensus.cc:491] T 6d6e462c8bfa47eaa43d3695c69d9c0a P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:48.101862 2854 raft_consensus.cc:513] T 6d6e462c8bfa47eaa43d3695c69d9c0a P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:48.102582 2854 leader_election.cc:290] T 6d6e462c8bfa47eaa43d3695c69d9c0a P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:48.103951 1191 tablet_bootstrap.cc:492] T 14cea6298852460ca8cfb74e321cc124 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:48.105754 885 tablet_service.cc:1514] Processing DeleteTablet for tablet 045ff4572f3347fc890b439d90cb0bbe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:47 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:48.106048 1191 tablet_bootstrap.cc:654] T 14cea6298852460ca8cfb74e321cc124 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:48.112326 1191 tablet_bootstrap.cc:492] T 14cea6298852460ca8cfb74e321cc124 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:48.113668 885 tablet_service.cc:1514] Processing DeleteTablet for tablet e90242bf273c42beb57ef160991a89ed with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:47 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:48.116118 1191 ts_tablet_manager.cc:1397] T 14cea6298852460ca8cfb74e321cc124 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.005s
I20250113 02:51:48.116883 1191 raft_consensus.cc:357] T 14cea6298852460ca8cfb74e321cc124 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:48.124039 1191 raft_consensus.cc:383] T 14cea6298852460ca8cfb74e321cc124 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:48.124287 1191 raft_consensus.cc:738] T 14cea6298852460ca8cfb74e321cc124 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:48.124801 1191 consensus_queue.cc:260] T 14cea6298852460ca8cfb74e321cc124 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:48.126659 1195 ts_tablet_manager.cc:1918] T 0b119704bc774ca9b4155c9619d4765b P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250113 02:51:48.126777 1195 log.cc:1198] T 0b119704bc774ca9b4155c9619d4765b P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/0b119704bc774ca9b4155c9619d4765b
I20250113 02:51:48.127099 1191 ts_tablet_manager.cc:1428] T 14cea6298852460ca8cfb74e321cc124 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.011s user 0.001s sys 0.001s
I20250113 02:51:48.128377 1191 tablet_bootstrap.cc:492] T b16dd8451f32478daf2ae118dbb9b304 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:48.130082 1191 tablet_bootstrap.cc:654] T b16dd8451f32478daf2ae118dbb9b304 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:48.133400 1198 ts_tablet_manager.cc:1939] T 542df00229f645788f1b821c0ce00c46 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting consensus metadata
I20250113 02:51:48.133972 1195 ts_tablet_manager.cc:1939] T 0b119704bc774ca9b4155c9619d4765b P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:48.134405 1198 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60230 (request call id 1509) took 2240 ms. Trace:
I20250113 02:51:48.134514 1198 rpcz_store.cc:276] 0113 02:51:45.893868 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:45.893916 (+ 48us) service_pool.cc:225] Handling call
0113 02:51:47.820270 (+1926354us) ts_tablet_manager.cc:1145] Deleting tablet 542df00229f645788f1b821c0ce00c46
0113 02:51:47.820278 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:48.011065 (+190787us) spinlock_profiling.cc:243] Waited 176 ms on lock 0x617000140050. stack: 00007f0bad094fd6 00007f0bad094d90 00007f0bac07448d 000000000096d544 000000000096d50d 000000000096ceda 00007f0bb883d2a1 00007f0bb88ca77c 00007f0bb8937fb3 00007f0bb9c2c463 00007f0bbc6d0eb7 00007f0bbc703584 00007f0bbc6e4690 00007f0bbc6e441d 0000000000a71ecc 00007f0bad0dbb64
0113 02:51:48.031285 (+ 20220us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:48.062809 (+ 31524us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:48.134379 (+ 71570us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":405209344,"tablet-delete.queue_time_us":1926218}
I20250113 02:51:48.134830 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1374) took 5255 ms. Trace:
I20250113 02:51:48.134900 1195 rpcz_store.cc:276] 0113 02:51:42.878937 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:42.878990 (+ 53us) service_pool.cc:225] Handling call
0113 02:51:47.921289 (+5042299us) ts_tablet_manager.cc:1145] Deleting tablet 0b119704bc774ca9b4155c9619d4765b
0113 02:51:47.921302 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:48.095747 (+174445us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:48.126550 (+ 30803us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:48.134810 (+ 8260us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5042158}
I20250113 02:51:48.135073 440 catalog_manager.cc:4872] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): tablet 542df00229f645788f1b821c0ce00c46 (table default.Table_b2598c65e9d04d8aa594a9f3d61d1e5a [id=e4f6f4ec007c454eba94263a37690f56]) successfully deleted
I20250113 02:51:48.135344 439 catalog_manager.cc:4872] TS 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095): tablet 0b119704bc774ca9b4155c9619d4765b (table default.Table_b95a9b1d12284c9aac701a8ef9c93482 [id=843aabcff7074acd98e5801b21282faa]) successfully deleted
I20250113 02:51:48.135690 1198 tablet_replica.cc:331] T 2470b010b571444eba3f132323038f33 P 4216a7f853d642b7bc3e5cff458f3aca: stopping tablet replica
I20250113 02:51:48.135877 1198 raft_consensus.cc:2238] T 2470b010b571444eba3f132323038f33 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:48.136037 1198 raft_consensus.cc:2267] T 2470b010b571444eba3f132323038f33 P 4216a7f853d642b7bc3e5cff458f3aca [term 1 FOLLOWER]: Raft consensus is shut down!
W20250113 02:51:48.136476 851 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.602s user 0.004s sys 0.042s
W20250113 02:51:48.136605 851 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.602s user 0.004s sys 0.042s
W20250113 02:51:48.136829 851 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:40095, user_credentials={real_user=slave}} blocked reactor thread for 602517us
I20250113 02:51:48.136898 1191 tablet_bootstrap.cc:492] T b16dd8451f32478daf2ae118dbb9b304 P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:48.137012 1191 ts_tablet_manager.cc:1397] T b16dd8451f32478daf2ae118dbb9b304 P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.003s
I20250113 02:51:48.138111 1192 tablet_bootstrap.cc:492] T d565b77f90784d87b64996772c938085 P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:48.138257 1192 ts_tablet_manager.cc:1397] T d565b77f90784d87b64996772c938085 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.340s user 0.000s sys 0.034s
I20250113 02:51:48.139053 1192 raft_consensus.cc:357] T d565b77f90784d87b64996772c938085 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:48.139297 1192 raft_consensus.cc:383] T d565b77f90784d87b64996772c938085 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:48.139366 1192 raft_consensus.cc:738] T d565b77f90784d87b64996772c938085 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:48.139577 1192 consensus_queue.cc:260] T d565b77f90784d87b64996772c938085 P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:48.143318 1195 tablet_replica.cc:331] T 8337f3c82d654ecdac2cbc1ffa88ef05 P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:48.143532 1195 raft_consensus.cc:2238] T 8337f3c82d654ecdac2cbc1ffa88ef05 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:48.143859 1198 ts_tablet_manager.cc:1905] T 2470b010b571444eba3f132323038f33 P 4216a7f853d642b7bc3e5cff458f3aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:48.145283 1192 ts_tablet_manager.cc:1428] T d565b77f90784d87b64996772c938085 P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20250113 02:51:48.151324 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_4ff2ad708f434e2292b3a0f9055d9d0f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:48.154093 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_c06ea026131b4309b2e58796efa73aa2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:48.154551 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c06ea026131b4309b2e58796efa73aa2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:48.154995 2827 raft_consensus.cc:491] T fe3d35d9aaf64188aaa3c1ac38d279d8 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:48.155122 2827 raft_consensus.cc:513] T fe3d35d9aaf64188aaa3c1ac38d279d8 P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:48.155680 2827 leader_election.cc:290] T fe3d35d9aaf64188aaa3c1ac38d279d8 P 4216a7f853d642b7bc3e5cff458f3aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467)
W20250113 02:51:48.156029 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4ff2ad708f434e2292b3a0f9055d9d0f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:48.157351 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_72fe5de3f5bd4766b19d415958c9e02b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:48.157788 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_72fe5de3f5bd4766b19d415958c9e02b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:48.164592 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e13f5bcdcee34225a46c6d2651e5ede9: Sending DeleteTablet for 3 replicas of tablet b815003a27f64809baaa2d43d6293cef
I20250113 02:51:48.165292 451 master_service.cc:946] ReplaceTablet: received request to replace tablet c548a79f394d4ab88a309ca6400a05a2 from {username='slave'} at 127.0.0.1:49258
I20250113 02:51:48.165937 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_53afceeb17dd4e71961773849182013d" } new_table_name: "default.Table_6af321230c60427d978d97149530ab64" modify_external_catalogs: true
I20250113 02:51:48.167902 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet b815003a27f64809baaa2d43d6293cef with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:48 UTC) from {username='slave'} at 127.0.0.1:60230
I20250113 02:51:48.168279 1192 tablet_bootstrap.cc:492] T 7fb282707196444690352dea3eb1387f P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
W20250113 02:51:48.170305 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f29e16fea2534c7dbe57639c7c6b37d6 consensus state has no leader
I20250113 02:51:48.170670 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_6803534f8cb245508e81a64bb59e2511"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:48.171100 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6803534f8cb245508e81a64bb59e2511 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:48.173044 2854 raft_consensus.cc:491] T a78081b7efe34af8a053f5272cfaf346 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:48.173287 2854 raft_consensus.cc:513] T a78081b7efe34af8a053f5272cfaf346 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:48.173871 2854 leader_election.cc:290] T a78081b7efe34af8a053f5272cfaf346 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:48.174938 742 tablet_service.cc:1514] Processing DeleteTablet for tablet b815003a27f64809baaa2d43d6293cef with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:48 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:48.175242 748 tablet_service.cc:1514] Processing DeleteTablet for tablet 943d3a125ac5406a83f3c8d80e073b6e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:25 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:48.175840 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49258:
table { table_name: "default.Table_d79dc368b229418d9ba63683c93bbcb4" } new_table_name: "default.Table_5496e0f8f8854c20956af920dc9fea4b" modify_external_catalogs: true
I20250113 02:51:48.175964 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_a802a89c341d4453a2b1ff4c2e23a712"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 02:51:48.176561 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a802a89c341d4453a2b1ff4c2e23a712 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 02:51:48.178933 1195 raft_consensus.cc:2267] T 8337f3c82d654ecdac2cbc1ffa88ef05 P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:48.179533 1195 ts_tablet_manager.cc:1905] T 8337f3c82d654ecdac2cbc1ffa88ef05 P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:48.180188 748 tablet_service.cc:1467] Processing CreateTablet for tablet 18a7bf846d0e473f8205254d6b40507f (DEFAULT_TABLE table=default.Table_82f0839904a94b9db9014e2aa6ad9db2 [id=46b0f129fff74f179f1df1e43a05e060]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:48.181958 751 tablet_service.cc:1467] Processing CreateTablet for tablet 7511fae0fc1a42f0bbbdbd3462e9b511 (DEFAULT_TABLE table=default.Table_d3095de5abfb4fb7ae50a24c7c9445ca [id=d3d8bcefe9284a76b106be405d618b69]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:48.182507 746 tablet_service.cc:1467] Processing CreateTablet for tablet c7185182c11747e8a511192e55ce60cf (DEFAULT_TABLE table=default.Table_61e9db1815324f248afc5bd24612a9d9 [id=c9e4ba35be09495bb2babf821103a366]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:48.182973 742 tablet_service.cc:1467] Processing CreateTablet for tablet 3b01e4e3d34c4a47b5ddfc10526ff033 (DEFAULT_TABLE table=default.Table_ffe02411b95246779d563a7896030496 [id=5b22897a0e534041a4e2d00ade06e7d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:48.183687 2854 raft_consensus.cc:491] T 415a1fa4b2f14cacb43937c8a372849a P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:48.183804 2854 raft_consensus.cc:513] T 415a1fa4b2f14cacb43937c8a372849a P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:48.184432 2854 leader_election.cc:290] T 415a1fa4b2f14cacb43937c8a372849a P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:48.185601 743 tablet_service.cc:1467] Processing CreateTablet for tablet d86232f5134145c0b03d5291ddffe1e2 (DEFAULT_TABLE table=default.Table_68db23b2977d4f0c9a7c2a96ee28fc80 [id=9b2200ab321441e6806ac819fad1fa24]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:48.186084 754 tablet_service.cc:1467] Processing CreateTablet for tablet 07cb5054ae4e471a982afc9c074cc044 (DEFAULT_TABLE table=default.Table_c4b56c0ef9c84e3494e2fc5f0436ce81 [id=70f5045f3e284dbf851df9af5d405530]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:48.186546 741 tablet_service.cc:1467] Processing CreateTablet for tablet 5ef00b2fb5964e60a33db0ad20556bac (DEFAULT_TABLE table=default.Table_0a219279fac1400295a14ee438d8c741 [id=42738d6be932409d8e2c803d76aadbd3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:48.186918 738 tablet_service.cc:1467] Processing CreateTablet for tablet fa11583bfa9f4ce2b791880389df7201 (DEFAULT_TABLE table=default.Table_433fdf8640d14c58aae8a24bcea4edf2 [id=941139ee8c4e409b9eb0bbac1db86ffa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:48.188051 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c7185182c11747e8a511192e55ce60cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:48.188639 1038 tablet_service.cc:1467] Processing CreateTablet for tablet 3b01e4e3d34c4a47b5ddfc10526ff033 (DEFAULT_TABLE table=default.Table_ffe02411b95246779d563a7896030496 [id=5b22897a0e534041a4e2d00ade06e7d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:48.188822 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fa11583bfa9f4ce2b791880389df7201. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:48.189060 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b01e4e3d34c4a47b5ddfc10526ff033. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:48.189239 1042 tablet_service.cc:1467] Processing CreateTablet for tablet 07cb5054ae4e471a982afc9c074cc044 (DEFAULT_TABLE table=default.Table_c4b56c0ef9c84e3494e2fc5f0436ce81 [id=70f5045f3e284dbf851df9af5d405530]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:48.189332 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ef00b2fb5964e60a33db0ad20556bac. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:48.189270 1035 tablet_service.cc:1467] Processing CreateTablet for tablet 1c6fa54a74b84ded9958c5b7c70f55e5 (DEFAULT_TABLE table=default.Table_e78e59f833614fad8928e0598bc8e7fa [id=d64776175f4f4194ac01a21b7e752432]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:48.189070 1032 tablet_service.cc:1467] Processing CreateTablet for tablet fa11583bfa9f4ce2b791880389df7201 (DEFAULT_TABLE table=default.Table_433fdf8640d14c58aae8a24bcea4edf2 [id=941139ee8c4e409b9eb0bbac1db86ffa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:48.189543 1042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07cb5054ae4e471a982afc9c074cc044. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:48.189966 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07cb5054ae4e471a982afc9c074cc044. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:48.190469 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d86232f5134145c0b03d5291ddffe1e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:48.190959 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7511fae0fc1a42f0bbbdbd3462e9b511. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:48.191368 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b01e4e3d34c4a47b5ddfc10526ff033. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:48.192183 1030 tablet_service.cc:1467] Processing CreateTablet for tablet fc715396e30943d4a9411cfb1007a2c2 (DEFAULT_TABLE table=default.Table_32a3dae98e444e1f8f79fefacd999119 [id=4f3b8002b7194e00b242cc7f4cf2cdb9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:48.192345 1036 tablet_service.cc:1467] Processing CreateTablet for tablet c7185182c11747e8a511192e55ce60cf (DEFAULT_TABLE table=default.Table_61e9db1815324f248afc5bd24612a9d9 [id=c9e4ba35be09495bb2babf821103a366]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:48.192647 1039 tablet_service.cc:1467] Processing CreateTablet for tablet d86232f5134145c0b03d5291ddffe1e2 (DEFAULT_TABLE table=default.Table_68db23b2977d4f0c9a7c2a96ee28fc80 [id=9b2200ab321441e6806ac819fad1fa24]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:48.192970 1026 tablet_service.cc:1467] Processing CreateTablet for tablet 5ef00b2fb5964e60a33db0ad20556bac (DEFAULT_TABLE table=default.Table_0a219279fac1400295a14ee438d8c741 [id=42738d6be932409d8e2c803d76aadbd3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:48.193150 752 tablet_service.cc:1467] Processing CreateTablet for tablet 1c6fa54a74b84ded9958c5b7c70f55e5 (DEFAULT_TABLE table=default.Table_e78e59f833614fad8928e0598bc8e7fa [id=d64776175f4f4194ac01a21b7e752432]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:48.192183 1024 tablet_service.cc:1467] Processing CreateTablet for tablet 7511fae0fc1a42f0bbbdbd3462e9b511 (DEFAULT_TABLE table=default.Table_d3095de5abfb4fb7ae50a24c7c9445ca [id=d3d8bcefe9284a76b106be405d618b69]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:48.188648 1034 tablet_service.cc:1467] Processing CreateTablet for tablet 18a7bf846d0e473f8205254d6b40507f (DEFAULT_TABLE table=default.Table_82f0839904a94b9db9014e2aa6ad9db2 [id=46b0f129fff74f179f1df1e43a05e060]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:48.193424 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c6fa54a74b84ded9958c5b7c70f55e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:48.193570 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fa11583bfa9f4ce2b791880389df7201. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:48.193580 455 catalog_manager.cc:5526] T 8f2fb58fc24d4692a3fe9a652e076aed P 4216a7f853d642b7bc3e5cff458f3aca reported cstate change: term changed from 0 to 1, leader changed from <none> to 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3). New cstate: current_term: 1 leader_uuid: "4216a7f853d642b7bc3e5cff458f3aca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } health_report { overall_health: HEALTHY } } }
I20250113 02:51:48.193979 455 catalog_manager.cc:5618] TS 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587) does not have the latest schema for tablet 8f2fb58fc24d4692a3fe9a652e076aed (table default.Table_a752553e5fcc4ed1a56d9d5197d6b885 [id=8edee52f05e24c03b088ce58f4b3f4d9]). Expected version 1 got 0
I20250113 02:51:48.194729 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc715396e30943d4a9411cfb1007a2c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:48.195524 744 tablet_service.cc:1467] Processing CreateTablet for tablet fc715396e30943d4a9411cfb1007a2c2 (DEFAULT_TABLE table=default.Table_32a3dae98e444e1f8f79fefacd999119 [id=4f3b8002b7194e00b242cc7f4cf2cdb9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:48.195794 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc715396e30943d4a9411cfb1007a2c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:48.197539 1036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c7185182c11747e8a511192e55ce60cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:48.197908 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ef00b2fb5964e60a33db0ad20556bac. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:48.198283 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7511fae0fc1a42f0bbbdbd3462e9b511. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:48.198662 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d86232f5134145c0b03d5291ddffe1e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:48.199034 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 18a7bf846d0e473f8205254d6b40507f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:48.199398 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c6fa54a74b84ded9958c5b7c70f55e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 02:51:48.202400 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49258:
name: "default.Table_c9694269deb3414591b71797d3e4595a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 02:51:48.208087 752 tablet_service.cc:1514] Processing DeleteTablet for tablet c821e1ffe83c4134837d4830895ec9d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:47 UTC) from {username='slave'} at 127.0.0.1:55398
I20250113 02:51:48.214676 442 catalog_manager.cc:4872] TS df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): tablet 9f8eb43a39774d2c8e9cb4e88a405b0f (table default.Table_bfda83bbe8bb4025b01edf1e38b1b73c [id=e900f4da41bf46bcace34b132f3ba977]) successfully deleted
I20250113 02:51:48.220055 895 tablet_service.cc:1514] Processing DeleteTablet for tablet c821e1ffe83c4134837d4830895ec9d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:47 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:48.220924 1192 tablet_bootstrap.cc:654] T 7fb282707196444690352dea3eb1387f P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:48.227041 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "303d911dfa2f4abbb053ead33db6d71c" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:48.232154 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 18a7bf846d0e473f8205254d6b40507f. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 02:51:48.240955 998 consensus_peers.cc:487] T 247af1a17df04b27898f86642336d51e P 4216a7f853d642b7bc3e5cff458f3aca -> Peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Couldn't send request to peer df69f48656b44bb890f76587ee69cc0f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250113 02:51:48.241461 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9269967dea94fc3a337546ad0842e11" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:48.241603 1058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8419fec408c40f6bd46c39de7b3dce9" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:48.241948 1058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "303d911dfa2f4abbb053ead33db6d71c" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:48.242728 1058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed68a2b7ebcf4841bef54798dd3ed80d" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
W20250113 02:51:48.244315 709 leader_election.cc:343] T f9269967dea94fc3a337546ad0842e11 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:48.251278 1191 raft_consensus.cc:357] T b16dd8451f32478daf2ae118dbb9b304 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:48.251500 1191 raft_consensus.cc:383] T b16dd8451f32478daf2ae118dbb9b304 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:48.251657 1191 raft_consensus.cc:738] T b16dd8451f32478daf2ae118dbb9b304 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:48.252009 1191 consensus_queue.cc:260] T b16dd8451f32478daf2ae118dbb9b304 P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:48.253216 1191 ts_tablet_manager.cc:1428] T b16dd8451f32478daf2ae118dbb9b304 P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.116s user 0.003s sys 0.000s
I20250113 02:51:48.253712 1191 tablet_bootstrap.cc:492] T dfea918c04c1408a8b86a44bb0911a7c P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:48.256618 1196 ts_tablet_manager.cc:1918] T e570465a3d4046b1ac19a3a034fe91f1 P df69f48656b44bb890f76587ee69cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250113 02:51:48.256903 1196 log.cc:1198] T e570465a3d4046b1ac19a3a034fe91f1 P df69f48656b44bb890f76587ee69cc0f: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-1/wal/wals/e570465a3d4046b1ac19a3a034fe91f1
W20250113 02:51:48.260027 709 leader_election.cc:343] T d8419fec408c40f6bd46c39de7b3dce9 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:48.260138 709 leader_election.cc:304] T d8419fec408c40f6bd46c39de7b3dce9 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:48.260511 2854 raft_consensus.cc:2743] T d8419fec408c40f6bd46c39de7b3dce9 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250113 02:51:48.260958 709 leader_election.cc:343] T 303d911dfa2f4abbb053ead33db6d71c P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
W20250113 02:51:48.261663 709 leader_election.cc:343] T ed68a2b7ebcf4841bef54798dd3ed80d P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:48.265233 1192 tablet_bootstrap.cc:492] T 7fb282707196444690352dea3eb1387f P 4216a7f853d642b7bc3e5cff458f3aca: No bootstrap required, opened a new log
I20250113 02:51:48.265389 1192 ts_tablet_manager.cc:1397] T 7fb282707196444690352dea3eb1387f P 4216a7f853d642b7bc3e5cff458f3aca: Time spent bootstrapping tablet: real 0.097s user 0.001s sys 0.005s
W20250113 02:51:48.266952 711 leader_election.cc:343] T f9269967dea94fc3a337546ad0842e11 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:48.267041 711 leader_election.cc:304] T f9269967dea94fc3a337546ad0842e11 P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:48.267436 1191 tablet_bootstrap.cc:654] T dfea918c04c1408a8b86a44bb0911a7c P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
W20250113 02:51:48.267664 711 leader_election.cc:343] T 303d911dfa2f4abbb053ead33db6d71c P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df69f48656b44bb890f76587ee69cc0f (127.0.104.2:38467): Illegal state: must be running to vote when last-logged opid is not known
I20250113 02:51:48.267781 711 leader_election.cc:304] T 303d911dfa2f4abbb053ead33db6d71c P 1186fa869eb14812832fa6562531f196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1186fa869eb14812832fa6562531f196; no voters: 4216a7f853d642b7bc3e5cff458f3aca, df69f48656b44bb890f76587ee69cc0f
I20250113 02:51:48.268491 2853 raft_consensus.cc:2743] T f9269967dea94fc3a337546ad0842e11 P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:48.268985 2853 raft_consensus.cc:2743] T 303d911dfa2f4abbb053ead33db6d71c P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 02:51:48.271001 1192 raft_consensus.cc:357] T 7fb282707196444690352dea3eb1387f P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:48.271274 1192 raft_consensus.cc:383] T 7fb282707196444690352dea3eb1387f P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:48.271351 1192 raft_consensus.cc:738] T 7fb282707196444690352dea3eb1387f P 4216a7f853d642b7bc3e5cff458f3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216a7f853d642b7bc3e5cff458f3aca, State: Initialized, Role: FOLLOWER
I20250113 02:51:48.271652 1192 consensus_queue.cc:260] T 7fb282707196444690352dea3eb1387f P 4216a7f853d642b7bc3e5cff458f3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:48.273319 1195 ts_tablet_manager.cc:1918] T 8337f3c82d654ecdac2cbc1ffa88ef05 P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250113 02:51:48.273429 1195 log.cc:1198] T 8337f3c82d654ecdac2cbc1ffa88ef05 P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/8337f3c82d654ecdac2cbc1ffa88ef05
I20250113 02:51:48.273784 1195 ts_tablet_manager.cc:1939] T 8337f3c82d654ecdac2cbc1ffa88ef05 P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:48.275956 1192 ts_tablet_manager.cc:1428] T 7fb282707196444690352dea3eb1387f P 4216a7f853d642b7bc3e5cff458f3aca: Time spent starting tablet: real 0.010s user 0.001s sys 0.002s
I20250113 02:51:48.276335 1192 tablet_bootstrap.cc:492] T c73c649a4ca44471af2cf6a77500b065 P 4216a7f853d642b7bc3e5cff458f3aca: Bootstrap starting.
I20250113 02:51:48.277920 1191 tablet_bootstrap.cc:492] T dfea918c04c1408a8b86a44bb0911a7c P 1186fa869eb14812832fa6562531f196: No bootstrap required, opened a new log
I20250113 02:51:48.278064 1191 ts_tablet_manager.cc:1397] T dfea918c04c1408a8b86a44bb0911a7c P 1186fa869eb14812832fa6562531f196: Time spent bootstrapping tablet: real 0.024s user 0.002s sys 0.001s
I20250113 02:51:48.278965 1191 raft_consensus.cc:357] T dfea918c04c1408a8b86a44bb0911a7c P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:48.279181 1191 raft_consensus.cc:383] T dfea918c04c1408a8b86a44bb0911a7c P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 02:51:48.279238 1191 raft_consensus.cc:738] T dfea918c04c1408a8b86a44bb0911a7c P 1186fa869eb14812832fa6562531f196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186fa869eb14812832fa6562531f196, State: Initialized, Role: FOLLOWER
I20250113 02:51:48.282716 1192 tablet_bootstrap.cc:654] T c73c649a4ca44471af2cf6a77500b065 P 4216a7f853d642b7bc3e5cff458f3aca: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:48.285938 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1375) took 5327 ms. Trace:
I20250113 02:51:48.286024 1195 rpcz_store.cc:276] 0113 02:51:42.958523 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:42.958701 (+ 178us) service_pool.cc:225] Handling call
0113 02:51:48.143261 (+5184560us) ts_tablet_manager.cc:1145] Deleting tablet 8337f3c82d654ecdac2cbc1ffa88ef05
0113 02:51:48.143272 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:48.247915 (+104643us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:48.273224 (+ 25309us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:48.285918 (+ 12694us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":35215,"tablet-delete.queue_time_us":5167501}
I20250113 02:51:48.286636 1191 consensus_queue.cc:260] T dfea918c04c1408a8b86a44bb0911a7c P 1186fa869eb14812832fa6562531f196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:48.288041 1191 ts_tablet_manager.cc:1428] T dfea918c04c1408a8b86a44bb0911a7c P 1186fa869eb14812832fa6562531f196: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20250113 02:51:48.288326 1191 tablet_bootstrap.cc:492] T 8412a588d0454ec78025e7ec30c38104 P 1186fa869eb14812832fa6562531f196: Bootstrap starting.
I20250113 02:51:48.257781 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe3d35d9aaf64188aaa3c1ac38d279d8" candidate_uuid: "4216a7f853d642b7bc3e5cff458f3aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:48.292318 1196 ts_tablet_manager.cc:1939] T e570465a3d4046b1ac19a3a034fe91f1 P df69f48656b44bb890f76587ee69cc0f: Deleting consensus metadata
I20250113 02:51:48.265852 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a78081b7efe34af8a053f5272cfaf346" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:48.266494 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d6e462c8bfa47eaa43d3695c69d9c0a" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:48.266723 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "415a1fa4b2f14cacb43937c8a372849a" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:48.269454 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed68a2b7ebcf4841bef54798dd3ed80d" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df69f48656b44bb890f76587ee69cc0f" is_pre_election: true
I20250113 02:51:48.270020 897 tablet_service.cc:1467] Processing CreateTablet for tablet 18a7bf846d0e473f8205254d6b40507f (DEFAULT_TABLE table=default.Table_82f0839904a94b9db9014e2aa6ad9db2 [id=46b0f129fff74f179f1df1e43a05e060]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:48.289603 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1376) took 5198 ms. Trace:
I20250113 02:51:48.271768 884 tablet_service.cc:1467] Processing CreateTablet for tablet 1c6fa54a74b84ded9958c5b7c70f55e5 (DEFAULT_TABLE table=default.Table_e78e59f833614fad8928e0598bc8e7fa [id=d64776175f4f4194ac01a21b7e752432]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:48.294229 1195 rpcz_store.cc:276] 0113 02:51:43.091563 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:43.091791 (+ 228us) service_pool.cc:225] Handling call
0113 02:51:48.289453 (+5197662us) ts_tablet_manager.cc:1145] Deleting tablet 1c3c3b7cf89e4f22b8a5dd7ba4f8f5ec
0113 02:51:48.289466 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:48.289583 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5197476}
I20250113 02:51:48.272960 880 tablet_service.cc:1467] Processing CreateTablet for tablet fa11583bfa9f4ce2b791880389df7201 (DEFAULT_TABLE table=default.Table_433fdf8640d14c58aae8a24bcea4edf2 [id=941139ee8c4e409b9eb0bbac1db86ffa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:48.294768 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1378) took 5201 ms. Trace:
I20250113 02:51:48.273274 899 tablet_service.cc:1467] Processing CreateTablet for tablet 07cb5054ae4e471a982afc9c074cc044 (DEFAULT_TABLE table=default.Table_c4b56c0ef9c84e3494e2fc5f0436ce81 [id=70f5045f3e284dbf851df9af5d405530]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:48.294857 1195 rpcz_store.cc:276] 0113 02:51:43.093185 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:43.093234 (+ 49us) service_pool.cc:225] Handling call
0113 02:51:48.294622 (+5201388us) ts_tablet_manager.cc:1145] Deleting tablet f86318637d5b4e378ce47e89a5d52b07
0113 02:51:48.294632 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:48.294753 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5201216}
I20250113 02:51:48.273659 894 tablet_service.cc:1467] Processing CreateTablet for tablet 5ef00b2fb5964e60a33db0ad20556bac (DEFAULT_TABLE table=default.Table_0a219279fac1400295a14ee438d8c741 [id=42738d6be932409d8e2c803d76aadbd3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:48.274072 890 tablet_service.cc:1467] Processing CreateTablet for tablet fc715396e30943d4a9411cfb1007a2c2 (DEFAULT_TABLE table=default.Table_32a3dae98e444e1f8f79fefacd999119 [id=4f3b8002b7194e00b242cc7f4cf2cdb9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:48.279446 885 tablet_service.cc:1467] Processing CreateTablet for tablet d86232f5134145c0b03d5291ddffe1e2 (DEFAULT_TABLE table=default.Table_68db23b2977d4f0c9a7c2a96ee28fc80 [id=9b2200ab321441e6806ac819fad1fa24]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:48.282043 892 tablet_service.cc:1467] Processing CreateTablet for tablet 7511fae0fc1a42f0bbbdbd3462e9b511 (DEFAULT_TABLE table=default.Table_d3095de5abfb4fb7ae50a24c7c9445ca [id=d3d8bcefe9284a76b106be405d618b69]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:48.296461 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1377) took 5204 ms. Trace:
I20250113 02:51:48.296543 1195 rpcz_store.cc:276] 0113 02:51:43.091702 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:43.096043 (+ 4341us) service_pool.cc:225] Handling call
0113 02:51:48.296338 (+5200295us) ts_tablet_manager.cc:1145] Deleting tablet d1f30873a02f4b61ac6940e44c332ad3
0113 02:51:48.296350 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:48.296445 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5200117}
I20250113 02:51:48.283152 895 tablet_service.cc:1467] Processing CreateTablet for tablet c7185182c11747e8a511192e55ce60cf (DEFAULT_TABLE table=default.Table_61e9db1815324f248afc5bd24612a9d9 [id=c9e4ba35be09495bb2babf821103a366]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:48.283535 2946 tablet.cc:1721] T c13e9a51c1e74b44b352bade78390ba5 P df69f48656b44bb890f76587ee69cc0f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250113 02:51:48.287707 893 tablet_service.cc:1467] Processing CreateTablet for tablet 3b01e4e3d34c4a47b5ddfc10526ff033 (DEFAULT_TABLE table=default.Table_ffe02411b95246779d563a7896030496 [id=5b22897a0e534041a4e2d00ade06e7d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 02:51:48.265146 898 tablet_service.cc:1514] Processing DeleteTablet for tablet b815003a27f64809baaa2d43d6293cef with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-13 02:51:48 UTC) from {username='slave'} at 127.0.0.1:37366
I20250113 02:51:48.299408 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1382) took 5116 ms. Trace:
I20250113 02:51:48.299504 1195 rpcz_store.cc:276] 0113 02:51:43.183454 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:43.183526 (+ 72us) service_pool.cc:225] Handling call
0113 02:51:48.299284 (+5115758us) ts_tablet_manager.cc:1145] Deleting tablet 7da6f8a4b89647968609f73f9ad5ba14
0113 02:51:48.299293 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:48.299386 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5115546}
I20250113 02:51:48.301038 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1383) took 5116 ms. Trace:
I20250113 02:51:48.301113 1195 rpcz_store.cc:276] 0113 02:51:43.184522 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:43.184581 (+ 59us) service_pool.cc:225] Handling call
0113 02:51:48.300885 (+5116304us) ts_tablet_manager.cc:1145] Deleting tablet ef078624960843eead2edd1e39b10118
0113 02:51:48.300897 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:48.301017 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5116118}
I20250113 02:51:48.302714 1195 tablet_replica.cc:331] T 7527e6e0b65e4a43b7f48899a34cdf7f P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:48.302943 1195 raft_consensus.cc:2238] T 7527e6e0b65e4a43b7f48899a34cdf7f P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250113 02:51:48.303136 1195 raft_consensus.cc:2267] T 7527e6e0b65e4a43b7f48899a34cdf7f P 1186fa869eb14812832fa6562531f196 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 02:51:48.303735 1195 ts_tablet_manager.cc:1905] T 7527e6e0b65e4a43b7f48899a34cdf7f P 1186fa869eb14812832fa6562531f196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 02:51:48.309624 1191 tablet_bootstrap.cc:654] T 8412a588d0454ec78025e7ec30c38104 P 1186fa869eb14812832fa6562531f196: Neither blocks nor log segments found. Creating new log.
I20250113 02:51:48.313438 1196 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37366 (request call id 1481) took 1074 ms. Trace:
W20250113 02:51:48.315883 1194 thread.cc:635] log-alloc [worker] (thread pool) Time spent creating pthread: real 0.604s user 0.021s sys 0.026s
W20250113 02:51:48.329726 1194 thread.cc:601] log-alloc [worker] (thread pool) Time spent starting thread: real 0.617s user 0.021s sys 0.027s
I20250113 02:51:48.347549 1195 ts_tablet_manager.cc:1918] T 7527e6e0b65e4a43b7f48899a34cdf7f P 1186fa869eb14812832fa6562531f196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250113 02:51:48.347687 1195 log.cc:1198] T 7527e6e0b65e4a43b7f48899a34cdf7f P 1186fa869eb14812832fa6562531f196: Deleting WAL directory at /tmp/dist-test-taskXJrxug/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736736671609957-416-0/minicluster-data/ts-0/wal/wals/7527e6e0b65e4a43b7f48899a34cdf7f
I20250113 02:51:48.348070 1195 ts_tablet_manager.cc:1939] T 7527e6e0b65e4a43b7f48899a34cdf7f P 1186fa869eb14812832fa6562531f196: Deleting consensus metadata
I20250113 02:51:48.349010 1195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55398 (request call id 1384) took 5141 ms. Trace:
I20250113 02:51:48.349118 1195 rpcz_store.cc:276] 0113 02:51:43.207160 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 02:51:43.207224 (+ 64us) service_pool.cc:225] Handling call
0113 02:51:48.302400 (+5095176us) ts_tablet_manager.cc:1145] Deleting tablet 7527e6e0b65e4a43b7f48899a34cdf7f
0113 02:51:48.302409 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0113 02:51:48.323389 (+ 20980us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:48.347383 (+ 23994us) tablet_metadata.cc:641] Metadata flushed
0113 02:51:48.348982 (+ 1599us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5094972}
I20250113 02:51:48.358000 2911 raft_consensus.cc:491] T 6a4c6dc08df744e6856bdf99ab65c575 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 02:51:48.358153 2911 raft_consensus.cc:513] T 6a4c6dc08df744e6856bdf99ab65c575 P df69f48656b44bb890f76587ee69cc0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1186fa869eb14812832fa6562531f196" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40095 } } peers { permanent_uuid: "df69f48656b44bb890f76587ee69cc0f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38467 } } peers { permanent_uuid: "4216a7f853d642b7bc3e5cff458f3aca" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42587 } }
I20250113 02:51:48.358768 2911 leader_election.cc:290] T 6a4c6dc08df744e6856bdf99ab65c575 P df69f48656b44bb890f76587ee69cc0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1186fa869eb14812832fa6562531f196 (127.0.104.1:40095), 4216a7f853d642b7bc3e5cff458f3aca (127.0.104.3:42587)
I20250113 02:51:48.359549 1195 tablet_replica.cc:331] T 9b2d0aa6f0e9456d95f4e48c360ab6e8 P 1186fa869eb14812832fa6562531f196: stopping tablet replica
I20250113 02:51:48.243120 1058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d6e462c8bfa47eaa43d3695c69d9c0a" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:48.243428 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a78081b7efe34af8a053f5272cfaf346" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca" is_pre_election: true
I20250113 02:51:48.243716 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "415a1fa4b2f14cacb43937c8a372849a" candidate_uuid: "1186fa869eb14812832fa6562531f196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4216a7f853d642b7bc3e5cff458f3aca"